Nightingale is an alert-centric open-source monitoring project. Like Grafana’s data source integration approach, Nightingale also integrates with multiple existing data sources. The difference is that Grafana focuses on visualization while Nightingale focuses on the alert engine and the processing and dispatch of alert events.

The Nightingale project was initially developed and open-sourced by Didi, and on May 11, 2022, it was donated to the China Computer Federation’s Open Source Development Committee (CCF ODC) — the first open-source project accepted by CCF ODC after its founding.

Code Repositories

• Backend (main repo): https://github.com/ccfos/nightingale
• Frontend: https://github.com/n9e/fe

How Nightingale Works

Many users have already collected metrics and log data on their own. In that case, they simply add their storage (VictoriaMetrics, ElasticSearch, etc.) as a data source in Nightingale, then configure alert rules and notification rules to produce and dispatch alert events.

The Nightingale project does not provide monitoring data collection itself. We recommend using Categraf as the collector — it integrates seamlessly with Nightingale.

Categraf can collect monitoring data from operating systems, network devices, various middleware, and databases, and push it to Nightingale via Remote Write. Nightingale then forwards the data to a time-series database (such as Prometheus or VictoriaMetrics) and provides alerting and visualization.

For some edge data centers, if the link to the central Nightingale server is poor and you want to improve alert availability, Nightingale also provides an edge-data-center alert-engine-downstream deployment mode. In this mode, even if the edge is cut off from the central side, alerting is unaffected.

In the diagram above, the link between DC A and the central DC is good, so the central Nightingale process serves as the alert engine. The link between DC B and the central DC is poor, so n9e-edge is deployed in DC B as the alert engine, performing alert evaluation on DC B’s data sources.

Alert Noise Reduction, Escalation, and Collaboration

Nightingale focuses on the alert engine — producing alert events and flexibly dispatching them according to rules, with 20 built-in notification channels (phone, SMS, email, DingTalk, Feishu, WeCom, Slack, etc.).

For more advanced needs such as:

• Wanting to centralize events from your company’s multiple monitoring systems on a single platform for unified noise reduction, response, and data analysis.
• Wanting to support on-call scheduling and practice the On-call culture, supporting alert acknowledgment, escalation (to avoid misses), and collaborative handling.

Nightingale is not a good fit. We recommend an on-call product like Flashduty that consolidates alerts from both cloud and on-prem monitoring systems and uniformly performs noise reduction, dispatch, and response.

Resources & Communication Channels

• 📚 Nightingale Introduction PPT helps you understand Nightingale’s key features (PPT link at the end of the article).
• ❤️ Ask a question & report a bug — write clearly the version, problem description, reproduction steps, screenshots, etc., otherwise the community won’t be able to help.
• 🌟 Add my WeChat: picobyte (note: Nightingale support group) to be pulled into a WeChat group. If you’ve taken Nightingale into production, please contact me to be pulled into the senior monitoring users group.

Key Features Overview

• Nightingale supports alert rules, mute rules, subscription rules, and notification rules, with 20 built-in notification channels and customizable message templates.
• Supports event pipelines for Pipeline-based processing of alert events — handy for integrating with your own systems, e.g., attaching extra metadata to events or relabeling events.
• Supports the business-group concept with permissions for managing different rule types in categorized fashion.
• Many databases and middleware have built-in alert rules ready to import; Prometheus alert rules can also be imported directly.
• Supports alert self-healing — automatically triggers a script after an alert to execute predefined logic, e.g., cleaning disk or capturing the scene.

• Nightingale archives historical alert events, supporting multi-dimensional query and statistics.
• Supports flexible aggregation and grouping to give an at-a-glance view of the company’s alert distribution.

• Nightingale comes with metric descriptions, dashboards, and alert rules for common operating systems, middleware, and databases — though these are community-contributed and of varying quality.
• Nightingale directly receives data via Remote Write, OpenTSDB, Datadog, Falcon, and many other protocols, so it can integrate with various Agents.
• Nightingale supports Prometheus, ElasticSearch, Loki, TDEngine, and other data sources for alerting.
• Nightingale can easily embed your internal systems such as Grafana, CMDB, etc., and you can even configure the menu visibility of these embedded systems.

• Nightingale supports dashboards with common chart types and ships with some built-in ones. The screenshot above shows one of them.
• If you are already used to Grafana, we still recommend using Grafana for visualization — Grafana is deeper here.
• For machine-related monitoring data collected by Categraf, we recommend using Nightingale’s built-in dashboards because Categraf follows Telegraf’s naming convention, which differs from Node Exporter.
• Because Nightingale has the business-group concept and machines can belong to different business groups, sometimes you want to view only machines in the current business group on a dashboard. So Nightingale’s dashboards can interact with business groups.

Thanks to Many Trusting Companies

Nightingale has many enterprise users. Below is a selection, in no particular order.

Open Source License

The Nightingale project is open-sourced under the Apache License 2.0.