Nightingale supports integration with various data sources including Prometheus, ElasticSearch, Grafana Loki, etc. After configuring a data source, Nightingale can query and display its monitoring data, and run alerts against it.
Nightingale supports many data sources. Early-supported data sources such as Prometheus, VictoriaMetrics, and ElasticSearch support both querying / charting and alerting. As the project evolved and Nightingale positioned itself as an alerting engine, newer data sources such as ClickHouse, MySQL, and Postgres support alerting only, not querying / charting.
Whether you want to view data from a data source or run alerts against it, you must first configure the data source. In Integrations Center - Data Sources, click Add, choose the data source type, fill in the data source address, username, password, etc., and click Save.
When configuring a data source, besides filling in the connection address, another key point is to choose the associated alert engine. If your data source is in an edge IDC and you have a dedicated n9e-edge there, choose that n9e-edge as the associated alert engine.
In the data source configuration form, each field has a tooltip (the small question-mark icon next to each form field — hover over it to see usage hints), so we don’t repeat them here.
After the data source is configured, go to Data Query - Metrics - Ad-hoc Query to query the TSDB. If you can get data back, the data source is correctly configured.
Common Questions
1. Nightingale’s configuration file config.toml already specifies the data source writer address. Do I still need to configure it again on the web page?
Yes. The writer address in config.toml is for the data forwarding pipeline, whereas the data source configuration on the web page is for querying and alerting. They are different concepts. Furthermore, the writer address should be a remote write address, whereas the data source configuration on the page is usually the data source’s base address. In addition, many users don’t use Nightingale to forward metrics at all, so they have not configured the writer address in config.toml — only the data source on the page.
2. I want to use edge mode to alert on a TSDB in an edge IDC, but the central n9e cannot reach the edge TSDB. Can Nightingale still be used for unified alerting in this case?
Yes. Such an edge TSDB still needs to be added on the page; when adding, choose “Save” instead of “Test and Save”, so that the central Nightingale will not validate connectivity and will save successfully. When configuring the data source, use the TSDB’s internal address, and choose an n9e-edge alert engine that can connect to the TSDB. At runtime, n9e-edge will use the TSDB’s internal address for queries and alerting.
In this scenario the edge TSDB can still alert, but its data cannot be queried from the Nightingale UI. The reason is that UI queries go through the central n9e, and since the central n9e cannot reach the edge TSDB, it cannot query.
