Nightingale supports Ad-hoc queries — you can query data source data directly from the UI. The menu is under Data Query: pick Metrics to query metric data, or Logs to query log data.

Metric Queries

Below is a metric query example:

This page is similar to Prometheus’ graph page and supports querying time-series metric data. Of course we have added some enhancements, such as built-in metrics, query history, and so on. The screenshot above shows a range vector in Table view. In that case Nightingale does an extra step: it computes the time delta between consecutive points (the +15 shown on the far right), which helps you spot data loss. For example, if most deltas follow a regular pattern matching the collection interval but two of them are several times larger, that indicates a collection or transport failure.

A common question from newcomers: why is there no data when I first enter this page? That is expected — you need to enter a PromQL query first to see data. The page does not show any data by default. PromQL is prerequisite knowledge for Prometheus and Nightingale. We recommend learning the basics first; see: PromQL tutorial series.

If you use Categraf, try querying the metric cpu_usage_active. If you get data back, the data source is correctly configured. If you use Node-Exporter, query node_load1 instead — if you get data back, the data source is correctly configured.

Log Queries

Log queries primarily support the ElasticSearch data source. When configuring an ElasticSearch data source, the version field can be confusing for many users. If your ElasticSearch is version 6.x, choose 6.0+; if it is 7.x, choose 7.0+; if it is even newer, also choose 7.0+. If you run into incompatibilities, please open an issue.

After configuring the data source, you can query in Data Query - Logs. Below is a log query example:

It looks very similar to Kibana’s log query page. Nightingale supports both querying by index pattern, and querying indexes directly without creating an index pattern (wildcards supported). However, querying indexes directly is not a recommended practice, and this feature may be removed in the future. The query syntax supports both KQL and Lucene (i.e. query string) — concepts that ElasticSearch users will be familiar with, so we won’t elaborate here.

Once the data source is successfully configured, the next step is the main course: configure alert rules to experience Nightingale’s alerting engine.