Nightingale supports log alerting. You can configure alert rules against log data in ElasticSearch, Loki, ClickHouse and other data sources. Nightingale periodically queries the data source and triggers an alert when the data meets the configured threshold. The main difference between log alerts and metric alerts is the way the query conditions are written. All other fields are common, so be sure to read the metric alerting documentation first; we won’t repeat that content here.

How ElasticSearch Alerting Works

ElasticSearch supports several query syntaxes, including DSL, KQL, Lucene, EQL, SQL, etc. As an alerting engine, Nightingale’s essence is to let users configure a query, periodically query the data source, evaluate the returned data against a threshold, and trigger an alert if the threshold condition is met.

The first supported query syntax is Lucene (a.k.a. query_string). After fetching the data, Nightingale performs basic statistics — for example, counting the number of matching log lines, or computing aggregations (average, max, percentiles, etc.) on a specific field. The result is then compared against the user-configured threshold, and an alert is triggered if the threshold is met.

ElasticSearch Alert Configuration

First, configure the data source — i.e. specify which ElasticSearch data sources this alert rule applies to. This is essentially the same logic as for metric alerts.

🟢 In the data source type dropdown, only types for which you have configured data sources are shown. That is, if you have only configured Prometheus data sources, you won’t see ElasticSearch, TDEngine, ClickHouse and other types when creating an alert rule.

Then configure the query and statistics:

First, select an index — wildcards are supported. In the example above I configured fc*. The most important field is the filter condition; I configured message.status:>100, which filters logs whose message.status field is greater than 100. This filter uses Lucene syntax, which differs from Kibana’s query syntax. Logs always have a date field; you need to tell Nightingale which field that is, then configure a time interval. In the example I set 5 minutes, so Nightingale filters the most recent 5 minutes of data based on the date field.

Below that is the statistical analysis method. In the example, count is selected, meaning we count log lines without any Group By.

Finally there is the threshold check, comparing the count result against the threshold. If the condition is met, an alert event is produced. In the example, the threshold is > 0, meaning the alert fires whenever count is greater than 0.

After a short wait, you can see the resulting alert events:

ElasticSearch Filter Conditions

What other syntaxes are available for the filter condition (message.status:>100 in the example above)? Click the small question-mark icon next to the filter field, and a side panel will display sample filter syntaxes. Some typical examples:

• status:active — query records where the status field contains active
• title:(quick OR brown) — query records where the title field contains quick or brown
• author:"John Smith" — query records where the author field contains the exact phrase John Smith
• count:[1 TO 5] — range query, inclusive (i.e. includes 1 and 5)
• date:[2022-01-01 TO 2022-12-31] — date range query
• age:>=10 — numeric comparison, greater than or equal to 10

Note: to avoid mistakes, do not add spaces around the colon : after the field name. Different conditions can be joined with AND, OR — e.g. status:active AND age:>=10. For more syntaxes, see the ElasticSearch official documentation.