Overview

Annotation Enrich by Query Data is a way of enriching alerts. After an alert fires, it queries a datasource for related information via an associated query and appends the result to the alert as annotations, providing richer alert data.

• Query returned data: append the data to the alert’s annotation
• No data returned: send the original alert normally

This mechanism enriches alert information so operators can quickly locate the root cause.

Use Cases

Case 1: Metric alert queries DB data

For metric-type alerts, you can query a DB for more information when the alert fires — e.g. query logs from a Doris datasource.

Case 2: Service response timeout alert queries metric data

For application-level 5XX alerts, you can query CPU / memory / host-load metrics when the alert fires and append them to the alert message for faster diagnosis.

Case 3: Log alert queries metric data

For error-log-type alerts, you can query experiential metric data — e.g. query Prometheus metrics that may correlate with the error log.

Configuration

Steps

1. Configure the data query

Select the datasource type and configure the query. The datasource can be selected here, or you can enable “use the alert event’s datasource”. Choosing alert-event mainly makes configuration and preview easier — the query can reference variables from the alert event.

For a Prometheus datasource, an example query:

up{instance=～"{{$labels.instance}}.*"}

For DB datasources, add a limit clause in the SQL and specify the database name. Example:

SELECT * FROM test_db.ewall_event WHERE event_date > '2025-11-20 17:00:01' and event_type != '{{$labels.app}}' limit 2;

The result returned is an array. Strongly recommended: limit the result count in the query, otherwise the resulting alert message will be huge and unreadable.

For Doris queries, strongly recommended: specify a concrete time range — otherwise it’s a full-table scan. Example:

SELECT scope_name,message FROM service_app.app_log WHERE message LIKE '%{{$labels.provider}}%' AND timestamp >= DATE_SUB(NOW(), INTERVAL 2 MINUTE) AND timestamp <= NOW() LIMIT 2

2. Data preview

Here you can see the actually-executed query and the returned data. If the preview shows no data, check whether the variables have been rendered correctly.

3. Data append

The data-append section is for adding the query result to the alert message. Configure the key / value of the data to append. For the value, Go template syntax is supported to combine query results.

Typical configuration:

实例标签: {{ $labels.instance }}
查询结果:{{ range $item := $A }} - {{ $item.handler }} ({{ $item.code }}) {{ end }}

查询1结果:
{{ range $item := $A }} - {{ $item.job }} ({{ $item.value }}) {{ end }}

查询2结果:
{{ range $item := $B }} - {{ $item.job }} {{ toTime $item.timestamp }} ({{ $item.value }}) {{ end }}

$labels refers to labels in the alert event; $A and $B refer to the corresponding query results. The field names in the query result can be found by looking at the data preview above.

4. Test

Test the overall configuration.

Notes

Query performance

• The query runs every time an alert fires — make sure it performs well
• Avoid complex aggregations or large time-range queries

Result count

• Avoid returning too much data, which produces unreadable alert messages

Datasource consistency

• If “use the alert event’s datasource” is enabled, you must ensure that the datasource used in the alert rule matches the datasource type used in the query processor. For example, if the alert rule uses a Prometheus datasource, the associated event processor must also use a Prometheus-type datasource.