Inhibit by Query Data — an advanced inhibit mechanism that runs a data query when an alert fires and inhibits the alert based on the query result.
Overview
“Inhibit by Query Data” is an advanced alert-inhibit mechanism. When an alert fires, the system first runs the configured data query and decides whether to inhibit the alert based on the result:
- Result has data: inhibit the alert, no notification sent
- No result: send the alert notification normally
This makes the inhibit strategy more flexible and intelligent, helping to avoid redundant alerts under specific conditions.
Use Cases
Case 1: Inhibit based on real-time state
When a service is under maintenance or has a known issue, query the maintenance marker data to inhibit related alerts.
Example: if the monitoring system contains the metric maintenance_mode{service="app"} with value 1, inhibit all alerts for that service.
Case 2: Cascading inhibition based on dependencies
When an upstream service is already alerting, inhibit related alerts on downstream services to avoid alert storms.
Example: when a database server is already alerting, inhibit connection-failure alerts on all application services that depend on that database.
Case 3: Dynamic threshold checks
Combine real-time query data to implement more complex inhibit logic.
Example: inhibit a single-node alert only when other nodes in the cluster are healthy, to avoid wrongly inhibiting alerts during a global failure.
Configuration
Configuration page
Steps
1. Choose the inhibit type
In the alert rule’s inhibit configuration, select “Inhibit by Query Data”.
2. Configure alert filter conditions
Define which alerts should be evaluated for inhibition. Two filter modes are supported:
Label filter
Match by alert labels.
- Label: pick or enter a label name
- Operator:
==(equal),!=(not equal),=~(regex match),!~(regex non-match) - Value: the value to match
Example:
Label: rulename
Operator: ==
Value: 告警
Attribute filter
Match by alert attributes.
Example:
Attribute: 业务组
Operator: ==
Value: n9e
Tip: multiple filter conditions can be added; all of them must be satisfied (AND).
3. Configure the data query
Define the data query to run. If the query returns at least one row, the alert will be inhibited.
Datasource type
Select the datasource type. Currently Prometheus is supported.
Datasource
Select a specific datasource instance.
Query configuration
Prometheus query
Enter a PromQL expression in the “metric” input.
Template variables:
{{$labels.xxx}}: reference a label value from the alert{{$value}}: reference the alert’s triggering value{{$event}}: reference the entire alert event object
Example:
mem_active{ident="{{$labels.ident}}"}
This queries mem_active for the host with the same ident label as the alert event; if data is returned, the alert is inhibited.
4. Save the configuration
After completing the above, click “Save” to apply.
Configuration Examples
Example 1: Inhibit based on host liveness
Scenario: when the host’s mem_active metric has data (i.e. the host is alive), inhibit network alerts for that host.
Configuration:
-
Label filter
rulename == 网络告警 -
Query
- Datasource type: Prometheus
- Datasource: prom
- PromQL:
mem_active{ident="{{$labels.ident}}"}
Logic: if the host’s memory-activity metric can be queried, the host is running normally; the network alert is likely a false positive and is inhibited.
Example 2: Inhibit alerts during a maintenance window
Scenario: inhibit alerts for a service that is under maintenance via a maintenance-mode marker.
Configuration:
-
Label filter
service == web-app -
Query
- Datasource type: Prometheus
- Datasource: prom
- PromQL:
maintenance_mode{service="{{$labels.service}}"} == 1
Logic: if the service’s maintenance-mode marker is 1, inhibit all related alerts.
Notes
Query performance
- The query runs every time an alert fires — make sure it performs well
- Avoid complex aggregations or large time-range queries
Template variables
- Template variables are replaced with actual values when the query runs
- If the label is missing,
{{$labels.xxx}}is replaced with an empty string - Confirm in advance that the alert event contains the required labels
Filter conditions
- Label filter and attribute filter are combined with AND — all must match
Datasource availability
- If the query fails, the alert is not inhibited and will be sent normally
Best Practices
- Test before going live: use the “Test” feature to verify the configuration works as expected
- Use attribute filters wisely: attribute filters help pinpoint exactly which alerts should be inhibited
- Avoid over-inhibition: make sure the inhibit logic does not accidentally kill important alerts
- Document the configuration: explain the inhibit logic in the rule’s notes for team maintainability
FAQ
Q: Are inhibited alerts saved to history?
A: Yes — inhibited alerts are saved in the historical alert records.
Q: How can I debug the query configuration?
A: Use the “Test” button to simulate an alert and view the query result and inhibit decision.
Q: Will a query timeout affect alert delivery?
A: No. If the query times out or fails, the alert is sent normally and not inhibited.
