Nightingale v9 alert mute rules: suppress alert notifications within specified labels / data sources / time windows, handling planned maintenance, known issues, and noise during non-critical periods.
Overview
Mute rules suppress alert notifications within specified labels / data sources / time windows. Events that hit a mute rule will not be pushed and will not appear in the active alerts list, but the alert engine still evaluates them (this does not affect alert state determination itself).
Sidebar path: Alerts → Rule Management → Mute Rules tab, URL /alert-mutes.
Applicable scenarios:
- Planned maintenance: during upgrade/restart windows, related alerts should be temporarily silent;
- Known issues being handled: during postmortem or when not yet root-fixed, avoid repeated notifications;
- Low-priority alerts during non-critical periods: temporarily silence non-core alerts at night / on holidays;
- Test environment noise: long-term mute high-frequency alerts in dev/test environments, only view them via dashboards.
Key difference from Global Mute (PLUS-only): mute rules are bound to a business group and only take effect within that business group; global mute takes effect across all business groups and is the “big switch” for Admins.
How It Works: a Funnel of Filters
A mute rule works like a funnel — events must satisfy all filter conditions to be suppressed. The logical relationship is business group AND data source type AND data source AND event severity AND all label conditions AND time window — all required.
If an event hits only some of the conditions, the rule does not take effect.
Form Fields
| Section | Field | Required | Description |
|---|---|---|---|
| Basic Info | Mute Reason | No (recommended) | Write down “why mute + responsible person + expected recovery time”; this is the title shown in the list |
| Filter Conditions | Business Group | Yes | Automatically bound to the current business group, not editable |
| Data Source Type | No | Empty = no type restriction (Prometheus / Elasticsearch / MySQL …) | |
| Data Source | No | Empty = all instances of that type | |
| Event Severity | No | Empty = no severity restriction (S1 / S2 / S3) | |
| Event Labels | At least 1 | Label matching condition; multiple rows can be added — see “Matching Operators” below | |
| Mute Duration | Mute Time Type | Yes | Fixed time / Periodic time |
| Start/End Time | Depends on type | Fixed: choose start/end time window; Periodic: choose weekdays + HH:MM range |
Matching Operators (6 Types)
Event label conditions support 6 operators, all targeting the label value:
| Operator | Meaning | Example |
|---|---|---|
== |
Equality match | rulename == Test server multi-business-group alert |
=~ |
Regex match | ident =~ flashcat|v63-192.168.0.2 |
!= |
Equality non-match (negation) | ident != pushgw-flashcat02-192.168.0.2 |
!~ |
Regex non-match (negation) | ident !~ flashcat (mute all that do not contain the flashcat keyword) |
in |
Belongs to a list | ident in (flashcat01-192.168.0.2, pushgw-flashcat02-192.168.0.2) |
not in |
Does not belong to a list | ident not in (...) |
Multiple event label conditions are AND-combined — an event must satisfy all label conditions to be muted.
Want OR semantics? Split different conditions into multiple mute rules (the event is muted if any rule hits).
Mute Duration: Fixed vs. Periodic
Fixed Time
- Mute duration: quick options like 1h / 6h / 24h / 7d;
- Mute start / end time: auto-calculated from the duration; can also be manually edited to any range;
- The default duration quick option is 1h; for sub-minute precision, manually edit the start/end fields.
Periodic Time
Mute every week at fixed time slots, taking effect long-term. Commonly used for “long-term silence during nightly maintenance windows”, “no noise during weekday lunch breaks”, etc.
- Choose weekdays (multi-select Monday to Sunday);
- Choose start/end time (HH:MM of the day);
- Multiple time slots can be added.
Important: a muted event’s TriggerTime must fall within the mute time range. If the event was triggered before the window but persists into it, it will not be muted — mute looks at the trigger moment.
Hands-on: Temporarily Mute a Known Alert
Classic usage — create a new mute rule with labels pre-filled from an active alert:
- Go to Active Alerts → click the event title to open the details drawer;
- At the bottom of the details drawer, click the Mute button → it auto-navigates to
/alert-mutes/add, with the event’s labels pre-filled into the form’s event label condition; - Adjust the mute duration (default 1h), fill in the mute reason;
- Save.
This event and future events matching the same labels will be suppressed.
FAQ
Q1: After saving the mute rule, the corresponding alert event is still in the active alert list — is this normal?
A: Mute rules only take effect on newly triggered alerts. Existing active alerts will not be immediately removed, and will disappear only after they naturally recover. To make it disappear immediately, manually delete the event in the details drawer (note: only do this when you’re sure the metric will no longer be reported, otherwise the rule will produce new events).
Q2: What’s the relationship among mute rules, subscription rules, and global mute?
A:
- Global mute (PLUS): cross-business-group mute, evaluated first; hits are filtered out;
- Business-group mute rules (this page): only take effect on the current business group;
- Subscription rules (docs): the opposite — subscribe to events from other business groups;
The relationship is: global mute filters first, then local mute filters, and finally the remaining events are notified / subscribed to other business groups. Any mute hit suppresses the event; there’s no repeated muting.
Q3: How do I write a periodic mute that spans days (e.g., 22:00 to 06:00 the next day)?
A: Split it into two segments: Monday to Sunday 22:00-23:59 + Monday to Sunday 00:00-06:00. The current UI does not support a single segment spanning days.
Q4: After a mute rule takes effect, will muted events still be evaluated by the alert engine? Will this increase engine load?
A: They are still evaluated — mute rules only filter notifications and event persistence. The engine still runs PromQL every evaluation cycle as usual. Mute imposes virtually zero burden on the engine itself; the difference is only at the event back-end processing step.
Q5: How do I bulk configure mute rules? For example, ops want to mute all core businesses on weekends?
A: There’s no bulk entry in the UI; two approaches:
- Use the API to write a script for bulk creation;
- PLUS users can use Global Mute — one rule covers all business groups, much lighter than N business-group mutes.