Nightingale v9 alert mute rules: suppress alert notifications within specified labels / data sources / time windows, handling planned maintenance, known issues, and noise during non-critical periods.

Overview

Mute rules suppress alert notifications within specified labels / data sources / time windows. Events that hit a mute rule will not be pushed and will not appear in the active alerts list, but the alert engine still evaluates them (this does not affect alert state determination itself).

Sidebar path: Alerts → Rule Management → Mute Rules tab, URL /alert-mutes.

Applicable scenarios:

  • Planned maintenance: during upgrade/restart windows, related alerts should be temporarily silent;
  • Known issues being handled: during postmortem or when not yet root-fixed, avoid repeated notifications;
  • Low-priority alerts during non-critical periods: temporarily silence non-core alerts at night / on holidays;
  • Test environment noise: long-term mute high-frequency alerts in dev/test environments, only view them via dashboards.

Key difference from Global Mute (PLUS-only): mute rules are bound to a business group and only take effect within that business group; global mute takes effect across all business groups and is the “big switch” for Admins.

How It Works: a Funnel of Filters

A mute rule works like a funnel — events must satisfy all filter conditions to be suppressed. The logical relationship is business group AND data source type AND data source AND event severity AND all label conditions AND time windowall required.

If an event hits only some of the conditions, the rule does not take effect.

Form Fields

Section Field Required Description
Basic Info Mute Reason No (recommended) Write down “why mute + responsible person + expected recovery time”; this is the title shown in the list
Filter Conditions Business Group Yes Automatically bound to the current business group, not editable
Data Source Type No Empty = no type restriction (Prometheus / Elasticsearch / MySQL …)
Data Source No Empty = all instances of that type
Event Severity No Empty = no severity restriction (S1 / S2 / S3)
Event Labels At least 1 Label matching condition; multiple rows can be added — see “Matching Operators” below
Mute Duration Mute Time Type Yes Fixed time / Periodic time
Start/End Time Depends on type Fixed: choose start/end time window; Periodic: choose weekdays + HH:MM range

Matching Operators (6 Types)

Event label conditions support 6 operators, all targeting the label value:

Operator Meaning Example
== Equality match rulename == Test server multi-business-group alert
=~ Regex match ident =~ flashcat|v63-192.168.0.2
!= Equality non-match (negation) ident != pushgw-flashcat02-192.168.0.2
!~ Regex non-match (negation) ident !~ flashcat (mute all that do not contain the flashcat keyword)
in Belongs to a list ident in (flashcat01-192.168.0.2, pushgw-flashcat02-192.168.0.2)
not in Does not belong to a list ident not in (...)

Multiple event label conditions are AND-combined — an event must satisfy all label conditions to be muted.

Want OR semantics? Split different conditions into multiple mute rules (the event is muted if any rule hits).

Mute Duration: Fixed vs. Periodic

Fixed Time

  • Mute duration: quick options like 1h / 6h / 24h / 7d;
  • Mute start / end time: auto-calculated from the duration; can also be manually edited to any range;
  • The default duration quick option is 1h; for sub-minute precision, manually edit the start/end fields.

Periodic Time

Mute every week at fixed time slots, taking effect long-term. Commonly used for “long-term silence during nightly maintenance windows”, “no noise during weekday lunch breaks”, etc.

  • Choose weekdays (multi-select Monday to Sunday);
  • Choose start/end time (HH:MM of the day);
  • Multiple time slots can be added.

Important: a muted event’s TriggerTime must fall within the mute time range. If the event was triggered before the window but persists into it, it will not be muted — mute looks at the trigger moment.

Hands-on: Temporarily Mute a Known Alert

Classic usage — create a new mute rule with labels pre-filled from an active alert:

  1. Go to Active Alerts → click the event title to open the details drawer;
  2. At the bottom of the details drawer, click the Mute button → it auto-navigates to /alert-mutes/add, with the event’s labels pre-filled into the form’s event label condition;
  3. Adjust the mute duration (default 1h), fill in the mute reason;
  4. Save.

This event and future events matching the same labels will be suppressed.

FAQ

Q1: After saving the mute rule, the corresponding alert event is still in the active alert list — is this normal?

A: Mute rules only take effect on newly triggered alerts. Existing active alerts will not be immediately removed, and will disappear only after they naturally recover. To make it disappear immediately, manually delete the event in the details drawer (note: only do this when you’re sure the metric will no longer be reported, otherwise the rule will produce new events).

Q2: What’s the relationship among mute rules, subscription rules, and global mute?

A:

  • Global mute (PLUS): cross-business-group mute, evaluated first; hits are filtered out;
  • Business-group mute rules (this page): only take effect on the current business group;
  • Subscription rules (docs): the opposite — subscribe to events from other business groups;

The relationship is: global mute filters first, then local mute filters, and finally the remaining events are notified / subscribed to other business groups. Any mute hit suppresses the event; there’s no repeated muting.

Q3: How do I write a periodic mute that spans days (e.g., 22:00 to 06:00 the next day)?

A: Split it into two segments: Monday to Sunday 22:00-23:59 + Monday to Sunday 00:00-06:00. The current UI does not support a single segment spanning days.

Q4: After a mute rule takes effect, will muted events still be evaluated by the alert engine? Will this increase engine load?

A: They are still evaluated — mute rules only filter notifications and event persistence. The engine still runs PromQL every evaluation cycle as usual. Mute imposes virtually zero burden on the engine itself; the difference is only at the event back-end processing step.

Q5: How do I bulk configure mute rules? For example, ops want to mute all core businesses on weekends?

A: There’s no bulk entry in the UI; two approaches:

  • Use the API to write a script for bulk creation;
  • PLUS users can use Global Mute — one rule covers all business groups, much lighter than N business-group mutes.

References

快猫星云 联系方式 快猫星云 联系方式
快猫星云 联系方式
快猫星云 联系方式
快猫星云 联系方式
快猫星云