The mute rules in Nightingale monitoring (menu entry: Alert - Rule Management - Mute Rules TAB) are usually used for the following scenarios:

• Mute expected alerts in advance, typically for some maintenance actions, such as restarting a machine, mute the related alerts of this machine in advance
• Some issues cannot be fixed in a hurry. They are already known, and continuous alert notifications are meaningless. Just temporarily mute them.

Principle

After an alert event is generated by the alert engine, before being persisted to the DB, it will first go through the judgment of the mute rule. If it matches the mute rule, it will not be persisted to the DB and cannot notify the user. The working timing is shown in the figure below:

A mute rule essentially configures a bunch of filter conditions to filter out the alert events you want to mute. How to filter? Obviously, filter based on the attributes and labels of the alert event. For example:

• Which data source generated the event
• The severity of the event
• The labels of the event

For example, the following example:

• Data source type: Prometheus. Only alert events with data source type Prometheus will be muted.
• Data source: not configured, indicating no restriction
• Event severity: all three levels are checked, indicating that alert events at all levels will be muted
• Event labels: two labels are configured, equivalent to: ident in ("10,1.2.3", "10.1.2.4") and rulename =~ "down"

All of the above filter conditions are overall in an and relationship, that is, only when all conditions hit a certain event, that event will be muted.

FAQ

1. I have configured a mute rule, why can the related alert events still be seen?

Usually because: the event was generated first, and then the mute rule was configured. Mute rules are remedies after the fact and cannot affect events that have already been generated.

2. Multiple conditions in event labels are also in an and relationship, but users don’t understand

As shown in the figure below, the user configured two items in the event label filter, both with the label key being ident:

The user’s intention is that either of 10.1.2.113 and 10.1.2.114 should be muted, but contrary to expectations, this is an and relationship, equivalent to: ident = "10.1.2.113" and ident = "10.1.2.114". Obviously, this condition will never hit any event. In fact, the user should use the in operator, as shown below:

3. The effective scope of mute rules is limited to the current business group

This is actually prompted on the page. To avoid mis-operation, the effective scope of mute rules is limited to the current business group. That is, mute rules can only mute alert events under the current business group, and alert events under other business groups will not be affected.

That is: if the mute rule and the alert rule are under different business groups, the mute rule will not take effect on it.

If mute rules were globally effective, it would be quite dangerous. For example, if a user casually configured a mute rule whose filter conditions match all alert events, then all alert events of the company would be muted.