Overview

Nightingale uses an architecture where alert rules and notification rules are separated:

• Alert rule: Detects metric anomalies and produces alert events.
• Notification rule: Defines the notification channel, recipients, and delivery strategy for alert events.

This design supports per-severity notification strategies. For example: Critical alerts are sent via phone, SMS, and DingTalk; Info alerts are sent only via email.

Architectural Benefits

Compared with early versions that configured notifications directly inside alert rules, the current architecture has the following benefits:

Feature	Description
Alert severity coupled with notification media	When alert suppression is enabled, different severities can be routed to different notification media.
Flexible media integration	Provides generic HTTP and script-based sending; HTTP parameters, headers, and body can be customized.
Centralized notification configuration	Alert rules and notification rules are decoupled — changing the notification strategy only requires editing the notification rule, and the change takes effect for all associated alert rules.
Custom message templates	Multiple message templates can be configured for the same notification medium, satisfying different teams’ needs.

Workflow

The flow for sending alert events is shown below:

Flow explanation:

1. The alert rule detects an anomaly and produces an alert event.
2. The alert event determines its delivery strategy based on the associated notification rule.
3. The notification rule sends the alert via the configured media to the configured recipients.

Multiple alert rules can be associated with the same notification rule, enabling unified management of notification strategies.

Notification Rule Configuration

Configuration

A notification rule supports configuring independent notification media for each alert severity. An example configuration is shown below:

Configuration items:

• Effective severity: Specifies the alert severities for which this medium applies (Critical / Warning / Info).
• Notification medium: Selects the channel used to send the alert.
• Recipients / receiving team: Specifies the individual or team receiving the notification.

Built-in Notification Media

The system ships with the following notification media, ready to use:

Notification Medium Variable Configuration

Use Case

When multiple teams need to use the same type of notification medium but with different parameters (for example, the DBA team and the Big Data team use different WeCom bots), this can be achieved via variable configuration.

How to Configure

Taking the WeCom bot as an example, the built-in WeCom notification medium contains the following two variable parameters:

Variable name	Description
Key	Unique identifier of the WeCom bot
Bot Name	Bot name (for identification / notes)

Reference the variables in the notification medium’s HTTP configuration (the variable definitions are like function parameters; the section below is like the function implementation, which can reference those parameters):

After configuration, each team can simply choose the WeCom medium when creating a notification rule and fill in their own bot Key.

Associating User Contact Information

For scenarios such as SMS and phone that require each user’s personal contact info, the system supports auto-fetching the contact information from the user Profile.

Configuration steps:

1. In the notification medium’s variable configuration, set the contact field to the appropriate type (e.g. Phone).

2. After configuration, the system will automatically:
   • Allow the notification rule configuration page to choose contacts or teams as the recipient.
   • Allow the {{ $sendto }} variable to be used in the HTTP request body or query string to reference the recipient’s contact info.

Note: {{ $sendto }} is a built-in system variable used to dynamically obtain the contact info of the notification recipient.

Media Integration Guide

IM Media Configuration

Please refer to the following documents to integrate IM notification media:

Notification medium	Configuration document
DingTalk	Integrate DingTalk alerts
WeCom	Integrate WeCom alerts
Feishu	Integrate Feishu alerts
DingTalk (with @user)	Integrate DingTalk alerts — configure @user
Multi-medium integrated config	Integrate DingTalk, Feishu and WeCom notifications
Aliyun SMS	Integrate Aliyun SMS

Tip: The WeChat video account “SRETALK” provides video tutorials on Feishu alert integration — search for it to watch.

Email Alert Configuration

Email alert configuration consists of 4 steps:

Step 1: Configure the SMTP server

1. Go to System Configuration > Notification Media menu
2. Select the built-in Email notification medium and click Edit
3. Fill in the SMTP server configuration:

Required parameters:

Parameter	Description
Server	SMTP server hostname or IP
Port	SMTP server port (typically 25, 465 or 587)
Username	SMTP authentication username
Password	SMTP authentication password or authorization code
From	Email address used to send alert emails

Step 2: Configure user receiving email

1. Click the user avatar at the top-right of the page
2. Enter the Personal Info configuration page
3. Fill in the receiving email under contact information

Step 3: Create the email notification rule

1. Go to the Notification Rules menu
2. Click New Notification Rule (or edit an existing rule)
3. Configure the notification rule:

Configuration items:

Item	Description
Effective severity	The alert severities that need email notification
Notification medium	Choose Email
Recipients	The users or teams receiving the alert

4. Click Test notification, pick a historical alert event to test sending
5. Associate the notification rule in the alert rule

Step 4: Customize email message template (optional)

1. Go to the Message Template menu
2. Select the Email message template to edit

Template variables:

Variable	Description
content	Email body content
subject	Email subject

Note: Message templates use Go template syntax — you can customize the email format according to your business needs.

Common Questions

Q1: I configured a notification rule but the alert wasn’t sent. How do I troubleshoot?

A: In this order:

1. Did the alert fire? Check active alerts to see if the event exists.
2. Notification rule binding: In the alert rule editing page, is this notification rule selected in the “Notification Configuration” section?
3. Effective severity: Does the notification rule’s “Effective severity” cover the alert event’s severity?
4. Recipient contact info: In the user’s profile, are the fields for the corresponding channel (email / phone / IM webhook) filled in?
5. Notification record: In the active alert detail drawer, click “Notification record” to see the actual send result.

Q2: I configured a DingTalk bot in the notification medium but it doesn’t receive messages?

A:

• Has the DingTalk bot’s “security settings” been configured with custom keyword / signing / IP allowlist? These must match the Nightingale side.
• Is the bot Webhook URL complete (including the access_token=xxx parameter)?
• Can the Nightingale server reach oapi.dingtalk.com over the network?

Q3: Can different business groups use different notification rules?

A: Yes. Notification rules support an “applicable business group” setting — multiple rules each handle their own business group, and alert events are routed to the matching rule based on business group membership.

Q4: If I modify a notification rule, will already-sent alerts be re-sent?

A: No. Notification rule changes only take effect for newly triggered events — already-sent notifications will not be replayed.

References

• Notification rule details
• Notification medium overview
• Message template
• Alert rules
• Active alerts
• User management (contact info)