The event processor is a concept introduced in Nightingale v8. After an alert event is generated, before sending the notification, you can use event processors to perform additional processing on the alert event. The open-source version supports 5 types of processors: Relabel, Callback, Event Update, Event Drop, AI Summary. Different processors can form a pipeline to flexibly process alert events. Scenarios include:

• Integrate with the internal CMDB to attach richer information to alert events
• Call DeepSeek’s interface to do intelligent analysis on alert events, then attach the analysis result to the alert event
• Send all alert events to your own system, equivalent to mirroring a copy, for subsequent analysis and processing
• Some specific alert events can be dropped, for example, some recovery events don’t need to send notifications

There are multiple concepts involved here: notification rules, event processors (Processor), event processing pipelines (Pipeline). Let me explain briefly:

• A notification rule can configure multiple event processing pipelines (Pipeline), executed in order
• An event processing pipeline (Pipeline) can configure multiple event processors (Processor), also executed in order

From the screenshot above, you can see that the entry menu is at Notification - Notification Rules. There may be many notification rules. When adding or editing a specific notification rule, you can see an Event Processing configuration area where multiple pre-created event pipelines (Pipeline) can be referenced. So where do you add, delete, modify event pipelines (Pipeline)? The entry is hidden — in the small gear on the right side of Event Processing, click to expand a drawer, in the drawer you can add, delete, modify event pipelines (Pipeline).

When creating or editing an event pipeline (Pipeline), a new drawer will be expanded. In this new drawer, you edit the Pipeline. We can configure multiple event processors (Processor) in the Pipeline:

Click Usage Instructions next to the event processor type field to view the usage documentation for the event processor.

Relabel Processor

The Relabel processor is similar to the Relabel operation on monitoring metrics in Prometheus, except in Nightingale, it is the Relabel of alert events. Alert events also have label fields, and there is a need to do some processing on labels, so the Relabel processor is provided here.

For specific usage instructions of the Relabel processor, click Usage Instructions next to the event processor type field on the Nightingale page to view.

Callback Processor

After an event is triggered, Nightingale can notify external systems via Callback, and external systems can perform automated processing based on event content. For example, I have seen some companies develop their own alert notification systems, not using Nightingale’s notification mechanism, but directly sending all alert events to their own systems through the Callback processor.

Here is a simple demonstration:

1. First create a “notification rule”, because the Callback processor belongs to a certain Pipeline, and a Pipeline belongs to a certain notification rule.
2. In the “notification rule”, reference the event processing Pipeline. The Pipeline needs to be created in advance (on the notification rule editing page, click the small gear to the right of the processor to open the drawer, and in the drawer create, edit Pipeline). The screenshot below is the details page of a Pipeline, with a Callback processor inside.

The http://10.99.1.107:8888/print here is a test program of mine that can print received HTTP requests, convenient for demonstration. This program is also an open-source small program, the address is github gohttpd.

After creating the Pipeline, go back to the notification rule page, and in the event processing area, select the Pipeline just created.

Next, you can configure an “alert rule” for testing. Test whether the generated alert can be received by a third-party program.

To see the effect as soon as possible, create an alert rule that is sure to trigger the threshold, then in the notification rule, select the notification rule just created:

Wait a moment and observe whether the program http://10.99.1.107:8888/print receives the callback HTTP request. The result I see in my environment is as follows:

As can be seen from the figure above, the HTTP request contains the information of the alert event. The content is as follows:

{
	"id": 1097371,
	"cate": "prometheus",
	"cluster": "prom",
	"datasource_id": 1,
	"group_id": 2,
	"group_name": "DBA-Postgres",
	"hash": "54f5543591c6dc0e30139cae196a1eee",
	"rule_id": 54,
	"rule_name": "测试事件回调",
	"rule_note": "",
	"rule_prod": "metric",
	"rule_algo": "",
	"severity": 2,
	"prom_for_duration": 0,
	"prom_ql": "cpu_usage_active{ident=\"ulric-flashcat.local\"} > 0",
	"rule_config": {
		"queries": [{
			"from": 0,
			"prom_ql": "cpu_usage_active{ident=\"ulric-flashcat.local\"} > 0",
			"range": {
				"display": "now-undefineds to now-undefineds",
				"end": "now-undefineds",
				"start": "now-undefineds"
			},
			"severity": 2,
			"to": 0,
			"unit": "none"
		}]
	},
	"prom_eval_interval": 15,
	"callbacks": [],
	"runbook_url": "",
	"notify_recovered": 1,
	"target_ident": "ulric-flashcat.local",
	"target_note": "",
	"trigger_time": 1749180264,
	"trigger_value": "33.06867",
	"trigger_values": "",
	"trigger_values_json": {
		"values_with_unit": {
			"v": {
				"value": 33.06867479671808,
				"unit": "",
				"text": "33.07",
				"stat": 33.06867479671808
			}
		}
	},
	"tags": ["__name__=cpu_usage_active", "cpu=cpu-total", "ident=ulric-flashcat.local", "rulename=测试事件回调"],
	"tags_map": {
		"__name__": "cpu_usage_active",
		"cpu": "cpu-total",
		"ident": "ulric-flashcat.local",
		"rulename": "测试事件回调"
	},
	"original_tags": ["", "", "", ""],
	"annotations": {},
	"is_recovered": false,
	"last_eval_time": 1749180264,
	"last_sent_time": 1749180264,
	"notify_cur_number": 1,
	"first_trigger_time": 1749180264,
	"extra_config": {
		"enrich_queries": []
	},
	"status": 0,
	"claimant": "",
	"sub_rule_id": 0,
	"extra_info": null,
	"target": null,
	"recover_config": {
		"judge_type": 0,
		"recover_exp": ""
	},
	"rule_hash": "dc128d86d65326499bd03ecfbe56e4c3",
	"extra_info_map": null,
	"notify_rule_ids": [3],
	"notify_version": 0,
	"notify_rules": null
}

Test is normal. If you have similar needs, you can use this kind of Callback processor to integrate, doing some automated logic in your own program.

Event Update Processor

Among event processors, there is also an Event Update processor. The configuration method is the same as Callback, and the working logic is also very similar. The differences are as follows:

When Nightingale calls the Callback address, it does not pay attention to the HTTP Response. When calling Event Update, it uses the content of the HTTP Response as the new alert event for subsequent processing.

So, Event Update, as its name suggests, is used to modify alert events. It is usually used to attach some additional information to alert events, for example:

• Hand the event to AI for analysis, get some conclusive information, attach to the event
• Query some meta-information in CMDB, attach to the event

Note that the structure of the alert event cannot be modified arbitrarily. For example, directly adding a field at the top level of the JSON will not be recognized by the subsequent process. It is usually recommended to attach new content to the annotations field. In the Callback processor example above, annotations is empty so the data structure cannot be seen. In fact, annotations is a map structure, with both map keys and map values being string types. When you attach content, you also need to follow this structure.

Advanced users can also modify other fields of the Event, but you need to be clear about the impact of your modification on subsequent processing. Ordinary users only need to attach content to the annotations field, then serialize the entire new Event to JSON and put it in the body of the HTTP Response.

Event Drop Processor

Among event processors, there is also an Event Drop processor. As the name suggests, it is used to drop alert events. For example:

• In some scenarios, although alert events are generated, you don’t want to go through the subsequent notification logic. You can use the Event Drop processor to drop this event.

To drop some alert events, you definitely need to filter. Usually, you can use various fields such as labels, annotations, severity to filter. This filtering rule may be very complex. So how to design this feature to be so flexible? We thought of a slightly complex but extremely flexible way: the user directly uses go template syntax to configure a template. In the template, you can reference the alert event and use if and other syntax to filter. As long as the final rendering result of this go template is true, this event will be dropped.

For specific usage instructions, click Usage Instructions next to the Event Drop event processor type field on the Nightingale page to view.

AI Summary Processor

The documentation for the AI Summary processor is also complete, as shown below:

Click Usage Instructions next to the AI Summary event processor type field to view the usage documentation for the AI Summary processor. There is also a small question mark icon to the right of each field below. Hover over it with the mouse to see related prompt instructions.