夜莺-Nightingale
夜莺V6
项目介绍 架构介绍
快速开始 快速开始
黄埔营
安装部署 安装部署
升级
采集器 采集器
使用手册 使用手册
API API
数据库表结构 数据库表结构
FAQ FAQ
开源生态
Prometheus
版权声明
第1章:天降奇兵 第1章:天降奇兵
第2章:探索PromQL 第2章:探索PromQL
第3章:Prometheus告警处理 第3章:Prometheus告警处理
第4章:Exporter详解 第4章:Exporter详解
第5章:数据与可视化 第5章:数据与可视化
第6章:集群与高可用 第6章:集群与高可用
第7章:Prometheus服务发现 第7章:Prometheus服务发现
第8章:监控Kubernetes 第8章:监控Kubernetes
第9章:Prometheus Operator 第9章:Prometheus Operator
参考资料

Nightingale allows you to customize your notification content using notification templates when alerts are triggered.

This article provides a detailed guide on configuring notification templates using DingTalk notifications. Before using the notification templates, you need to set up the DingTalk Group Robot and ensure that DingTalk is checked in the notification medium of the alert rules.

Notification Template001

Adjusting Notification Templates

Core Function of Notification Templates

In simple terms, the core function of the notification template is to filter and adjust the generated alert event information and ultimately send the filtered and adjusted alert event information to IM communication tools for display.

Description of Alert Event Fields

By comparing the fields of Flashcat’s AlertCurEvent structure, as shown in the code fields below:

Property Type Description
Id int64 Unique identifier
Cate string Data source type (host, prometheus, etc.)
Cluster string Belonging cluster
DatasourceId int64 Data source
GroupId int64 Business group
GroupName string Business group name
Hash string Event hash value
RuleId int64 Rule
RuleName string Rule name
RuleNote string Rule remarks
RuleProd string Type of the rule (host, metric)
Severity int Event level
PromForDuration int Duration of the rule, in seconds
PromQl string PromQl
RuleConfig string Configuration of the alert rule
RuleConfigJson interface{} Configuration of the alert rule
PromEvalInterval int Execution frequency, in seconds
Callbacks string Callback function
CallbacksJSON []string Callback addresses
RunbookUrl string Runbook link
NotifyRecovered int Whether to notify recovery
NotifyChannels string Notification media
NotifyChannelsJSON []string Notification media
NotifyGroups string Alert receiving group
NotifyGroupsJSON []string Alert receiving group
NotifyGroupsObj []*UserGroup Alert receiving group objects
TargetIdent string Unique identifier of the machine
TargetNote string Target remarks
TriggerTime int64 Alert timestamp
TriggerValue string Alert trigger value
Tags string Tags
TagsJSON []string Tags
TagsMap map[string]string Tag mappings
Annotations string Additional information
AnnotationsJSON map[string]string Additional information
IsRecovered bool Whether recovered
NotifyUsersObj []*User Detailed information of notified users
LastEvalTime int64 Last evaluation time
LastSentTime int64 Last alert send time
NotifyCurNumber int Current alert notification send count
FirstTriggerTime int64 First alert trigger time
ExtraConfig interface{} Additional configuration
Status int Status
Claimant string Claimer
SubRuleId int64 Subscription rule
ExtraInfo []string Additional information

Sample Alert Template

Below is a default notification template for DingTalk. The {{.XXX}} represents the variables referenced in the template. By comparing with the AlertCurEvent structure, it can be seen that each field in this structure can be referenced in the alert template. The previous text also mentioned the correspondence between alert information and AlertCurEvent, so you should be able to understand the specific information to be displayed in the alert template.

- **规则备注**: {{.RuleNote}}
{{- end}}
{{- if not .IsRecovered}}
- **当次触发时值**: {{.TriggerValue}}
- **当次触发时间**: {{timeformat .TriggerTime}}
- **告警持续时长**: {{humanizeDurationInterface $time_duration}}
{{- else}}
{{- if .AnnotationsJSON.recovery_value}}
- **恢复时值**: {{formatDecimal .AnnotationsJSON.recovery_value 4}}
{{- end}}
- **恢复时间**: {{timeformat .LastEvalTime}}
- **告警持续时长**: {{humanizeDurationInterface $time_duration}}
{{- end}}
- **告警事件标签**:
{{- range $key, $val := .TagsMap}}
{{- if ne $key "rulename" }}
	- {{$key}}: {{$val}}
{{- end}}
{{- end}}
{{$domain := "http://10.99.1.209" }}   
[事件详情]({{$domain}}/alert-his-events/{{.Id}})|[屏蔽1小时]({{$domain}}/alert-mutes/add?busiGroup={{.GroupId}}&cate={{.Cate}}&datasource_ids={{.DatasourceId}}&prod={{.RuleProd}}{{range $key, $value := .TagsMap}}&tags={{$key}}%3D{{$value}}{{end}})|[查看曲线]({{$domain}}/metric/explorer?data_source_id={{.DatasourceId}}&data_source_name=prometheus&mode=graph&prom_ql={{.PromQl}})

Testing Alert Template

After configuring the DingTalk Group Robot and checking DingTalk in the notification medium of the alert rules.

Notification Template002

Notification Template003

After triggering an alert, you can wait for a moment and receive the alert sent by the DingTalk robot.

Notification Template004

Usage Methods of Different Types of Variables in Alert Template

Numeric Types

For numeric type variables, we can directly reference them and perform mathematical operations or use built-in functions for calculations, for example:

恢复时间:{{timeformat .LastEvalTime}}

In this example, we reference the numeric type LastEvalTime and use the custom function timeformat to convert the timestamp into a formatted time string.

String Types

For array and Map type variables, you can access the elements within them using indices and keys, for example:

监控指标1: {{range .TagsJSON}}
    - {{.}}
{{end}} 
监控指标2:{{$metric := index .TagsMap "__name__"}}
{{if and (eq "disk_free" $metric)  (gt (len .TagsMap) 5) }}
机器:{{index $labels "ident"}} 分区:{{index .TagsMap "device"}}
{{else}}
{{.TagsMap}}
{{end}}

In this example, we use the range function to iterate through the elements of the .TagsJSON array while accessing the value of the Map type variable .TagsMap using the key “name”.

Conditional Statements and Custom Functions

In addition to basic variable types, Go templates also support the use of conditional statements and custom functions. For example, we can use if-else statements for conditional judgment, use comparison operators, and define our own custom functions to extend the functionality of the template.

监控指标2:{{$metric := index .TagsMap "__name__"}}
{{if and (eq "disk_free" $metric)  (gt (len .TagsMap) 5) }}
机器:{{index $labels "ident"}} 分区:{{index .TagsMap "device"}}
{{else}}
{{.TagsMap}}
{{end}}

{{- if .IsRecovered -}}
恢复时间:{{timeformat .LastEvalTime}}
{{- else -}}
触发时间: {{timeformat .TriggerTime}}
触发时值:  {{humanize (formatDecimal .TriggerValue 0)}}
{{- end -}}
发送时间: {{timestamp}}

In the above example, the if-else statement is used for conditional judgment, using eq to check for equality, gt to check if greater than, and using the custom variables $metric and $labels, while timeformat, humanize, formatDecimal, and timestamp are calls to custom functions. Additionally, if you want the generated text to be more compact, you can use the - sign. By placing multiple - signs between if, else, and end, they serve to remove line breaks and spaces between the conditional judgment statements and related content. (Debugging tip: The notification template editor allows for previewing). For more details on Go templates, you can refer to the official documentation.

Appendix

What Alert Event Information is Generated by Flashcat?

This is very simple to check, with just 4 steps:

  1. Use the nc command on the server to start a port.

Notification Template005

  1. Then configure the callback address in the alert rules to the server where nc is located and the started port.

Notification Template006

  1. After triggering the alert, you can view the alert event information on the server side using nc.

Notification Template007

快猫星云 联系方式 快猫星云 联系方式
快猫星云 联系方式
快猫星云 联系方式
快猫星云 联系方式
快猫星云
OpenSource
开源版
Flashcat
Flashcat