Overview

Callback is an event processor in the Nightingale alerting system. It lets you send alert event information to an external service via an HTTP call. When an alert event passes through this processor, the event data is POSTed in JSON format to the configured URL, which can notify an external system or trigger an automated workflow. Unlike the Event Update processor, the Callback processor does not modify the event content — it is mainly used for event notification and integration.

How It Works

1. Event received: the processor receives an alert event
2. Serialization: the event object is converted to JSON
3. HTTP call: the JSON is POSTed to the configured URL

Configuration

Basic Configuration

URL (required)

• Description: HTTP endpoint of the external service
• Format: full HTTP/HTTPS URL
• Example: https://your-service.com/api/alert-webhook

Advanced Configuration

Authentication

• Auth username: HTTP Basic auth username
• Auth password: HTTP Basic auth password
• When to use: when the external service requires authentication

HTTP Headers

• Description: custom HTTP request headers
• Format: key/value pairs
• Default: Content-Type: application/json
• Example:

  X-API-Key: your-api-key
  Authorization: Bearer your-token
  X-Custom-Header: custom-value
  

HTTP Proxy

• Description: HTTP proxy server address
• Format: http://proxy-host:port or https://proxy-host:port
• When to use: when the external service must be reached through a proxy

Callback Timeout

• Description: HTTP request timeout
• Unit: milliseconds (ms)
• Default: 10000 (10 seconds)
• Recommendation: tune based on the external service’s response time

TLS InsecureSkipVerify

• Description: whether to skip TLS certificate verification
• Default: off (verify certificates)
• Note: enable only for testing or with private certificates

External Service Interface Spec

Your external service must satisfy the following requirements:

Request format

• Method: POST
• Content-Type: application/json
• Body: a full alert event JSON object

Response format

• Status code: 200 recommended to indicate success
• Body: optional; response content will be logged into Nightingale logs

Event object structure example

{
  "rule_name": "规则名称",
  "severity": 2,
  "tags": [
    "host=server01",
    "service=web"
  ],
  "tags_map": {
    "host": "server01",
    "service": "web"
  },
  "annotations": {
    "summary": "告警摘要",
    "description": "详细描述"
  }
}

Use Cases

Case 1: ITSM ticket creation Automatically create incident tickets in your service-management system.

Case 2: Automated operations Trigger automated remediation scripts or operations actions.

Case 3: Data sync Sync alert data to an external monitoring or analytics system.

Real-World Example

Python service example

Below is a simple example using Python’s standard library, showing how to build an external service that receives Callback requests.

1. Create the Python service file (callback_receiver.py)

from http.server import HTTPServer, BaseHTTPRequestHandler
import json
import logging
from datetime import datetime
from urllib.parse import urlparse

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class CallbackHandler(BaseHTTPRequestHandler):
    def do_POST(self):
        """Handle POST requests"""
        try:
            # Parse request path
            parsed_path = urlparse(self.path)
            
            if parsed_path.path == '/api/alert-webhook':
                self.handle_alert_callback()
            else:
                self.send_error(404, "Not Found")
                
        except Exception as e:
            logger.error(f"Error handling request: {str(e)}")
            self.send_error(500, "Internal Server Error")
    
    def handle_alert_callback(self):
        """Handle alert callback from Nightingale"""
        try:
            # Read request body
            content_length = int(self.headers.get('Content-Length', 0))
            if content_length == 0:
                self.send_error(400, "Empty request body")
                return
                
            post_data = self.rfile.read(content_length)
            event = json.loads(post_data.decode('utf-8'))
            
            print(json.dumps(event, ensure_ascii=False, indent=2))
            
            response = {
                "status": "success",
                "message": "Alert received and printed"
            }
            
            response_data = json.dumps(response, ensure_ascii=False).encode('utf-8')
            self.wfile.write(response_data)
            
        except json.JSONDecodeError as e:
            logger.error(f"JSON parsing error: {str(e)}")
            self.send_error(400, "Invalid JSON")
        except Exception as e:
            logger.error(f"Error processing event: {str(e)}")
            self.send_error(500, "Internal Server Error")
    
    def log_message(self, format, *args):
        """Custom log format"""
        logger.info(f"{self.address_string()} - {format % args}")

if __name__ == '__main__':
    server_host = '0.0.0.0'
    server_port = 5001
    
    print("Alert callback receiver starting...")
    print(f"Webhook endpoint: http://localhost:{server_port}/api/alert-webhook")
    print("Press Ctrl+C to stop")
    print("=" * 80)
    
    try:
        server = HTTPServer((server_host, server_port), CallbackHandler)
        server.serve_forever()
    except KeyboardInterrupt:
        print("\nStopping service...")
        server.shutdown()
        print("Service stopped")

2. Start the service

python callback_receiver.py

After startup, the service listens on http://localhost:5001.

3. Configure in Nightingale

On the Callback processor configuration page, set:

• URL: http://localhost:5001/api/alert-webhook

Differences from the Event Update Processor

FAQ

Q: What if the external service is unavailable?

A: The processor logs an error and continues to pass the original event downstream — the alert pipeline is not interrupted.

Q: Is retry supported?

A: Automatic retry is not supported in the current version; implement retry on the external side if needed.

Q: How do I debug configuration issues?

A: Click the “Test” button on the page to verify the configuration, and check Nightingale logs.

Q: What if response time is too long?

A: Increase the Timeout setting appropriately, or improve the external service’s response performance.

Notes

• Ensure high availability of the external service to avoid impacting alert handling
• Set the timeout reasonably to balance response speed and stability
• Continuously monitor performance and availability of the external service
• Thoroughly test the configuration and the external service before using in production

References

• Event Label Relabel
• Script Execution
• Event Pipelines Overview
• Execution Records
• Notification Channel: Callback