Overview

PagerDuty is the most popular alert response / On-Call management platform overseas. Nightingale delivers alerts to PagerDuty via Events API v2, and PagerDuty handles on-call scheduling, escalation, mobile push / phone / SMS notifications, and the full incident lifecycle.

Nightingale’s PagerDuty notification channel uses two types of credentials at the same time:

1. API Key (REST API Access Key) — configured on the notification channel. Its purpose is so that “while you are creating a notification rule, Nightingale can call the PagerDuty REST API to list all Services in your account”, which lets you pick from a dropdown;
2. Integration Key (Routing Key) — belongs to a specific Service → Integration, and is the credential that actually authenticates events delivered to Events API.

When you create a notification rule, Nightingale uses the API Key from step 1 to fetch the Integration Key automatically — you only need to tick the target Service in the dropdown, no need to manually paste the 32-character routing key.

• Use case: an overseas team uses PagerDuty for on-call scheduling/escalation; or multi-cloud / multi-monitoring setups want to consolidate to PagerDuty for unified post-incident reporting.
• Prerequisites:
  1. A PagerDuty account, with at least one Service, and at least one Events API v2 integration under that Service;
  2. A PagerDuty REST API Key (user-level or account-level).
• The whole setup has three steps: (1) Create Service + Integration + API Key in PagerDuty → (2) Create a PagerDuty notification channel in Nightingale → (3) Pick the target Service in the notification rule.

Step 1: PagerDuty side

1.1 Create the Service and Integration

1. Sign in to PagerDuty (https://<your-subdomain>.pagerduty.com).
2. Go to Services → Service Directory → New Service, and follow the guide:
   • Name (such as Nightingale);
   • Escalation Policy: pick an on-call policy;
   • Integration: pick Events API v2;
3. Once created, open the Service. In the Integrations tab you’ll see the integration you just created. Copy its Integration Key (Routing Key, 32-char hex) — you don’t need to paste this into Nightingale yourself, but a screenshot helps for troubleshooting.

1.2 Create a REST API Access Key

1. Click your avatar at top-right → My Profile → User Settings → API Access → Create API User Token.
   • If you are an account admin, you can also go to Integrations → API Access Keys → Create New API Key to create an account-level Key;
   • Permission: read-only is enough (Nightingale only calls the /services / /integrations list endpoints).
2. Copy the generated API Key (looks like y_NbAkKc66ryYTWUXYEu, 20 characters).

Warning: this API Key is NOT the Integration Key (Routing Key).

• The API Key is so Nightingale can fetch the Service list for you;
• The Routing Key is the “secret” used per event delivery; Nightingale fetches and uses it for you automatically inside the notification rule.

Step 2: Create a PagerDuty notification channel in Nightingale

1. Sign in to Nightingale → left menu Notification → Notification Channels.

2. In the type panel on the left, click PagerDuty (URL /notification-channels/add?ident=pagerduty).

3. Fill in the form:

   

   Section	Field	Notes
   Basic	Name	e.g. PagerDuty
   PagerDuty	API Key	Paste the REST API Access Key from Step 1.2
   PagerDuty	Proxy	If Nightingale is deployed in an internal network and needs an HTTP proxy to reach api.pagerduty.com / events.pagerduty.com, set the proxy address here
   PagerDuty	Timeout	Default 5000 ms
   PagerDuty	Retries	Default 3; transient network issues are retried automatically

4. Click Save.

PagerDuty does not need URL or request body configuration — the event delivery URL is fixed at https://events.pagerduty.com/v2/enqueue, and the request body structure is built internally by Nightingale (not from a user-defined HTTP template).

Step 3: Pick the Service in the notification rule

1. Left menu Notification → Notification Rules → New (or edit an existing rule).

2. In the “Notification Configuration” area:

   • Notification Channel: pick the PagerDuty channel you just created;
   • Services: a multi-select dropdown in the form Service Name / Integration Name. Nightingale uses the API Key on the channel to fetch all Services + Integrations visible to your account, ready for you to pick;
     • Once picked, Nightingale calls the API again to fetch the corresponding Integration Key automatically, used for sending events later;
     • If the dropdown is empty: check the API Key validity and that the network can reach api.pagerduty.com (see Q1).
   • Applicable Severity / Time Range / Tags: filter as needed.

3. After saving, click “Notification Test” — a fresh incident should immediately appear in the PagerDuty console, and it will be auto-resolved once the alert recovers.

What the event looks like inside PagerDuty

How Nightingale maps event fields to PagerDuty fields:

Nightingale field	PagerDuty field	Notes
Non-recovery event	event_action: trigger	Triggers a new incident
Recovery event	event_action: resolve	Resolves the incident matching dedup_key
$event.Hash	dedup_key	Alert fingerprint; same fingerprint trigger then resolve closes it
$event.RuleName	payload.summary	Incident title
$event.Cluster	payload.source	Source
$event.Severity	payload.severity	1→critical, 2→error, 3→warning; critical triggers phone escalation
$event.Target	payload.component	Component, useful for dispatching
$event.TagsJSON / AnnotationsJSON etc.	payload.custom_details	All tags, annotations, rule_id, rule_note, prom_ql, datasource, etc. passed through as-is
Nightingale event detail page link	links	The Incident detail page can jump back to Nightingale

FAQ

Q1: The Services dropdown is always empty?

A:

• Wrong / disabled API Key: regenerate one in PagerDuty;
• No network connectivity: the Nightingale server must be able to reach api.pagerduty.com (China-region cloud hosts are often blocked and need a proxy);
• Insufficient permission: the user behind the API Key must at least have access to GET /services and GET /services/{id}/integrations. The default User API Token is enough;
• Pagination hit by too many Services: Nightingale paginates automatically. If the list is still incomplete, check whether some Services in PagerDuty are scoped to a Team that the API Key’s user is not a member of.

Q2: The alert was triggered but no Incident appeared in PagerDuty?

A:

• In Nightingale’s “Notification Logs”, check the Response of that event. Common results:
  • status_code:202, response:{"status":"success",...} → success;
  • status_code:400, response:{"errors":["Event object is invalid"],...} → invalid event body (usually a Nightingale bug — please file an issue);
  • status_code:401/403 → wrong routing key or the Integration is disabled.
• Check that the Service’s Integration is Active is checked in PagerDuty.

Q3: The alert recovered but the PagerDuty incident is still open?

A: Resolve relies on dedup_key matching, and Nightingale uses $event.Hash as the dedup_key. If:

• The recovery event’s Hash differs from the trigger event (the alert rule ID or dimension labels changed), PagerDuty cannot find the matching incident;
• The trigger event hadn’t been processed by PagerDuty when the resolve was sent — occasionally producing an out-of-order “resolve before trigger”. Nightingale already orders by event time to minimize this.

Q4: A critical alert didn’t trigger phone escalation?

A: PagerDuty by default only does phone/SMS escalation on severity: critical. Nightingale’s mapping:

• Nightingale Severity 1 → critical (yes);
• Nightingale Severity 2 → error (email/App push);
• Nightingale Severity 3 → warning. If you want Severity 2 to also escalate, in PagerDuty’s Event Rules, manually rewrite error-severity events to critical.

Q5: Can a single notification rule deliver to multiple Services at once?

A: Yes. The Services dropdown is multi-select; each selected Service receives its own copy of the event (each routing_key triggering its own Service’s escalation policy). Note that PagerDuty bills by event count.

Q6: Pricing?

A: PagerDuty bills by licensed users (Seats) + event usage, with 600 events per month on the free tier. See PagerDuty Pricing. For high alert volumes, prefer doing aggregation/suppression on the Nightingale side before egress.

Q7: PagerDuty is slow from a Mainland China account?

A: PagerDuty service nodes are mainly in the US/EU. China-region cloud hosts have high latency or packet loss when connecting directly. We recommend:

• Configure an outbound proxy on the channel form (such as your company’s overseas HTTP proxy);
• Or deploy a forwarding node overseas.

References

• PagerDuty Events API v2 Docs
• PagerDuty · REST API Access Keys
• PagerDuty · Services and Integrations