How to map the .Alerts.Firing json object in Grafana Alerting to a Custom structure for PagerDuty Ingestion

Grafana Cloud
1

I am trying to integrate grafana with PagerDuty to present alerting data structure in a particular way.
I am trying to use the Notification Template to emit a custom json structure for the Alert for ingestion into PagerDuty.
The current object that is forwarded to Pagerduty is a firing object, e.g:

{
  "details": {  
    "firing": { alert kv's here }
}

I want to map this in its place:

{
  "payload": {
    "custom_field": { alert kv's here }
}

I have been trying to update the notification template to map these objects but struggling to find the right combination. Any help appreciated.

1 Like
2

Hey @alexralph, did you find a solution for your issue? If yes, how did you do it?

Thanks

1 Like
3

It seems we’re running into the same thing after upgrading from Grafana 8 → 10.

Hard to google on, this one. Did any of you figure it out?

4

Same here! If anyone has solved this I’d appreciate a link or hint :slight_smile:

5

Define all details in PD contact point, e.g.:
image
and of course define notification templates used in those details.

6

Thanks for the response, but there is something deeper going on here that I can’t understand. Grafana is somehow sending the v1 Events API format still. When PagerDuty shows me the message it received it looks like this:

{
  "client": "Grafana",
  "client_url": "http://xxx/grafana/",
  "contexts": [
    {
      "href": "http://xxx/grafana/",
      "text": "External URL",
      "type": "link"
    }
  ],
  "description": "[FIRING:1]  (Test Warning 000000001 Operations B pageops-warning)",
  "event_type": "trigger",
  "incident_key": "xxx",
  "service_key": "xxx",
  "details": {
    "firing": "
Value: [no value]
Labels:
 - alertname = Test Warning
 - datasource_uid = 000000001
 - grafana_folder = Operations
 - ref_id = B
 - target = pageops-warning
Annotations:
 - grafana_state_reason = NoData
Source: http://xxx/grafana/alerting/grafana/bdn3727vgcr28b/view?orgId=1
Silence: http://xxx/grafana/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTest+Warning&matcher=datasource_uid%3D000000001&matcher=grafana_folder%3DOperations&matcher=ref_id%3DB&matcher=target%3Dpageops-warning&orgId=1
Dashboard: http://xxx/grafana/d/21Ln2GUGz?orgId=1
Panel: http://xxx/grafana/d/21Ln2GUGz?orgId=1&viewPanel=14
",
    "lol": "wat",
    "num_firing": "1",
    "num_resolved": "0",
    "resolved": "",
    "severity": "warning"
  }
}

I have Grafana v10.x installed and from what I can tell from the code this format was removed a while ago. For example, EventType/event_type shouldn’t be sent and CustomDetails/custom_details should be.

Did I miss some sort of upgrade step which upgrades the alerting package for Grafana? This is the struct I would expect Grafana to send: alerting/receivers/pagerduty/pagerduty.go at v10.0.x · grafana/alerting · GitHub

1 Like