PagerDuty contact point response status 400 Bad Request

Grafana Alerting
1

Hi everyone,
I’m trying to provision PagerDuty contact point using ConfigMap in kubernetes cluster.
I generated integration key in PagerDuty, and deployed the configmap and mount it to Grafana container /etc/grafana/provisioning/alerting/pagerduty-contact-point.yaml.
New contact point has been created in the Grafana, but when I try to test it or send any alert to PagerDuty I got the following error:

logger=alerting.notifier.pagerduty t=2023-02-08T18:11:00.037869248Z level=info msg="notifying Pagerduty" event_type=trigger
logger=alertmanager org=1 t=2023-02-08T18:11:00.723299933Z level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="pagerduty/pagerduty[0]: notify retry canceled due to unrecoverable error after 1 attempts: send notification to Pagerduty: webhook response status 400 Bad Request"

If I create contact point via UI using the same integration key it works good.
ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
     app: grafana
  name: grafana-contact-point
  namespace: my-namespace
data:
  pagerduty-contact-point.yaml: |-
    apiVersion: 1
    contactPoints:
      - orgId: 1
        name: pagerduty
        receivers:
          - uid: pagerduty_cp
            type: pagerduty
            settings:
              integrationKey: $GRAFANA_PAGERDUTY_INTEGRATIONKEY

$GRAFANA_PAGERDUTY_INTEGRATIONKEY - secret value from env vars.
Grafana version 9.1.6

2

I found the solution in this Github thread: [Bug] Notification test fails with PagerDuty integration · Issue #9036 · grafana/grafana · GitHub my problem was that I was using an “API key” when I should have been using a “Integration key” which is found under Integrations > Service Directory:

image
image2358×668 88.9 KB

Might help anyone finding this thread from Google.

1 Like