Setting PagerDuty Severity from alert labels

Grafana Alerting
,
1

Using Grafana 10.4.1 (AWS Managed Grafana), I am creating an alert and setting a label called ‘severity’ that I want to use in the Contact Point, which sends it to PagerDuty. I expect it to set the ‘Severity’ field for PagerDuty to the value of the label set in the alert. Below are the steps I’ve gone through.

In the alert settings I add a label called ‘severity’ and set it to ‘critical’:

image
image876×575 36.6 KB

Then I create a Notification Template in Contact Points which should return the value of the Alert Label or ‘Info’ as a default if the Alert label is not set.

image
image422×218 3.38 KB

Next, in the Contact Point that is configured to send to PagerDuty, I try to use that template to set the ‘severity’ value.

image
image732×181 12.6 KB

When I test an alert with that label set to “info”, I expect the PagerDuty alert to be set to “Low” urgency based on the settings in PagerDuty where I set the notification policy.

image
image881×730 29.2 KB

But instead, it’s getting set to the (default?) “High” urgency.

image
image1334×737 26.5 KB

When I look at the details of what got sent to PagerDuty, I don’t see the top-level field for ‘severity’. It exists in the labels but I don’t think PagerDuty looks there to determine the urgency of the alert.

Received this event via the events API
{
  "client": "Grafana",
  "client_url": "https://grafana-workspace.us-east-1.amazonaws.com/",
  "contexts": [
    {
      "href": "https://grafana-workspace.us-east-1.amazonaws.com/",
      "text": "External URL",
      "type": "link"
    }
  ],
  "description": "[FIRING:1] TestAlert Grafana info ",
  "event_type": "trigger",
  "incident_key": "obfuscated",
  "service_key": "obfuscated",
  "details": {
    "firing": "
Value: [no value]
Labels:
 - alertname = TestAlert
 - instance = Grafana
 - severity = info
Annotations:
 - summary = Notification test
Silence: https://grafana-workspace.us-east-1.amazonaws.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTestAlert&matcher=instance%3DGrafana&matcher=severity%3Dinfo
",
    "num_firing": "1",
    "num_resolved": "0",
    "resolved": ""
  }
}
2

I’ve modified the Notification Template a bit, as shown below, but it still doesn’t set the Severity field before sending it to PagerDuty.

{{- define "alert.severity" -}}
  {{ range .Alerts }}
    {{- with .Labels.severity }}
      {{ . }}
    {{ else }}
      Info
    {{ end -}}
  {{ end }}
{{- end -}}
3

It doesn’t work due to the wrong hyphen “-” placement. Try this one:

{{ define "alert_severity" -}}
  {{ range .Alerts -}}
    {{ with .Labels.Severity -}}
      {{ . }}
    {{- else -}}
      critical
    {{- end -}}
  {{ end -}}
{{ end -}}

it works for me. Make sure your output in the preview looks like this:

====================================>critical<====================================

Without any additional lines or spaces.