How To Use Alert Message Templates in Grafana

I am attempting to create a message template for my alerts. My goal is a simplified message using labels. I will paste my template below and the message I receive in slack. The ‘variable’, ‘location’, etc are names of labels.

{{ define "Alerting Template" }}
Alert: {{ .Labels.variable }} on {{ .Labels.location  }} is above threshold of {{ .Labels.threshold }}. For more detail, {{ .PanelURL }}.
{{ end }}

This is the output:

Alert: <no value> on <no value> is above threshold of <no value>. For more detail, <no value>.

I’ve tried to follow the default template and the documentation here. Is there something I am missing?

I am currently using OS 8.5 (and will soon be upgrading to 9).

1 Like

welcome to the :grafana: community @baileyheit1!

This is a really common question and I’m still learning how to access the values out of the alert query too. I hope this is a helpful example - I tried to make it very detailed for the benefit of other community users, even though I can tell you already know how to do some of these steps :wink:

With an alert query that looks like this - I created this with our sample TestDataDB datasource and manually added some labels that are similar to the ones you mentioned.

Click on the preview alert button to see the labels and values in the map:

To refer to labels and values from the query add a new annotation to the alert like this. Name the annotation AlertValues

{{ with $values }}
{{ range $k, $v := . }}
   Location: {{$v.Labels.location}}
   Variable: {{$v.Labels.variable}}
   Alerting value: {{ $v }}
{{ end }}
{{ end }}

Here it is in the context of the alert definition:

Then you can add a message template called myalert to create a single alert message and print the annotation. Notice that you will reference the annotation by name (.Annotations.AlertValues) in the sample below:

{{ define "myalert" }}
  [{{.Status}}] {{ .Labels.alertname }} 
  {{ .Annotations.AlertValues }}
{{ end }}

Here it is in context of the UI:

Next, you can add a message template for the full message, I called this one mymessage. Notice how the mymessage template references the myalert template that I created in the last step:


{{ define "mymessage" }}
  {{ if gt (len .Alerts.Firing) 0 }}
    {{ len .Alerts.Firing }} firing:
    {{ range .Alerts.Firing }} {{ template "myalert" .}} {{ end }}
  {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
    {{ len .Alerts.Resolved }} resolved:
    {{ range .Alerts.Resolved }} {{ template "myalert" .}} {{ end }}
  {{ end }}
{{ end }}

Lastly, you can add the mymessage template to the body of your slack contact point under Optional Slack Settings:

Alert summary:
{{ template "mymessage" . }}

In the UI, it looks like this:
Screen Shot 2022-07-20 at 5.22.40 PM

Now you should have an alert to your slack contact point that looks something like this:

A side note related to using Alert.Labels . . .

Something that can get confusing is that .Labels mentioned in the Template Data docs refer to the labels attached to the alert object NOT the query response labels. For those, we need to get into the values map in the exercise above:

4 Likes

Thank you very much! It works like expected but it’s pitty that the documentation is not very clear :confused:

1 Like

this is awesome! isn’t it odd that templates are under contact points?

Hi, thanks for your detail instructions. those helps me a lot.
However, I would like to send the alert to webhook which did not have any option to pairing with custom message/template that we already created. I have seeking many threads or in config file but still did not get any idea to use custom template into webhook alert (not like optional slack settings).
Have you try with webhook? thanks in advance

@melori.arellano do you know what tweaks are needed when not using a classic query as the current example returns two results with a reduce and use C for results alert:

I have tried $values.C in the alert in place of $values but the results in the email just ignoring the options:

Thanks

@ardianre these instructions are meant to be for slack. The webhook has a different format the json body sent uses the valueString property for the metric values returned by the alert: Webhook notifier | Grafana documentation

@mrdibbley Can you share an example of the reduce condition? I tried an example that uses input C as the alert condition and it still worked for me:

When I preview the alert, the values are in var=B even though it references the query result C:

My alert returns only the results of expression B (based on query C):

Screen Shot 2022-08-01 at 7.20.55 PM

@melori.arellano sure I’ve got it working with the following:



Thanks

@mrdibbley thank you for sharing this! Can you also show me what shows in the box when you click Preview Alerts button? That will show me what the values look like.

@melori.arellano sure here you go:

Thanks

1 Like

@mrdibbley I’m just getting back from some time away. Just to confirm - is this working for you now using $values.B?