How To Use Alert Message Templates in Grafana

Hello there, I hope you are doing fine. I’ve been following your methods, however I am not able to get certain values fetched.

Here’s a little context for you.

Problem: Receiving alerts on google hangouts, however certain values are empty.

Fig:

Alert notification (Google Groups Chat)

As you can see in alert notification received, I am not getting instance and job labels, however I do get those in the value array. What am I doing wrong here?

Thanks

@milinpatel13298 the Message field should have a reference to the message template you created. If you refer back to the solution for this post, there’s an example of how to do that. The example is for slack, but the syntax will be the same for email.

The default alert template will print Annotations, so that is why you’re seeing them without your customized alert template.

From the screenshot, it looks like you created the “myalert” template which formats a single alert, but not the “mymessage” template which will iterate through and print “my alert”. Can you try creating the “mymessage” template from the solution post and referencing it in your slack contact point?

Hi @melori.arellano i have gone through this doc for my alert it is unfortunate that im not able to get result from my alert can you help me out plz.

below i have provided details.

Here is my query preview

Screenshot from 2022-11-16 17-39-01

And here is the result im getting on slack
Screenshot from 2022-11-16 17-42-49

can you plz help.

And here is the test result from slack.

**Firing**
Value: [ var='B0' metric='value' labels={labels.client_id=119049, labels.job_id=483374} value=1 ], [ var='B1' metric='value' labels={labels.client_id=164840, labels.job_id=1779411} value=1 ], [ var='B2' metric='value' labels={labels.client_id=154959, labels.job_id=1779374} value=1 ], [ var='B3' metric='value' labels={labels.client_id=147525, labels.job_id=1779369} value=1 ], [ var='B4' metric='value' labels={labels.client_id=154959, labels.job_id=1779347} value=1 ], [ var='B5' metric='value' labels={labels.client_id=170033, labels.job_id=1779262} value=1 ], [ var='B6' metric='value' labels={labels.client_id=170033, labels.job_id=1779205} value=1 ], [ var='B7' metric='value' labels={labels.client_id=154959, labels.job_id=1779202} value=1 ], [ var='B8' metric='value' labels={labels.client_id=170033, labels.job_id=1779195} value=1 ], [ var='B9' metric='value' labels={labels.client_id=170033, labels.job_id=1779190} value=1 ], [ var='B10' metric='value' labels={labels.client_id=170033, labels.job_id=1779188} value=1 ], [ var='B11' metric='value' labels={labels.client_id=165276, labels.job_id=1779165} value=1 ], [ var='B12' metric='value' labels={labels.client_id=170033, labels.job_id=1779155} value=1 ], [ var='B13' metric='value' labels={labels.client_id=67722, labels.job_id=1532293} value=1 ], [ var='B14' metric='value' labels={labels.client_id=67722, labels.job_id=1532292} value=1 ], [ var='B15' metric='value' labels={labels.client_id=67135, labels.job_id=1532209} value=1 ], [ var='B16' metric='value' labels={labels.client_id=87740, labels.job_id=1232704} value=1 ], [ var='B17' metric='value' labels={labels.client_id=111156, labels.job_id=1232666} value=1 ], [ var='B18' metric='value' labels={labels.client_id=79632, labels.job_id=1141244} value=1 ]
Labels:
- alertname = TrendHarvestAlert
- severity = warning
Annotations:
- AlertValues = {{ with $values }}
{{ range $k, $v := . }}
  Location: {{ index $values.B0.Labels “labels.client_id” }}
  Alerting value: {{ $v }}
{{ end }}
{{ end }}

i just want to get client_it and job_id with value on alert

Do I understand that all of the above applies only to Slack? Props to melori.arellano for taking the time, and did a good job of documenting the templating for Slack. In the forums. But this 79 posts five-month back-and-forth topic needs to be consolidated and put into documentation.

What about the sixteen (16) OTHER Contact Point types that AM supposedly supports? Where is the documentation for those, Grafana? And why do 16 different CP’s have different ways of templating? :face_with_raised_eyebrow: You are asking people who need to send to multiple CPs to learn different syntax for each one.

If Grafana does not document HOW to configure for the other Contact Points, including which code to put into which Alert field or Template field, it seems to me that it only theoretically supports them. In practical terms, Grafana does not, in truth, support them, and is padding the numbers of supported platforms, and wasting a lot of users time.

5 Likes

Great info. Would you happen to know how to get rid of the very top text in the Slack alert? My alerts look slightly similar, and the top text is too redundant for me. I’m not sure which portion of the alert it is to make it blank.

I had the same question…I cannot seem to customize this text, plus there are always two rows that contain the same text:

image

Compare the above with Slack alerts coming from InfluxDB:

image

@melori.arellano boa tarde, tudo bem?
Você pode dá um exemplo como montar mensagem de alerta para o Google
Hangouts Chat ?
Desde ja agradeço

@melori.arellano @grant2 Do you know of a solution for this one? The issue is with Classic Condition where you don’t know all of the keys e.g. B0 in advance

Also where does the “with $values” syntax come from? I am not able to find “with” in golang, Grafana or Prometheus docs

@melori.arellano Thank you for your reply. I was able to make this work following your steps (successfully receiving the emails now). I do have one final question - is it currently possible to format the Annotations (newline, etc). I tried to use some HTML syntaxes but they didn’t work.

alert-email

@melori.arellano @grant2
Actually, one more question - I have also realized if I choose to templatize my alert messages like the steps mentioned above, I do not have the ability to send other default annotations in the email anymore (Summary, description, Runbook URL, etc). Am I understanding this correctly?

@milinpatel13298

Yes, if you create a custom template for your alert messages, you should be able to include (in that custom template) the default annotations like Summary, Description, SilenceURL, etc.

@grant2
Thank you for replying so quickly.
Would that be by modifying this to add {{ .Annotations.Summary }} and such? If so, how are you able to handle the space in “Runbook URL”?

1 Like

Here is my template:

{{ define "slack.body" }}
{{- range .Alerts -}}
{{ if gt (len .Annotations) 0 }}
*Summary*: {{ .Annotations.summary }}
*Description*: {{ .Annotations.description }}
{{ end }}
{{ end }}
{{ end }}

For Runbook URL, I have never used/tried that but acc. to this, the syntax is runbook_URL and it should take the URL defined here (note that there is no _ in this syntax).

@grant2
Thanks again!
That’s interesting!
So, then theoretically, I should be able to use the label/variables in my Summary and Description annotations and then bring them into my email message without having to creating a custom AlertValues annotation from the example shown by @melori.arellano, and only needing to create a myalert and mymessage template instead ?

1 Like

@milinpatel13298 I posted a similiar question here. Give me a hint If you succed. Thanks.

Hi @grant2
I was trying to test this so I went and created a Grafana Cloud instance for myself since I do not have admin privileges at work. I have realized that it is possible to parse label values without even needing to create any custom message templates. All I did was use the variables in the default Description annotation (something like below)

{{ with $values }}
{{ $values.B }} value found for {{ $values.B.Labels.state }}. Please investigate.
{{ end }}

This is the preview Alerts output:
[ var=‘B’ labels={state=AL} value=-0.72494383 ], [ var=‘C’ labels={state=AL} value=1 ]

Now, the only difference I could tell between these two Grafana instances is that one is “Cloud Pro” and the other is “Open Source” and the other difference is that the Cloud Pro is v9.2.7-0f96ed148c (0f96ed148c) while the Open Source is v9.1.5 (df015a9301).

Do you know if they fixed this in v9.2.7 (therefore, being a version issue) or is this a cloud vs open source issue?

Also, tagging @melori.arellano in case she has an answer and @georgerobinson since I noticed he is working on some documentation improvements related to this.

I really hope that it’s a version issue so I can just upgrade and have the issue resolved rather than having to create these custom message templates since they don’t have much formatting options like the way the default alert messages look.

Hi! I’m afraid the preview button cannot be used to preview custom labels and annotations (with or without templating) because the preview button is an estimation of the alert condition in the user interface, and doesn’t use alerting to expand custom labels and annotations. We know that this is an issue and are planning to address it soon.