How To Use Alert Message Templates in Grafana

[melori.arellano]

I’ll be away for a few days, but I’ll come back and help. What are you getting in the email alert? There are a few parts to get message templating working, to summarize:

Create the annotation with your alert labels and values in the alert rule
Create the message template that will print the alert data created by the annotation
Configure your contact point to utilize your message template

Can you share a screenshot of the annotation you’ve created, the message template, and the contact point? It’s possible that if all of the steps are correct, and your email message uses the default format, it needs to be set up to use the template.

As a test, I just changed the contact point for the sample alert I’ve been sending to slack and re-routed it to my email contact point. All of the template data came through with no other changes, so this process should work for you.

[milinpatel13298]

Hi @melori.arellano!
Thank you for your reply.
No problem, whenever you can.

I think I have missed the second and third step and will look into that.
However, now I’m curious that even if the default email template is set to send whatever information/annotations are set in the “Add details for your alert” section, should it not process those variables and send them neatly anyways, rather than needing to go through step 2 and step 3?
Also, without step 2 and step 3, should it be expected to see the variable values populated at least when viewing these Summary and Annotations in the alerts UI?

For some reason, I’m not able to upload a screenshot, neither here nor on the GitHub Issues page.

[melori.arellano]

@milinpatel13298 I used the default template for my mail test and it did print the annotations. The difference is that they weren’t customized to use the formatting I specified in the message template.

If there is data from the alert, you should see the same information as preview shows when using {{$value}}.

This table is really helpful: How to template annotations and labels | Grafana documentation

[milinpatel13298]

@melori.arellano Thank you for your response.
Like I mentioned in my earlier response, I did try all of those combinations but now of them work for me.
Could you elaborate what you meant by “Create the message template that will print the alert data created by the annotation”?
Also, I’m still not sure whether I should expect these values to populate in the Alerts UI or just the actual email because either of them is not working for me?

[grant2]

I know your question was directed to @melori.arellano , but here is my take on your question…

@milinpatel13298

Could you elaborate what you meant by “Create the message template that will print the alert data created by the annotation”?

A message template is something like this:

Inside of the above message template, we have {{ .Annotations.summary }} and {{ .Annotations.description }}, which means we will display this data:

This results in a Slack alert that looks like this:

[milinpatel13298]

@grant2
Thank you for your answer. I’m clear now that I have the default message set up currently as I get whatever new annotations I add in my alerts.
It’s just that they don’t seem to be populating the variable values like I have described in my earlier comment (neither in the Alerts UI nor in the email body).

Edit: Based on @mleduc 's post/question I do understand now, that maybe, I should not expect the values to be populated in the alerts UI after all. Is that a safe assumption?

[hect0rg]

Does anyone know how I can alter this to allow me to only iterate through Expression C for example? It is repeating the info three times in my alerts so I am assuming that it is going through each of the variables as it goes through all the values.

{{ with $values }}
{{ range $k, $v := . }}
   Location: {{$v.Labels.location}}
   Variable: {{$v.Labels.variable}}
   Alerting value: {{ $v }}
{{ end }}
{{ end }}

[grant2]

@hect0rg Welcome to the forum. Please post the full output when you click Preview Alerts.

[hect0rg]

Hi @grant2

This is what it looks like.

[grant2]

Maybe try this? (to obtain the values in expression C)

Location: {{ $values.C.Labels.location }}
Variable: {{ $values.C.Labels.variable }}

[hect0rg]

Ok I am not sure what i did differently this time because I had tried that in the past with no luck. This time it worked. Thanks for the help!

[jlu1]

Hello, sorry to insist but I didn’t get any solution for my issue. Could someone please give me a hand with this? I’m still trying to get the “hostname.keyword” and “remoteEmail.keyword” values from my classic condition output inside the labels field, but I can’t make it work.

After being able to fetch these values I think all my problems will be solved.

I appreciate any help, best regards.

[grant2]

@jlu1
Please post the full output when you click Preview Alerts.

[jlu1]

Hi grant, thanks for the quick answer. I actually got a workaround using the reReplaceAll function. My issue now is that I’m not receiving any data in Slack.
If I use the default message I get the information I’m looking for (leaving Text body empty):

**Firing**
Value: [ metric='value' labels={hostname.keyword=xx, remoteEmail.keyword=xx@xx.com} value=32 ], [ metric='value' labels={hostname.keyword=xx, remoteEmail.keyword=xx@xx.com} value=8831 ], [ metric='value' labels={hostname.keyword=xx, remoteEmail.keyword=xx@gmail.com} value=22 ]
Labels:
- alertname = Failed Logins
- notification = loginFailedSlack
Annotations:
- length =
[  Server: xx, User email: xx@xx.com, Failed logins: 32 ]
[  Server: xx, User email: xx@xx.com, Failed logins: 8831 ]
[  Server: xx, User email: xx@gmail.com, Failed logins: 22 ]
Source: http://grafana.staged-by-discourse.com/alerting/gvUNplWVk/edit
Silence: http://grafana.staged-by-discourse.com/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DFailed+Logins%2Cnotification%3DloginFailedSlack

And I want to get the Annotations “length” part, so I create a template as:

{{ define "myalert" }}
  {{ .Annotations.length }}
{{ end }}

And my text body as:

Alert summary:
{{ template "myalert" . }}

But I’m receiving an empty message. As you can see from the message above this Annotation is not empty, and here I also show it to you:

Thanks for the help

[grant2]

What do you have in your Slack contact point boxes shown below?

[jlu1]

Here’ the title and the text body.

[grant2]

Try this…

1. Remove the Title completely (should be blank like in my screenshot)
2. Change the Text Body to:

{{ template "myalert" . }}

[jlu1]

I got the same result unluckily

[milinpatel13298]

@milinpatel13298

[ var=‘last_process_count’ labels={process=process1, server=SANDBOX} value=7 ], [ var=‘server_health_condition’ labels={process=process1, server=SANDBOX} value=0 ]

[ var=‘last_process_count’ labels={process=process2, server=SANDBOX} value=7 ], [ var=‘server_health_condition’ labels={process=process2, server=SANDBOX} value=0 ]

@melori.arellano @grant2 Sorry for the bother again, I still haven’t been able to figure this out.
What should ideally work with the above generated alert labels in my Alerts preview.
Also, I was able to get our contact point configuration from my admin and here is what that looks like:

Message - empty
Subject - {{ template “default.title” . }}
Is any change recommended in this config, specifically the Message field?

[miguel12347]

@melori.arellano @grant2 Hello, could the templates be used to send the content of a log in json format? I want to notify whenever there is an error type log and send the content of the error to Slack. I have my logs centralized in loki

this would be the content of my log

If you could help me please, I’ve been researching about it for days but I have no result