How To Use Alert Message Templates in Grafana

@jlu1 the dot is needed to provide the context.

Are you getting any notifications to slack, even a blank one?

Check out my post at the top of this thread, if you see your label shown in the alert preview output, then you can use it in the alert by following the steps there.

Yes, I receive a blank one if I use the dot.
image

But I couldn’t find a way of retrieving any data.

Hello,

This explanation was very helpful, thanks @melori.arellano !

I would also want to ask is there a relation between - the threshold which we set inside alert rules and the panels inside a dashboard. I am asking because I would like to display the same threshold on both graphs.

Thanks for the guide @melori.arellano, it has got me most the way but now stuck again. How are you handling long value strings or conversions to % when using these templates?

I have an AlertValue that works, but gives me a very long number value and I was trying to trim it down using one of the humanize template functions, but whenever I put the function in the AlertValue it breaks the alert.

{{ with $values }}
{{ range $k, $v := . }}
   <b>Deployment:</b> {{$v.Labels.deployment}} 
   <b>Success Rate (%):</b> {{ $v | humanizePercentage }}
{{ end }}
{{ end }}

I have also tried it as:

{{ humanizePercentage $v }}

Also seems like the latest Grafana release has removed/broken HTML support in the annotations.

@jlu1 The problem could live in two places:

  1. The alert is not generating any values to add into your message template

Check the output of the alert by expanding the alert information on the alerting page, it should show the values that have been captured from the alert. Notice that it shows the annotation configuration and then when you expand the nested alert you’ll see the created labels and the expected value for the annotation:

  1. The message template configuration isn’t capturing the right values from the annotation in your alert. Then you’ll want to try testing the message template itself to make sure that it is properly passed to slack.

Hi @melori.arellano, you are right about the first case - I’m not receiving any data in my “check” Annotation.

Could you please help me on how to extract the “hostname.keyword” and “remoteEmail.keyword” from the Value field? Is this even possible? What would be the way of extracting those values in the case I have an array of these set of values as an output? For instance:
[ metric='value' labels={hostname.keyword=a, remoteEmail.keyword=aaa@aaa.com} value=9999 ], [ metric='value' labels={hostname.keyword=b, remoteEmail.keyword=bbb@bbb.com} value=1111 ]

Thank you very much for for help.

I tested this out on my instance using the sample data in our docs Examples of template functions

humanizePercentage

TemplateString { humanizePercentage $value }. ← this didn’t work, because in the alert template, the value was in $values.B.Value

From my alert preview, I can see that the value is in $values.B.Value

This worked for me:

{{$values.B.Value | humanizePercentage }}

Screen Shot 2022-09-19 at 4.44.33 PM

@dselena this is a good question. With the new unified alerting in Grafana 8+ alerts are decoupled from dashboards so the thresholds don’t transfer over automatically.

1 Like

How does it work with multipe values returned? I have multipe values returned which is why your initial template was so helpful. So of the queries return dozens of results e.g.

[ var=‘B0’ metric=‘server1:9198 / learnapplogs’ labels={instance=server1:9198, job=logstash_exporter, pipeline=logs} value=0.2932612419621986 ],
[ var=‘B1’ metric=‘server2:9198 / learnapplogs’ labels={instance=server2:9198, job=logstash_exporter, pipeline=logs} value=0.2856948269581056 ]

Ideally this would somehow work:

{{ with $values.B.Value | humanize }}
{{ range $k, $v := . }}
Instance: {{$v.Labels.instance}}
Pipeline: {{$v.Labels.pipeline}}
Message Count: {{ $v }}
{{ end }}
{{ end }}

Dear @melori.arellano,

Thank you for your response! For manual set, you mean to create alert and with the same configuration for visual threshold on the panel or there is another way to connect both thresholds manually?

Thank you in advance!

So my alert Preview looks something like the following:

[ var=‘B0’ metric=‘17145’ labels={} value=4 ], [ var=‘B1’ metric=‘10040’ labels={} value=4 ], [ var=‘B2’ metric=‘Machine2’ labels={} value=4 ],

And I want to access the metric variable in the dollar sings area. How would that be possible?

{{ with $values }}
{{ range $k, $v := . }}
$$$$$$$
Alerting value: {{ $v }}
{{ end }}
{{ end }}

Hello @melori.arellano

Using custom labels break the HTML formatting for the email notifications in Grafana alerting. This is apparently a known issue discussed here.

Would you by any chance have any suggestions as to a potential workaround, other than what is discussed in the link above (editing the HTML template on Grafana server).

The problem we’re having is that we have data we want to suppress in the email message, such as:

Value: [ var='B' labels={instance=<XXXX> } value=53.46666666679084 ], [ var='C' labels={instance=<XXXX>} value=1 ]

Thanks!