Including Query Labels in Messaging Templates

  • What Grafana version and what operating system are you using?
    GrafanaOnline 8.2.2

  • What are you trying to achieve?
    Trying to include Grafana Query Labels in Unified Alerting Message Template

  • How are you trying to achieve it?
    Using the basic Message Template

{{ define "alert" }}
  [{{.Status}}] {{ .Labels.alertname }}

  Labels:
  {{ range .Labels.SortedPairs }}
    {{ .Name }}: {{ .Value }}
  {{ end }}

  {{ if gt (len .Annotations) 0 }}
  Annotations:
  {{ range .Annotations.SortedPairs }}
    {{ .Name }}: {{ .Value }}
  {{ end }}
  {{ end }}

  {{ if gt (len .SilenceURL ) 0 }}
    Silence alert: {{ .SilenceURL }}
  {{ end }}
  {{ if gt (len .DashboardURL ) 0 }}
    Go to dashboard: {{ .DashboardURL }}
  {{ end }}
{{ end }}
  • What happened?
    The labels appearing in the email message are the labels created when creating/editing the alert - not the labels generated by the alert query.

  • What did you expect (want) to happen?
    I want the email message to include the labels from the query as well as the labels defined in the alert.

  • Can you copy/paste the configuration(s) that you are having problems with?
    See above.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?
    Message templating | Grafana Labs
    This resourced provides examples of data that can be inserted into the Alert Messages - but does not provide a comprehensive set of variables which can be referenced by the Go Templating language, and specifically does not mention how to include labels generated by queries as opposed to Labels created for organizing Alerts.

2 Likes

i’m stuck also on this problem. an simple {{ print . }} show all the query labels but i do not know how to access this values :S

volker.

So - I found the option to include a specific message to users that “might” be able to include label data (might as I’ve not yet been able to see the passed message included in the output of a templated message). The message is in an inconvenient and strange place, however.

When creating a notification group there is an option to include a message to the group - and pre-unified alerting documentation indicates the message “can” use template variables. The notification group message option is provided as an example for including variables such as ${}.

I do not quite understand the strategy behind defining a specific message from within a Notification Group. Doesn’t that then mean that each specific query label requires a different notification group?

It would seem to make more sense to have an option within the alert itself for a message body that could picket up by the message template for inclusion in the resulting alert message.
If someone knows what I am missing - I am all ears.

The capability to include label data within alert messages will be even more important as Grafana’s Unified Alerting pairs up with Grafana On Call capabilities announced yesterday.

While Grafana On Call call escalation is a very exciting addition to Grafana Cloud, escalation only works well when escalation messages clearly and succinctly define the issue being escalated.

@skydiablo If I understand correctly, {{print .}} is listing query label names - but not values. Correct? If so, that is further than I’ve been able to get. Could you share more details? Seems to me that you are close. We need something like {{print .}}.value although that seems to be a bit off.

{{ print . }} dump all available variables they pass to the template. so i can see an map with all labels like:

{firing map[alert:site_offline alertname:Standort Offline] map

2021-11-10 10:02:58.903005623 +0000 UTC
m=+163186.232816138 0001-01-01 00:00:00 +0000 UTC
http:///alerting/SrGluVHnk/edit
19370ac361655d63
http:///alerting/silence/new?alertmanager=grafana&matchers=alert%3Dsite_offline%2Calertname%3DStandort+Offline

[ metric=‘ping_response_time{
host=“0.0.0.0”,
instance=“0.0.0.0:48507”,
job=“netzueberwachung-ping”,
latitude=“0.0000”,
longitude=“0.0000”,
name=“FooBar”,
probe=“success”,
timeout=“3”
}’
labels={
name=ping_response_time,
host=0.0.0.0,
instance=0.0.0.0:48507,
job=netzueberwachung-ping,
latitude=0.0000,
longitude=0.0000,
name=Foobar,
probe=success,
timeout=3
}
value=301
], [ metric='ping_response_time{ … //many more elements from the query result

but i do not know how to access this labels ?

That is pretty much what I thought you were doing. I am not that familiar with GO templating and am not sure where you are placing {{print .}}. I will try adding {{print .}} as the only item in the message template and see what results. Let me know if that is not correct.

alerts → contact points → add template:

name: test_message
content:

{{ define "test_message" }}
  {{ print . }}
{{ end }}

alerts → contact points → Contact points:

message:

{{ template "test_message" . }}

this an short description how to get my result, i’m currently use the grafana-default-email contact point.
volker.

1 Like

@skydiablo Perfect - thank you. Testing now. I think we can figure this out.

1 Like

A different thread reminded me of the generic use for the notification policy message block.

The message block in notification policy can be used to send specific text or define the message template to be used for this policy. While not a fan of defining which template to be used at the notification group level, which would leads to the creation of more notification policies for different types of messages. In my mind notification policy would better define who gets messages, handing off message content details elsewhere. Perhaps understanding templating logic can resolve that issue.

But… if the message block in Notification Policy is used to define the policy - how do you add query labels to the alert message itself. Working with @skydiablo and will report out once we figure it out.


@skydiablo I researched message templates yesterday and found this "Default Template in the code for Alerting on Github.

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] 
{{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}
{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}
{{ define "__text_alert_list" }}{{ range . }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}{{ if gt (len .GeneratorURL) 0 }}Source: {{ .GeneratorURL }}
{{ end }}{{ if gt (len .SilenceURL) 0 }}Silence: {{ .SilenceURL }}
{{ end }}{{ if gt (len .DashboardURL) 0 }}Dashboard: {{ .DashboardURL }}
{{ end }}{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }}
{{ end }}{{ end }}{{ end }}
{{ define "default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "default.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "__text_alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "__text_alert_list" .Alerts.Resolved }}{{ end }}{{ end }}
{{ define "__teams_text_alert_list" }}{{ range . }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
{{ if gt (len .GeneratorURL) 0 }}Source: {{ .GeneratorURL }}
{{ end }}{{ if gt (len .SilenceURL) 0 }}Silence: {{ .SilenceURL }}
{{ end }}{{ if gt (len .DashboardURL) 0 }}Dashboard: {{ .DashboardURL }}
{{ end }}{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }}
{{ end }}
{{ end }}{{ end }}
{{ define "teams.default.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "__teams_text_alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "__teams_text_alert_list" .Alerts.Resolved }}{{ end }}{{ end }}

I’m not quite sure I understand all of the nested calls - but activated it with a custom notification policy referencing:

{{template "teams.default.message" . }}

The results were disappointing, not only did they not include the desired values, but the results were not formatted as I would iike.

I did more research this morning and ran into this Community post with the same type issues.
In the post, @mattabrams indicates that message templating only works with multi-dimensional queries. Which seems odd, but could point out a reason we are having issues.

Towards the end of the thread @janczewskimarcin indicates a solution involving:

  • assigning a query label to a defined label in the alert definition
  • referencing the label variable in a message template

I believe the solution also involved the use of a multi-dimensional alert rather than a classic alert as well.

I am going to do some work down that route today to see if I can come up with a solution.

@skydiablo here is the documentation reference regarding variables requiring the use of multi-dimensional queries: Create Grafana managed alert rule | Grafana Labs

This older GitHub post is interesting although hopefully no longer relevant: ngalert don’t work with prometheus datasource · Issue #35540 · grafana/grafana (github.com)

this is currently the case, as far as I understand. will see if the alerting folks can jump in on this thread

@mattabrams Thanks - that is what I am investigating. The interesting thing, though is that all of my alert messages are now broken, where they were working before as classic alerts. By broken, I mean the HTML is not being generated and there are no “buttons” for clickable links. I also realize i have both a “Grafana” alert manager and a “grafanacloud-tenant-ngalertmanager” Alert Manager listed in Contact Points where templates are stored. I’m not sure how long there have been two, nor whether both are valid.

I will have to open a support ticket if I cannot figure out why the original classic alerts are suddenly broken and cannot figure out how to fix them.

Feedback on both variable content (as opposed to variable name) and using message templates from the alert team would be great.

I found this YouTube tutorial very helpful.
For me, using “Query | Reduce | Math” to create the alert was a big help (classic condition doesnt seem to populate some tokens) . This allows {{ $values.B.Value }} and {{ $labels.instance }} to be used in the Contact templates and Alarm Annotations.
I also wrote my own template as the default is fine for debugging, but a bit noisy for production.

2 Likes

I found that same tutorial. I can’t believe it’s pretty much the only thing available to have this make sense. This makes me wanna go back to Grafana 5. :frowning:

1 Like