Like documented here:
The $labels variable contains the labels from the query. For example, a query that checks if an instance is down might return an instance label with the name of the instance that is down. For example, suppose you have an alert rule that fires when one of your instances has been down for more than 5 minutes. You want to add a summary to the alert that tells you which instance is down.
So I don’t want to edit thousands of alerts to add $labels to some alert label that I can use in a notification template and then track if these are not changed, etc.
How can I access the same content that I’d get from $labels but in the notification template?
For each Alert, I can see .Labels but that’s the alert’s labels, not the label for the metric that triggered the alert in the first place.
For example, if I’m monitoring memory usage per instance, I want the instance name in the notification but that’s buried in .ValueString which is not useful.
Hi,
What is your template that you used for alerting?
Perhaps, it can be simplified using
> $values.B.Labels.instance
in the Annotation for the instance name or
> $values.B.Value
for the metric value that you want to show.
, then just show the annotations Sortedpairs as follow:
{{ range .Annotations.SortedPairs -}}
- {{ .Name }}: {{ .Value }}
{{ end }}
{{- end -}}
Regards,
Fadjar
$values is not a valid variable inside notification templates.
If I understand it right, it refers to the metric’s values and contains metadata from the datasource (e.g. labels).
.Values only has the query ID and the value, which doesn’t say anything to anyone.
Hi…
Sorry, my wrong not complete explanation
You can put the $values.B.Labels.instance not in the template, instead in the Annotation at the Alert Rule. Put your Key, such as Node then the value using {{ $values.B.Labels.Instance }} and
Value in the key then {{ $values.B.Value }} to show the Metric value. You can look at this…
The B, you can find at the query:
You can see that I need to show the target name using {{ $values.B.Labels.target }}
And if you want to show the value you can use: {{ $values.B.Value }}
Then you can this snippet below in the template.
{{ range .Annotations.SortedPairs -}}
- {{ .Name }}: {{ .Value }}
{{ end }}
{{- end -}}
Below is my complete template for telegram message
{{- /* Telegram message to use: {{ template "telegram.message" . }} */ -}}
{{ define "__alerts_list" -}}
{{ range . }}
{{if ne (index .Labels "alertname") "" -}}
{{ if eq .Status "firing" }}🔴 {{ else }}🟢 {{ end }}
{{- if ne (index .Labels "severity") "" -}}
{{ index .Labels "severity" }} {{ end -}}
{{ index .Labels "alertname" }} 🕙 {{ .StartsAt.Format "15:04:05 🗓️ 2006-01-02" }}{{ end -}}
{{ if len .Annotations }}
Details:
{{ range .Annotations.SortedPairs -}}
- {{ .Name }}: {{ .Value }}
{{ end }}
{{- end -}}
-------------------------------
{{- end -}} {{- /* range */ -}}
{{- end -}} {{- /* define __alerts_list */ -}}
{{ define "__telegram.title" -}}
{{ if ne (index .CommonLabels "severity") "" }} <u><b>P{{ index .CommonLabels "severity" }}</b></u> {{ end -}}
{{ if ne (index .CommonLabels "alertname") "" -}}
[{{ index .CommonLabels "alertname" }}]
{{- end -}}
{{- end -}}{{- /* define __telegram */ -}}
{{ define "telegram.message" }}
{{ if gt (len .Alerts.Firing) 0 }}
🚨 ALARM (#{{ .Alerts.Firing | len }})
{{- template "__alerts_list" .Alerts.Firing }}{{ end -}}
{{ if gt (len .Alerts.Resolved) 0 }}
✅ RESOLVED {{ template "__telegram.title" . }} (#{{ .Alerts.Resolved | len }})
{{- template "__alerts_list" .Alerts.Resolved }}{{ end }}
{{- end -}}
Below, the result of the telegram message:
Regards,
Fadjar
1 Like
Thanks, really appreciate the advice.
One downside of it all is that I have to edit thousands of alerts and keep them updated to have this functionality back from pre-unified alerting.
Is this the official way of doing things now? I expected the notification template to be able to access all the information I needed to format a notification but it seems I have to spread template logic in two places.
You can follow discussion in here
Fortunately, I have no more than 20 alert rules, so it still manageable for me.
Or you can use the Alert API
Regards,
Fadjar
1 Like
Understood. Thanks again!
Have you looked at provisioning your alerts?
Yes, but it’s a complicated situation with many stakeholders with different technical skills… As a maintainer of the whole thing I’m trying to make it work the best I can without making it more difficult for users… if they have to remember to add these templates or they somehow break them… it makes the platform look fragile.
Ideally, if I could get access to the values in ValueString in json format, that would suffice.
1 Like
so others contribute to this ecosystem by carving alert templates themselves?
No, they contribute with alerts only.
I caved and went with a Go template inside the alert’s annotation that gets used by the notification template… essentially, per-alert templates. I’ve spent countless hours fixing lots of alerts and there are still errors due to human error. I was trying to avoid this situation but it seems unavoidable with Grafana 10.
@giovannitirloni it somewhat depends on the use case, but there are good reasons to template annotations and there are good reasons to template the notification. I’ve written a blog post about this here.
The general advice is that if you are changing how alerts are presented in a notification, template the notification. If you are adding information about an alert (i.e. explaining what it means and how to respond to it) then template custom annotations, such as the Summary and Description annotations.
1 Like
