How to keep ValueString in notification templates in Grafana 9.3 and later

Grafana Alerting
1

Here is a custom notification template that keeps the ValueString from Grafana 8.0 to 9.2, should you want to keep using it in Grafana 9.3 and later.

{{ define "custom.text_alert_list" }}{{ range . }}
Value: {{ or .ValueString "[no value]" }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}{{ if gt (len .GeneratorURL) 0 }}Source: {{ .GeneratorURL }}
{{ end }}{{ if gt (len .SilenceURL) 0 }}Silence: {{ .SilenceURL }}
{{ end }}{{ if gt (len .DashboardURL) 0 }}Dashboard: {{ .DashboardURL }}
{{ end }}{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }}
{{ end }}{{ end }}{{ end }}

{{ define "custom.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "custom.text_alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "custom.text_alert_list" .Alerts.Resolved }}{{ end }}{{ end }}

and here is an example of that notification template when used in a Slack message:

Screenshot 2023-03-06 at 16.51.20
Screenshot 2023-03-06 at 16.51.201324×522 56 KB

1 Like
2

Hi georgerobinson,

i have a question regarding the notification template.
How can I display the exact values that triggered the alarm in the Google Chat notification? (Sometimes these can be several values)
example: (Exact values for rx and tx in alarm)
image

myalert:

{{ define "myalert" }}
<font color="#FFB600"><b>[{{.Status}}]</b></font>: {{ .Labels.alertname }}
--------------------------------
<font color="#FFB600"><b>[Value]</b></font>: {{ or .ValueString "[no value]" }}
--------------------------------
{{ if gt (len .Annotations) 0 }} <font color="#FFB600"><b>[Annotations]</b></font>:{{ range .Annotations.SortedPairs }}
{{ .Name }}: {{ .Value }}{{ end }}
--------------------------------
<font color="#FFB600"><b>[Dashboard]</b></font>: <a href="{{ .DashboardURL }}">{{ .DashboardURL }}</a>
{{ end }}
{{ end }}

mymassage:

{{ define "mymessage" }}{{ if gt (len .Alerts.Firing) 0 }} <font color="#FF2A00"><b>**Firing**</b></font>
    {{ range .Alerts.Firing }} {{ template "myalert" .}} {{ end }}{{ end }} {{ if gt (len .Alerts.Resolved) 0 }} <font color="#17D600"><b>**Resolved**</b></font>
    {{ range .Alerts.Resolved }} {{ template "myalert" .}} {{ end }}{{ end }}{{ end }}

Or is it possible to see somewhere how the Deafult alerting was in the Grafana 8 version ?

THX!
br

3

Hi! It looks like you want to use the values field of each alert as documented here Reference | Grafana documentation. I’d recommend reading the documentation through first and then if you have any further questions I’d be happy to help. Customize notifications | Grafana documentation

4

Hi georgerobinson,

can you help me with this in a little more detail. I have a value output in the chat, which also contains labels. Here we have too much labels information. Can I somehow reduce this to the most important?

e.g

grafana template = …[Value]: {{ or .ValueString “[no value]” }}…
Google Chat = …

[firing]: test

[Value]: [ var=‘C0’ metric=‘api test’ labels={**name**=probe_success, container=…
[Annotations]:
summary:

if I have several entries in the list , so C0, C1 and C2 … then the lebels becomes infinitely long in the message and totally unclear -thanks
BR.

5

Hi! :wave: You can reduce it, but can you tell me how you want it to look? Which labels are important to you?

6

so here would be the whole message for Values, here we would want for example only the info of the Instance=https://…? (

[Value]: [ var='C0' metric='api' labels={__name__=probe_success, container=test, endpoint=http, instance=https://test.com/, job=kube-prom-stack-prometheus-test, namespace=monitoring, pod=kube-prom-stack-prometheus-testtxzvd, service=kube-prom-stack-prometheus-test, target=server} value=0 ],

changed:
[Value]: [ var='C0' metric='api' instance=https://test.com/ ],

Thx!

7

So the ValueString cannot be changed because its just text, but you can create your own text with custom notification templates.

Looking at the example you shared it seems you just want the instance label, and not the value value=0?

In either case, would you rather not have something like this:

2 firing alerts(s):
probe_success: instance=https://test.com/ value=0
probe_success: instance=https://test2.com/ value=0

1 resolved alert(s):
probe_success: instance=https://test3.com/ value=1

You can then forget about var='C0' and metric='api' which I don’t think are that useful?

8

I would like to try then like you did in your example, can you help me one more tip for the correct spelling in the template, how would I adjust something like that for example so that I have such an alerting.

9

There are a couple of changes I would recommend:

  1. Instead of using Classic Conditions change the alert rule to use Reduce expression and then either a Threshold or Math expression. This will allow you to use $labels in both annotations and notification templates.

  2. Add a summary description to your alert rule with the following template:

instance={{ index $labels "instance" }} value={{ index $values "B" }}

You should change "B" to the Ref ID of the Reduce expression if its different.

  1. Create a custom notification template like this and use it in your contact point:
{{ define "alert.summarize" -}}
{{ index .Labels "alertname" }}: {{ index .Annotations "summary" }}
{{- end }}

{{ define "alerts.summarize" -}}
{{ if .Alerts.Firing }}{{ len .Alerts.Firing }} firing alert(s)
{{ range .Alerts.Firing -}}
{{ template "alert.summarize" . }}
{{ end }}{{ end }}

{{ if .Alerts.Resolved }}{{ len .Alerts.Resolved }} resolved alert(s)
{{ range .Alerts.Resolved -}}
{{ template "alert.summarize" . }}
{{ end }}{{ end }}
{{ end }}
10

Hi @georgerobinson. Please I need help with value string in grafana v9.4 I used Valuestring: {{ .ValueString }} in my custom grafana template, and got the output below

Valuestring: [ var=‘D0’ metric=‘test04/ngrv-frontend’ labels={name=kube_deployment_status_replicas_available, container=kube-state-metrics, deployment=nsrv-frontend, endpoint=http, instance=10.244.1.140:8080, job=kube-state-metrics, namespace=test04, pod=kube-prometheus-kube-state-metrics-78bc9b4b5c-479wx, service=kube-prometheus-kube-state-metrics} value=2 ]

My question is how do I get the value of the metric to show in my alerts output and value which is 2. I want it to look like this

‘test04/ngrv-frontend’ = 2

11

Hi! There is an example of how to do this in the message above from March 16 ^^. I’ve marked it as the solution.

1 Like