Documentation of Alert Templating in Serious Need of Improvement!

Hi! It is in the template code. You see how in my example I renamed each template. For example, __subject was renamed to custom.subject.

To use the template you can follow this documentation here Use notification templates | Grafana documentation

1 Like

ok thanks I hope it works now I edited the template code with this:

{{ define "custom.message" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ if gt (.Alerts.Resolved | len) 0 }}, RESOLVED:{{ .Alerts.Resolved | len }}{{ end }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}
{{ define "custom.alert_list" }}{{ range . }}
Value: {{ or .ValueString "[no value]" }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}{{ if gt (len .GeneratorURL) 0 }}Source: {{ .GeneratorURL }}
{{ end }}{{ if gt (len .SilenceURL) 0 }}Silence: {{ .SilenceURL }}
{{ end }}{{ if gt (len .DashboardURL) 0 }}Dashboard: {{ .DashboardURL }}
{{ end }}{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }}
{{ end }}{{ end }}{{ end }}
{{ define "default.title" }}{{ template "custom.message" . }}{{ end }}
{{ define "default.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "custom.alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "custom.alert_list" .Alerts.Resolved }}{{ end }}{{ end }}
{{ define "teams.custom.alert_list" }}{{ range . }}
Value: {{ or .ValueString "[no value]" }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
{{ if gt (len .GeneratorURL) 0 }}Source: [{{ .GeneratorURL }}]({{ .GeneratorURL }})
{{ end }}{{ if gt (len .SilenceURL) 0 }}Silence: [{{ .SilenceURL }}]({{ .SilenceURL }})
{{ end }}{{ if gt (len .DashboardURL) 0 }}Dashboard: [{{ .DashboardURL }}]({{ .DashboardURL }})
{{ end }}{{ if gt (len .PanelURL) 0 }}Panel: [{{ .PanelURL }}]({{ .PanelURL }})
{{ end }}
{{ end }}{{ end }}
{{ define "teams.default.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "teams.custom.alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "teams.custom.alert_list" .Alerts.Resolved }}{{ end }}{{ end }}

and specified it in the contact point like this:

In your contact point you’ll only want to execute custom.message from the Message field so just {{ template "custom.message" . }}

1 Like

ok thank you so much for your support :blush:

hello
i need help with this template thing.
i have been trying to only send one .PanelURL for the alerts that have multiple labels in them for example my test alert after reduce, produces 2 labels based on eks region and if they are over a threshols, they should fire. so when my alert fires, i get one message containing the values for both of them and all the links are duplicates. so imaging what are the alerts with more labels look like. i have tried many different templatings to no avail. here is the screenshot with the default message.
can you help me with this?

@georgerobinson I am really sorry to brought this topic again, I have just specified {{ template "custom.message" . }} as you said and the other details have gone, here is the screenshot:


Can you use this template and see if it works?

{{ define "custom.subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ if gt (.Alerts.Resolved | len) 0 }}, RESOLVED:{{ .Alerts.Resolved | len }}{{ end }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}

{{ define "custom.alert_message" }}{{ range . }}
Value: {{ or .ValueString "[no value]" }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}{{ if gt (len .GeneratorURL) 0 }}Source: {{ .GeneratorURL }}
{{ end }}{{ if gt (len .SilenceURL) 0 }}Silence: {{ .SilenceURL }}
{{ end }}{{ if gt (len .DashboardURL) 0 }}Dashboard: {{ .DashboardURL }}
{{ end }}{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }}
{{ end }}{{ end }}{{ end }}

{{ define "custom.title" }}{{ template "custom.subject" . }}{{ end }}

{{ define "custom.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "custom.alert_message" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "custom.alert_message" .Alerts.Resolved }}{{ end }}{{ end }}
2 Likes

Thank you so much it worked now:

I am really sorry for taking your precious time.

1 Like

I came back to see if anything has been done to improve documentation for Alert Manager (AM) templating after starting the topic in November. Hey, looks like Grafana took its users issues to heart! Clicked the link to see what’s new – silly me. Documentation is for the next version of Grafana. For docs for the latest stable release, they send you down a rabbit hole to old, unimproved documentation. :rofl:

But wait, there is more! The documentation for the next release says point blank:

"You cannot use notification templates to:

  • Change how images are included in notifications, such as the number of images in each notification or where in the notification inline images are shown (fairly important in a well formatted message, but could be looked past, maybe)
  • Change the design of notifications in instant messaging services such as Slack and Microsoft Teams (critical to showing and highlighting critical information in the message, cutting out chaff, and making an alert readable)
  • Change the data in webhook notifications, including the structure of the JSON request or sending data in other formats such as XML (Teams contact point IS a webhook notification, IDK about Slack)
    :exploding_head:

So – Grafana offers you AM Contact Points for both Slack and Teams, arguably the most popular business oriented IM services in use. But the alerts that come out of AM into MS-Teams are incomprehensible, mysterious junk. (I don’t know about Slack, we don’t use it.) But Grafana now admits in its “improved” documentation that there is no way to make the messages intelligible (and therefore actionable) to the recipients in Slack or Teams.

Welp, at least Graf is no longer BS-ing us about usable Alerts to those platforms. We need waste no more time on the effort. I suppose that could be considered an improvement. The junk you get in Teams (or Slack) is what you get, basta, end of story, you cannot improve it.
:man_shrugging:
I give up!

Hi! Thanks for the feedback! I think there is some misunderstanding on some of the points here. I hope I can clarify it below!

  • Change the design of notifications in instant messaging services such as Slack and Microsoft Teams does not refer to the text in the notification, but refers to changing structural design elements of the notification. For example, in the case of Microsoft Teams this would be adding or removing cards from the Adaptive Cards message rather than changing the text in the notification.

  • Change the data in webhook notifications, including the structure of the JSON request or sending data in other formats such as XML. This sentence is about changing the data in webhook notifications using the webhook contact point. I can understand the confusion here, we will update the docs to make this clearer.

Best regards,

George

anyone? im stuck with this issue

so since no one answered, this is what i could do to workaround the issue. here is my message template which is working for me so far. i may need to add more things to it later if need be. hope it could help others.

{{ define "Message-Template" }}
{{ $summary:= "" }}
{{ $description:= "" }}
{{ $runbook:= "" }}
{{- if gt (len .Alerts.Firing) 0 }}
{{- range .Alerts.Firing }} 
{{ if gt (len .PanelURL ) 0 }}<{{ .PanelURL }}|Alerting>: {{ else }}*Alerting*: {{ end }}{{ if gt (len .Annotations.AlertValues) 0 }}{{- .Annotations.AlertValues }}{{ end -}}
{{ if gt (len .Annotations.summary) 0 }} {{ $summary = .Annotations.summary }}{{ end -}}
{{ if gt (len .Annotations.description) 0 }}{{ $description = .Annotations.description }}{{ end -}}
{{ if gt (len .Annotations.runbook_url) 0 }}{{ $runbook = .Annotations.runbook_url }}{{ end -}}
{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 }}
{{- range .Alerts.Resolved }}
*OK* - {{ if gt (len .Annotations.AlertValues) 0 }}{{- .Annotations.AlertValues }}{{ end }}
{{- end }}
{{- end }}
{{- if gt (len $summary ) 0 }}
*Summary*: {{ $summary }}{{ end -}}
{{- if gt (len $description ) 0 }}
*Description*: {{ $description }}{{ end -}}
{{- if gt (len $runbook ) 0 }}
<{{ $runbook }}|Runbook URL>{{ end -}}
{{ end }}

Hi!
Wanted to ask for help @georgerobinson
Do You have any idea how Value field could be sorted into more readable information?
For example in old grafana (I think version 7, but not sure because recently started to work here) the Alert notification was coming to teams in something like this:

Now after upgrade, Grafana sending everything in one string and that is not ok if there are more than one Value :

I have tried to customise alert and message, but nothing works there. Is there any solution? Using Grafana v9.3.0
Thanks!

Please feel free to have a look at templates I shared to help members setup Grafana alerting.

1 Like

Thanks, but this is not helping for me, as Teams does not support HTML? Or I need to upgrade to 9.3.2?

1 Like

Teams is actually supported in latest version (contact point name is Microsoft Teams).
You only have to adapt the markdown characters in the templates, you can look at the difference between Telegram (supports HTML) and Slack (own markdown).
Teams markdown seems very similar to Slack: Use Markdown formatting in Teams - Microsoft Support

1 Like

Upgraded to 9.3.6 and still Value field prints out like sh :smiley: t
Can someone please explain how to print values in normal view, not like this? At least sort them


@bLd759 @grant2 @georgerobinson please check

Hi @twocats, it looks like you might be using a custom template as the Value field changed in 9.3.0?

Hi! Thanks for checking out my issue.
Yes, we been using custom alert and Value is defined as Value: {{ or .ValueString “[no value]” }}
Tried another ways, but nothing changes this field. What should I do? Please suggest

Can you share the template, and how you are using the template from your contact point(s)?