I came back to see if anything has been done to improve documentation for Alert Manager (AM) templating after starting the topic in November. Hey, looks like Grafana took its users issues to heart! Clicked the link to see what’s new – silly me. Documentation is for the next version of Grafana. For docs for the latest stable release, they send you down a rabbit hole to old, unimproved documentation.
But wait, there is more! The documentation for the next release says point blank:
"You cannot use notification templates to:
Change how images are included in notifications, such as the number of images in each notification or where in the notification inline images are shown (fairly important in a well formatted message, but could be looked past, maybe)
Change the design of notifications in instant messaging services such as Slack and Microsoft Teams (critical to showing and highlighting critical information in the message, cutting out chaff, and making an alert readable)
Change the data in webhook notifications, including the structure of the JSON request or sending data in other formats such as XML (Teams contact point IS a webhook notification, IDK about Slack)
So – Grafana offers you AM Contact Points for both Slack and Teams, arguably the most popular business oriented IM services in use. But the alerts that come out of AM into MS-Teams are incomprehensible, mysterious junk. (I don’t know about Slack, we don’t use it.) But Grafana now admits in its “improved” documentation that there is no way to make the messages intelligible (and therefore actionable) to the recipients in Slack or Teams.
Welp, at least Graf is no longer BS-ing us about usable Alerts to those platforms. We need waste no more time on the effort. I suppose that could be considered an improvement. The junk you get in Teams (or Slack) is what you get, basta, end of story, you cannot improve it.
I give up!
Hi! Thanks for the feedback! I think there is some misunderstanding on some of the points here. I hope I can clarify it below!
Change the design of notifications in instant messaging services such as Slack and Microsoft Teams does not refer to the text in the notification, but refers to changing structural design elements of the notification. For example, in the case of Microsoft Teams this would be adding or removing cards from the Adaptive Cards message rather than changing the text in the notification.
Change the data in webhook notifications, including the structure of the JSON request or sending data in other formats such as XML. This sentence is about changing the data in webhook notifications using the webhook contact point. I can understand the confusion here, we will update the docs to make this clearer.
so since no one answered, this is what i could do to workaround the issue. here is my message template which is working for me so far. i may need to add more things to it later if need be. hope it could help others.
{{ define "Message-Template" }}
{{ $summary:= "" }}
{{ $description:= "" }}
{{ $runbook:= "" }}
{{- if gt (len .Alerts.Firing) 0 }}
{{- range .Alerts.Firing }}
{{ if gt (len .PanelURL ) 0 }}<{{ .PanelURL }}|Alerting>: {{ else }}*Alerting*: {{ end }}{{ if gt (len .Annotations.AlertValues) 0 }}{{- .Annotations.AlertValues }}{{ end -}}
{{ if gt (len .Annotations.summary) 0 }} {{ $summary = .Annotations.summary }}{{ end -}}
{{ if gt (len .Annotations.description) 0 }}{{ $description = .Annotations.description }}{{ end -}}
{{ if gt (len .Annotations.runbook_url) 0 }}{{ $runbook = .Annotations.runbook_url }}{{ end -}}
{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 }}
{{- range .Alerts.Resolved }}
*OK* - {{ if gt (len .Annotations.AlertValues) 0 }}{{- .Annotations.AlertValues }}{{ end }}
{{- end }}
{{- end }}
{{- if gt (len $summary ) 0 }}
*Summary*: {{ $summary }}{{ end -}}
{{- if gt (len $description ) 0 }}
*Description*: {{ $description }}{{ end -}}
{{- if gt (len $runbook ) 0 }}
<{{ $runbook }}|Runbook URL>{{ end -}}
{{ end }}
Hi!
Wanted to ask for help @georgerobinson
Do You have any idea how Value field could be sorted into more readable information?
For example in old grafana (I think version 7, but not sure because recently started to work here) the Alert notification was coming to teams in something like this:
Teams is actually supported in latest version (contact point name is Microsoft Teams).
You only have to adapt the markdown characters in the templates, you can look at the difference between Telegram (supports HTML) and Slack (own markdown).
Teams markdown seems very similar to Slack: Use Markdown formatting in Teams - Microsoft Support
Upgraded to 9.3.6 and still Value field prints out like sh t
Can someone please explain how to print values in normal view, not like this? At least sort them
Hi! Thanks for checking out my issue.
Yes, we been using custom alert and Value is defined as Value: {{ or .ValueString “[no value]” }}
Tried another ways, but nothing changes this field. What should I do? Please suggest
Hi! Your custom template is printing the ValueString, which will always look like your original screenshot. If you want to do something different you can instead iterate over $values. We have an example of that in the new default template alerting/default_template.go at main · grafana/alerting · GitHub
Hi! This is a special kind of alert called DatasourceError. It means Grafana was not able to query your datasource to evaluate the alert. This can happen if the datasource is misconfigured, unreachable from Grafana, or was down at the time this alert fired.
Hi! Thanks for reply. Sounds strange, as we have everything in k8s and if there something is down, we get repots and alerts. But, if alert is on thanos datasource, and alerting from grafana, how this cam be fixed? Need to fix all alerts?
Hi! I would first check that your Grafana server can talk to your Thanos datasource. When there is a DatasourceError alert, the reason for the error will be in the annotations.
t
