MS Teams alerting

[twocats]

Hi! Will try one more time describe my issue.
I have Grafana 9.3.6 via kube-prometheus-stack and bunch of alerts (almost 500) configured. Alerts are sending to teams private channel via webhook, and there is one thing I can not make work normally. Have tried everything from different posts from community, nothing really works.
My issue is following, alert dashboards are configured to check servers, for example call processing queue, there are 10 servers. When queue is over 100, there is alert sent to teams, but in teams alert is so ugly You can break eyes to see which servers are firing alert.

My goal is to sort them like:
SERVER1 = VALUE
SERVER2 = VALUE
SERVER3= VALUE

At the moment, all my tries are without success. Have tried new template, have tried to add to alert code from this post and also tried what @georgerobinson suggested to add to message this :

{{ range .Alerts }}
{{ range $k, $v := .Values }}
{{ $k }} = {{ $v.Labels }} {{ $v.Value }}
{{ end }}
{{ end }}

I am not going to paste everything I have tried, but maybe have some idea, where to find solution?

[georgerobinson]

Hi! I understand you created a template called myalert. Can you show me where you use this template from your contact point? You can find more information about this here Use notification templates | Grafana documentation

[twocats]

Hi!
So, my message for now looks like this (reverted back and have not saved otrher options, because none of them were working):

{{ define "custom.alert_message" }}{{ range . }}
Value: {{ or .ValueString "[no value]" }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
{{ if gt (len .DashboardURL) 0 }}Dashboard: {{ .DashboardURL }}\n{{ end }}
{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }}\n{{ end }}
{{ end }}{{ end }}
{{ define "custom.subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ if gt (.Alerts.Resolved | len) 0 }}, RESOLVED:{{ .Alerts.Resolved | len }}{{ end }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}

{{ define "custom.title" }}{{ template "custom.subject" . }}{{ end }}

{{ define "custom.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "custom.alert_message" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "custom.alert_message" .Alerts.Resolved }}{{ end }}{{ end }}

And contact point looks like this:

@georgerobinson Do You need with code You provided?

[georgerobinson]

Thanks!

I see that you are using Value: {{ or .ValueString "[no value]" }}. Did you mean to change this to range over the values in $values? I don’t see anywhere else where $values is being used in your template.

[twocats]

Sorry, as I said, I deleted previous tries and left only working template.
I tried something more or less like this as You advised in other post:

{{ define "custom.alert_message" }}{{ range . }}
{{ range .Alerts }}
{{ range $k, $v := .Values }}
{{ $k }} = {{ $v.Labels }} {{ $v.Value }}
{{ end }}
{{ end }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
{{ if gt (len .DashboardURL) 0 }}Dashboard: {{ .DashboardURL }}\n{{ end }}
{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }}\n{{ end }}
{{ end }}{{ end }}
{{ define "custom.subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ if gt (.Alerts.Resolved | len) 0 }}, RESOLVED:{{ .Alerts.Resolved | len }}{{ end }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}

{{ define "custom.title" }}{{ template "custom.subject" . }}{{ end }}

{{ define "custom.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "custom.alert_message" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "custom.alert_message" .Alerts.Resolved }}{{ end }}{{ end }}

Also, besides to this, I tried to add Alertvalues to Annotation and Summary field with such code:

{{ with $values }}
{{ range $k, $v := . }}
   Location: {{$v.Labels.server}}
   Alerting value: {{ $v }}
{{ end }}
{{ end }}

This kind of gave me what I needed

but I cant get these values to send over teams in message. This screenshot is from alerting/list
Looks like with this I am pretty close, but my tries to write code for message did not work out

[georgerobinson]

The second screenshot, that looks like an Annotation called AlertValues, is that correct? If so, I think you might be confusing two separate templating systems in Grafana: templating labels and annotations, and templating notifications.

[twocats]

I am sorry, if I am not describing properly what I need, as I am like beginner with whole Grafana, alerting etc. But, You want to say that is not the same? As I understand from alerts in teams and dashboard, they are same - value. If You see my first screenshot in this post, there is firing alert and Values are printed out in ugly way (ok, there ir var, metric and labels, but still, values (no?), but they are same as screen from Annotation called AlertValues. E1, E2, E3 etc. I want to send these values via teams in ordered way. If I am not correct, please explain, as I have not understood whole thing

[georgerobinson]

Hi! Let’s start from the beginning. It looks like you are using Classic Conditions, but your alert is working on multi-dimensional data because I can see multiple servers in the Teams message.

1. I would suggest is that you change your alert from Classic Conditions to Reduce expression with a RefID B and a Threshold expression with a RefID C. This will create one alert per server, rather than a single alert for all servers.

2. I would recommend creating a Summary annotation for this alert rule that contains the following:

{{ $labels.server }} = {{ $values.B }}

3. I would recommend creating the following notification template:

{{ define "simple.message" -}}
{{ if .Alerts.Firing -}}
{{ len .Alerts.Firing }} firing alert(s)
{{ template "alerts.summarize" .Alerts.Firing }}
{{- end }}
{{- if .Alerts.Resolved -}}
{{ len .Alerts.Resolved }} resolved alert(s)
{{ template "alerts.summarize" .Alerts.Resolved }}
{{- end }}
{{- end }}

{{ define "alerts.summarize" }}
{{ range . -}}
- {{ index .Annotations "summary" }}
{{ end }}
{{ end }}

This should give you the following message:

1 firing alert(s)

- server1 = value1

1 resolved alert(s)

- server2 = value2

[twocats]

Thanks! I will try and let You know, if this was the solution. One thing about classic conditions, this alert indeed is using classic condition, but we have most of alerts with expressions

[twocats]

Hi @georgerobinson , sorry for not providing feedback, was busy with other tasks.
I want to say huge thanks for this, Your solution fixed my issue

[twocats]

Hi @georgerobinson , ran into one small issue, which again could not fix:
If I use Your provided simple message, I do not have any links added to message:

Tried to modify message template, but I got only like this:

My main question, is there any way to add one link per message, not per alert? I am looking forward to keep only darshboard link.
Also one more, there is button View URL in the end of message, which forwards to the alert configuration itself, I am wondering, is there any use case and even is it possible to put Dashboard URL under that button?

[georgerobinson]

@twocats

My main question, is there any way to add one link per message, not per alert?

Yes it’s possible for Dashboard URL, but whether the link goes to the right place depends on how you have configured grouping in your notification policies.

For example, if you have a notification that contains an alert from Dashboard A and another alert from Dashboard B, but you only want one link, which one do you want? The link to Dashboard A or the link to Dashboard B?

The SilenceURL is different for each alert so I would recommend either including it for all alerts or for none of them.

[twocats]

@georgerobinson As far as I know, we use notifications from one Dashboard, in 90%, there are not combined notifications, so answer is the link form Dashboard A. To be more precise, we have one notification per panel in most cases. So for this case I would be very happy if I could change button to Dashboard URL. Is there any documentation? Could not find any info about this

And in this company alerts goes to Incident Managers and they do not use silence, they pass information further and we can silence alert if needed through Grafana GUI.

[georgerobinson]

You could write something like this (I’ve used ... here to represent the rest of the template from the original example):

{{ define "simple.message" -}}
...
Dashboard URL: {{ (index .Alerts 0).DashboardURL }}
{{- end }}

This template however requires the following to work:

1. All alerts are linked to a dashboard
2. Each notification contains alerts from the same dashboard. If you have two alerts firing from different dashboards then these must be delivered as separate notifications

[twocats]

@georgerobinson

Dashboard URL: {{ (index .Alerts 0).DashboardURL }}
{{- end }}

Thanks! This worked again but link is not as link, it is as plain text.

Asked before, but is it possible to reconfigure “View URL” button to not open alert configuration, but for example this dashboard URL?

[georgerobinson]

Ah, when using Microsoft Teams links need to be formatted as Markdown:

Dashboard URL: [{{ (index .Alerts 0).DashboardURL }}]({{ (index .Alerts 0).DashboardURL }})

[twocats]

Worked, thank You very much!