A different thread reminded me of the generic use for the notification policy message block.
The message block in notification policy can be used to send specific text or define the message template to be used for this policy. While not a fan of defining which template to be used at the notification group level, which would leads to the creation of more notification policies for different types of messages. In my mind notification policy would better define who gets messages, handing off message content details elsewhere. Perhaps understanding templating logic can resolve that issue.
But⌠if the message block in Notification Policy is used to define the policy - how do you add query labels to the alert message itself. Working with @skydiablo and will report out once we figure it out.
@skydiablo I researched message templates yesterday and found this "Default Template in the code for Alerting on Github.
{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
{{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}
{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}
{{ define "__text_alert_list" }}{{ range . }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}{{ if gt (len .GeneratorURL) 0 }}Source: {{ .GeneratorURL }}
{{ end }}{{ if gt (len .SilenceURL) 0 }}Silence: {{ .SilenceURL }}
{{ end }}{{ if gt (len .DashboardURL) 0 }}Dashboard: {{ .DashboardURL }}
{{ end }}{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }}
{{ end }}{{ end }}{{ end }}
{{ define "default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "default.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "__text_alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "__text_alert_list" .Alerts.Resolved }}{{ end }}{{ end }}
{{ define "__teams_text_alert_list" }}{{ range . }}
Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}
{{ if gt (len .GeneratorURL) 0 }}Source: {{ .GeneratorURL }}
{{ end }}{{ if gt (len .SilenceURL) 0 }}Silence: {{ .SilenceURL }}
{{ end }}{{ if gt (len .DashboardURL) 0 }}Dashboard: {{ .DashboardURL }}
{{ end }}{{ if gt (len .PanelURL) 0 }}Panel: {{ .PanelURL }}
{{ end }}
{{ end }}{{ end }}
{{ define "teams.default.message" }}{{ if gt (len .Alerts.Firing) 0 }}**Firing**
{{ template "__teams_text_alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**Resolved**
{{ template "__teams_text_alert_list" .Alerts.Resolved }}{{ end }}{{ end }}
Iâm not quite sure I understand all of the nested calls - but activated it with a custom notification policy referencing:
{{template "teams.default.message" . }}
The results were disappointing, not only did they not include the desired values, but the results were not formatted as I would iike.
I did more research this morning and ran into this Community post with the same type issues.
In the post, @mattabrams indicates that message templating only works with multi-dimensional queries. Which seems odd, but could point out a reason we are having issues.
Towards the end of the thread @janczewskimarcin indicates a solution involving:
- assigning a query label to a defined label in the alert definition
- referencing the label variable in a message template
I believe the solution also involved the use of a multi-dimensional alert rather than a classic alert as well.
I am going to do some work down that route today to see if I can come up with a solution.
@skydiablo here is the documentation reference regarding variables requiring the use of multi-dimensional queries: Create Grafana managed alert rule | Grafana Labs
This older GitHub post is interesting although hopefully no longer relevant: ngalert donât work with prometheus datasource ¡ Issue #35540 ¡ grafana/grafana (github.com)
