Documentation of Alert Templating in Serious Need of Improvement!

First, I have been trying for days to figure out what should be a very simple, straight-forward task - Create a human readable message of a firing alert. In my case the alerts are going to either email or MS-Teams. Grafana official documentation is un-clear and inadequate. The video posted does not touch on message formatting at all. (Template data | Grafana documentation)

Just searching on this forum, I find MANY people frustrated and calling for help on this issue, and few if any responses from Grafana. There are some forum members suggesting other posts and posts at 3rd party locations. Expecting people to go elsewhere than Grafana.com for so basic a function as formatting messages is absurd. IF those posts are accurate, then their explanations need to be incorporated into Grafana official documentation, not left hidden away in the forums where users might or might not find them, or linked on a third party site.

Second, it appears that very significant coding is required to format alerts so that they provide concise, human readable messages, to whit:

{{ define "myalert" }}
  [{{.Status}}] {{ .Labels.alertname }}

  Labels:
  {{ range .Labels.SortedPairs }}
    {{ .Name }}: {{ .Value }}
  {{ end }}

  {{ if gt (len .Annotations) 0 }}
  Annotations:
  {{ range .Annotations.SortedPairs }}
    {{ .Name }}: {{ .Value }}
  {{ end }}
  {{ end }}

  {{ if gt (len .SilenceURL ) 0 }}
    Silence alert: {{ .SilenceURL }}
  {{ end }}
  {{ if gt (len .DashboardURL ) 0 }}
    Go to dashboard: {{ .DashboardURL }}
  {{ end }}
{{ end }}

Step 2: Configure a template to render entire notification message.

{{ define "mymessage" }}
  {{ if gt (len .Alerts.Firing) 0 }}
    {{ len .Alerts.Firing }} firing:
    {{ range .Alerts.Firing }} {{ template "myalert" .}} {{ end }}
  {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
    {{ len .Alerts.Resolved }} resolved:
    {{ range .Alerts.Resolved }} {{ template "myalert" .}} {{ end }}
  {{ end }}
{{ end }}

Are you kidding – WTH Grafana?! No explanation of how this code was derived. No commenting of the code so that we can modify it for our individual needs. No explanation of syntax, or options that are possible. This is not an enterprise solution - this is a science experiment!

Additionally, the documentation does not explain WHERE to place this coding:

Do I place it in the Alert Rule (if so, which of the six options does one choose? No info, no hint provided)

image

OR maybe i past code into the Contact Point message?

image

I’ve wasted a whole lot of time with trial and error – is my code wrong, or am I putting it in the wrong place - it is anyone’s guess because as there is next to no documentation.

@praveenmayantha is absolutely correct – Alert Manager message formatting is a s**tshow!

Grafana - Please thoroughly document what you have in place now, and please strongly consider cutting out the ridiculous coding required to do something as simple as format a message.

5 Likes

Thanks for this post, I’m feeling the same as you regarding the templating of alerts… It’s been a frustating and hard path since I started thinking about sending some human-readable alerts, and yet I have no solution for it :sleepy:.

Hopefully this post has some relevance and they start doing something. I totally agree with you that the documentation (at least for this topic) is awful.

2 Likes

@meloriarellano please see above, and pass on to whoever are responsible for documentation on the @grafana-labs-team

Also please see:

1 Like

This post was flagged by the community and is temporarily hidden.

@leodegariopasakdal , @eledobleefe , @thepalbi , @tolzhabayev , @nataliabernarte , @muhammadshahzeb

I agree. I got many headache trying to figure out on how the templating really works. Good feature, but it needs more documentation.

Hello! I am an engineer in the Alerting team at Grafana. Thank you for your feedback and we appreciate that a lot of users are frustrated with the documentation on templating alert notifications.

I, with the help of another engineer in the team, will be working on improving this documentation over the following weeks. Having read a number of topics on the community forum I think I have a good understanding of some of the most frequent issues people have and how we can improve the documentation to help with these issues.

I would love to collect feedback on frustrations with documentation on templating alert notifications if possible and we will attempt to address it with the planned work.

4 Likes

Can you please help me on the above, how to configure the custom mail subjects and body?

We are expecting the custom email subject and body like below, can some one please help how can I get the custom email subject and body whenever any issue happen.

Actual Subject: [FIRING:1]

Expecting Subject: [CRITICAL]:[Environment] <Service_Name> down On <Host_Name/Server_Name>

EX1: [CRITICAL]:[DEV] MySQL down (C-DEV6-403) On tesDev101 server

EX2: [CRITICAL]:[QA] MySQL down (C-QA-403) On tesQA101 server

EX3: [CRITICAL]:[PROD] MySQL down (C-Prod-Db) On testProd201 server

Can we have example how can we get the custom email sub name like [CRITICAL]:[Environment] down <Service_Name> On ?
Can we have example how can we get the custom message or body ?

Hi! Can I ask if Critical, Environment, Service Name and Host Name are labels? If so it will be possible to iterate labels in the email subject template.

Hi @georgerobinson

Thanks for the reply back. Yes, they are actually labels but in the subject we are expecting only simple line instead of other unnecessary stuff.

Instead of Firing, sub name should be start with Severity (Critical/Warning/Alert) like above I have mentioned. Can we please remove the State(Firing) and replace with Severity would be good and user friendly.

We are getting the below Very Long subject unnecessarily and unable to understand that’s why we requesting the above subject.

[FIRING:1] MySQL SQL Thread Is Stopped DEV_ALL_TRACKS SLAVE_OF_test C-STACK-DEV MariaDB AVRkMGSVkz mysql_slave_status_master_server_id /agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0 mysqld_exporter C-STACK-DEV Experimental /agent_id/07f2228c-ac31-…

Even Body, also has some unnecessary content instead of simple and understandable content.

We dont need this much content in the Body.

Firing: 1 alert for alertname=MySQL SQL Thread Is Stopped APP_TYPE=DEV_ALL_TRACKS DB_SERVICE_TYPE=SLAVE_OF_test ENV=C-STACK-DEV SERVICE_RUNNING=MariaDB alert_rule_uid=AVRkMGSVkz name=mysql_slave_status_master_server_id agent_id=/agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0 agent_type=mysqld_exporter environment=C-STACK-DEV grafana_folder=Experimental instance=/agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0 job=mysqld_exporter_agent_id_07f2228c-ac31-4b13-ba0b-a69d54e943f0_mr machine_id=/machine_id/4e4c02654583449ea8e09f4b826d25be master_host=http://test123402.db.test.com node_id=/node_id/761ebe55-a19e-4c36-8f32-d09e86b2a092 node_name=test123 node_type=generic percona_alerting=1 replication_set=C-DEV service_id=/service_id/02048fcd-e2ca-4df8-b9c0-6a7ba705d01a service_name=C-DEV4-RO service_type=mysql severity=critical template_name=mysql_replication_sql_running
Firing MySQL SQL Thread Is Stopped
Value: [ var=‘A’ labels={APP_TYPE=DEV_ALL_TRACKS, DB_SERVICE_TYPE=SLAVE_OF_test123402, ENV=C-STACK-DEV, SERVICE_RUNNING=MariaDB, name=mysql_slave_status_master_server_id, agent_id=/agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0, agent_type=mysqld_exporter, environment=C-STACK-DEV, instance=/agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0, job=mysqld_exporter_agent_id_07f2228c-ac31-4b13-ba0b-a69d54e943f0_mr, machine_id=/machine_id/4e4c02654583449ea8e09f4b826d25be, master_host=http://test123402.db.test.com, node_id=/node_id/761ebe55-a19e-4c36-8f32-d09e86b2a092, node_name=test123, node_type=generic, replication_set=C-DEV, service_id=/service_id/02048fcd-e2ca-4df8-b9c0-6a7ba705d01a, service_name=C-DEV4-RO, service_type=mysql} value=2 ]
description: MySQL Replication Not Running on test123:C-DEV4-RO

rule: MySQL SQL Thread Is Stopped

summary: MySQL Replication Not Running on test123:C-DEV4-RO
Labels:

  • alertname: MySQL SQL Thread Is Stopped
  • APP_TYPE: DEV_ALL_TRACKS
  • DB_SERVICE_TYPE: SLAVE_OF_test123402
  • ENV: C-STACK-DEV
  • SERVICE_RUNNING: MariaDB
  • agent_id: /agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0
  • agent_type: mysqld_exporter
  • environment: C-STACK-DEV
  • grafana_folder: Experimental
  • instance: /agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0
  • job: mysqld_exporter_agent_id_07f2228c-ac31-4b13-ba0b-a69d54e943f0_mr
  • machine_id: /machine_id/4e4c02654583449ea8e09f4b826d25be
  • master_host: test_hosy
  • node_id: /node_id/761ebe55-a19e-4c36-8f32-d09e86b2a092
  • node_name: test123
  • node_type: generic
  • percona_alerting: 1
  • replication_set: C-DEV
  • service_id: /service_id/02048fcd-e2ca-4df8-b9c0-6a7ba705d01a
  • service_name: C-DEV4-RO
  • service_type: mysql
  • severity: critical
  • template_name: mysql_replication_sql_running

I think we need only below body enough for user understanding.

description: MySQL Replication Not Running on test123:C-DEV4-RO

rule: MySQL SQL Thread Is Stopped

summary: MySQL Replication Not Running on test123:C-DEV4-RO
Labels:

  • alertname: MySQL SQL Thread Is Stopped
  • APP_TYPE: DEV_ALL_TRACKS
  • DB_SERVICE_TYPE: SLAVE_OF_test123402
  • ENV: C-STACK-DEV
  • SERVICE_RUNNING: MariaDB
  • agent_id: /agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0
  • agent_type: mysqld_exporter
  • environment: C-STACK-DEV
  • grafana_folder: Experimental
  • instance: /agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0
  • job: mysqld_exporter_agent_id_07f2228c-ac31-4b13-ba0b-a69d54e943f0_mr
  • machine_id: /machine_id/4e4c02654583449ea8e09f4b826d25be
  • master_host: test_hosy
  • node_id: /node_id/761ebe55-a19e-4c36-8f32-d09e86b2a092
  • node_name: test123
  • node_type: generic
  • percona_alerting: 1
  • replication_set: C-DEV
  • service_id: /service_id/02048fcd-e2ca-4df8-b9c0-6a7ba705d01a
  • service_name: C-DEV4-RO
  • service_type: mysql
  • severity: critical
  • template_name: mysql_replication_sql_running

Please remove clumsy content and give an options that user can customize body and subject.

Hi! Can you show me the template you have created for the email subject? How do you want to subject to look when there are two or more alerts in a notification?

Hi @georgerobinson

I tried {{ .severity }} or {{ .Labels.severity }} or {{ $Labels.severity }} it would send the test but not render the subject in the email and I see the following in the /srv/logs/grafana.logs file:
level=warn msg="failed to template email message" err="template: :1:11: executing \"\" at <.labels.severity>: can't evaluate field labels in type *channels.ExtendedData"

But none of them are working fine and tried search in the Grafana document but there are examples found in the docs. Can you please share us the simple examples for both Email Subject and Body with settings and screenshots would be very helpful?

Can’t we send an email alert for single server instead of grouping of multiple servers?

Hi! It looks like the template is attempting to print the labels without first iterating over the alerts. For example:

{{ define "email.subject" }}
{{ range .Alerts }}{{ .Labels.alertname }}{{ end }}
{{ end }}

and then in the email contact point set to the subject to:

{{ template "email.subject" . }}

There is an example of it here Message templating | Grafana documentation.

Hi @georgerobinson

Thank you so much.

Finally, the email Subject working fine. But can you please give me the same for Message Body as well? We want body message like below.

summary: MySQL Replication Not Running on test123:C-DEV4-RO
Labels:

  • alertname: MySQL SQL Thread Is Stopped
  • APP_TYPE: DEV_ALL_TRACKS
  • DB_SERVICE_TYPE: SLAVE_OF_test123402
  • ENV: C-STACK-DEV
  • SERVICE_RUNNING: MariaDB
  • agent_id: /agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0
  • agent_type: mysqld_exporter
  • environment: C-STACK-DEV
  • grafana_folder: Experimental
  • instance: /agent_id/07f2228c-ac31-4b13-ba0b-a69d54e943f0
  • job: mysqld_exporter_agent_id_07f2228c-ac31-4b13-ba0b-a69d54e943f0_mr
  • machine_id: /machine_id/4e4c02654583449ea8e09f4b826d25be
  • master_host: test_hosy
  • node_id: /node_id/761ebe55-a19e-4c36-8f32-d09e86b2a092
  • node_name: test123
  • node_type: generic
  • percona_alerting: 1
  • replication_set: C-DEV
  • service_id: /service_id/02048fcd-e2ca-4df8-b9c0-6a7ba705d01a
  • service_name: C-DEV4-RO
  • service_type: mysql
  • severity: critical
  • template_name: mysql_replication_sql_running

Hi! You should be able to use the template I shared for the subject as a reference for the message. It’s more or less the same, you just need to iterate over the labels like so:

{{ range .Alerts }}
{{ range .Labels.SortedPairs }}
{{ .Name }}: {{ .Value }}
{{ end }}
{{ end }}

Hi @georgerobinson

Thank you so much for the help. I don’t have much knowledge in to the scripting, any how I will try with the above examples.

But I am not getting like below

  • alertname: MySQL SQL Thread Is Stopped
  • APP_TYPE: DEV_ALL_TRACKS
  • DB_SERVICE_TYPE: SLAVE_OF_test123402
  • ENV: C-STACK-DEV

I am using the script below script.

{{ define “email.body” }}
{{ range .Alerts }}
{{ range .Labels.SortedPairs }}
{{ .Name }}: {{ .Value }}
{{ end }}
{{ end }}
{{ end }}

But I am getting the mail like below instead of separate lines.

  • alertname: MySQL SQL Thread Is Stopped
  • APP_TYPE: DEV_ALL_TRACKS* DB_SERVICE_TYPE: SLAVE_OF_test123402* ENV: C-STACK-DEV

Can I get separate line like above?

Hi @georgerobinson

I have tried go emojis in the mail subject, but its not working and I am getting the mail sub like below.
image

And my code is:

{{ define “alert_severity_prefix_emoji” -}}
{{- if ne .Status “firing” -}}
:white_check_mark:
{{- else if eq .CommonLabels.severity “critical” -}}
:red_circle:
{{- else if eq .CommonLabels.severity “warning” -}}
:warning:
{{- end -}}
{{- end -}}

To use an emoji in an email I think you need to use the actual emoji in the template. The short name :red_circle: will not be changed into an emoji in Grafana.

@georgerobinson Can you please share me the example? How can we insert an actual emoji into the Template like below?

I think instead of :warning you need to use the actual :warning:

emoji