Using Grafana Template for Email Subject. Email Notifications are sent to recipients with no subject in some cases

Grafana Alerting
1

Hi,
The Grafana Alert emails in some cases are firing with (No Subject). I am using a template for the email subject.

Template Description:

{{ define "Email.Subject" }}
  {{ range .Alerts }}
    {{ $alert := . }}
    {{ range .Labels.SortedPairs }}
      {{ if eq .Name  "alertname" }}
        Production Alert -  {{ .Value }}
      {{end}}
    {{ end }}
  {{ end }}
{{ end }}

Example:
Based on the Alert Rule after the evaluation period, it collects the list of errors and send out the email notification. I receive one email that has either 10 or less firing instances. The first email has the correct Subject line with the Alert Name.
Within a minute I receive a second email which has more than 10 Firing instances. This second email has no subject. It has the following text “(no subject)”

Grafana Version: 10.2.3
Alert Manager: Grafana

Contact Points Settings:

  1. Integration: Email
  2. Optional Settings: Single Mail for all recipients
  3. Subject: {{template “Email.Subject” .}}
  4. Notification settings: Disable Resolved Message Enabled.

Notification Policy:
Matching Labels is selected for each type of Alert.
Continue matching subsequent sibling nodes: Enabled
Override grouping: Enabled
Group by: **grafana_folder, alertname

Anybody has faced similar issue, please let me know how to resolve this. Really appreciate your help on this. Thanks

2

I would say that you are reaching some email subject limits - e.g. max length. Don’t ask me what’s the actual max length limit - that depends on used email servers, which delivers that email (for example, gmail: 998 characters).
There is definitely some limit, so be smart and create short subject in all cases = don’t print everything, e.g. just first X alerts. Generally, all contact points have some limits, so be prepared for disaster when everything is failing and everything will be alerting.

3

@jangaraj Thanks for the response.
Just to clarify, this limitation is part of the email servers and not on the Grafana config? Please let me know.

4

Yes, but that’s guess. I don’t have idea what kind of email servers are processing your emails exactly and what’s their limits. I don’t see a reason why Grafana should have a problem to generate subject with X (e.g. 1M) characters with this setup.

5

I changed the template and I was able to get the subject line in the email for firing instances that have more than 10. But it fails again when the instances are more than 30. Looks like its an issue on how to use the range function and pull the Alert Name. Any better ideas on how this can be achieved please let me know. Thanks

Template:
{{ define “Email.Subject” }}
{{ range .Alerts }}
Production Alert - {{ .Labels.alertname }}
{{ end }}
{{ end }}

6

As I said - report only first X alerts in the subject. For example I report only first 5 alert via MS team:

{{ define "generic.msteams.alerts" }}
{{- $alerts := . -}}
{{ if gt (len $alerts) 5 }}
First 5 alerts only:
{{ $alerts = slice $alerts 0 5 }}{{- else -}}
{{ end -}}
  {{- range $alerts }}
    {{- template "generic.msteams.alert" . -}}
  {{- end -}}
{{- end -}}

Or disable grouping - then million failing instances = million alert emails (and not one email with million alertnames)

1 Like
7

@jangaraj Morning, The req was to group all firing instances in a single mail based on the alert. I was able to resolve this issue by changing the template. Now I could see a mail with 90 Firing instances grouped into a single mail and the Subject line is populated with the Alert Name. You can close this issue now.

New Template:
{{ define “Email.Subject” }}
{{ range $index, $alert := .Alerts }}
{{ if eq $index 0 }}
Production Alert - {{ .Labels.alertname }}
{{end}}
{{ end }}
{{ end }}

1 Like