Detailed instructions on how to manage alerts for "DataSourceNoData" errors?

Grafana Alerting
1

  • What Grafana version and what operating system are you using?

    • Grafana 12.3.0 container on Ubuntu 24.04

  • What are you trying to achieve?

    • I’m using summaries to improve alerting in my Grafana setup, but when one of my datasources fails because of “NoData”, I get a normal summary with [no value] instead of labels and variables. I want to create customized alerts for the DatasourceNoData errors, and I understand that it creates a separate alert, but it’s not clear how to achieve it.

  • What happened?

    • These are the notifications I receive when there is NoData:

    image
    image625×226 16.2 KB

  • Can you copy/paste the configuration(s) that you are having problems with?

    • Notification template (for summary):

      {{ define "alert_summary" }}
  {{ if gt (len .Alerts.Firing) 0 }} 
    {{ len .Alerts.Firing }} firing: 
    {{ range .Alerts.Firing }} 
    {{ .Annotations.summary }} 
    {{ end }} 
  {{ end }} 
  {{ if gt (len .Alerts.Resolved) 0 }} 
    {{ len .Alerts.Resolved }} resolved: 
    {{ range .Alerts.Resolved }} 
    {{ .Annotations.summary }} 
    {{ end }} 
  {{ end }} 
{{ end }}

    • Notification template (for title):

    { define "alert_title" }}
  🚨 {{ .Status | title }} Alert: {{ .CommonLabels.alertname }}
{{ end }}

I’m hoping someone can explain to me how I define a generic alert for all the DatasourceNoData that applies for all my alerts, and that has it’s own format.

2
3

Thank you, @jangaraj. I am creating a notification policy, as you suggested. Would you, if possible, suggest a syntax that can be used in my scenario? I tried this in my notification template, but it isn’t working:

{{ define "test_summary" }}
  {{ if eq (index .Labels "alertname") "DatasourceNoData" }} 
    No data coming for alert: {{ .Labels.rulename }}
  {{ end }} 
  {{ if gt (len .Alerts.Firing) 0 }} 
    {{ len .Alerts.Firing }} firing: 
    {{ range .Alerts.Firing }} 
    {{ .Annotations.summary }} 
    {{ end }} 
  {{ end }} 
  {{ if gt (len .Alerts.Resolved) 0 }} 
    {{ len .Alerts.Resolved }} resolved: 
    {{ range .Alerts.Resolved }} 
    {{ .Annotations.summary }} 
    {{ end }} 
  {{ end }} 
{{ end }}

image

4

Sorry, you must say what is not working explicitly. Don’t think that someone will know that implicitly.

5

I wasn’t able to wrap this around. As you suggested somewhere else, I created different notification policies to route DatasourceNoData and DatasourceError messages to another slack channel. By any chance, do you know if there is a way to edit the summary for those specific errors? I manage 2 different environments, and I want to include the some internal info to make it easy to identify.

6

not sure it will work but have you tried to template the summary annotation within the alert rule itself ?

7

Yes, sir, I tried it. I even use it for the summary, and the resolved summary, but I am not sure if there is a way to template a “DataSourceNoData” error, since what I understand is that those are completely separate alerts by default. Showing my current config:

image
image1791×673 60.4 KB

8

Hi @diegomadiedo1

Chiming in on this issue in case it helps.

For reference, the following documentation explains how the system works:

The architecture is as follows:

  1. An alert is handled by one (or more) notification policies, depending on matching labels.
  2. The notification policy routes the alert to one or more contact points.
  3. A contact point has a template assigned for a particular property, such as the description or title.

Someone can explain to me how I define a generic alert for all the DatasourceNoData that applies for all my alerts

I think you got this right but anyway…

  • define a notification policy that handles all DatasourceNoData alerts using the label alertname=DatasourceNoData at the root of the notification policy tree
  • disable the notification policy option Continue matching subsequent sibling nodes.
  • select a contact point in the notification policy, and your desired template to the contact point

It’s recommended to define generic templates instead of custom templates for specific alerts. This is not always desired, but it is recommended when possible. So you were on the right track but your template has some errors.

A template always receives a collection of alerts, so (index .Labels "alertname") is empty. This should instead look something like:

{{ range .Alerts.Firing }}
  {{ if eq (index .Labels "alertname") "DatasourceNoData" }}
    No data coming for alert: {{ .Labels.rulename }}
  {{ else }}
    {{ .Annotations.summary }}
  {{ end }}
{{ end }}

.... same for Resolved alerts

Related docs:

Templating is tricky.

When building custom templates, I recommend firing expected alerts and preview the notification template in Grafana.

Hope this helps!

1 Like