Detailed instructions on how to manage alerts for "DataSourceNoData" errors?

Grafana Alerting
1

  • What Grafana version and what operating system are you using?

    • Grafana 12.3.0 container on Ubuntu 24.04

  • What are you trying to achieve?

    • I’m using summaries to improve alerting in my Grafana setup, but when one of my datasources fails because of “NoData”, I get a normal summary with [no value] instead of labels and variables. I want to create customized alerts for the DatasourceNoData errors, and I understand that it creates a separate alert, but it’s not clear how to achieve it.

  • What happened?

    • These are the notifications I receive when there is NoData:

    image
    image625×226 16.2 KB

  • Can you copy/paste the configuration(s) that you are having problems with?

    • Notification template (for summary):

      {{ define "alert_summary" }}
  {{ if gt (len .Alerts.Firing) 0 }} 
    {{ len .Alerts.Firing }} firing: 
    {{ range .Alerts.Firing }} 
    {{ .Annotations.summary }} 
    {{ end }} 
  {{ end }} 
  {{ if gt (len .Alerts.Resolved) 0 }} 
    {{ len .Alerts.Resolved }} resolved: 
    {{ range .Alerts.Resolved }} 
    {{ .Annotations.summary }} 
    {{ end }} 
  {{ end }} 
{{ end }}

    • Notification template (for title):

    { define "alert_title" }}
  🚨 {{ .Status | title }} Alert: {{ .CommonLabels.alertname }}
{{ end }}

I’m hoping someone can explain to me how I define a generic alert for all the DatasourceNoData that applies for all my alerts, and that has it’s own format.

2
3

Thank you, @jangaraj. I am creating a notification policy, as you suggested. Would you, if possible, suggest a syntax that can be used in my scenario? I tried this in my notification template, but it isn’t working:

{{ define "test_summary" }}
  {{ if eq (index .Labels "alertname") "DatasourceNoData" }} 
    No data coming for alert: {{ .Labels.rulename }}
  {{ end }} 
  {{ if gt (len .Alerts.Firing) 0 }} 
    {{ len .Alerts.Firing }} firing: 
    {{ range .Alerts.Firing }} 
    {{ .Annotations.summary }} 
    {{ end }} 
  {{ end }} 
  {{ if gt (len .Alerts.Resolved) 0 }} 
    {{ len .Alerts.Resolved }} resolved: 
    {{ range .Alerts.Resolved }} 
    {{ .Annotations.summary }} 
    {{ end }} 
  {{ end }} 
{{ end }}

image

4

Sorry, you must say what is not working explicitly. Don’t think that someone will know that implicitly.

5

I wasn’t able to wrap this around. As you suggested somewhere else, I created different notification policies to route DatasourceNoData and DatasourceError messages to another slack channel. By any chance, do you know if there is a way to edit the summary for those specific errors? I manage 2 different environments, and I want to include the some internal info to make it easy to identify.

6

not sure it will work but have you tried to template the summary annotation within the alert rule itself ?

7

Yes, sir, I tried it. I even use it for the summary, and the resolved summary, but I am not sure if there is a way to template a “DataSourceNoData” error, since what I understand is that those are completely separate alerts by default. Showing my current config:

image
image1791×673 60.4 KB