Alerting Notification Policy - DatasourceError / DatasourceNoData

Grafana Cloud
,
1

  • What Grafana version and what operating system are you using?
    Grafana Cloud v8.3.5

  • What are you trying to achieve?
    We have alerts configured with Alert state if execution error or timeout = Error and want to be able to define a different notification policy for any error alerts so that a different team are notified of errors. The existing notification channel should not receive these alerts.

  • How are you trying to achieve it?
    Alerts defined as Alert state if execution error or timeout = Error
    Existing Notification Policy using a filter of alertname != DatasourceError to filter out these alerts.
    Once the above is proven to work, then the intention is to create a new Notification Policy to route these alerts to a dedicated channel

  • What happened?
    DatasourceError notifications are still received into the existing notification channel

  • What did you expect to happen?
    DatasourceError notifications should not be received in the existing notification channel, but “genuine” alerts should continue to be routed.

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?
    Alerting: Create DatasourceError alert if evaluation returns error by gerobinson · Pull Request #41869 · grafana/grafana · GitHub

2

Hi Patrick,

Just for clarification, you are attempting to edit an existing rule you had? I believe you are hitting a known issue we are working on right now. Do you mind opening a ticket with our Support team describing this?

3

Hi Aaron,

Apologies for the dely in replying. I was editing an existing rule, but it would not be a large task to delete this and create a new rule from scratch if you feel that this would be a worthwhile exercise.

Cheers,

Patrick

4

Hi Patrick,

What version of Grafana Cloud are you on? You can find this information by logging into your instance and hovering over the question mark icon at the bottom left.

5

Hi Aaron,

Version 8.3.5

Cheers,
Patrick

6

Hey Patrick,

Sorry about that, I missed that in your original post. There is currently a bug affecting 8.3.5 that is causing issues with rule editing and creation. It is fixed in 8.3.6, which is slated for release next week. Once you see the version change, if you’re still having issues, please open a Support ticket for further review.

Otherwise, you should be all set with the release next week.

1 Like