Following a migration to Grafana 9 (from 8), I have switched to unified alerting and had to update all the rules.
Generally speaking it is working, however multiple times per day I’m getting dozens of notifications “DATASOURCE NO DATA”, because it seems that Grafana can’t evaluate the rule (Prometheus).
I’m not sure what happens, but this is really disrupting the monitoring.
Is there a workaround to this behavior? Maybe like a timeout / retry in case of a severed communication with the datasources?