Data Source-Managed Alerts Auto-Resolve on No Data — How to Prevent This?

Grafana Alerting
,
1

Hi all,

In Grafana Cloud, I’m using Linux server integration alerts with the “data source-managed” rule type. I noticed that if a server goes down and no data is returned, the alert (e.g. NodeFilesystemAlmostOutOfSpace) gets automatically resolved, even though the issue still exists.

This behavior causes real problems to be missed.

Is there a way to change how these alerts handle “no data”(like with “Grafana-managed” alerts)? Or any workaround to keep alerts active ?

Thanks in advance!

2

That’s a normal behavior of the Prometheus ruler, it relies on the fact that metric is always written. If you want to alert on missing metric you need to create another rule that would assert just that and use something like absent_over_time