Firing alerts get resolved with no data

Grafana Alerting
1

I’m getting Firing alerts resolved with grafana_state_reason=MissingSeries annotation despite setting No data hadling → Alert state if no data to either to Keep Last State or Alerting.

Do I understand it wrong how it should work? I want the alerts to stay in Firing state when corresponding series gets missing in query result.

We are running v11.3.

Thank you.

2

Hi Alex,

This is the expected behaviour according to the docs.

Stale alert instances:

An alert instance is considered stale if its dimension or series has disappeared from the query results entirely for two evaluation intervals.
Stale alert instances that are in the Alerting , No Data , or Error states transition to the Normal state as Resolved.

grafana_state_reason:

Stale alert instances in the Normal state include the grafana_state_reason annotation with the value MissingSeries.

3

Hi, thanks.

But there are configuration options (in section “3. Set evaluation behavior” of an alert rule) Configure no data and error handling that can be used to override this behavior, I think. I tried “Alert state if no data or all values are null” to set to:

  • Alerting - Sets the alert instance state to Pending and then transitions to Alerting once the pending period ends. If you sent the pending period to 0, the alert instance state is immediately set to Alerting
  • Keep Last State - Maintains the alert instance in its last state. Useful for mitigating temporary issues.

Neither worked for me. I had alert Alerting (Firing), series disappeared, alert state went to Normal and I received the notification Resolved | MissingSeries.

Thanks!

4

This is expected. NoData and options to execute it as X is related to a situation when query does not return any points. Only in this case it is classified as no data.
A situation when a series is missing is not considered as NoData. In this case, the alerting keeps the last state of the alert instance for 2 rule evaluation cycles and if it’s still missing, it evicts it from the state with transition Normal (MissingSeries).
In Grafana 12 we will let users configure that number of evaluations.

5

Thank you, Yuriy. I misunderstood what NoData means.

May I hope that the new option in Grafana 12 will allow users to set really large number (or even infinite) so it would effectively allow keep the last state forever?

Thanks.