Grafana alert not firing despite alert condition firing

[aaronsc]

Hello, I recently upgraded from Grafana v9.5.0 to 11.4.0. Before this update, my alerts worked perfectly fine. If a node went down, the alerts went into a firing state and I got an email.

After this update, all of my alerting rules are in a ‘Normal’ state despite alert conditions for the rule clearly being in a firing state. This is resulting in me not being alerted if a node has a problem.

How can a rule not be in a ‘Firing’ state if one of the conditions is in a ‘Firing’ state?

For an experiment, I created a new rule that only monitors the status of one of my nodes that is currently down.

Screenshot of the alerting rule for this node:

As you can see, the rule is in a Normal state.

If I click view on the alert rule, this is what I see:

In the bottom right, you can see that the alert condition is FIRING.

Rule configs:

As you can see, the condition is CLEARLY firing but the actual rule is still in a normal state. Can someone help me figure out why the rule is not firing like it should be?

Thank you for your time.

[aaronsc]

Still looking for a cause for this.

[yuriy.tseretyan]

1. What does state history says?
2. Go to “Active notifications” and see if there is any activity. There should be groups of alerts that are active
3. Go to contact points page, find the contact point and try “test” it and check if you receive the notification.

If it does not help. Check if there are logs for errors. If nothing suspicious (e.g. errors). Enable debug logs and check for messages like Putting alert, flushing and “Notify success” or “Notify failed”