Grafana v8.1.2 (103f8fa094) Alert Always Pending


I have installed 8.1.2 and moved to the ngalert alerting so can get metrics sent out within the alert email BUT I have an issue with alerts not firing; if the timings are set Evaluate = 1m and for = 1m30s it fires after or around that period of time but if the timings are set to anything bigger they go into pending state - wait for the eval period to finish then add another of the same into pending state, when the next eval period happens it then drops the older pending from pending and adds a fresh one.

The query is monitoring memory on a single server - Added the screen grabs of what is happening below:

Can anyone tell me whats going on with this as its proper melting my head :slight_smile:

As you can see for the pic above there are 2x pending for the same server and these just rotate

Any help with this esp if I am not understanding how this works would be greatly received

With Grafana v8.2, we are struggling at the same place. (@stuartpelton ) Did you find any solutions?

Just in case this is still a topic:
The behavior is caused by the used labels. The alert includes the label values to calculations. If the label changes, the alert jumps back to pending, even if continuous values are below/above limits.

For us, moving the values from labels to text content solved the issue.

@gn0m3r4l apologies for the late reply, yes I came to the same conclusion, the metric changed so created a new alert.