Alert state never changes to Firing

Grafana Alerting
1

I’ll begin with noting this is an upgraded instance running in OKD. The oldest deployment I see has it running 7.3.3 so it’s been upgraded from that version to current, 9.2.0 over 11 total deployments.

Here is the issue. I create a simple rule looking for disk used over 75% using the following

  1. PromQL - disk_used_percent{job=“linux_collect”} > 50
  2. Expression - reduce last of A (query), strict
  3. Expression - Math, $B > 75

Running query returns (4) values that are over 75. Preview Alerts reflects the same showing (4) volumes with Alerting state. Saving the rule is successful and it shows in the list.

After this though…the state of the alert never changes from Normal to Alerting. There are older state changes recorded prior to upgrading to v8 style alerting though I honestly see nothing in state changes for other alerts since late 2020.

This is an inherited instance and I’m not sure why state changes don’t occur as other instances I manage function as expected…both upgraded and created from version 9.x to begin with.

Any clues as to something I may have missed would be appreciated.

2

They say there are no stupid questions, but this one was.

Answer to my issue - this is a test instance and there was a value in OKD setting execute_alerts to false in the legacy [alerting] section. Thanks to me for finally discovering this and I hope this helps someone else.