We’re setting up grafana alerting based on metrics we have in a timescaledb database. The alerts within grafana seems to operate correctly (opening and closing based on the thresholds, etc). However, once we set up an opsgenie contact point and notification policy, we have noticed that the alerts (in opsgenie, not in grafana) close every 5 minutes (via a grafana api call into opsgenie). Our alerts have a group interval of 3m (ie. query every 3 minutes) and a “For” of 6m (ie. hold threshold for two consequtive checks).
Our notification policy has grouping disabled because we want each instance of an alert to be an individual alert in opsgenie (although we tried grouping as well but got the same result).
I am not sure why alerts that have been active for multiple hours/days, always send a close status to opsgenie every 5 minutes. This results in a constant barrage of alerts opening and closing in opsgenie which is clearly not the desired behaviour.
We are running grafana version 9.5.14
Any help is appreciated.
Thanks.
1 Like
Try to check alert state history and see what’s the for closing reason there.
There is no state history, or rather, grafana maybe doesn’t keep state history logs for very long. This alert has been active for over 11 days. The alert is not closing in grafana, which is why I do not understand why grafana alert manager is closing the opsgenie alert every 5 minutes just to reopen it when it’s actual alert state is not changing within grafana.
1 Like
How state history looks like, pls?
So it seems to be related to the “Group Interval”. For some reason, that interval is acting like the “Repeat Interval”. If I set my group interval to n minutes, then my alerts always close n minutes after opening, even though the alert state is not changing. Our alerts are always unique (ie. grouped by a unique id field) but I have also tried disabling grouping (with …) and get the same behaviour.
So what I can’t figure out now is why the alerts in opsgenie are getting closed every Group Interval but the alert in grafana is not (ie, it stays in alert state, as it should)
There is no state history. It enters alert state and stays there.
1 Like
