My alert has been firing since Jan 10 and the notification policy is set with a Repeat Interval of 52w. However, I am receiving a notification for this alert approximately every 120 hours. Here are the last two timestamps from the notifications.
Here is where I set the Repeat Interval:
I am unable to find an explanation for this behavior. Any help would be appreciated!!
Your Grafana config is correct. Unfortunately, there is Prometheus alert manager bug:
opened 07:27PM - 25 Jul 23 UTC
type/bug
area/alerting
needs investigation
### What happened?
When I create an alert with a very high repeat interval(mult… iple weeks), I am getting realerts every 5 days.
This is very similar to the alertmanager defaults described in this issue: (the default data.retention is 5 days)
https://github.com/prometheus/alertmanager/issues/2934
In alertmanager this value can be increased using a command flag you can add when you run it. I am not able to find any options in Grafana to change this value.
### What did you expect to happen?
Alerts to repeat at the interval set, and not sooner.
### Did this work before?
No
### How do we reproduce it?
1. Create an alert that will be matched with a notification with a repeat interval greater than 5 days
2. Alert will repeat at 5 days.
### Is the bug inside a dashboard panel?
_No response_
### Environment (with versions)?
Grafana version v9.5.1 (bc353e4b2d)
### Grafana platform?
A downloaded binary
### Datasource(s)?
_No response_
