What Grafana version and what operating system are you using?
Grafana OSS v8.2.1
What are you trying to achieve?
Send notifications for multi-dimensional alerts
How are you trying to achieve it?
We are trying to play around with the new Unified Alerting from Grafana 8. We created a Grafana managed alert based on InfluxDB data-source with multi-dimensional capabilities.
We have a request returning multiple series (memory usage of a particular binary), that is reduced with a Max operation. We then configured a threshold for the alert and assigned a “team” label to the alert.
We want to send alert notifications through mail so we created a “test-email” contact-point. A notification policy routes all alerts matching with tag “team=my-team” to the “test-email” contact-point.
What happened?
The alert properly triggers with every combination of tags that triggered it but we receive a new email every 4 hours, with the exact same combination of tags and the exact same status (the alert fired and never resolved)
What did you expect to happen?
The alert should trigger once, and we should receive a single email, not every 4 hours with the exact same state.
Can you copy/paste the configuration(s) that you are having problems with?
Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
Did you follow any online instructions? If so, what is the URL?
Did you figure out how to resolve the issue? We have the exact same problem using:
Grafana 8.4.6
InfluxDB with the Flux Querying Language
Multiple Series are returned so several “child” alerts are created for each site
First alert that is being triggered is legitimate but then approximately every 4 hours (to be very exact it’s always 4 hours and 5 minutes) the alert is being fired again, although it hasn’t been resolved in the mean time.
Updated to Grafana 8.5.1 yesterday, because I saw that there were a lot of changes regarding alerting. However, the upgrade didn’t resolve the issue, still the same problem.
Oh my god… That was actually very easy and straight forward. Thank you so much vshaev!
Again, to sum it up:
Go to your alerting page and then to “Notification policies”
There is one root policy at the top which you can edit
Expand “Timing options”
Under Repeat interval you can set a value, when an already firing alert should be resent. If you don’t set a value it will default to 4 hours
Alternatively, you can leave the root policy as it is and instead edit your custom notification policy. Enable the option “Override General Timings” and set the repeat interval there.
