Hi everyone. I create an alert to understand how it works. Usually when the alert went firing the alert notification is send, but when the alert is normal the notification sometimes is send and sometimes not. It’s not constant, and for this i can’t understand what’s the problem. Maybe one time is all ok and then no notication arrives.
Thank you.
and this is the notification police
Hi! Can you share more information about the problem you are trying to solve? Grafana does not always send notifications whenever an alert changes from firing/resolved and back again, it depends on your Alertmanager timings (Group wait, Group interval and Repeat interval).
Unlike some other systems, Alertmanager turns discrete events (firing and resolved alerts) into a continous series of notifications, dictated by Group wait, Group interval and Repeat interval. Grafana does not send notifications for every event every time.
You’re right, it’s hard to explain what i mean. I start creating an alert; once it’s firing i received the alert notification and the same when the alert return into normal state. If the alert goes firing again and then he return to normal in the same way, i don’t receive the notification, unless the alert has no data and i receive the resolved notifcation because of no data.
It sounds like your specific example is dependent on Group interval.
When the alert goes into firing, Alertmanager waits Group wait seconds before sending a firing notification.
When the first firing alert is resolved, Alertmanger sends a resolved notification.
Depending on how much time has passed since the alert was resolved, when the alert goes into firing again it can sometimes wait Group interval seconds, not Group wait seconds. This depends on the amount of time that has passed since the alert last resolved, or if there are two or more alerts in the same group, since the last alert in the group resolved.
The default configuration for Group wait is 30 seconds and for Group interval is 5 minutes.