Send notification whenever the alert is in the Firing state after every evaluation

Grafana Alerting
1

Hello,

Grafana version: 9.5.2

I have an alert that checks some data every hour. I set the evaluation interval to 1h, and the “for” set to 0, as I want to receive a Notification as soon as the condition is met.

The problem is, if the alert state is “Firing” for 2 consecutive evaluations, I receive only one notification.

How can I change this behavior, I want to receive a notification for every evaluation where the condition is met.

Thanks

2

hi @clevernessisamyth

do you have any alert grouping config in place ?

e.g.:

image
image1529×1281 64.2 KB

3

Hello @antonio.merello,

Thanks for the reply.

I don’t have any alert group setup; I use the default.

4

This is not supported by the Grafana Alerting. However, as in many complex and flexible systems, there are workarounds (or better say “hacks”) that can simulate the behavior you want to achieve.
Below are ways that I would not recommend using but they will let you do what you want to achieve.

  1. Produce a new state every time. Add some label to the result that changes every evaluation. Every evaluation you will make state manager create a new state and if it is alerting, send to the notification pipeline. By adjusting parameter group_wait in the notification policies you can control how fast you will get the alert. This will create flapping alerts effect unless you set contact points to not send resolve notifications.
  2. Set repeat_interval for your notification policy to be around the eval interval of the alert rule. group_wait will also help control the delay of the first notification. It will not guarantee that the alert is sent after each evaluation but will trigger notification at similar intervals unless the alert is resolved.
1 Like
5

Thank you very much!