Notification policies and Hard and Soft States

[mb19]

Hello there,

I am trying to manage alert notifications in Grafana, I have a dashboard that occasionally has “0 values” because telegraf has reading errors and that is recorded in the data source from which grafana reads.

I don’t know if you are familiar with Icinga2 which has “hard and soft states” when detecting a problem with a host/service, it re-checks the object a number of times before sending notifications.

My question is, is it possible to recreate this behavior with Grafana and only send notifications after an alert rule that is in “firing” has been evaluated, for example, twice? I want to be able to differentiate between temporary read failures and persistent failures.

By the way, I think the “Mute timings” option would not work since these errors are completely random over time.

Thanks a lot!

For exmaple, here I don’t want any notification:

But if the gap had been bigger I would want a notification

[jangaraj]

Check Pending period in your alert config:

If alert is evaluated every minute and pending period is 5 minutes = 5 consecutive checks must be failing (firing) to generate alert.
It’s good alerting practice to use pending period, because almost everything may have a short term problems (e.g. one because something is just rebooting, reconfiguring, …)

[mb19]

Thank you very much, but I think I’m a little lost, where is that option?

Could it be because of the version? Mine is “v9.4.7”

[jangaraj]

It has name for in your case.

[mb19]

thank you so much!