Alerting when metrics every 15 minutes and null data

Hello,

I am trying to set up alerting for an elasticsearch datasource. I have set up Anomaly Detection on elastic and I am retrieving the anomaly values to print it and alert on Grafana (in order to have a centralized alerting system). When there are anomalies, I have a data point with a value between 0 and 100 every 15th minute (10:00,10:15,10:30, etc) and when there are none, the data point is a null value.

I’d like to alert when two consecutive data point are over 80. My problem here is the null values. I need to tell the grafana alerting that there is a point every 15 min but the alerts triggers for the null values inbetween. I tried setting up Drop NNN and Keep Last behavior but in that case, if there is a null value and the alert was firing, it does not go back to OK. I also tried to set up the Interval to 15m but Grafana detects a null value 40s later (I don’t understand this behavior).

TLDR; I need an alert that fires if there are 2 data points over 80 at 10:15 and 10:30 and to go back to OK if the following data point at 10:45 is null.

Not sure how to implement this out of the box, however, a third-party alerting service might help here. For SIGNL4, we have implemented something like this:

  • Wait for two events matching a certain condition within a period, then trigger an alert
  • If there is an OK condition, then close the alert
  • Plus, only alert users within certain times, e.g. only wake users up if it is really necessary
    In this case you need to send an event for a null value from Grafana, which should be possible, I think.
    I hope this helps you.

You can do so by setting

(right under pending period setting). As for the interval and pending period settings - I think that setting interval to 15m for the group and pending period to 30m should be alright (I think).