I am trying to set up alerting for an elasticsearch datasource. I have set up Anomaly Detection on elastic and I am retrieving the anomaly values to print it and alert on Grafana (in order to have a centralized alerting system). When there are anomalies, I have a data point with a value between 0 and 100 every 15th minute (10:00,10:15,10:30, etc) and when there are none, the data point is a null value.
I’d like to alert when two consecutive data point are over 80. My problem here is the null values. I need to tell the grafana alerting that there is a point every 15 min but the alerts triggers for the null values inbetween. I tried setting up Drop NNN and Keep Last behavior but in that case, if there is a null value and the alert was firing, it does not go back to OK. I also tried to set up the Interval to 15m but Grafana detects a null value 40s later (I don’t understand this behavior).
TLDR; I need an alert that fires if there are 2 data points over 80 at 10:15 and 10:30 and to go back to OK if the following data point at 10:45 is null.
Not sure how to implement this out of the box, however, a third-party alerting service might help here. For SIGNL4, we have implemented something like this:
Wait for two events matching a certain condition within a period, then trigger an alert
If there is an OK condition, then close the alert
Plus, only alert users within certain times, e.g. only wake users up if it is really necessary
In this case you need to send an event for a null value from Grafana, which should be possible, I think.
I hope this helps you.
(right under pending period setting). As for the interval and pending period settings - I think that setting interval to 15m for the group and pending period to 30m should be alright (I think).
