How to alert upon non-negative delta of number of log entries in an interval?


I have a graph that counts number of certain log entries in Elasticsearch (like errors).
Now I want to be alerted when the difference between the last value and the previous value is not negative (i.e. when I check for number of error entries within 5m interval, then if there are no new errors the value should be lower in the next 5m interval, unless it is already 0).

Query settings:

Query: (kubernetes_container_name.keyword:“container1” OR kubernetes_container_name.keyword:“container2”) AND ((NOT exists:level.keyword) OR level.keyword:error)
Metric (1): Count
Group By: Date Histogram
Query options: Min interval/Interval: 5m

Alert settings:

Evaluate every: 1m
For: 0m
WHEN diff() OF query(errors, 5m, now) IS ABOVE -1
AND last() OF query(errors, 5m, now) IS ABOVE 0

Now when I inspect query, the last data within 15m interval are:
1, 2, 0, 2

yet the rule test evaluates both conditions as FALSE.


It seems it’s kind of bug. This is not working in 7.5.7 but using the same I get some alerts in 8.0.6.