I currently track different customer (>450) app outages by querying for a specific logged message in my Graph Panel using Elasticsearch as my data source.
To try and get an alert whenever there is an additional customer added to my Graph, I have the following alert condition set up:
avg() OF query(B, 5m, now) IS ABOVE 0
The problem, however, is that I also get false alerts when a customer outage event has concluded, no longer logging what I’m querying for to get my count and the value DECREASES.
What can I do to have the alert trigger only when there is a positive increase in the total count?