- What Grafana version and what operating system are you using?
Grafana v8.0.3
- What are you trying to achieve? What happened? What did you expect to happen?
We have a cron job that runs every 2 hours, creating a CronSucceeded event at the end of each run. We would like to be alerted when there has been no CronSucceeded event in the past 3 hours.
- How are you trying to achieve it?
Since the event is a counter, we expect to see increasing steps, which we do:
They “reset” every once in a while because a new host takes over.
Now, I don’t really care which host is performing the job. So I add an aggregator—sum(application) group by.
Okay, they’re now a combined series. But why did the y-axis scale explode? That’s not my actual question, but I’m bringing it up in case it hints at a relevant problem.
So I decided to abandon that wonkiness and do a delta() for each instead:
(I cannot embed more than 2 images as a new user.)
Screen Shot 2022-01-02 at 06.20.14|626x500
Wonderful, a y-value of 1 every 2 hours is what I expect to see. But okay, now can I do the sum without the y-scale blowing up? Apparently yes.
Screen Shot 2022-01-02 at 06.22.16|626x500
Now, my actual question. I want to set up an alert that looks for an event (y=1) in the past 3 hours, i.e. query(A, 3h, now).
(I cannot embed more than 2 links as a new user.)
upload://awGy3IyfZPyW7hF6lxisEzlHFSe.jpeg (See reply below.)
But this alert has been going off seemingly randomly. (Of course it’s not random, but I don’t understand.) If I zoom into one of the alerted situations…
upload://yqRH4ufh8Ba5SiN3EoSAnKs0nJ9.jpeg (See reply below.)
…I can guess that the last() value after each of those peaks might indeed be 0, not 1. So I probably have to do this another way. (Another side question: Why is last() my only option for WHEN?)
How do I set up the alert I want? To repeat: Alert when there has been no CronSucceeded event in the past 3 hours.
If someone could shed some light on that as well as my intermediate questions, I’d be indebted. Thank you in advance.
- Can you copy/paste the configuration(s) that you are having problems with?
N/A
- Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
I don’t believe so.
- Did you follow any online instructions? If so, what is the URL?
Nope.