Monitor weekly job with counter

Grafana Time Series Panel
1

Hey,
I have a weekly job that produce results and increase a relative counter. once the job done the counter value remains the same until the next (successful) run.
I want to have alert when there was no change in the counter value.
Thanks for the helpers!
Amir

2

Welcome @amirlasry

What is your datasource and can you please share your query that you have created to capture the count?

3

Thanks @grant2 ,
query is: delta(my_metric_of_kind_counter [1w])
datasource is Prometheus

4

Maybe this?

changes

So using something like this in Alert manager will trigger if no changes for the time interval changes(metric_name[5m]) = 0

5

Thanks, and what time period to use for evaluating the rule (assuming I don’t false alarms and getting alarm only if a week passed and no change in the counter (I have a weekly job that I want to monitor it’s results))?

6

If the job runs weekly, then wouldn’t you want to evaluate it every 1 week (or 7 days)?

7

so how do I ensure detect no value change it time? (job run every sunday)

8

Can you create an Alert Rule with your Prometheus datasource as the source, and then post your query in the box and test it to see if it yields results?

image
image2376×1049 185 KB

Once that is working, you would go on to create a Reduce expression in B and then a Math expression in C, and then alert off of expression C.

But for now, let’s just focus on getting query A working.

9

Thanks, managed :+1: