Hi All,
I am pulling Job data from Prometheus and working with custom queries that give me the hour and minute that the Job ended/started. These Jobs run once per day and start/end around the same time. I want to know if these times deviate which is why I am monitoring them.
My main issue deals with new days. How do I ensure that I am not reading yesterday’s Job? For example, yesterday the Job ended 10 minutes late and today the Job is ending on time. For 10 minutes (if querying the last 24h) I will be getting both end times.
Ideally I would like to “reset” my Grafana dashboard at a specific time (maybe midnight depending on the batch of jobs) and start reading any new data I get. I also want the capability to go back to previous days to view the status’/metrics from that day.
Example Query: (hour(timestamp(0 < JobSecondsSinceEnd{} < 300)) + round((minute(timestamp(0 < JobSecondsSinceEnd{} < 300)) - (0 < JobSecondsSinceEnd{} < 300)/60))/100)
Result: 13.35 or 13:35
In the example, I am retrieving the timestamp (hour and minute) of when the JobSecondsSinceEnd is less than 5 minutes. This is the sampling interval we are using on Prometheus. I understand this does not get the minimum timestamp, feel free to help me with that part as well.
Example Graph of JobSecondsSinceEnd{} for a Job running once per day:
I was thinking about only querying the last 20h that way the statistics will begin to reset before the batch of Jobs starts for day. But this does not seem like the most reliable method.
I appreciate any help that people can give. I am very new to Grafana and Prometheus so I am hoping there is a simple solution somewhere out there.
Thank you.