I have a batch job which is scheduled to run once on each week day. On Monday the batch runs at 6:00 AM and on remaining week days it runs at 8:30 AM.
Currently working on setting up on Grafana alert which will be triggered if batch fails. The job runs in the AWS environment and Grafana receives the status details (successful, failed metrics) through CloudWatch datasource.
The alert should be moved to firing status once the batch fails, and moved back to Normal status in either of below scenarios.
- Batch is manually run successfully.(OR)
- Batch runs successfully on the next day.
Both these cases will indicated by a message in the logs.
Given the above scenarios, the alert query needs to fetch at least data for Time range of 24hr to identify the status of batch for the previous day.
Based on this I was thinking of below alert condition:
failure_count > 0 && success_count < 1 && manual_success_count < 1
This condition works for the batch failure and success on the same day.
However this condition is not going to work in the below scenario for multiple days:
Let’s say
- Batch failed to run at 8:30 AM on Tuesday morning and alert is moved to firing status. (failure_count = 1)
- Issue was fixed and the batch was run manually at 4:00 PM and the alert moved to Normal status. (manual_success_count = 1)
- On Wednesday morning 8:30 AM, the batch has failed again. (failure_count = 1)
In this scenario the alert condition is not met as the manual_success_count = 1 in the last 24 hours.
I am new to Grafana and not sure whether this can be achieved. Tried to go through the documentation and different sources on web but so far could not find something useful. Please help.