I need some advice regarding alerting.
I am running a MS SQL Server query.
I want it to alert every 24 hours at 7am if the count of the query is >0. It’s working fine so far using mute timings but I am evaluating the query every 1min for 5min with a small gap in the mute timing so that it alerts at 7am but this is causing a strain on my db as their are many queries that are running so often.
Is there a way for me only run the query every 24 hours or something so that it alerts at 7am and doesn’t cause such a strain?
Hi! Grafana is not CRON-like. It’s based on Prometheus, which is designed to evaluate alerts continuously. This means it’s not possible to tell Grafana to evaluate the rule at 7am every day, for example. In this case, I’m not sure Grafana is the best fit for your use case and you might want to consider using something else.
I’m currently testing this. Only problem is if the evaluation time is 24h and you put a smaller time in the ‘for’ period then it gives an exclamation thing… Will test it and see of it works but I think I tested it previously and it didn’t alert at 7am…
@ashramsarup07 I have exactly same requirement as you have I need to trigger the alert exactly at 7am based on the query, can you please let me know how you did that. Also if possible can you please share some screenshot for the configuration you did?
I setup the alert to evaluate every 1 minute for 5 minutes - and added a mute timing like shown below. (This mute timing is for 7am daily in south africa as it uses UTC… Also this mute timing is set like this to alert only from monday to friday at 7am if the threshold is met.)
I have one query if you evaluate every 1 minute for 5 minutes, agree that you have set mute timings so it will not trigger the alert but the query will still happen right ?
@ashramsarup07 But there is one issue I am seeing let say if the query is executing during the mute timings and there is a scenerio of alerting and it stays alerting when mute timings is over in that case it does not trigger the alert
I’m currently facing the same situation — I’m also running an MS SQL Server query in Grafana and want it to alert only once every 24 hours, ideally around 7 AM, without putting a heavy load on the database.
Did you manage to find a solution for this setup?
I’d really appreciate it if you could share how you handled the daily scheduling or query timing.
You could configure the alert rule to be evaluated every 24h, also the alert query should look for data in the last 24h. Ideally you should add a label to the query based on the date as well, so every evaluation would generate new alerts based on current date, it’d avoid grafana sending same alert next days. So there will be new alert for next day.
Also, would be good to configure the notification policy used by your alert to wait 24h+ before sending updates (this is the group_interval config, it’ll make your alert send once a day since new updates to old alert will not be sent before 24h), also updates the repeat_interval config to every 24h+ (so before it triggers, the next evaluation should mark old alert as resolved).
Remember to mark NoData state as Normal state, as the “Missing series evaluations to resolve” to 1 instead of the default 2, it’s supposed to mark as resolved the alerts with missing timeseries in the next evaluation.
Unfortunately, there is no way to run it at an specific time, but at least you’ll have it running once a day consistently at same time.
