I need some advice regarding alerting.
I am running a MS SQL Server query.
I want it to alert every 24 hours at 7am if the count of the query is >0. It’s working fine so far using mute timings but I am evaluating the query every 1min for 5min with a small gap in the mute timing so that it alerts at 7am but this is causing a strain on my db as their are many queries that are running so often.
Is there a way for me only run the query every 24 hours or something so that it alerts at 7am and doesn’t cause such a strain?
Hi! Grafana is not CRON-like. It’s based on Prometheus, which is designed to evaluate alerts continuously. This means it’s not possible to tell Grafana to evaluate the rule at 7am every day, for example. In this case, I’m not sure Grafana is the best fit for your use case and you might want to consider using something else.
You could do a mute timing for all the time except when you want the alert
Did you even read what I said above?
Sorry. Working and doing this. How about the evaluation time? It may not be exactly accurate but you could do the evaluation every 24 hours
I’m currently testing this. Only problem is if the evaluation time is 24h and you put a smaller time in the ‘for’ period then it gives an exclamation thing… Will test it and see of it works but I think I tested it previously and it didn’t alert at 7am…
@ashramsarup07 I have exactly same requirement as you have I need to trigger the alert exactly at 7am based on the query, can you please let me know how you did that. Also if possible can you please share some screenshot for the configuration you did?
@ashramsarup07 can you please reply ?
Apologies, I have a backlog of emails.
I setup the alert to evaluate every 1 minute for 5 minutes - and added a mute timing like shown below. (This mute timing is for 7am daily in south africa as it uses UTC… Also this mute timing is set like this to alert only from monday to friday at 7am if the threshold is met.)
-hope this helps.
@ashramsarup07 Thanks for the reply.
I have one query if you evaluate every 1 minute for 5 minutes, agree that you have set mute timings so it will not trigger the alert but the query will still happen right ?
Yes the query will still be running. The mute timing will only stop the alert from sending.
is there a way to even stop executing the query too ?
No. Please see the comment above by [georgerobinson] on October 11.
@ashramsarup07 But there is one issue I am seeing let say if the query is executing during the mute timings and there is a scenerio of alerting and it stays alerting when mute timings is over in that case it does not trigger the alert