Hi Guys,
I'm evaluating use of Alerts through Grafana as apposed to using other tools.
Basically, I have set up grafana to pull metrics from Prometheus and have set up alerts (integrated with slack for notifications).
The Query is a rate calculation that is set to evaluate every 30s.
Alert is set for avg() of query(A, 5m, now). I understand that this runs every 30s and evaluates the overlap from the previous metric.
1. Is there anyway to apply a condition or a count over a sliding window.. for eg., I do not want to alert for a once-off spike, however, I want to raise an alarm when it happens 3 times over 30 minutes?
2. Is there a way to silence the alert for a time range?
3. Is there a way to clear the state (automated house-cleaning)
4. Is there a way to save the alerts into a data store?
Any support is highly appreciated !
Best Regards
Varaga