I'm evaluating use of Alerts through Grafana as apposed to using other tools. Basically, I have set up grafana to pull metrics from Prometheus and have set up alerts (integrated with slack for notifications). The Query is a rate calculation that is set to evaluate every 30s. Alert is set for avg() of query(A, 5m, now). I understand that this runs every 30s and evaluates the overlap from the previous metric. 1. Is there anyway to apply a condition or a count over a sliding window.. for eg., I do not want to alert for a once-off spike, however, I want to raise an alarm when it happens 3 times over 30 minutes? 2. Is there a way to silence the alert for a time range? 3. Is there a way to clear the state (automated house-cleaning) 4. Is there a way to save the alerts into a data store?
Any support is highly appreciated !