Every time an alert reaches the threshold it is routed to an external service which is added as a webhook contact point. I checked in the service logs, Grafana is calling the service every time (for repeated attempts as well).
Multiple rules configured with the alert evaluation behaviour -
Evaluation every 1h for 0
No changes in notification policy timing.
Every alert fires twice within the same minute even.
Desired: They should be firing once every hour.
How to resolve this? Does the service need to send an acknowledgement back to Grafana for it stop sending the payload again?
PS: The alert stays in alerting state the whole time after changing from Normal once. Shouldn’t it go back to Normal state whie evaulating and then become alerting again?