Same alert firing twice within fraction of seconds - webhook

Grafana Alerting
1

Version: 9.2.2
Every time an alert reaches the threshold it is routed to an external service which is added as a webhook contact point. I checked in the service logs, Grafana is calling the service every time (for repeated attempts as well).

Multiple rules configured with the alert evaluation behaviour -
Evaluation every 1h for 0
No changes in notification policy timing.

Every alert fires twice within the same minute even.

Desired: They should be firing once every hour.

How to resolve this? Does the service need to send an acknowledgement back to Grafana for it stop sending the payload again?

PS: The alert stays in alerting state the whole time after changing from Normal once. Shouldn’t it go back to Normal state whie evaulating and then become alerting again?

2

The issue was because of multiple instance of Grafana setup in kubernetes which were running independently of each other, therefore each of them was receiving the request, processing and sending the alerts.