Grafana Alert sent 3 alert notifications at once

Grafana Alerting
21

Hi @dmytrodrachov ,
I’m experiencing the same issue with AWS Managed Grafana.
Were you able to resolve this issue together with AWS Support?

22

Hey @localhorst unfortunately no. They have a bug, but eta is unknown.

24

hi there. have the same problem with HA Grafana (not AWS), but notification channel is email, so have 2 emails for 1 firing alert.

did you find the solution?

25

Same problem here with Amazon Managed Grafana firing 3 separate email via SNS.

26

the same issue - Amazon Managed Grafana firing 3 slack the same notifications… in Gragana it shows only one…
Hi @dmytrodrachov did AWS support solve your issue?

27

Slack notification same issues. 3 messages

28

Same issue also for me, is there anyone who knows how to fix?

29

I’m also having this problem with Amazon Managed Grafana. I hope a fix gets implemented soon

30

Same here with Amazon Managed Grafana. 3 alerts notifications instead of one.

31

Hello, I found a solution for my specific case. The problem was that I had 3 grafana replicas, so I recieved 3 alerts.
I scaled the pod to 1, and now it work as I expected, I recieve just one alert notification

32

@munafoa Just a quick question over here ? is your grafana locally deployed or you are using Amazon Managed Grafana?

33

This doesn’t sound like you are using Amazon Managed Grafana, but maybe you can confirm. Thanks! @munafoa

34

Hello guys, no i’m not using Amazon Managed Grafana

35

I just spoke with AWS support, this issue is as suspected which is the HA mode which the architecture is built upon for Amazon Managed Grafana. There is no hard date set in place, but they are hopeful to have a fix in place by next month August 2024. In the mean time, with sns as a contact point it might be possible to implement a deduplication message with TTL in place. Havent tried it, but will report back if we decide to test it out.

36

Update: Amazon has began implementing a fix to prevent multiple alerts (this week) the week of Sept 18. I can confirm I am now only receiving one alert rather than three. My workspace is located in us-east-1. The patch is implement automatically by AWS nothing on your end should need to be done.

More information and updates about this change can be found here Notification deduplication for Unified Alerting · Issue #47 · aws/amazon-managed-grafana-roadmap · GitHub