Alert Notification firing twice on Teams

[zeeshankhan]

Hi Team,

I have configured alert to verify and fire on every 1 hour but some how two alerts notification are getting fired at the same time.

These are my configuration screenshots:

Can anyone please assist me on this.

Regards,
Zeeshan

[georgerobinson]

Hi! What are the Timing options in your notification policy?

[zeeshankhan]

PFA

[georgerobinson]

Thank you! And to confirm you have no other notification policies?

[zeeshankhan]

along with that I have added this matching labels policy.

[niyantaz]

Facing similar issue with Slack. Alerts firing twice under 5mins through webhook. Any resolution?

[salmagor]

I have the same issue - anyone solved this yet?

[georgerobinson]

Are you running Grafana HA?

[zeeshankhan]

No I am not using Grafana HA.

[georgerobinson]

OK! What we can do is look at the debug logs to understand why this might be happening.

1. Enable debug logging in Grafana. You can do this with:

[log]
level = debug

2. Reproduce the duplicate notifications with Microsoft Teams

3. After receiving duplicate notifications look for the following logs:

DEBUG[06-13|20:50:55] Received alert                           logger=ngalert.notifier.alertmanager LOG15_ERROR= component=alertmanager orgID=1 component=dispatcher alert=Test[f66efaf][resolved]
DEBUG[06-13|20:50:55] flushing                                 logger=ngalert.notifier.alertmanager LOG15_ERROR= component=alertmanager orgID=1 component=dispatcher aggrGroup="{}:{alertname=\"Test\", grafana_folder=\"Test Folder\"}" alerts=[Test[f66efaf][resolved]]
DEBUG[06-13|20:51:26] Notify success                           logger=ngalert.notifier.alertmanager LOG15_ERROR= component=alertmanager orgID=1 component=dispatcher receiver=grafana-default-email integration=email[0] attempts=1

Of course, instead of receiver=grafana-default-email integration=email[0] it should show your Microsoft Teams integration instead.

If Grafana is sending duplicate notifications you should see multiple Notify success logs at the same time.

[zeeshankhan]

I am not sure if I can enable debugging. I am using azure managed grafana and I do not have the config file.

[zeeshankhan]

I am using azure managed grafana ver 9.4.12.

[georgerobinson]

In such case you might have better luck contacting Azure support as without debug logs I’m not sure how else we can figure out why you are receiving duplicate notifications.

[zeeshankhan]

Can you please look into my other request ?Change timezone Notification Policies / Edit mute timing(Azure managed Grafana(ver9.4.12))