One alert sending multiple notifications to Google Chat space

Grafana Alerting
1

Hi all, I apologize if this is a duplicate topic but I am having difficulty finding any similar examples of this issue. Any help is appreciated! -

For each alert I have setup in Grafana that uses a GChat contact point, when an alert is fired (or resolved) Grafana sends 3 identical alert chat messages to the space. While not world-shattering, this does make notifications annoying and more difficult to read. Please see example image below.

If I test a contact point, it behaves normally and only sends 1 message. Additionally, I do not believe this is an example of a “flapping” alert, as the alert moves between firing and resolved in an expected time frame.

Running Grafana v9.0.0

Thanks in advance!

image
image409×897 36.6 KB

2

have you find a solution for this? I have the exact same problem.

3

Hi @mcaxtr
By default Grafana groups alerts by grafana_folder and alertname. You can disable this grouping:

Alerting -> Notification Policies -> <your_policy> -> Click Override grouping and set Group by to Disable

Grafana disable alerts grouping
Grafana disable alerts grouping1165×1037 45.3 KB

After that you should get only one notification per triggered/resolved alert check.

Note: Default repeat notification interval is 4h and you can also override that.

Best regards,
ldrascic

4

Hello @ldrascic,

Thanks for replying my message.

This is a different issue though. It seems to be related with a bug in latest versions of grafana for HA setup. The messages gets duplicated by the number of instances in the cluster. You can read more here.

Best regards,
mcaxtr