Slack notifications spamming for 1 and only alert 🤔 Grafana Alerting in HA mode?

Grafana Alerting
,
1

Hi community :wave:

I have a very strange issue :thinking:

Context

  • In my organization, we are using Grafana alerting (so far, so good :smile: )
  • We have recently setup HA for alerting (I hope I am not misleading with that information ^^)
  • Our Grafana instances are deployed with the Grafana-Operator on Kubernetes.

Observations

Observation1

  • A given alert has been in the firing state for more than 3d on 1 Grafana instance
  • On the other instance, that we can hit sometimes while going through the ALB ingress in front of both of them, we can see that the same alert is reported as “firing” only for 16h

→ :question: 1st question: How come?

image
image1280×558 115 KB

image
image1489×680 67 KB

:memo: Note:

  • The alert is based on a Prometheus query.
  • The Prometheus data source for this metric and alert is an ALB endpoint. Behind are 2 Prometheus servers.
  • We are aware that the data source does no “stickiness” regarding the Prometheus server targeted … We plan to use Thanos queries + Thanos-sidecars to address this issue later (This is not the main debate of this topic, I think :sweat_smile: )

Observation2

Here are some Grafana internal metrics from our 2 instances

image
image1920×1191 160 KB

→ :question: Why don’t we have the same amount of alerts in total

:memo: Both instances are using the same MariaDB instance as a backend.

Observation3

Here are some AlertManager cluster metrics

image
image2226×1476 322 KB

→ :question: I have to admit, I don’t know what metric I am supposed to look and I don’t see anything obvious :sweat_smile: Except the amount of messages sent/received that match the period where the alert was spamming out Slack channel and till we silenced it.

Does any of you have suggestions regarding that issue?
I can provide more info details if required :innocent:

Thanks for your time and help :pray: