I have 2 instances(9.4.7) and set up HA alerting. backend db is PG.
during the helm upgrade for grafana, the firing alerts would be retriggered. I consider when pod gets inited, it cannot touch/detect well with rest of pods at startup.
in grafana, we have a lot of monitoring. It’s hard to make a batch silence for all alerts as they are matched by different tags.
I even enarge the “for” param to 5m, it’s still retriggered.
It is easy, set just smart condition, e. g.
alertname !~ Ihopethistringisnotusedinanyalertname
Please note I used negative matching, not matching.
oh, this is good as a bending kick. But still not a formal way.