We’ve been using silences as a way to suppress alerts for certain customers. A 10 year silence doesn’t sound perfect, but at the same time it’s a simple, operator accessible way to stop alerts based on labels which remains visible at an appropriate level.
Now though we’re refreshing our overall environment and building with Terraform, and as such Silences can’t be set via Terraform. I get the fact that you can’t / shouldn’t set a “temporary” resource in that model.
So what else should I do to squash alerts for these few specific labels? I could exclude the labels in the source query, but that would require a code redeployment as we will have a read-only prod environment (bar silences I guess… huh, now that’s another issue to think about!). We could also filter by using notification polices to catch and stop exceptions before they reach a live contact point. Again, would be a code deployment, but maybe that’s unavoidable, and fairly rare.
On the Terraform side, we could hack in a null provider to run a script that does inject these silences. Icky, but again, I get why the grafana provider does not implement silences based on their defined use case.
Any other thoughts about this would be appreciated.
(One thing I don’t understand though, is why the silences API call isn’t part of the provisioning API either way. If it’s exposed at /api/alertmanager/grafana/api/v2/silences, why can’t I also reach it under /api/v1/provisioning/silences? I suppose it’s the same logic as for the TF provider, but I feel a little anxious going all the way into the first API endpoint.)