Hi,
I have deployed kube-prometheus-stack to my production k8’s EKS cluster.
I have used terraform to deploy dashboards, alerts, contact points and a notification policy.
Prometheus and alert manager are both sts’s.
The issue I have is that from time to time, my contact points and alert template disappear. All my dashboards and alerts are still present.
When I go to admin under the alerting tab. The following message pops up.
No admin configuration available.
Inspecting the browser logs, I see the following.
Everything else is working just fine. However with contact points and alert templates missing, it make my monitoring system pretty useless.
I am happy to provide any additional information you require to assist me with this issue.
Thanks