Hi all,
I’m trying to set up a Grafana 8 stack with Loki and Prometheus with docker-compose file. As Grafana 8 provides nice unified alerting, I got a little bit confused at its correct usage and its difference with prometheus-Alertmanager. As my understanding, there are 4 possible ways to set up the alerting for this stack, I hope people can correct me if my understanding is wrong. :
- create and mount notifiers.yml into ‘/etc/grafana/provisioning/notifiers/’ directory. but for me in Grafana UI interface ‘contact points’ tab page it doesn’t show anything, seems not loaded. There are people having the same issue here:
Notification policies and contact points are missing after upgrade to 8.3.1 · Issue #42957 · grafana/grafana · GitHub
Provisioning contact points - create contact point(s) in Grafana UI.
Both 1 and 2 don’t need to add alertmanager config in Loki&Prometheus config file, and don’t need to set up alertmanager container at 9093. 1 should be the persist way of 2. - set up alertmanager container at port 9093, add alertmanager config in Loki&Prometheus config file.
- set up alertmanager container at port 9093, add alertmanager config in Loki&Prometheus config file, and in Grafana UI ‘contact points’ create a new contact point with type ‘Alertmanager’.
If my understanding is correct, the way 1 is the best choice for me, I can persist the notification channel definition through notifiers.yml file, and as Grafana said, this unified alerting should replace a separate prom Alertmanager which sounds really great, but it doesn’t work for me now.
I also don’t understand well what does it mean by create a contact point with Alertmanager type, it sounds duplicate, if we have prom Alertmanager, we don’t need to set contact point in Grafana anymore. For example if my notification channel is through Slack, I can define it in Alertmanager config file, or define it in /etc/grafana/provisioning/notifiers.yml file, why need both of them?
Can people help me to make this clear?