Log-based alerts are firing but not seen in notification policy and not getting delivered

Grafana
, , ,
1

Hello, we run self-hosted setups of Grafana v10.4.6, Mimir 2.13.0 and Loki v3.1.0. For alerting, we use Mimir-Alertmanager and send our alerts to Slack + Pagerduty.

When we set log-based alerts (Grafana Alerts > Datasource Managed Alerts), the alerts are showing up as “Firing” on Grafana. But for some reason, they are not getting delivered on Slack or Pagerduty.

Upon further checking, the notification policy shows “0 instances” (Grafana > Alerting > Notification Policies). Check attached picture. I am suspecting this is a grafana bug, but I want to be sure that i am not making any configuration bugs from my side.

notif_policy_0instance
notif_policy_0instance759×121 6.35 KB

  • I already double-check to ensure that there is no label mismatch between the alerts and notification policy

Logs of Mimir-Alertmanage, Loki-Ruler and Grafana arenot showing anything abnormal. Need some help in finding source of this issue. Thanks

2

Check whether Alertmanager receives any alerts from ruler. There should be something like “Received alert” prometheus-alertmanager/dispatch/dispatch.go at aafa445c5501735423c4e10cd5e70c86a0b75516 · grafana/prometheus-alertmanager