Hi,
In Grafana 9.1.7 I have created a Mimir managed alerting rule as described here. The rule itself works, but when firing it does not send alerts to the webhook I have specified as default contact point. The default contact point works fine, when Grafana managed alerts I have set up fire it will send alerts to the webhook. Does the Mimir managed alerting need some additional configuration to make it work?
I’m experiencing the same issue here…
Grafana alerts from dashboards are working fine.
Managed alert from Mimir/Loki triggers but no notification send
did anyone manage to solve this problem?
Did anyone solve this?
I am experiencing the same issue.
Hi! 
You cannot send alerts from Mimir to Grafana. Grafana is just a user interface when using Prometheus and Mimir. You’ll need to run an Alertmanager and tell Mimir to send alerts to it instead. You can of course still use Grafana to view the alerts in that Alertmanager and the rules in Mimir.
Hey George!
Thanks for your answer.
I am using the Slack webhook.
I configured rules using both Mimir and Grafana ruler, and webhooks using the Grafana and Mimir Alertmanager (through the Grafana UI). However, only when using Grafana Alertmanager the webhook works.
I will add that for some reason I can not send a test notification for the webhooks I configured with the Mimir Alertmanager.
Am I not suppossed to do it this way (through the UI)?
Thanks!
You can see the alerts from Mimir in the Alertmanager, correct? If so, I would turn on debug logging on the Mimir Alertmanager and try to see what’s happening. You’ll want to look for logs like flushing and Notify success if it’s working. Otherwise, I would recommend looking for error logs to help track down the issue.
Thanks, I’ll try that.
I can see the alerts in the Mimir Alertmanager configuration files + object storage.
Mmm. What do you mean by configuration files? You should be able to see the alerts in the Alertmanager UI, or in Grafana (when selecting the Mimir Alertmanager from the drop-down).
I’m not accessing the Alertmanager directly.
I created alerts in Grafana, using the Mimir ruler (there’s a choice wether to use Grafana/Mimir or Loki/Mimir or loki recording rule. I chose Mimir or Loki). Then I can see these alerts in Grafana under “Alert rules → Mimir / Cortex / Loki”.
Sorry for the confusion. When I said config file, I meant I can see the webhook template in the Mimir Alertmanager local storage. There’s also “notifications” file yet it is empty…
I think maybe I’m missing something 
Ah. You’re talking about alert rules, not alerts. No, I was asking about the alerts. Can you see those in the Mimir Alertmanager under the “Groups” page?
Yes exactly, my bad.
No I can’t see them there 
Only under Grafana Alertmanager.
Maybe they’re using the Grafana Alertmanager? How can I configure alerts to use the Mimir Alertmanager?
I thought using a notification policy with a label will work, but now I understand that it doesn’t.
It’s not possible to send alerts from Mimir to Grafana Alertmanager. It you can see them there, it sounds like they have been created as Grafana Managed Alerts instead of in Mimir?
But then where can I configure Mimir alerts?
Is configuring alert rules under “Alert rules → Create alert rule → Mimir or Loki alert” not the same as configuring Mimir alerts?
By the way, thanks for all the help, really appreciate it 
This is correct. It should be impossible to see firing alerts from alert rules in Mimir in the Grafana Alertmanager because Grafana cannot receive alerts from other rulers, whether that’s Prometheus, Mimir or Loki.
Perhaps you could share a screenshot of the Alert rules page and the Groups page?
You’re right, only the Grafana Managed alert appears under Grafana in Groups. But the Mimir alerts don’t appear at all!!
Attaching pictures:
Are you using Grafana Cloud or Mimir open source? If the latter, is Mimir configured correctly to use the Alertmanager? (Optional) Grafana Mimir Alertmanager | Grafana Mimir documentation
Just wanted to update that we solved it!
A few points-
Thanks for your help George!
