We are using Grafana alert manager and there are certain alert rules those are supposed to send alert on Slack but we are missing most of the alerts. However if I test slack alert, the alert is sent to Slack instantly. Earlier we thought alerts are not generated, but then on alert manager can see that they are being fired properly as per condition.
To cross verify, we added “email” contact point also and there we can see all alerts are properly being sent to email. So following things are confirmed :
- There is no issue with slack notification from Grafana as can send TEST notification successfully.
- Alerts are being fired when they should, so alerts and firing is also fine.
I am not sure where to look, there is nothing relevant in logs either.
In routing policy, default is set to SLACK, then in sub policy have added slack again and then mail notification. I have no matchers set as want to send all alerts to slack.
In logs mostly seeing this :
logger=ngalert.sender.router rule_uid=wAVSpqm4k org_id=1 t=2023-06-20T22:04:04.632679518Z level=info msg="Sending alerts to local notifier" count=1
logger=ngalert.sender.router rule_uid=f633c787-5984-47f2-92c8-75b534435213 org_id=1 t=2023-06-20T22:04:06.621795387Z level=info msg="Sending alerts to local notifier" count=4
When I run slack test notification, then I see this :
logger=ngalert.notifier.slack t=2023-06-20T22:07:52.489233162Z level=error msg="Missing receiver"
logger=ngalert.notifier.slack t=2023-06-20T22:07:52.489408306Z level=error msg="Missing group labels"
So what I am missing ?