Hi folks!
I hope you are having lovely day. I am having so much pain around this current issue that I am facing so I would be so grateful if you could take some of your time to help me if you can.
Around 2 weeks ago, our Grafana stop sending us notifications generally (both for firing and resolved alerts) towards Teams. We were using version 10.4. Going throughout several documentations and forum answers, we concluded that it might be the problem on Teams and so it was! Webhooks which we used to use were depreciated and we were urged to move to Workflows. All of that was set up and test notification was sent and well received (although there is an error in Grafana when it that webhook validation failed, but I believe that has to do something with Workflows returning wrong status code although POST was successful).
The following day, we started received FIRING alerts for our system and there were bunch of them (one per every minute). This was unusual behavior because, before notifications stopped coming in the first place, we were only getting one notification per alert and afterwards (if resolved) we would get resolved notification. So this got us by surprise because we didn’t configure/change anything in Grafana itself except the webhook URL which now pointed to Workflows server. Ultimately, we realized it is a group interval that needed to be increased in order for us not to get FIRING alerts each time we evaluated the alert. Now that that is settled, we are facing the issue that we never get RESOLVED state for that alert. We have tried everything:
- brought version to 11.0. in order to see whether it is a deprecation problem,
- tried creating new notification policies and contact points,
- tried reshaping notification template,
- tried bringing down group interval time in order to see whether RESOLVED state would be sent,
and nothing of it worked.
Has anyone had this type of problem and resolved it? If so, we would be immensely grateful if anyone can help us.