Microsoft Teams Notification Problem

I’m having an issue starting at around noon EST today with Grafana sending alerts to Microsoft Teams channels. When an alert gets triggered, it’s spamming the channel with ~15 alerts, waiting a little bit and spamming with more alerts. This has been setup and working for a long time now. I was on Grafana version 9.2.4 and I’ve updated to version 9.3.2 and still having the same problem. Looking at the Grafana logs, I’m seeing this:

Failed to send test alert.: send notification to Teams: webhook failed validation: {"version":"1.1","content":{"headers":[{"key":"Content-Type","value":["text/plain; charset=utf-8"]}]},"statusCode":200,"reasonPhrase":"OK","headers":[],"trailingHeaders":[],"requestMessage":null,"isSuccessStatusCode":true}

In that error message it says that the status code was 200 and that isSuccessStatusCode is true. It also says that it failed to send the alert, however the alert comes trough. I’m guessing Grafana for some reason is not able to validate, it sends the notification, Grafana thinks it failed and then tries again? I have Grafana setup in a docker container and I’ve tried upgrading, restarting the container, restarting the VM, and creating new webhooks. I’m unable to fix the issue.

I’ve also tried to send a test notification from the contact point inside the Grafana gui and get the same error:


But again, The notification comes through to my Teams channel with no problems… Can anyone help?

@wymangr reported this issue in github as well - it turns out this was a change made on the Microsoft side and is being rolled back per this issue in their repo.

Posting here for anyone else that encountered this. Thanks for letting us know @wymangr!