Occasional pending error alert states for azure monitor metrics

Hello Grafana Community,

I’m experiencing some strange behavior with alerting states for certain graphs on our locally installed grafana 9.2 instance.

Specifically we are using the azure monitor datasource and have multiple alerts configured for the availability metric provided by azure insights resources. The alerting works most of the time but I’m getting occasional pending error states, which I don’t understand. Here is a screenshot how this looks:

Screenshot 2022-10-21 at 10.50.32

Its always the same behavior: pending error state and one minute later it switches back to ok. It does not trigger any alert because I configured a 2 minute evaluation time frame, but it looks really bad if you review these graphs:

Screenshot 2022-10-21 at 11.11.50

Sometimes I don’t see them for several days and sometimes they occur multiple times a day. I have multiple of these alerts configured, which are all affected but usually not at the exact same time. So I would rule out any general connection issue of my local grafana instance, cause I would assume if my internet access to azure is not working, I would have error messages for all graphs at the same time.

Any ideas what could cause these error states? Thanks in advance!