Alerts Close when updated via API and other alerting issues

Grafana Configuration
, , ,
1

Hi, having some issues provisioning and maintaining the Alerts via the API.

Grafana Ver 9.0.4 running in container
DataSources:

  • Azure Metrics / Logs
  • SQL Azure PaaS
  • SQL Azure VM

I can successfully create and update alerts and dashboards using the API’s.
But am experiencing an issue, when the alerts are updated via the API, it causes their alerting state to reset, go to pending, then alerting again.
Also have noticed the number of alerts drops and increases during deployment.
I am not doing any deletions, only POST new alerts, or PUT existing alerts.

These are the APIs i hit

  "PUT" { "/api/v1/provisioning/alert-rules/$($AlertModel.uid)" }
  "POST" {  "/api/v1/provisioning/alert-rules" }

I configured Prometheus to scrape Grafana Metrics, and pushed some annotations so i could see whats happening during deployment time:

image
image1874×977 90.9 KB

You can see in the Legend panel, the Range of change, is only caused by a deployment happening, the alerts are actually constantly in this state.

The quantity of Normal alerts fluctuates between 127 - 181
But there is actually this many Alerts
image

I am making an assumption that the difference of 54 alerts, is because some of the alerts create their own dimensions based of the queries and that’s getting reset at deployment time.

And the alerts going from Alerting State, to OK, Pending, Alerting again, i’ve no idea what is causing that.

image
image638×796 93 KB

And to top off the alerting issues, we have DataSourceNoData and DataSourceErrors constantly firing at random points that i also cannot debug.

image
image1858×975 79 KB

Would really appreciate some help if possible please. Thanks

2

can you confirm you’re using these api endpoints?

Here are the newest stable API endpoints for provisioning Grafana alerts:

https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/pkg/services/ngalert/api/tooling/api.json

3

I can confirm i am using those API endpoints for provisioning alerts.
The only other API endpoints i use are :

  • /api/alertmanager/grafana/config/api/v1/alerts for POST Notification Policies

  • api/folders for GET / POST Folders