Grafana v8.3.4 upgrade to v9.3.1. DatasourceError alerts for prometheus backend dashboards

Grafana Configuration
1

We upgraded grafana from 8.3.4 to 9.3.1 on GKE and all the alerts setup for prometheus datasource dashboard started to trigger false alerts with: #12280: [FIRING:1] DatasourceError Alerting (alert msg..). Is there some details on how to get around this issue so that we can upgrade without triggering all the alerts that’s configured in prometheus backed dashboards?

  • What Grafana version and what operating system are you using?
    v8.3.4 upgrading to v9.3.1 on GKE 1.21.14-gke.3000 with Postgresql 10 backend

  • What are you trying to achieve?
    We are trying to upgrade grafana v8.3.4 to v9.3.1

  • How are you trying to achieve it?
    In the StatefulSet for grafana, change: image: grafana/grafana:8.3.4 to image: grafana/grafana:9.3.1

  • What happened?
    Alerts configured for dashboards with prometheus datasource started to trigger:
    #12280: [FIRING:1] DatasourceError Alerting (alert on monitoring)

  • What did you expect to happen?
    We expected to not get the “DatasourceError” alerts for all the alerts that are configured for prometheus backend dashboards.

  • Did you follow any online instructions? If so, what is the URL?
    I followed this instruction. The only change that was made was to upgrade our Postgres from 9.6 to 10, before upgrading to grafana 9.3.1

2

After upgrade to v9.3.1, under the Alerting page, I see following errors:

Errors loading rules
Failed to load the data source configuration for prometheus-server-1: Unable to fetch alert rules. Is the prometheus-server-1 data source properly configured?
Failed to load the data source configuration for prometheus-server-2: Unable to fetch alert rules. Is the prometheus-server-2 data source properly configured?

I’ve confirmed that the datasources prometheus-server-1 and prometheus-server-2 are working properly. I can pull metrics from those prometheus servers.