Significant Increase in Read IOPS After Upgrading Grafana from 8.5.x to 10.1.8

Grafana Alerting
1

Hello Grafana Community,

I recently upgraded my Grafana instance from version 8.5.x to 10.1.8, and since the upgrade, I’ve noticed a significant increase in the read IOPS for the Aurora Reader instance, which is causing a noticeable increase in costs.

Here are some details about my setup:

Grafana Version: 10.1.8 (upgraded from 8.5.x)
Database: Aurora MySQL
Deployment: Kubernetes, using the Aurora Reader instance to fetch data
Issue: A sharp increase in read IOPS since upgrading. The IOPS seem higher even for queries that previously didn’t cause this much load.

I have reviewed the following:

  1. Queries executed by Grafana (such as fetching data from data_source and alert_notification_state tables)
  2. Dashboard settings and refresh intervals
  3. The performance of the Aurora instance before and after the upgrade

Has anyone else experienced a similar issue after upgrading from Grafana 8.5.x to 10.1.8? If so, what steps did you take to reduce the read IOPS? Are there any specific settings or optimizations I might have missed?

Any advice or guidance would be greatly appreciated!

Thanks in advance!

2

I have the same problem on 10.2.6. Grafana is reading something from the disk at 40-50 megabytes per second, which is strange and suspicious.