Grafana 10.1.x with Redshift Data Source 504 timeout

Hello we upgrade our own hosted Grafana cluster on AWS from 10.0.3 to 10.1.1 and the Redshift (v 1.12.1) datasource stopped working - timeout with 504.

Redshift datasource is configured to use assume role as it is located on another AWS account. We had to roll back to 10.0.3 for the Redshift queries to start working again.

Anyone come across this issue?

Checking Cloudwatch Log on Redshift AWS account we noticed a lot of rate exceeded / throttling error after the upgrade to 10.1.x like below

errorCode|ThrottlingException
errorMessage: Rate exceeded
eventCategory: Management
eventName: ListStatements
eventSource: redshift-data.amazonaws.com
|eventType: AwsApiCall|

There is also a significant jump in the number of requests / activities at VPC endpoint for redshift-data (10x more in 10.1.x compared with 10.0.3)

An update i think we tracked it down to Async Query Support that been enabled by default on 10.1.x.

Going to try and turn off this feature toggle and see how it goes

see Upgrading to Grafana 10.1.1 causes all Redshift queries to timeout. · Issue #250 · grafana/redshift-datasource · GitHub