Grafana version 9.3.1
I’ve got a TimescaleDB data source set up to show recent data from the last 3 days really quickly. It’s working well with a 1-second refresh rate, and the queries finish in 200-400 ms.
Usually, my users focus on looking at data within the same day, so the TimescaleDB source is perfect. But I’m thinking about adding BigQuery into the mix for storing data going back several years – stuff that’s too much for the fast TimescaleDB. Sometimes, users want to go back months and check out specific data points for specific times.
I’ve set up a combination of these sources, with TimescaleDB in Query A and BigQuery in Query B.
But here’s the snag. While the TimescaleDB query is quick, around 200-400 ms, the BigQuery query takes more than 2 seconds.
That’s usually okay with the 1-second refresh rate, as users have to choose specific dates and times using the date range finder to access the BigQuery data, which turns off the 1-second refresh.
The problem comes when I narrow down to the last 15 minutes. Even then, the total query time is over 2 seconds. It seems like the mixed data source insists on running both queries one after the other. Even when the TimescaleDB query (Query A) is done, it hangs on until the BigQuery query (Query B) finishes before it fills in the chart.
I even tried using an IF-statement in the BigQuery query to run it only when the time range is outside what TimescaleDB covers. But the total query time still goes over 2 seconds because of BigQuery’s slower performance.
So, my big question is: How can I set up these mixed data sources to run independently? I want the queries to work on their own timelines, so the charts get filled in promptly, even when I’m using short time spans with the slower BigQuery data source.