Time to update visualisation on my dashboard with infinity plugin and csv file

Hi! New to Grafana and Ive just installed an open source version (latest) on a vSphere VM (16vCPU and 64BG RAM) as well as an EC2 instance C5.9xLarge (36 vCPU and 72GB RAM)

I’m using the infinity plugin to pull in historic storage performance stats from a CSV file

~600,000 odd lines - across a 3 month time period at 5 min intervals…
22 volumes, collecting
R IOPS
W IOPS
R Throughput
W Throughput

The CSV file is static - no changes to the data. Each are configured like so

On my Dashboard I have 4 times series visualisations - one for each of R-IOPS, W-IOPS, R-TP, W-TP

"format": "timeseries",
"parser": "backend",
"source": "url",
"type": "csv",
"url": "http://10.....ed.csv",
"method": "GET"

Each Visualisation pulls in the Volume Name, the timestamp and metric i want to chat.

  "pluginVersion": "11.4.0",
  "targets": [
    {
      "columns": [
        {
          "selector": "timestamp",
          "text": "time",
          "timestampFormat": "2006/01/02 15:04:05",
          "type": "timestamp"
        },
        {
          "selector": "Name",
          "text": "",
          "type": "string"
        },
        {
          "selector": "Volume Read IOPS",
          "text": "Read IOPS",
          "type": "number"
        }

I use transformation to go from wide, to long format for the required metric

  "transformations": [
    {
      "id": "prepareTimeSeries",
      "options": {
        "format": "multi"
      }
    }

When I change the time to view or zoom in, it takes a while for the charts to update… about 45 seconds (VM) 30 secs EC2 Instance)…

Is that a reasonable amount of time… personally I would have expected much quicker.

Is the delay here due to the infinity plugin pulling the data set in and reading each time?
Thanks

Reading 600k in any tool will take a long time

yep that’s a fair call - so does it read the CSV file any time a chart’s time period to view is changed?
Given it’s static data - there is no need to continuously re-read

I would check your browser’s network tab to see if it does that.

Thank you for the tip - Yes I can see 4 queries - all at ~50MB in size - the same size as the CSV.
When you go from the 90 day view to 2 days - 4 queries all at ~50MB, and same when zooming into 30 min view - 4 queries @~50MB each
Thanks

1 Like

it is by design, limited when using inline.

like, data warehouse, maybe you can trim your csv with pre defined aggregations instead of doing aggregation on the fly, if that is what you are doing.