Dashboard times out when more than about 24 hours of data is queried

Grafana Dashboards
1

  • Grafana version and what operating system are you using?
    Ubuntu 22.04, Linux container, 4 cpu’s, 4GB of RAM
    Grafana: v10.2.3
    InfluxDB: v2.7.4

  • What are you trying to achieve?
    I use the “Proxmox VE Cluster [Flux]” dashboard to analyse metrics for a Proxmox Cluster and want to view the metrics for the last 30 days.

  • What happened?
    The dashbard works fine for 24 hours selection, even 2 days, but for more than that period it either only returns partial data or nothing.

  • What did you expect to happen?
    I expect it to return all the data and display it in the graphs.

  • Can you copy/paste the configuration(s) that you are having problems with?
    No

The queries the dashboard uses can be found here: [Proxmox Cluster [Flux] | Grafana Labs](https://Proxmox VE Cluster [Flux])

2

Welcome @lifeboy
What happens if you try that same query outside of grafana directly against influxdb

3

Which query do you mean? Is there a way to see the query in the dashboard before it runs?

4

The queries for the dashboard

5

I can’t find where thay query is? Could you be more specific please? I’m a grafana user, not a developer.

6

Try pasting your Flux query into Influx Data Explorer and see if it times out when the time selected is >24 hours.

1 Like
7

I’ve searched without success to find where this query is that the dashboard uses. If someone can point me to the location or send a screenshot of where to find it, it would be great. I’m not unfamiliar with querying databases, but the dashboard doesn’t reveal it secrets easily it seems! :wink:

8

When you set up influx DP as a data source, it would have asked you for the URL. That URL is where you can find influx data explorer. It’s a separate program and has a separate login… usually it’s https://your-ip-address;8086

9

Indeed, I’m aware of that. Here is screen where the data source is.

image
image1060×806 43.9 KB

However, the query that populates the dashboard is not in there. Under the dashboard setting (select the “gear” icon top right) has this screen, which is the json for the dashboard I presume.

image
image1060×806 67.4 KB

However, it’s doesn’t contain any queries.

Die data source (InfluxDB) is at lcoalhost:8086 and I can log in there and do queries.

10

Aha! When I go to a particular graphs in the dashboard and opt to edit that, I see the queries for that graph. So the dashboard actually contains many queries (which I assumed must be the case anyway). Here is one that times out when I select “30 days” for example:

image
image1218×853 112 KB

Here’s the query:

from(bucket: "${Bucket}")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "system")
  |> filter(fn: (r) => r["_field"] == "cpu")
  |> filter(fn: (r) => r["object"] == "qemu")
  |> filter(fn: (r) => r["_value"] > 0)
  |> filter(fn: (r) => contains(value: r["nodename"], set: ${server:json}))
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
//  |> yield(name: "mean")

If I paste that into InfluxDB, I get an error.

 compilation failed: error @7:62-7:68: invalid expression @7:60-7:61: $
11

Having discovered this in the dashboard, I see there is a “query options” button with a “query inspector” feature. I’m using that now to see what I can learn about the problem.

12

The query explorer shows this:

A

Status: 500. Message: context deadline exceeded (Client.Timeout or context cancellation while reading body),context deadline exceeded (Client.Timeout or context cancellation while reading body)
13

You need to copy your query from Grafana and paste into Influx Data Explorer.

1 Like
14

I may be wrong, but the above syntax works for Grafana but not directly in Influx

15

Influx may not like the $

16

Manually enter the bucket name {bucket} in the query. Right now it is dynamic

I encouage you to understand the full dashboard before copy pasting. It could have a delete all buckets embedded and you willbe on unemployment line

17

I really don’t need to understand the full dashboard if I’m just a user of it, do I? I’m not able to use the product within the parameters that it provides. Unfortunately there seems no way to contact the creator of the dashboard, @mephisto, in the environment where the dashboards are posted. That is unfortunate, since it’s really a great way to view the statistics of a proxmox cluster.

The bottom line is (I think) that the rows returned are likely too many when I select a period of 30 days, since it works for 2 days and mostly for 7 days too. The error message “Client.Timeout or context cancellation while reading body” makes it clear that the query takes too long. So in my mind the solution is simple: Increase the default timeout. How that can be done is not clear however.

18

Correct: The $ is doesn’t like is this one:
filter(fn: (r) => contains(value: r["nodename"], set: ${server:json}))
But ${server:json} is an array it seems, so I can’t simply replace that with a server name?

19

Look at variables section it will tell you what all the $ are

20

Ok, progress! In the InflucDB “Data Explorer” I run the query below:

image
image937×252 28.5 KB

It completes in just over 120 seconds. Grafana doesn’t wait that long.

So we’re back to my original question: How does one increase the query time limit in Grafana?