How can I increase the dataproxy timeout on Hosted Grafana?

Grafana Cloud
1

  • What Grafana version and what operating system are you using?
    Cloud Pro (Licensed)
    v9.3.2-45365 (ef5286dd77) browsing on Google Chrome, Windows 10

  • What are you trying to achieve?
    Increase the dataproxy timeout, as my ML training is constantly Erroring

  • How are you trying to achieve it?
    I dont know. That’s why i’m asking

  • What happened?
    Nothing

  • What did you expect to happen?
    Nothing… I didn’t do anything yet

  • Can you copy/paste the configuration(s) that you are having problems with?
    I’m using hosted Grafana, so i’m not sure exactly how to SSH into my hosted Grafana instance, or if it’s even possible?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    Training failed:

image
image1006×537 29.3 KB

  • Did you follow any online instructions? If so, what is the URL?

Negative

The reason i’m asking is because Grafana support staff told me to increase the dataproxy timeout.

2

Hi @cx_182 ! I’m not sure if you are still working on this issue, but if so can you confirm the data source that your Machine Learning jobs are using?

And regarding your question here:

I’m using hosted Grafana, so i’m not sure exactly how to SSH into my hosted Grafana instance, or if it’s even possible?

SSH access to the hosted Grafana server is not provided. You can interact with your Grafana instance via the main user interface at your hosted Grafana URL or using the API:

3

I face the same problem. I have connected Grafana to an InfluxDB data source.

I have extended the config /etc/grafana/grafana.ini as below and restarted the grafana-server service.

[dataproxy]
timeout = 1200

Yet if I run large queries in Grafana (which I expect to take several minutes) I get the timeout error below.

Post “http://localhost:8086/api/v2/query?org=main-org”: net/http: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers)

It seems the client i.e. Grafana runs into a timeout. But my config changes have no effect.

Grafana version: 10.2.2

As a reference, the problem is discussed in this Grafana GitHub issue.

4

How do you know that’s a Grafana timeout which has been reach? It can be some network timeout on the OS level, e. g. tcp keepalive

5

Good question. I have a very simple setup: (1) InfluxDB + Grafana on a bare metal home server (2) I call from a browser on my laptop.

InfluxDB itself has dashboards. The queries behind the InfluxDB dashboards take about two minutes but work fine i.e. no TCP timeouts from my home server. I only get the timeouts in the Grafana dashboards (which use InfluxDB as data source).

Both InfluxDB service and Grafana service run on the same bare metal machine. Is it possible to have a TCP timeout between these services on the same machine? Or did you mean a timeout between my chrome browser client and the Grafana server? Below the TCP settings on my home server. TCP keepalive is 2 hours.

sysctl net.ipv4.tcp_fin_timeout
net.ipv4.tcp_fin_timeout = 60

sysctl net.ipv4.tcp_keepalive_time
net.ipv4.tcp_keepalive_time = 7200

sysctl net.ipv4.tcp_keepalive_intvl
net.ipv4.tcp_keepalive_intvl = 75

sysctl net.ipv4.tcp_keepalive_probes
net.ipv4.tcp_keepalive_probes = 9