Define timeout for Prometheus queries in Grafana

Grafana Prometheus
1

  • What Grafana version and what operating system are you using?
    Grafana 8.5.3 deployed pm k8s with Promtheus-stack helm chart.

  • What are you trying to achieve?
    Increase the timeout of promtheus quries

  • How are you trying to achieve it?
    we increased the timeout in prometheus datasource configuration and in addition in grafana.ini we increased the dataproxy timeout ,
    but the timeout still 30s

  • What happened?
    still getting time out after 30s

  • What did you expect to happen?
    timeout to be changed according to when we defined (bigger than 30s)

  • Can you copy/paste the configuration(s) that you are having problems with?

 grafana.ini:
    dataproxy:
      logging: true
      timeout: 900
      keep_alive_seconds: 900
      dialTimeout: 900
      tls_handshake_timeout_seconds: 900

  additionalDataSources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-stack-kube-prom-prometheus.monitoring:9090/
        access: proxy
        isDefault: true
        jsonData:
          timeInterval: 30s
          queryTimeout: 900s
          manageAlerts: false
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

Grafana UI error:

image
image1920×1063 156 KB

2

What is the solution for this issue. even i am seeing same error when data is pulled from Prometheus to Grafana Dashboard

3

Do you put the grafana behind proxy?

If yes, you need to increase the proxy time out.
If you use nginx, add this snippet in server:

        proxy_read_timeout 600;
        proxy_connect_timeout 600;
        proxy_send_timeout 600;

Regards,
Fadjar