Define timeout for Prometheus queries in Grafana

  • What Grafana version and what operating system are you using?
    Grafana 8.5.3 deployed pm k8s with Promtheus-stack helm chart.

  • What are you trying to achieve?
    Increase the timeout of promtheus quries

  • How are you trying to achieve it?
    we increased the timeout in prometheus datasource configuration and in addition in grafana.ini we increased the dataproxy timeout ,
    but the timeout still 30s

  • What happened?
    still getting time out after 30s

  • What did you expect to happen?
    timeout to be changed according to when we defined (bigger than 30s)

  • Can you copy/paste the configuration(s) that you are having problems with?

 grafana.ini:
    dataproxy:
      logging: true
      timeout: 900
      keep_alive_seconds: 900
      dialTimeout: 900
      tls_handshake_timeout_seconds: 900
  additionalDataSources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-stack-kube-prom-prometheus.monitoring:9090/
        access: proxy
        isDefault: true
        jsonData:
          timeInterval: 30s
          queryTimeout: 900s
          manageAlerts: false
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

Grafana UI error:

What is the solution for this issue. even i am seeing same error when data is pulled from Prometheus to Grafana Dashboard

Do you put the grafana behind proxy?

If yes, you need to increase the proxy time out.
If you use nginx, add this snippet in server:

        proxy_read_timeout 600;
        proxy_connect_timeout 600;
        proxy_send_timeout 600;

Regards,
Fadjar