Loki queries within grafana dashboards

Hi all,

I have quite simple queries which return data for 90 days without problem when used in explorer mode. But the same queries (some of them) ends with:

#### <html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx</center> </body> </html>

when used on grafana dashboard.
Further inspection of failed query stats shows that it ends usually after 1 minute.

I have re-set various timeouts:

  • grafana (loki datasource timeout 120s)
    timeout = 300
  • nginx (loki-distributed gateway)
    proxy_read_timeout 300s;
  • loki
    http_server_read_timeout: 300s
    http_server_write_timeout: 300s
    query_timeout: 3m

but this happening still.

Did I forgot to something?

Best regards,

And answering to myself - yes I totally forgot that access to my grafana is managed by ingress so:

nginx.ingress.kubernetes.io/proxy-connect-timeout: “360”
nginx.ingress.kubernetes.io/proxy-read-timeout: “360”
nginx.ingress.kubernetes.io/proxy-send-timeout: “360”

solved the issue.