What Grafana version and what operating system are you using?
Grafana v9.3.2 Alpine image, running on Azure App Service
What are you trying to achieve?
I’ve got Grafana (9.3.2 Alpine container) deployed to an Azure App Service, connected with a Loki datasource. Everything has been working ok, except I’m having issues with long running queries to Loki.
For longer Loki queries, I’m receiving a time out from the Azure App Service side:
1 queries with total query time of 4.00 min status: 504 statusText: "Gateway Timeout"
This always times out after 4 minutes, and unfortunately I’ve found it’s an Azure limit, that cannot be increased:
Is it possible to handle a query timeout from an external loadbalancer without increasing the timeout on the loadbalancer, but instead by sending a TCP Keep-Alive for the connection?
I was looking into the dataproxy settings and found the keep_alive_seconds setting, which I thought may work for this use case: Configure Grafana | Grafana documentation
Unfortunately experimenting with this value hasn’t helped so was wondering if anyone else has experienced something similar and knows of a way around it before I switch Infrastructure setup.
Here’s my relevant proxy config settings:
[server] # The full public facing url you use in browser, used for redirects and emails # If you use reverse proxy and sub path specify full url (with sub path) root_url = <The fqdn for my loadbalancer> [dataproxy] # This enables data proxy logging, default is false logging = true # How long the data proxy waits to read the headers of the response before timing out timeout = 30 # How long the data proxy waits to establish a TCP connection before timing out dialTimeout = 10 # How many seconds the data proxy waits before sending a keepalive probe request. keep_alive_seconds = 15
Thanks in advance!