Reverse proxy - delayed connect error: 111

  • What Grafana version and what operating system are you using?
    Grafana 9.3.1 installed on Ubuntu 22.04 via apt-get

  • What are you trying to achieve?
    Simple reverse proxy without subpath

  • How are you trying to achieve it?
    Following the documentation Run Grafana behind a reverse proxy | Grafana Labs
    With an nginx proxy

  • What happened?
    Grafana works well, but arround one of ten request (including requesting html, js, css, imgs, /api …) to the proxy fails with the following error :

upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111

The http request that fails change at every reload, sometimes it can’t request the js files, sometimes the css and sometimes everything loads well but some /api request fails

  • What did you expect to happen?
    All requests to the grafana server should succeed

  • Can you copy/paste the configuration(s) that you are having problems with?

# grafana.ini (params that are not listed here are still commented
app_mode = production

[server]
http_port = 3000
domain = grafana.**********.com

# tried to enable/disable subpath
# nginx grafana.conf
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

upstream grafana {
  server community.grafana.com;
}

server {
  listen 80;

  root /usr/share/nginx/html;
  index index.html index.htm;
  server_name grafana.****************.com;

  location / {
    proxy_set_header Host grafana.***************.com;
    proxy_pass http://grafana;
  }

  # Proxy Grafana Live WebSocket connections.
  location /api/live/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $http_host;
    proxy_pass http://grafana;
  }
}
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    The following error is shown in “popovers” inside Grafana UI, but is not visible in logs (grafana nor nginx)
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111

Thanks for your help.

For those who has a similar problem :
The issue was caused by my load balancer.