Grafana with reverse proxy SSL on, the websocket not work

Grafana Configuration
1

  • Grafana version
    Grafana v10.0.3 (eb8dd72637)
    with docker

  • trying to achieve
    I want use grafana with loki live stream log feature
    grafana is behind nginx (reverse proxy) and ssl on

  • How trying to achieve it?
    I reference doc
    Run Grafana behind a reverse proxy | Grafana Labs
    and enable ssl on nginx

  • What happened?
    when ssl not enable
    grafana with loki live stream log can work

when I enable ssl
loki live stream log stop working

Screenshot_20230810_095823
Screenshot_20230810_095823783×511 29.9 KB

and I get log in loki

msg="Error in upgrading websocket" err="websocket: the client is not using the websocket protocol: 'websocket' token not found in 'Upgrade' header"

nginx config

map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}


upstream grafana {
  server grafana-1:3000 max_fails=1 fail_timeout=1s;
  server grafana-2:3000 max_fails=1 fail_timeout=1s backup ;
}

server {

  listen 3000 ssl;
  
  ssl_certificate     /etc/letsencrypt/live/${HOST_FQDN}/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/${HOST_FQDN}/privkey.pem;
  error_page 497 301 =307 https://$host:$server_port$request_uri;
  proxy_connect_timeout 10m;
  proxy_send_timeout 10m;
  proxy_read_timeout 10m;



  location / {
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
  }
  location /api/live/ {
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
  }
}

how do I update config to accomplish this ?
thanks

2

That looks like a problem Grafana<->Loki. Is Loki behind Nginx as well or is there direct connection Grafana-Loki?

3

loki behind nginx too

I find problem

I see chrome error is

WebSocket connection to 'wss://192.168.100.152:3000/api/datasources/proxy/uid/loki-web/loki/api/v1/tail?query=%7Bjob%3D%22fluent-bit%22%7D%20%7C%3D%20%60%60' failed:

look like uri not match config location

so I add another nginx location for loki

server {
  
  listen 3000 ssl;
  http2 on;
  
  ssl_certificate     /etc/letsencrypt/live/${HOST_FQDN}/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/${HOST_FQDN}/privkey.pem;
  error_page 497 301 =307 https://$host:$server_port$request_uri;
  proxy_connect_timeout 10m;
  proxy_send_timeout 10m;
  proxy_read_timeout 10m;

  location / {
    proxy_set_header Host $http_host;
    proxy_pass http://grafana;
  }

  location /api/live/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $http_host;
    proxy_pass http://grafana;
  }
  
  location /api/datasources/proxy/uid/loki-web/loki/api/v1/tail {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $http_host;
    proxy_pass http://grafana;
  }

and problem solved