Traefik: prometheus live websocket connection not working?

Grafana Live
, ,
1

Grafana is running behind reverse proxy. Recently we switched our runtime stack from podman to (single node) k3s and thus the reverse proxy from nginx to traefik. TLS is enabled on Traefik Ingress, but not on grafana (Traefik always does TLS termination).

We have dashboards with Alert list panels. The Alert list panel gets it’s data from a prometheus datasource. When we were using nginx as reverse proxy, the alert list got updated immediately as soon as any alert changed.

With Traefik as reverse proxy the auto updating of the Alert list panels does not work any more.

In the logs I find many entries like this:

{"duration":"10.457473ms","handler":"/api/live/ws","level":"info","logger":"context","method":"GET","msg":"Request Completed","orgId":1,"path":"/api/live/ws","referer":"","remote_addr":"A.B.C.D","size":0,"status":-1,"t":"2023-08-18T13:15:41.561089762Z","time_ms":10,"uname":"XXX","userId":1}

The remote_addr is the actual address of my laptop, but the status -1 looks a bit suspicous to me.

The Traefik IngressRoute for grafana is configured like this:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`grafana.my.domain.com`)
      services:
        - name: grafana
          port: 80

What I think (please correct me if wrong):

  • for immediate update in the Alert list panels Grafana live API is used
  • Grafana live API uses /api/live/ws endpoint which uses websockets
  • websockets get broken due to TLS termination by traefik
  • status: -1 in Grafana logs comes from not working websocket connection

What I tried as well:

  • use IngressRouteTCP instead of IngressRoute and enable TLS passthrough.
    • The X-Forwarded-* headers are not there anymore and thus remote_addr in the logs is changed to the IP of traefik pod.
    • status is still -1
  • Run Grafana with TLS enabled, configure IngressRoute to use scheme: https and use a ServesTransport in Traefik to access grafana with TLS enabled.
    • Grafana is accessible as when not running with TLS enabled
    • still TLS is terminated at ingress
    • still status is -1

Is there a configuration I missed to get immediate live update working through traefik in k3s?

2

Is there really no one with experience in live update & traefik as reverse proxy in the community? Would be so great if we could solve this.