Grafana is running behind reverse proxy. Recently we switched our runtime stack from podman to (single node) k3s and thus the reverse proxy from nginx to traefik. TLS is enabled on Traefik Ingress, but not on grafana (Traefik always does TLS termination).
We have dashboards with Alert list panels. The Alert list panel gets it’s data from a prometheus datasource. When we were using nginx as reverse proxy, the alert list got updated immediately as soon as any alert changed.
With Traefik as reverse proxy the auto updating of the Alert list panels does not work any more.
In the logs I find many entries like this:
remote_addr is the actual address of my laptop, but the
-1 looks a bit suspicous to me.
The Traefik IngressRoute for grafana is configured like this:
apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: labels: app: grafana name: grafana spec: entryPoints: - websecure routes: - kind: Rule match: Host(`grafana.my.domain.com`) services: - name: grafana port: 80
What I think (please correct me if wrong):
- for immediate update in the Alert list panels Grafana live API is used
- Grafana live API uses
/api/live/wsendpoint which uses websockets
- websockets get broken due to TLS termination by traefik
status: -1in Grafana logs comes from not working websocket connection
What I tried as well:
IngressRouteand enable TLS passthrough.
X-Forwarded-*headers are not there anymore and thus
remote_addrin the logs is changed to the IP of traefik pod.
- Run Grafana with TLS enabled, configure
scheme: httpsand use a
ServesTransportin Traefik to access grafana with TLS enabled.
- Grafana is accessible as when not running with TLS enabled
- still TLS is terminated at ingress
Is there a configuration I missed to get immediate live update working through traefik in k3s?