Grafana 13.x: ERR_CONNECTION_RESET while viewing (higher volume) logs

Grafana
, , ,
1

Hi there,

we’re using Loki to collect logs from various sources. Since Grafana 13, trying to view logs provided through the Loki plugin often results in “No data” while there actually are logs. Browser dev tools show an ERR_CONNECTION_RESET at the same time. When lowering the result volume (time range, max rows), logs suddenly appear as expected. There’s no indication of an error on Grafana’s or Loki’s side, telling from the logs; Loki reports a valid line count, yet the result is not displayed. This was definitely not present up to (including) Grafana 12.

backend_srv.ts:143  POST https://<redacted>/grafana/api/ds/query?ds_type=loki&requestId=explore_8fv_1 net::ERR_CONNECTION_RESET

Grafana is operated as part of the kube-prometheus-stack in a managed Kubernetes 1.35 environment. The Grafana Helm Chart version is 12.7.1, the cluster uses Traefik 3.7 as ingress controller.

Interestingly when bypassing Traefik (e.g. port forward into the Grafana Pod) the issue does not occur.

Are there any known issues when operating this combination? Or else - have there been significant changes between Grafana 12 and 13 that might explain the behaviour? What might cause the ERR_CONNECTION_RESET errors? Any hints / ideas?

I’ll provide additional detail if needed.

Best wishes

Karsten

2

Since the issue disappears when bypassing Traefik and only occurs with larger responses, the problem is somewhere in the Traefik ↔ Grafana path rather than Loki itself.

One confirmed change worth noting: starting in Grafana v13.0, HTTP compression (server.enable_gzip) is enabled by default, whereas it was previously disabled.

As a first troubleshooting step, temporarily disable Grafana’s gzip compression and retest:

yaml

grafana.ini:
  server:
    enable_gzip: false

Then run helm upgrade and test with the same large response that was failing.

If that doesn’t help, check Traefik’s compression middleware, timeout settings, and access logs at the time of the reset to narrow down where the connection is being reset.

3

@infofcc3 Yep, that was exactly the culprit - thanks a lot for clarification. Setting enable_gzip: false did the trick. I’ve since reverted that setting to its (new) default and added a Traefik Middleware dropping the Accept header which basically has the same effect. So - issue solved. Again, thanks for your suggestions. :slight_smile: