We’re seeing these 2 specific errors in loki pods in K8s cluster onprem, singleBinary deploymentMode and storage in S3.
‘error loading cache generation numbers’ and ‘failed loading deletes for user’, no other messages show any other process is affected.
We do run a proxy between loki and S3, and as far as we can tell no_proxy config is complete and should cover grpc comms between components:
no_proxy: 100.122.16.0/20,10.96.0.0/12,.mgmt.domain.uk,.service.domain.uk,.svc,.cluster.svc, localhost, 127.0.0.1, loki, loki-headless, loki.app-base-loki.svc.cluster.local
Following various suggestions we’ve tried setting the compactor_grpc address to localhost and 127.0.0.1 and adding those to the no_proxy but without success.
Turning off retention stops the messages but we need retention of the logs.
- do these messages actually have any impact?
- Is there some other configuration we’re missing here?
What version are you running?
I see something in the 2.7.1 release note related to the error messages you are getting.
Hi Tony,
Versions are:
app. kubernetes. io / version 3.5.3
helm.sh/chart: loki-6.35.1
I did see something here with looked like it might be related: Enabling retention is incompatible with setting up a proxy through env vars in SingleBinary and SimpleScalable modes · Issue #17100 · grafana/loki · GitHub but that looks like it should have been fixed in helm chart version 6.35.0 loki/production/helm/loki/CHANGELOG.md at main · grafana/loki · GitHub if I’m reading it right.
Let me know if you need any other info.
Can you share the full logs on the GRPC errors? I wonder which component is reporting it.
I can’t upload the json export so here are a few lines as an example:
Common labels: {“app”:“loki”,“cluster”:“redacted”,“component”:“single-binary”,“container”:“loki”,“detected_level”:“error”,“instance”:“loki”,“job”:“app-base-loki/loki”,“namespace”:“app-base-loki”,“service_name”:“loki”,“stream”:“stderr”}
Line limit: “1000 reached, received logs cover 4.67% (2min 48sec) of your selected time range (1h)”
Total bytes processed: “6.07 MB”
2025-11-24 10:07:50.506 ts=2025-11-24T10:07:50.464019479Z caller=spanlogger.go:116 user=fake level=error msg=“failed loading deletes for user” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.506 level=error ts=2025-11-24T10:07:50.463921272Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.506 ts=2025-11-24T10:07:50.444338494Z caller=spanlogger.go:116 user=fake level=error msg=“failed loading deletes for user” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.506 level=error ts=2025-11-24T10:07:50.444262157Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.506 ts=2025-11-24T10:07:50.408883509Z caller=spanlogger.go:116 user=fake level=error msg=“failed loading deletes for user” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.506 level=error ts=2025-11-24T10:07:50.408818091Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.506 level=error ts=2025-11-24T10:07:50.407628929Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 ts=2025-11-24T10:07:50.484193083Z caller=spanlogger.go:116 user=fake level=error msg=“failed loading deletes for user” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.484106651Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.482562086Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 ts=2025-11-24T10:07:50.471008485Z caller=spanlogger.go:116 user=fake level=error msg=“failed loading deletes for user” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.470946847Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.469599788Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.461749464Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.443167123Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 ts=2025-11-24T10:07:50.435732155Z caller=spanlogger.go:116 user=fake level=error msg=“failed loading deletes for user” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.4356667Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.434620033Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 ts=2025-11-24T10:07:50.423128054Z caller=spanlogger.go:116 user=fake level=error msg=“failed loading deletes for user” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.423045014Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.491 level=error ts=2025-11-24T10:07:50.421616534Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.405 ts=2025-11-24T10:07:50.390922766Z caller=spanlogger.go:116 user=fake level=error msg=“failed loading deletes for user” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.405 level=error ts=2025-11-24T10:07:50.390835772Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
2025-11-24 10:07:50.405 level=error ts=2025-11-24T10:07:50.381556652Z caller=gennumber_loader.go:136 msg=“error loading cache generation numbers” err=“rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: reading server HTTP response: unexpected EOF””
