Loglines about 20 min delayed

Grafana Loki
, ,
1

Hello everybody!

I have loki setup with 6 instances (3 x target:read, 3x target:write) in a kubernetes cluster. I have fluent-bit sending service logs to loki. Data is stored in a minio cluster. Grafana, loki and minio are all part of the same k8s cluster, logshipping happens from the same cluster and an additional cluster.

My current problem is, that those logs appearing with a 20 minute delay in the grafana ui. It’s not a problem with timestamps. I also don’t have any error messages from fluent-bit or the loki ingestors.

it’s simply that when I query logs in the grafana ui, only logs about 20 minutes old (and older) are shown.

My guess would be, that the membership ring isn’t working properly and grafana shows only logs, that are allready persisted. But I can’t find a way to debug this.

Hints or tips would be much appreciated!

regards
sebastian

2

  1. What do you see if you curl the /ring endpoint for both reader and writer?

  2. Try to get into one of the reader container and see if you can hit one of the writer, and vice versa.

  3. Check your configuration for query_ingesters_within.