I have loki setup with 6 instances (3 x
target:write) in a kubernetes cluster. I have fluent-bit sending service logs to loki. Data is stored in a minio cluster. Grafana, loki and minio are all part of the same k8s cluster, logshipping happens from the same cluster and an additional cluster.
My current problem is, that those logs appearing with a 20 minute delay in the grafana ui. It’s not a problem with timestamps. I also don’t have any error messages from fluent-bit or the loki ingestors.
it’s simply that when I query logs in the grafana ui, only logs about 20 minutes old (and older) are shown.
My guess would be, that the membership ring isn’t working properly and grafana shows only logs, that are allready persisted. But I can’t find a way to debug this.
Hints or tips would be much appreciated!