Some logs are missing in Loki

Grafana Loki
1

Hi,

I’ve setup a Loki instance in a Kubernetes cluster, based on this chart : helm-charts/charts/loki-simple-scalable at main · grafana/helm-charts · GitHub
Logs are pushed by some Fluentd pods, then we consume them with Grafana for dashboarding and alerting purpose. We also use an Azure Storage account for persistence, instead of having logs stored directly in some volumes of this cluster.

The issue I’m facing is that some logs seem to be missing. At some times, the system can be under high traffic, with a good amount of logs being pushed continuously. But when querying (through Grafana, or using a LogCLI pod), some logs don’t show up at all. As if it was lost during the write process into Loki.

In Loki container logs, I see no particular error or warning. I don’t really know where to look to solve this issue. Do you have any hint on why it could happen ?

Thanks in advance

3 Likes
2

Hi redamattar,

have you able to find any answer for this issue? I too am facing the same issue.

3

I’m having the same problem.
I have fluentbit as client, output is set to cloudwatch logs and loki.
But when I compare the number of loglines per hour in cloudwatch and loki, there is a difference.
Some hours they match perfectly, but some hours there is a significant amount of missing logs in loki, around 40,000 loglines.

Have you resolved this now?