Err="failed to flush chunks: store put chunk: RequestError: send request failed

Grafana Loki
1

Hello everyone.
I have loki deployed in kubernetes .
And from time to time I get these errors:

level=error ts=2024-11-11T08:57:28.612339465Z caller=flush.go:178 component=ingester loop=7 org_id=fake msg="failed to flush" retries=0 err="failed to flush chunks: store put chunk: RequestError: send request failed\ncaused by: Put \"http://loki-minio.loki.svc:9000/chunks/fake/3adb375ae1647d87/1931a6c6f84%3A1931a6c712b%3A800f4713\": EOF, num_chunks: 1, labels: {}

How can I increase retries? Or if it is not possible, how can I solve this problem?

2

I’d first try and pin down whether it’s a minio problem or loki problem. If it’s intermittent it’s probably minio, and you should look for a solution there first.

3

Hello Tony.
Thank you very much for your reply.
These errors do not appear very often.
In the minio logs I only found these errors around this time:

Error: Marking loki-minio-0.loki-minio-svc.loki.svc.cluster.local:9000 offline temporarily; caused by Post "http://loki-minio-0.loki-minio-svc.loki.svc.cluster.local:9000/minio/lock/v7/lock": lookup loki-minio-0.loki-minio-svc.loki.svc.cluster.local: i/o timeout (*fmt.wrapError)

Error: Marking loki-minio-0.loki-minio-svc.loki.svc.cluster.local:9000 offline temporarily; caused by Post "http://loki-minio-0.loki-minio-svc.loki.svc.cluster.local:9000/minio/storage/export/v49/readversion?disk-id=6c97ecc5-661c-4372-9c84-df12f83d1ac2&file-path=fake%!F(MISSING)8c7560f724b78863%!F(MISSING)19224799699%!A(MISSING)19224d5211e%!A(MISSING)ca4bac67&read-data=false&version-id=null&volume=chunks": lookup loki-minio-0.loki-minio-svc.loki.svc.cluster.local: i/o timeout (*fmt.wrapError)

minio is in the same namespace as loki.

Maybe you have any ideas what this could be related to?

4

If I had to guess minio is probably overloaded. I don’t use it, you might want to check with minio community for tuning advise.

Alternatively, you can try to reduce the number of files set to minio by tuning maximum chunk age, idle time, and target size. But this will be more like a bandaid.

5

Ok, thank you very much.
I’ll try to write to the minio community.