Loki - Helm can write data to S3 but cannot get S3 objects!

Any idea what causes this issue?

level=error ts=2023-03-01T08:46:45.809733204Z caller=batch.go:720 org_id=fake msg=“error fetching chunks” err=“failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 7SDDM64KPCXEAGYN, host id: 67TezMmbJRevEIFmK4Eyb1Dd4lbdXjUcpMdezCz61fkU4W6hu5vBAx32oo8/c+zXItALVSdFFEs=”

level=error ts=2023-03-01T08:46:51.28874269Z caller=retry.go:73 org_id=fake msg=“error processing request” try=2 err=“rpc error: code = Code(500) desc = failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: FBTJ341CYYD66NG2, host id: VtY5FdJ64W/2xEu3gUoSa4+kzIhvAGmqlnNMHIov8hfGaVyERpK7Q1bmPBgA7rcaYiR5xxqzdf4=\n”

I’m using this file

I’m having the same problem, has this been resolved?

I’m using a microservices model with caching turned on, and both index and chunk are stored in alibaba’s oss.
I’m also having problems with my query, my grafana can only show me the most recent data, not the historical data. But I do have old data in my S3.
I’m not sure if it has anything to do with the error.

My configuration is as follows:

  bucketnames: local-loki-test
  endpoint: http://local-loki-test.oss-cn-beijing-internal.aliyuncs.com

I queried the request log by request ID and found that from the request parameter, it is ListObjects interface, but this request is a record of Getobject, i.e., the object of this request is local-loki-test instead of index_19651, if there is no local-loki-test object in the bucket, it will return 404. loki-test object in the bucket will return 404.
Is this problem related to loki?

have you tried changing to alibabacloud instead of aws? because I see that you are using AliCloud OSS. I encountered the same error months ago, but now successfully running it using this config

  endpoint: https://oss-{region}-internal.aliyuncs.com
  bucket: bucket-name
  access_key_id: access_key_id
  secret_access_key: secret_access_key

Thank you for your reply.
I have tried using the alibabacloud configuration but it throws a 403 exception.
The error message is as follows:

err="failed to flush chunks: store put chunk: SecondLevelDomainForbidden: Please use virtual hosted style to access.\n\tstatus code: 403

I also have an issue, not sure if it is related to the above error as he can query some data.
I can’t query old data via grafana UI or logCLI, but I can confirm that I have historical data in my backend storage as the monitoring data has been trending upwards. It looks like there are large chunks of data that are not available and it seems to be related to the index storage
Screenshot info below:

May I ask if you have encountered this issue?
Looking forward to your reply Thanks!