Any idea what causes this issue?
level=error ts=2023-03-01T08:46:45.809733204Z caller=batch.go:720 org_id=fake msg=“error fetching chunks” err=“failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 7SDDM64KPCXEAGYN, host id: 67TezMmbJRevEIFmK4Eyb1Dd4lbdXjUcpMdezCz61fkU4W6hu5vBAx32oo8/c+zXItALVSdFFEs=”
level=error ts=2023-03-01T08:46:51.28874269Z caller=retry.go:73 org_id=fake msg=“error processing request” try=2 err=“rpc error: code = Code(500) desc = failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: FBTJ341CYYD66NG2, host id: VtY5FdJ64W/2xEu3gUoSa4+kzIhvAGmqlnNMHIov8hfGaVyERpK7Q1bmPBgA7rcaYiR5xxqzdf4=\n”
I’m using this file
I’m having the same problem, has this been resolved?
I’m using a microservices model with caching turned on, and both index and chunk are stored in alibaba’s oss.
I’m also having problems with my query, my grafana can only show me the most recent data, not the historical data. But I do have old data in my S3.
I’m not sure if it has anything to do with the error.
My configuration is as follows:
I queried the request log by request ID and found that from the request parameter, it is ListObjects interface, but this request is a record of Getobject, i.e., the object of this request is local-loki-test instead of index_19651, if there is no local-loki-test object in the bucket, it will return 404. loki-test object in the bucket will return 404.
Is this problem related to loki?
have you tried changing to
alibabacloud instead of
aws? because I see that you are using AliCloud OSS. I encountered the same error months ago, but now successfully running it using this config
Thank you for your reply.
I have tried using the
alibabacloud configuration but it throws a 403 exception.
The error message is as follows:
err="failed to flush chunks: store put chunk: SecondLevelDomainForbidden: Please use virtual hosted style to access.\n\tstatus code: 403
I also have an issue, not sure if it is related to the above error as he can query some data.
I can’t query old data via
grafana UI or
logCLI, but I can confirm that I have historical data in my backend storage as the monitoring data has been trending upwards. It looks like there are large chunks of data that are not available and it seems to be related to the index storage
Screenshot info below:
May I ask if you have encountered this issue?
Looking forward to your reply Thanks!