Loki - Helm can write data to S3 but cannot get S3 objects!

abdel24hour · March 1, 2023, 8:53am

Any idea what causes this issue?

level=error ts=2023-03-01T08:46:45.809733204Z caller=batch.go:720 org_id=fake msg=“error fetching chunks” err=“failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 7SDDM64KPCXEAGYN, host id: 67TezMmbJRevEIFmK4Eyb1Dd4lbdXjUcpMdezCz61fkU4W6hu5vBAx32oo8/c+zXItALVSdFFEs=”

level=error ts=2023-03-01T08:46:51.28874269Z caller=retry.go:73 org_id=fake msg=“error processing request” try=2 err=“rpc error: code = Code(500) desc = failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: FBTJ341CYYD66NG2, host id: VtY5FdJ64W/2xEu3gUoSa4+kzIhvAGmqlnNMHIov8hfGaVyERpK7Q1bmPBgA7rcaYiR5xxqzdf4=\n”

I’m using this file

github.com

grafana/loki/blob/main/production/helm/loki/values.yaml

global:
  image:
    # -- Overrides the Docker registry globally for all images
    registry: null
  # -- Overrides the priorityClassName for all pods
  priorityClassName: null
  # -- configures cluster domain ("cluster.local" by default)
  clusterDomain: "cluster.local"
  # -- configures DNS service name
  dnsService: "kube-dns"
  # -- configures DNS service namespace
  dnsNamespace: "kube-system"
# -- Overrides the chart's name
nameOverride: null
# -- Overrides the chart's computed fullname
fullnameOverride: null
# -- Image pull secrets for Docker images
imagePullSecrets: []
kubectlImage:
  # -- The Docker registry

This file has been truncated. show original

everhopeful · October 24, 2023, 12:44pm

I’m having the same problem, has this been resolved?

everhopeful · October 24, 2023, 12:52pm

I’m using a microservices model with caching turned on, and both index and chunk are stored in alibaba’s oss.
I’m also having problems with my query, my grafana can only show me the most recent data, not the historical data. But I do have old data in my S3.
I’m not sure if it has anything to do with the error.

My configuration is as follows:

aws.
  bucketnames: local-loki-test
  endpoint: http://local-loki-test.oss-cn-beijing-internal.aliyuncs.com

I queried the request log by request ID and found that from the request parameter, it is ListObjects interface, but this request is a record of Getobject, i.e., the object of this request is local-loki-test instead of index_19651, if there is no local-loki-test object in the bucket, it will return 404. loki-test object in the bucket will return 404.
Is this problem related to loki?

ysf · October 25, 2023, 8:41am

have you tried changing to alibabacloud instead of aws? because I see that you are using AliCloud OSS. I encountered the same error months ago, but now successfully running it using this config

alibabacloud:
  endpoint: https://oss-{region}-internal.aliyuncs.com
  bucket: bucket-name
  access_key_id: access_key_id
  secret_access_key: secret_access_key

everhopeful · October 25, 2023, 11:30am

Thank you for your reply.
I have tried using the alibabacloud configuration but it throws a 403 exception.
The error message is as follows:

err="failed to flush chunks: store put chunk: SecondLevelDomainForbidden: Please use virtual hosted style to access.\n\tstatus code: 403

everhopeful · October 25, 2023, 11:42am

I also have an issue, not sure if it is related to the above error as he can query some data.
I can’t query old data via grafana UI or logCLI, but I can confirm that I have historical data in my backend storage as the monitoring data has been trending upwards. It looks like there are large chunks of data that are not available and it seems to be related to the index storage
Screenshot info below:

May I ask if you have encountered this issue?
Looking forward to your reply Thanks!