Using multiple s3 buckets for loki giving error

Grafana Loki
1

Hello,

we have implemented multiple buckets as storage

    storage_config:
      aws:
        s3: s3://us-east-1
        bucketnames: grafana-loki-us-east-1-##########,grafana-loki-us-east-1-##########-001,grafana-loki-us-east-1-##########-002,grafana-loki-us-east-1-##########-003,grafana-loki-us-east-1-##########-004,grafana-loki-us-east-1-##########-000
        # https://github.com/grafana/loki/issues/4779#issuecomment-981397844
        http_config:
          response_header_timeout: 5s
      boltdb_shipper:
        shared_store: s3
        active_index_directory: /var/loki/index
        cache_location: /var/loki/cache
        cache_ttl: 168h
        {{- if .Values.indexGateway.enabled }}
        index_gateway_client:
          server_address: dns:///{{ include "loki.indexGatewayFullname" . }}:9095
        {{- end }}
      index_queries_cache_config:
        redis:
          endpoint: redis:6379
          expiration: 168h

This was done to balance perfomance. Now query speed is better, but some queries are giving the below errror

failed to load chunk ‘fake/f6cae7010d24dfb1:18c146759f1:18c146ceef3:ceedcd1c’: failed to get s3 object: NoSuchKey: The specified key does not exist. status code: 404, request id: Y4F3CJFEY8029WMH, host id: NXQtBsCXyEBb9y6/Cft8KpituvCq72qJugVD4qq54d3HGhuN8/0Lk7Uw+9PXOU42qlorBQBc00XK9Nis81K5hia8KpBS0bU02I+ZrHcvLNg=
(Trace ID: b11b9e18bb4cde3f141d37e3e93d968d)

Any idea about the issue ?

2

I’ve never tried multiple s3 buckets before, so I could be speaking out of ignorance here. But from looking at your configuration it looks like your boltdb-shipper is also configured to go to multiple S3 buckets. Is that recommended? I’d try having a separate storage configuration for the index part only and see if that helps (test in a lab environment of course).

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.