Setting up Loki with Helm and Rook-CEPH S3 bucket backend

Grafana Loki
1

Team,

We are aiming towards installing Loki with CEPH S3 bucket as backend storage but loki-stack pod is unable to identify the S3 - Access and Secret keys, gives below errors in the pod logs:

level=error ts=2024-02-28T19:35:41.137726987Z caller=flush.go:146 org_id=fake msg="failed to flush user" err="store put chunk: InvalidAccessKeyId: The AWS Access Key Id you
provided does not exist in our records.\n\tstatus code: 403, request id: KQC43VCQ1XZDRHQ7, host id: SO66dNbWXsv/6l7fkZPDr1WCJwKpb0urwIps72VPX982qQpSLzrJOH8dSERt+oTLsyIIsF+XB
ee7RJehorjgFA=="

Below is the values.yaml used for loki:

loki:
  enabled: true  
  isDefault: true
  url: http://{{(include "loki.serviceName" .)}}:{{ .Values.loki.service.port }}
  auth_enabled: false
  commonConfig:
    path_prefix: /var/loki
    replication_factor: 1
  compactor:
    apply_retention_interval: 1h
    compaction_interval: 5m
    retention_delete_worker_count: 500
    retention_enabled: true
    shared_store: s3
    working_directory: /data/compactor  
  backend:
    persistence:
      storageClass: ceph-filesystem
      enableStatefulSetAutoDeletePVC: false
      size: 10Gi
    extraArgs:
        - '-config.expand-env=true'
    extraEnv:
      - name: S3_LOKI_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: ceph-bucket
            key: AWS_ACCESS_KEY_ID
      - name: S3_LOKI_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: ceph-bucket
            key: AWS_SECRET_ACCESS_KEY
  write:
    replicas: 2
    persistence:
      storageClass: ceph-filesystem
      enableStatefulSetAutoDeletePVC: false
      size: 10Gi
    extraArgs:
        - '-config.expand-env=true'
    extraEnv:
      - name: S3_LOKI_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: ceph-bucket
            key: AWS_ACCESS_KEY_ID
      - name: S3_LOKI_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: ceph-bucket
            key: AWS_SECRET_ACCESS_KEY
  read:
    replicas: 2
    persistence:
      storageClass: ceph-filesystem
      enableStatefulSetAutoDeletePVC: false
      size: 10Gi
    extraArgs:
        - '-config.expand-env=true'
    extraEnv:
      - name: S3_LOKI_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: ceph-bucket
            key: AWS_ACCESS_KEY_ID
      - name: S3_LOKI_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: ceph-bucket
            key: AWS_SECRET_ACCESS_KEY            
  config:
    #server:
    #  http_listen_port: 3100
    schema_config:
      configs:
        - from: 2020-05-15
          store: aws
          object_store: s3
          schema: v11
          index:
            prefix: index_
            period: 24h
    storage_config:   
      aws:
        s3forcepathstyle: true
        s3: s3://<AccessKey-Redacted>:<SecreyKey-Redacted>@rook-ceph-rgw-ceph-objectstore.rook-ceph.svc/ceph-bkt-5059ccef-9b41-4185-8dd4-902c91bfa32d
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  livenessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  datasource:
    jsonData: "{}"
    uid: ""

Please suggest.
Any lead would be appreciated.

Thank You !