Loki not pushing logs to s3

Grafana Loki
1

I am trying to setup loki locally by pulling docker image and connecting loki with s3 for logs storage.
I have this loki config yaml file.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0
  chunk_target_size: 1536000
  chunk_block_size: 262144

schema_config:
  configs:
    - from: 2023-01-01
      store: tsdb
      object_store: aws
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h
  aws:
    s3: s3://AKIAU5DVIDKCSUWJSOLM:2eJoL28TUmdlqCsmM4ZPpmnyvuJVL@ap-south-1
    bucketnames: sparrow-logs

compactor:
  working_directory: /loki/compactor
  shared_store: s3

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: true
  retention_period: 168h

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules
  ring:
    kvstore:
      store: inmemory
  enable_api: true

It doesn’t show any error. In fact it shows some logs like this

2024-08-05 09:37:16 level=info ts=2024-08-05T04:37:16.79647667Z caller=index_set.go:86 msg="uploading table index_19940"
2024-08-05 09:37:16 level=info ts=2024-08-05T04:37:16.796544212Z caller=index_set.go:107 msg="finished uploading table index_19940"
2024-08-05 09:37:16 level=info ts=2024-08-05T04:37:16.796556754Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19940"
2024-08-05 09:37:16 level=info ts=2024-08-05T04:37:16.796595795Z caller=table.go:318 msg="handing over indexes to shipper index_19940"
2024-08-05 09:37:16 level=info ts=2024-08-05T04:37:16.796608004Z caller=table.go:334 msg="finished handing over table index_19940"

But when I check my s3 bucket it remain empty. I have verified my s3 creds by uploading file from my terminal and they are correct.

2

Feels like you are still writing to local directory. I’d double check:

  1. See if chunks are indeed written to a local directory.
  2. Check /config endpoint from your container and compare the configuration.

Also your configuration has access key and secret key, might want to remove / change that.

3

I faced the same issue when using Loki Distributed version 0.80.1.

Any configuration changes were not working. After reviewing the documentation, I found that the S3 definition was incorrect and should be under common.storage.s3.

Here is an example of the correct configuration:

common:
  compactor_address: http://{{ include "loki.compactorFullname" . }}:3100  
  storage:
    s3:
      endpoint: s3.ap-southeast-1.amazonaws.com
      region: <YOUR_REGION>
      bucketnames: <YOUR_BUCKET_NAME>
      secret_access_key: <AWS_SECRET_ACCESS_KEY>
      access_key_id: <AWS_ACCESS_KEY_ID>
      insecure: false

and schemaConfig storageConfig parts looks like this

 schemaConfig:
    configs:
    - from: "2025-03-10"
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

  storageConfig:
    boltdb_shipper:
      shared_store: s3
      cache_ttl: 1h
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
    filesystem:
      directory: /var/loki/chunks