Loki Does Not Read Older Logs From AWS S3

Hello community,

I’ve recently started using Loki for my project and have encountered an issue. I’m confused because I can’t seem to read my older logs that Loki has stored in AWS S3.

The Loki instance has been running via Docker Compose for the past 5 days, successfully storing logs in my AWS S3 bucket. However, I’m unable to retrieve logs from Grafana for dates older than today.

My question is, what steps should I take to ensure that Loki can sync the older logs from AWS S3, especially when logs have been removed in Loki due to the retention feature?

Here’s my Loki config file:

auth_enabled: false

  http_listen_port: 3100

  path_prefix: /tmp/loki
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
      store: inmemory

        enabled: true
        max_size_mb: 100

  max_look_back_period: 0s
      enabled: true
      max_size_mb: 500
      ttl: 24h

    - from: 2020-10-24
      store: boltdb-shipper
      object_store: aws
      schema: v13
        prefix: loki_index_
        period: 24h

    active_index_directory: /loki/index
    shared_store: aws
    cache_location: /loki/boltdb-cache
    s3: s3://access_key:secret_access_key@region
    bucketnames: be-app-log

  working_directory: /loki/boltdb-shipper-compactor
  shared_store: aws
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

  split_queries_by_interval: 0
  allow_structured_metadata: true
  retention_period: 24h

This is how log logs are stored in object store, S3 in your case. If the logs are older, they are deleted. This is why you can only see the last 24h of logs in Grafana.

Do you mean the logs in S3 are also deleted after 24 hours?

As of now, I can still see my older logs in my S3 bucket, but they have been compressed to GZIP. From my understanding, the retention period won’t delete the logs stored in AWS S3, as I would need to configure a separate retention policy in my S3 buckets to delete the logs.

Please correct me if I’m wrong.

The docs for retention_period say “Retention period to apply to stored data”

There are some other settings that do affect how and when data is deleted, like retention_delete_delay

1 Like

Could you please inform me which object store you are using to store the logs from Loki, @b0b ?

Have you checked whether the logs in your object store are being removed by Loki once they have surpassed the retention_period?

We use S3 as well. Not actually sure how to check what the oldest files in an S3 bucket is :grimacing:

But e.g. Total number of objects is not growing at the moment. Actually it is going down. Everything looks like I would expect it to look if data retention worked as intended.

I see. May I know how you set your retention period?

I think this is all the config related to data retention

  working_directory: /var/loki/retention
  shared_store: s3
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  retention_period: 1200h
1 Like

Well noted, @b0b . Thank you for your assistance! :slight_smile:

@airelljordan , have you find the way to read the older logs, which is in the bucket, but i am not able to read.

so for example i have a s3bucket and i put the data for 5 days, and then i changed the bucket, or change the complete setup, but i want that older 5 days logs to get read again in the new setup, is there any possibility to read that