Loki Does Not Read Older Logs From AWS S3

Grafana Loki
,
1

Hello community,

I’ve recently started using Loki for my project and have encountered an issue. I’m confused because I can’t seem to read my older logs that Loki has stored in AWS S3.

The Loki instance has been running via Docker Compose for the past 5 days, successfully storing logs in my AWS S3 bucket. However, I’m unable to retrieve logs from Grafana for dates older than today.

My question is, what steps should I take to ensure that Loki can sync the older logs from AWS S3, especially when logs have been removed in Loki due to the retention feature?

Here’s my Loki config file:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

chunk_store_config:
  max_look_back_period: 0s
  chunk_cache_config:
    embedded_cache:
      enabled: true
      max_size_mb: 500
      ttl: 24h

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: aws
      schema: v13
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    shared_store: aws
    cache_location: /loki/boltdb-cache
  aws:
    s3: s3://access_key:secret_access_key@region
    bucketnames: be-app-log

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: aws
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

limits_config:
  split_queries_by_interval: 0
  allow_structured_metadata: true
  retention_period: 24h
2

This is how log logs are stored in object store, S3 in your case. If the logs are older, they are deleted. This is why you can only see the last 24h of logs in Grafana.

4

Do you mean the logs in S3 are also deleted after 24 hours?

As of now, I can still see my older logs in my S3 bucket, but they have been compressed to GZIP. From my understanding, the retention period won’t delete the logs stored in AWS S3, as I would need to configure a separate retention policy in my S3 buckets to delete the logs.

Please correct me if I’m wrong.

5

The docs for retention_period say “Retention period to apply to stored data”

There are some other settings that do affect how and when data is deleted, like retention_delete_delay

1 Like
6

Could you please inform me which object store you are using to store the logs from Loki, @b0b ?

Have you checked whether the logs in your object store are being removed by Loki once they have surpassed the retention_period?

7

We use S3 as well. Not actually sure how to check what the oldest files in an S3 bucket is :grimacing:

But e.g. Total number of objects is not growing at the moment. Actually it is going down. Everything looks like I would expect it to look if data retention worked as intended.

8

I see. May I know how you set your retention period?

9

I think this is all the config related to data retention

compactor:
  working_directory: /var/loki/retention
  shared_store: s3
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
limits_config:
  retention_period: 1200h
1 Like
10

Well noted, @b0b . Thank you for your assistance! :slight_smile:

11

@airelljordan , have you find the way to read the older logs, which is in the bucket, but i am not able to read.

so for example i have a s3bucket and i put the data for 5 days, and then i changed the bucket, or change the complete setup, but i want that older 5 days logs to get read again in the new setup, is there any possibility to read that