S3 Retention Configuration

Grafana Loki
,
1

Hello,

I am using the following version of loki:

Maintainer: Grafana Labs <support@grafana.com>
Architecture: amd64
Version: 2.7.1

loki, version HEAD-e0af1cc (branch: HEAD, revision: e0af1cc8a)
  build user:       root@5004faa13e2e
  build date:       2022-12-09T19:23:40Z
  go version:       go1.19.2
  platform:         linux/amd64

I have setup S3 backend and loki is writing as normal. However, I did not setup retention and my S3 bucket is growing. I only need 1 month of logs (744h). Given the crucial nature of the configuration, I’m not sure which setting I need to use to achieve this. Is it possible to achieve retention with the retention guide or should I setup lifecycle rule on S3 bucket itself?

The problem with the lifecycle rule is that the seed file is not updated and there is a risk of deletion. I’d be really glad if you can help me setup retention on AWS S3.

Here is my configuration:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_server_max_send_msg_size: 8388608
  grpc_server_max_recv_msg_size: 8388608

ingester:
  chunk_idle_period: 2h
  chunk_target_size: 1572864
  chunk_encoding: snappy
  max_chunk_age: 2h
  wal:
    enabled: true
    dir: /data/loki/wal
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
      heartbeat_timeout: 10m

schema_config:
  configs:
  - from: 2021-09-27
    store: boltdb-shipper
    object_store: aws
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
  aws:
    s3: s3://REDACTED
    bucketnames: REDACTED
  boltdb_shipper:
    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: aws

chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

compactor:
  working_directory: /data/loki/boltdb-shipper-compactor
  shared_store: aws

limits_config:
  reject_old_samples: false
2

We’ve used the retention period as set out in the retention guide under the limits_config block. This is easier to manage in our use case instead of managing lifecycle policies within Azure blob storage in our case
We’re also looking into a tenanted type approach to be able to use runtime configs and use the various environments as tenants and apply retention per env

3

You can do both. Loki’s retention policy allows you to set retention per tenant if you wish to do so, but you can still use S3 bucket lifecycle policy as the last line of defense just in case.

This is what we do:

  1. In our Loki cluster we take in logs from both production and development environments, and we use Loki’s retention policy to set long retention for production environments, and override it with short retention for development environments. Cost saving.
  2. In S3 bucket we have lifecycle policy that retires all objects 10 days after the longest retention policy in Loki (retires meaning it gets versioned).
  3. We have another S3 lifecycle policy that permanently deletes an object 10 days after it’s versioned.
4

How about loki_cluster_seed.json file? It hasn’t been touched since the installation and I’m afraid if I enable lifecycle policy, this will get deleted.

Since I’m using relatively old version of loki (2.7.1), retention is achieved using table manager. I’ve added the following configuration but I still do not see bucket reducing its size. What am I doing wrong?

table_manager:
  retention_deletes_enabled: true
  retention_period: 30d

compactor:
  working_directory: /data/loki/boltdb-shipper-compactor
  retention_enabled: true
  shared_store: aws

limits_config:
  reject_old_samples: false
  retention_period: 30d
5
  1. You can configure lifecycle policy to ignore the seed file.
  2. The seed file is used by analytics (not critical, as far as I understand). So you can ignore it and wait for it to be re-created, or disable analytics.
6

Try disabling table manager and use compactor only. See Log retention | Grafana Loki documentation.

7

Thank you! I ignored the seed file and setup lifecycle policy on S3. There was a considerable improvement on bucket size. Everything seems to be working with the configuration above and S3 lifecycle policy.

Best,
Eren