Setting up Loki Retention for Cost Optimization

Grafana Loki
,
1

Logs quietly increase costs over time. If you collect and retain everything without limits even in non-production or test clusters, storage costs will become noticeably higher after one or two years. Especially with object stores such as S3, costs grow linearly with stored volume. You should therefore design a retention period that fits each environment and workload so that you can manage total storage and query performance together.

In Loki, retention is handled by the Compactor. The default is to retain forever, and a retention policy is applied only when you enable retention_enabled. If the object store has its own lifecycle policy, its period should be longer than Loki’s retention period.

The Compactor’s algorithm to apply retention is as follows:

  • For each day or table (one table per day with a 24h index period):
    • Compact multiple index files in the table into per-tenant index files. The result of compaction is a single index file per tenant per day.
    • Traverse the per-tenant index and use the tenant configuration to identify the chunks that need to be removed.
    • Remove the references to the matching chunks from the index and add the chunk references to a marker file on disk.
    • Upload the new modified index files.

Loki Helm values example

Below enables the Compactor and applies a default 30-day retention. You can extend per-environment differences later using retention_period or retention_stream.

loki:
  compactor:
    working_directory: /tmp/loki/retention
    compaction_interval: 10m
    retention_enabled: true
    retention_delete_delay: 10m
    retention_delete_worker_count: 100
    delete_request_store: s3

  limits_config:
    retention_period: 30d

  • retention_enabled should be set to true. Without this, the Compactor will only compact tables.

  • delete_request_store should be set to configure the store for delete requests. This is required when retention is enabled.

  • working_directory is the directory where marked chunks and temporary tables will be saved.

  • compaction_interval dictates how often compaction and or retention is applied. If the Compactor falls behind, compaction and or retention occur as soon as possible.

  • retention_delete_delay is the delay after which the Compactor will delete marked chunks.

  • retention_delete_worker_count specifies the maximum number of goroutine workers instantiated to delete chunks.

Required checks

  1. In the Compactor logs, confirm removing messages such as
    level=info ts=20xx-xx-xxT03:57:42.850907902Z caller=index_set.go:269 table-name=loki_index_20384 msg=“removing source db files from storage” count=6
  2. After the deletion delay has passed, verify that the number of S3 objects and total size decreases.
    image
    image767Ă—508 17.8 KB

    In the example above, applying a retention policy to a Loki setup that previously had no retention significantly reduced the bucket size.

Conclusion

Designing Loki retention around the Compactor lets you directly control stored log volume and cost. Combine a company-wide default with label-based granularity and avoid conflicts with object store lifecycle policies to reduce storage costs and query load in a stable way. By optimizing retention together with labeling strategy and caching, you can achieve both cost savings and performance stability.

3 Likes