How does retention's non-retroactive behaviour impact old logs?

Grafana Loki
1

Hello,

I’m running Loki 2.9.15. I had the pod up for a few months with no retention enabled, so naturally I accumulated close 2TB of logs.

Then, two days ago, I set retention to 14 days. However, no log has been deleted at all. That is expected for logs more recent than 14 days, but my months old logs are not being deleted either.

I looked at the docs and in the latest version of the docs I see the statement:

Changes to your retention period are not retroactive, that is, they are not applied to logs that have already been ingested.

Does that mean that all the logs that were ingested prior to my change of the retention period will never be deleted by the compactor? If yes, does that apply to Loki 2.9 as well?

If that is the case, then what is the recommended approach to delete logs ingested before the change in the retention period?

FWIW, this is my config for the schema, compactor, and limits:

...
schema_config:
  configs:
  - from: '2020-10-24'
    index:
      period: 24h
      prefix: index_
    object_store: filesystem
    schema: v11
    store: boltdb-shipper
  - from: '2026-08-05'
    index:
      period: 24h
      prefix: index_
    object_store: filesystem
    schema: v13
    store: tsdb
...
compactor:
  retention_enabled: true
  shared_store: filesystem
  working_directory: /loki/compactor
...
limits_config:
  allow_structured_metadata: true
  ingestion_burst_size_mb: 15.0
  ingestion_rate_mb: 4.0
  per_stream_rate_limit: 4MB
  per_stream_rate_limit_burst: 15MB
  retention_period: 14d
  split_queries_by_interval: '0'

Thanks

2

I am not sure what that line actually means, from the logic I don’t think there is such limitation, but I could be wrong. We use S3 as storage, and we’ve always had a secondary lifecycle policy on the S3 bucket to remove files older than 10days + global retention period in case compactor fails for whatever reason.

Are your old files still present? If so it’s pretty safe to remove older files (both index and chunks) under your loki storage, assuming nothing else is writing to the directory, of course. I’d recommend you to create a backup just in case, and start with files older than, say, 1 year, and see how that goes.

3

Thanks @tonyswumac, I think that should be a safe way to proceed. Out of curiosity, should the deletion API endpoint be a good approach in this case as well?

5

I actually don’t use the deletion API very often. I only use it when I need to delete an entire log stream. If you don’t have a large number of log streams it may work.