Retention period for different log stream

Grafana Loki
1

Hi guys,

Do we have any features or solutions to save logs with different retention period? This maybe a common case that for some services we wish to save their logs for a longer period whereas other are not.

Regards

2

You can override it like this:

limits_config:
  retention_period: 744h
  retention_stream:
  - selector: '{namespace="dev"}'
    priority: 1
    period: 24h

You can also override it per tenant. See more here: Log retention | Grafana Loki documentation

3

Appreciate for the reply!!

However I’m using s3 bucket as storage. The doc says I can only manage retention policy from aws side (we are using table-manager to do retention for current configuration). Does loki have control of retention under such case? Or the “select stream” is a feature only supported by compactor

4

You can absolutely manage retention policy from Loki. You should want to use compactor instead of table manager (see Log retention | Grafana Loki documentation), and if you are not at a version where compactor is recommended you should consider upgrading as well.

If you are using S3 as storage, this would be my recommendation:

  1. First ,determine what your absolute max retention period is (let’s say for example this is 365 days).
  2. Enable versioning for S3, and in S3 life cycle policy set object retirement to 375 days, then set 10 days to permenantly delete versioned objects. This gives you 375 + 10 days to recover something if you ever need it
  3. In compactor, set default retention to 365 days.
  4. Override for stream or tenant as needed.
1 Like
5

That would be great!

One last question here, the compactor will delete logs chunks inside s3 instead of only delete index files right? I’m trying to save storage cost.

6

Yes, compactor will remove chunk files. I still recommend also putting in S3 lifecycle policy just as a safety net.

1 Like