I’m running into an issue with the Loki compactor and hope to get some advice from the community. We’re using Google Cloud Storage (GCS) as our object store, and we’ve enabled a bucket-level retention policy for compliance reasons.
The goal is to ensure our logs are immutable for their required retention period. However, this seems to be conflicting with the compactor’s need to merge and delete older files.
The compactor is consistently failing with the following error in a test instance:
level=error ts=2025-08-28T07:09:09.992745174Z caller=compactor.go:691 msg="failed to compact files" table=loki_index_20328 err="googleapi: Error 403: Object 'tback-loki-xxxxxxxxxxxxxxx/index/loki_index_20328/1756362652-loki-0-1754916784061287323.tsdb.gz' is subject to bucket's retention policy or object retention and cannot be deleted or overwritten until 2025-08-29T00:01:25.074671-07:00, retentionPolicyNotMet"
It’s clear what’s happening – the compactor is trying to delete an index file that is locked by our GCS retention policy.
My question is: How are others handling the requirement for immutable log storage with Loki? Is there a recommended configuration or a different approach to achieve compliance that doesn’t break the compaction process?
I think it’s possible to use a different GCS storage for index, so essentially you’d configure index to store to one bucket, and chunks to another, then apply different policies. Test in your dev environment first, of course, and if you do decide to migrate you’ll probably need to configure a new index configuration (test migration plan too).
Just curious, Loki logs are already immutable (as long as you don’t expose the delete endpoint). What sort of policies are you applying to your storage?
I think it’s possible to use a different GCS storage for index, so essentially you’d configure index to store to one bucket, and chunks to another, then apply different policies.
Loki 2.8 introduced TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki. More detailed information about TSDB can be found under the manage section.
[…]
Prior to Loki 2.0, chunks and index data were stored in separate backends: object storage (or filesystem) for chunk data and NoSQL/Key-Value databases for index data. These “multistore” backends have been deprecated, as noted below.
Does anybody know more? I’d be more than happy to move indexes and log chunks to different buckets again.
Just curious, Loki logs are already immutable (as long as you don’t expose the delete endpoint). What sort of policies are you applying to your storage?
I know that neither I nor loki change the data. However, it be helpful in compliance certification processes to just have Bucket Retention Policy in place that says that a file in the bucket needs to retained for 1 year (or for my test case for 1 day).
This doesn’t seem to work because the index files seem to get deleted and recreated a few times on the first day. Then they’re not touched anymore until loki (or a bucket policy) reaps them.
