Can the Index in Object Storage be completely maintained by the Lifecycle Policy?

Grafana Loki
1

Hi,

Our Loki is currently saving its logs to object storage (an oci bucket to be precise).

We have set Loki’s retention to infinite and let a lifecycle policy of the object storage be in charge of deleting the logs older than a retention window. The reason for this is because Loki was saturating our bucket requests.
The side effect of this configuration is that the Index, also in object storage, keeps growing and its older metadata is not being cleaned up.

My question is, for both BoltDB shipper and TSDB Index implementations, can we safely enable the lifecycle policy so it can delete the Index folders that are older than our Lifecycle Policy retention period or will that cause problems to Loki?

From my understanding each Index folder, besides the newest not yet compacted ones, represents the log metadata for a specific day (under default settings), so if for example the retention is 10 days, it’s safe to delete all the Index folders older than 10 days.
Is that correct or is the only way to clean the Index to temporarily enable Loki’s retention again?

2

If your compactor is working properly, you should end up with 1 index file per day. And yes, you should be able to use object storage lifecycle policy to remove the old index files (test first of course). Preferably you should remove the index files a day before the chunk files, otherwise you could see errors when querying the very oldest logs when index file is still present but chunk files no longer are.

1 Like