Hi Loki Team/Community!
Hopefully this is an easy one to answer. We have Loki running in microservice mode, index is in DynamoDB and chunks in S3. Currently, we are retaining 4 weeks worth of logs and I have set lifecycle policies on the S3 bucket to automatically clean up stale logs.
We are at the point now where we would like to keep more logs in Loki. I assume a lot of users will want to keep logs for long periods of time for auditing and compliance reasons, and I am looking for the best way to handle this.
- Are there any performance considerations if we are keeping more logs?
1 month vs 1 year vs 10 years?
- If there are performance trade-offs, what kind of strategy can you recommend for backing up/restoring logs?
Let’s say I turn off lifecycle policy on the bucket, can Loki reindex files that already exist? Could I do this on-demand for a date range that I need to audit?
Would love to know how others have solved this.