I’m trying to load historical as well as current logs into a Loki instance.
One stumbling block I ran into was old logs being rejected. Part of the solution was to bump ingester.max_chunk_age to a large value of 8928h or ~1 year.
This seems to have helped, but we hope to keep these logs around indefinitely. The oldest logs are about four months old. Is there any risk of them being expired or expunged after another two months have elapsed? Or does max_chunk_age really only affect log ingestion?
(We have no plans for eliminating the original logs, of course, so losing them would be more of an annoyance than a disaster, but still. I’d prefer to avoid it if I could.)
max_chunk_age is one of the control for ingesters on how frequent to write to the backend storage. You should not set it to anything longer than a couple of hours. If you send it to 1 year that means you can have active log streams sitting in ingesters for prolonged period of time. I would not recommend that. It’s fine if you set it temporarily to ingest old logs, but you should look to change it back once you are done.
A related question: is there any other way to get old logs ingested, one that I missed? Preferably one that does not require us to change settings temporarily and then remember to return them to something sane.
There are two things you need to watch out for when ingesting old logs:
You have to ingest the logs in order from old to new.
You cannot ingest logs into a log stream that already has newer logs.
#2 is probably the most common issue most people run into. Let’s say you have a log stream {app="123"} that has logs from today, you cannot ingest logs from a week ago into the same label {app="123"}. But you can work around this easily by adding another label, say {app="123",ingest_attempt="1"}, effectively creating a new log stream, but still allow you to search with just the app label. This is usually what I do if I need to ingest older logs into Loki.
