I was watching the latest lock architecture and notice kafka is introduced and we remove ingestor replica. Then what’s the best practice on configuration of kafka commit interval - if it is too long, it’s good for query to build larger chunk but it’s harder to rebuild/replay data on bad days.
Also if there’s no ingestor replica, and if some ingestor has down time, wouldn’t it impact querier availability? I assume querier still directly hits ingestor to query most recent data
Kafka is replacing ingester replication as the durability mechanism. I haven’t seen documentation indicating the Kafka consumer commit interval affects chunk building → chunk creation still appears to be governed by settings such as chunk_target_size, max_chunk_age, and chunk_idle_period.
The commit interval seems primarily to affect recovery behavior: a longer interval generally means fewer offset commits but potentially more data to replay after an ingester restart, while a shorter interval reduces replay at the cost of more frequent commits.
Also if there’s no ingestor replica, and if some ingestor has down time, wouldn’t it impact querier availability? I assume querier still directly hits ingestor to query most recent data
I’d expect recent (hot) queries to continue depending on ingesters, since the newest data hasn’t yet been flushed to object storage. What I couldn’t determine from the GrafanaCON talk is how the new architecture maintains hot query availability during an ingester failure → for example, whether another ingester immediately takes over the Kafka partition or whether there’s another mechanism. That part would need clarification from the Loki team.
They should be relevant:
Imagine I configure offset interval 1s, but chunk age is 30minute. Let’s say at 15th minute, the ingestor hasn’t flushed data to object store, and there’s fail over happens. Then ingestor will start from 15minute-1s to build chunk, and lose previous 14minute59s data. Ingestor should only commit only after chunk has flushed to avoid lose data. So commit interval should impact chunk size