I have a question. Please help me answer it.
Loki’s ingester node occupies very high memory (the memory dumped by the GO program occupies 3GB, and the container memory occupies 6GB), but the throughput is low (15MB per second). The proportion of bytes.growSlice in the dumped memory is between 60% and 80%. in between, and most of them are being reordered by rebound (looking at the code, the chunks to be written are not in order, so they need to be reordered). Is it as expected?
It is important to mention that we have added a layer of MQ buffering to loki’s promtail node and loki’s distributor & injgester. But we believe that MQ is most likely to be first in, first out, and there shouldn’t be too many situations where the order is wrong.
I haven’t done a deep dive on Loki’s memory usage, hopefully someone more knowledgable can answer. But i believe ingesters assume that chunks are potentially out of order (there is a configuration for this), so they are reordered before flushing.