Hi, i´m testing Loki and using the helm chart loki-stack for deployment my log solution, before i was used the boltdb-shippper as storage, but i starting test with tsdb after last update, but i couldn´t apply the compactor, because it only accept with boltddb-shipper the compactor.
My helm chart version is: 2.9.10
Please, could someone help-me?
here is the log message
level=info ts=2023-05-09T13:05:04.81855817Z caller=modules.go:863 msg=“Not using boltdb-shipper index, not starting compactor”
I’ve just stumbled over a similar question. TSDB is supposed to be the successor of boltdb-shipper, but compactor officially does only support boltdb-shipper.
With table-manager being deprecated, that leaves TSDB without any retention mechanism - or does it?
With object-storage as backend we can still configure a retention time enforced by the backend storage, but with loki running on local filesystem storage, TSDB seems to be not usable.
TSDB still benefits from compaction, and certainly you still need to run compactor for retention.
How does TSDB benefit from compaction if compactor doesn’t support TSDB?
Please take my comment below with a grain of salt, as I haven’t fully gone through the TSDB code with a complete understanding yet.
But I believe the reason it benefits from compaction is reduction of number of index files. In loki you will have one index file per writer per configured period. If you have a small cluster this is probably not a big deal, but if you have a big cluster this adds up quickly. Compactor compacts the index files into one per configured period.
Hi again,
I think this is a misunderstanding. We know and understand what compaction is and why it is important (along with its built-in retention).
but looking at the documentation, and it clearly says
However retention through the Compactor is supported only with the boltdb-shipper store.
However I cannot find the error message @gleydsoncavalcantic mentioned in the current code (I can find it in 2.6.0). So my current assumption is that compactor does support tsdb and the documentation might just be out of date.
You could be right, I am not too familiar with the new format yet.