Compactor - uneven CPU load between instances

When I’m looking at the usage of our tempo compactors, some of them are idling, while some of them have almost 100% of CPU usage:

I hope that the load distribution could be distributed more evenly by tweaking the compactor configuration.

If I would change max_block_bytes, or compaction_window- would it force the compactor to ingest smaller blocks, and thus would better split the work between the instances?

Our current configuration looks like:

compactor:
  compaction:
    # 30 days
    block_retention: 720h
    # 500MB max block bytes
    max_block_bytes: 500_000_000
    compaction_window: 1h
    compaction_cycle: 30s
  ring:
    kvstore:
      store: memberlist
1 Like

We have the same issue here. 1 compactor consumes 100x cpu and 10x memory than the other 39 compactors.