Tempodb_compaction_outstanding_blocks kept Increasing after reducing compactor replica from 3 to 1

Grafana Tempo
1

Hi,
We are seeing tempodb_compaction_outstanding_blocks increasing continuously post reducing the compactor replica count from 3 to 1.

We faced this issue: #2560
Errors says: If multiple replicas are used, then they may be process same traces for compaction reutilizing in this error.

Article toomanyoutstandingblocks asking to increase compactor count, but we may face issue #2560 again.

Our Single Compactor Replica Utilization around 2.5GB and running smoothly in our Docker Swarm Cluster, while limits are below.

      resources:
        limits:
          memory: 4096M
        reservations:
          memory: 256M

Current Tempo Version: 2.7.0

Checked: #4067 (comment) , on Single Replica, will it be beneficial to decrease [compaction_window: <duration>] to 30 mins in reducing tempodb_compaction_outstanding_blocks?

image
image688×173 19.4 KB

Few Queries:

  1. How to manage/reduce tempodb_compaction_outstanding_blocks without compromising performance?

  2. Is block_list_length & tempodb_compaction_outstanding_blocks has any relation between them? In our case since few months block_list_lenght stable at 27k for 90 days retention data, but tempodb_compaction_outstanding_blocks increasing to 44k from 30th July 2025 on 90 Days retention data.

2

you need to scale up your compactors to manage the blocklist, without compaction blocklist will keep increasing.

if you increase ingestion, it will create more blocks, and to catch up with that you need to compact. at one point a block reaches the max block size and then it’s not compacted further.

tempodb_compaction_outstanding_blocks is a measure of compaction load, so if it’s not coming down and keeps growing that means that you are not compacting fast enough.

tempodb_blocklist_length is a measure you how many blocks you have in the backend, if your ingestion of retention is increased, it will increase but if your compaction is doing the job, it should come down as you compact more blocks.

If you not on a recent version, please try to upgrade, and also setup monitoring if you haven’t yet.