Difference between block_retention vs compacted_block_retention

Grafana Tempo
1

Google Answered:

In Grafana Tempo, block_retention and compacted_block_retention control how long data is stored. block_retention determines how long blocks (raw trace data) are kept before deletion, while compacted_block_retention determines how long compacted blocks (aggregated trace data) are kept after compaction.

My config:

       compactor:
         compaction:
           compaction_window: 5m0s
           block_retention: 8766h0m0s  # 1 Year
           compacted_block_retention: 1h  #Default

I want to keep Tracelog data for 1 Year so block_retention: 8766h0m0s. I am seeing Queries are slow & Block List Lenght is increasing. Currently its 95k.

As per Google Ans, I understand that, I am not keeping compacted blocks more than 1Hour, may be due to which all blocks are uncompacted due to which queries are slower and block list length is increasing.

If I change compacted_block_retention: 8766h0m0s # 1h Default 1 year, then queries:

  1. Will I able to retain data till 1 Year but in compacted format which will improve query performance & able to query last 1 year data?
  2. Can I change block_retention: 8766h0m0s to 14 days from 1 year to reduce block list length?
2

block_retention: the amount of time after the data is deleted, i.e it’s retention

compacted_block_retention: The compactor compacted_block_retention is used to keep a block in the backend for a given period of time after it has been compacted and the data is no longer needed. This allows queriers with a stale blocklist to access these blocks successfully until they complete their polling cycles and have up to date blocklists. Like the complete_block_timeout , this should be at a minimum 2x the configured blocklist_poll duration.

See more about compacted_block_retention at Polling | Grafana Tempo documentation

I want to keep Tracelog data for 1 Year so block_retention: 8766h0m0s. I am seeing Queries are slow & Block List Lenght is increasing. Currently its 95k.

As you add more blocks to the backend, the blocklist will increase.

checkout Tune search performance | Grafana Tempo documentation and Improve performance with caching | Grafana Tempo documentation for query and search perf tuning

1 Like
3

Thanks @surajsidh for your response

Right, we are facing high block list lenght in Prod, which is processing few hundred GB’s of Data/Day.

Recently reduced the Data Retention from 1 Year to 90 Days for Tempo and Block List Length dropped from 95k to 27k.

In Article: Tune search performance | Grafana Tempo documentation saw , while additional compactors more aggressively reduce the length of your blocklist and copies of data (if using RF=3).

Currently using single compactor replica in our docker swarm env: We earlier used 3 compactor replicas but faced this issue: Compaction Errors & High network usage by Tempo · Issue #2560 · grafana/tempo · GitHub .

Even with 3 Compactor replicas we were seeing: block lenght list paramter increased consistently to 90k from 60k in last 90 days few months ago.

From your above post I went through link which were helpful, narrowing down to block length list topic

Few Follow up Queries:

  1. How can we reduce block_list_length if we need to reduce based on best practices?
    1. What is the recommended block_list_length for 90 days retention period.
    2. Will higher block_list_length such as 27k affect performance?
4

I have answer similar questions here, please see those as well.

Compact and scale up compactors, use tempodb_compaction_outstanding_blocks to measure the pending work.

depends a lot on many factors, and workload, I can tell you that we have seen cluster with 1M+ blocklist

Yes, try to compact as much as you can. but at some points you need to look at the cost to compact and cost to query an extra block, sometimes it’s better to scale up read path, sometimes it’s better to scale up compactor. it’s a tradeoff but usually less blocks will means better performance but beyond a blocksize it might impact negatively.

imagine if you had a one block of 1TB, it would be bad for perf. so you need to find a balance and see what works for your workload.