Cannot query traces from Tempo after 48h

Grafana Tempo
1

Hello! I couldn’t query traces from Tempo datasource (storage configured as s3 bucket). Traces avaliable only for 48h.
Why Tempo could`t query my tracees from s3 bucket? In our production infrastructure, we need to be able to request traces for 30 days.
Should I set block_retention: 30 days to solve the problem ?
Compactor config:

compactor:
  replicas: 1
  config:
    compaction:
      block_retention: 48h
      iterator_buffer_size: 1000
      max_time_per_tenant: 5m
      compaction_cycle: 30s

Storage:

storage:
  trace:
    backend: s3
    s3:
      access_key: grafana-tempo
      bucket: grafana-tempo
      endpoint: "minio.company.io"
      insecure: false

My bucket structure:

image
image708×401 14.6 KB

tempodb_blocklist_length:
image
image1363×665 49.8 KB

Other metrics:
image
image1863×808 85.9 KB

image
image1354×813 65.8 KB

2

Hi, yes block_retention controls the life of files in the s3 bucket. The compactor is responsible for this task. Set it to block_retention: 720h for 30 days retention. Documentation

1 Like
3

@mdisibio Hello! Could your help me please to understand this metrics: tempo_distributor_ingester_append_failures_total(Panel Failed batch sent to ingesters)?
I couldn’t figure out about what kind of failures this metrics contains?
I see it everytime when we sent traces to Tempo, and I also check this metrics: tempo_receiver_refused_spans and tempo_discarded_spans_total, there are both = 0.

image
image1860×764 72.8 KB

4

Hi, the metric tempo_distributor_ingester_append_failures_total means the distributor component had trouble forwarding traffic to the ingesters. More detail will be in the distributor logs, possibly the error pusher failed to consume trace data. Based on your screenshot it looks like some traffic was ok because the bottom left panel Ingester Traces Created has data.

5

Just want to ask a follow up question regarding this one. In the document, there is another parameter to control the retentions

# Optional. Duration to keep blocks that have been compacted elsewhere. Default is 1h.
        [compacted_block_retention: <duration>]

It was quite obvious what this is used for. Could you please share some lights?

6

compacted_block_retention configures how long compacted blocks are kept in storage before deletion. When the compactor compacts blocks, it doesn’t delete them right away, but marks them as compacted. Compacted blocks are deleted afterwards asynchronously.

7

@mariorodriguez , thanks a lot for the reply. It is helpful to understand there is a separate garbage collection algorithm on this. But why keeping a retention on these blocks? Is it to try to protect the data when there could be transient failures/crashes?

8

Mainly to help with block list maintenance. Once a new block is created by compacting others, it can take a bit for all queriers to find it and update it in their block lists. Not deleting compacted blocks right away allows for queriers to fallback to those and adds resilience to the read path.