Hello! I couldn’t query traces from Tempo datasource (storage configured as s3 bucket). Traces avaliable only for 48h.
Why Tempo could`t query my tracees from s3 bucket? In our production infrastructure, we need to be able to request traces for 30 days.
Should I set block_retention: 30 days to solve the problem ?
Compactor config:
compactor:
replicas: 1
config:
compaction:
block_retention: 48h
iterator_buffer_size: 1000
max_time_per_tenant: 5m
compaction_cycle: 30s
Storage:
storage:
trace:
backend: s3
s3:
access_key: grafana-tempo
bucket: grafana-tempo
endpoint: "minio.company.io"
insecure: false
My bucket structure:
tempodb_blocklist_length:
Other metrics:
Hi, yes block_retention controls the life of files in the s3 bucket. The compactor is responsible for this task. Set it to block_retention: 720h for 30 days retention. Documentation
1 Like
@mdisibio Hello! Could your help me please to understand this metrics: tempo_distributor_ingester_append_failures_total(Panel Failed batch sent to ingesters)?
I couldn’t figure out about what kind of failures this metrics contains?
I see it everytime when we sent traces to Tempo, and I also check this metrics: tempo_receiver_refused_spans and tempo_discarded_spans_total, there are both = 0.
Hi, the metric tempo_distributor_ingester_append_failures_total means the distributor component had trouble forwarding traffic to the ingesters. More detail will be in the distributor logs, possibly the error pusher failed to consume trace data. Based on your screenshot it looks like some traffic was ok because the bottom left panel Ingester Traces Created has data.
