Hello! I couldn’t query traces from Tempo datasource (storage configured as s3 bucket). Traces avaliable only for 48h.
Why Tempo could`t query my tracees from s3 bucket? In our production infrastructure, we need to be able to request traces for 30 days.
Should I set block_retention: 30 days to solve the problem ?
Compactor config:
compactor:
replicas: 1
config:
compaction:
block_retention: 48h
iterator_buffer_size: 1000
max_time_per_tenant: 5m
compaction_cycle: 30s
Storage:
storage:
trace:
backend: s3
s3:
access_key: grafana-tempo
bucket: grafana-tempo
endpoint: "minio.company.io"
insecure: false
My bucket structure:
tempodb_blocklist_length:
Other metrics:
Hi, yes block_retention
controls the life of files in the s3 bucket. The compactor is responsible for this task. Set it to block_retention: 720h
for 30 days retention. Documentation
1 Like
@mdisibio Hello! Could your help me please to understand this metrics: tempo_distributor_ingester_append_failures_total(Panel Failed batch sent to ingesters)?
I couldn’t figure out about what kind of failures this metrics contains?
I see it everytime when we sent traces to Tempo, and I also check this metrics: tempo_receiver_refused_spans and tempo_discarded_spans_total, there are both = 0.
Hi, the metric tempo_distributor_ingester_append_failures_total
means the distributor component had trouble forwarding traffic to the ingesters. More detail will be in the distributor logs, possibly the error pusher failed to consume trace data
. Based on your screenshot it looks like some traffic was ok because the bottom left panel Ingester Traces Created
has data.