We’re hitting a cap of 2560 spans in our trace view (Our workload has ~6000 spans for a trace). Is there any configuration, environment variable, or workaround to increase this limit? Any guidance would be appreciated!
Thanks for responding!
I’m using Grafana Tempo to store traces and Grafana as the frontend. I understand Loki is different, but Tempo suits my use case since it’s optimized for trace data.
The issue I’m facing is the 2560 span limit, which I believe is a frontend constraint. Is there any way, through config, environment settings, or otherwise to increase or bypass this limit?
Appreciate any guidance!
After days of debugging and thinking the issue was with Grafana UI or Tempo, I finally found the real bottleneck: OpenTelemetry’s OTLP exporter and BatchSpanProcessor default limits.
If you’re using a Python script to generate and export spans via OTLP (like I was), here’s what you need to know:
By default, the OpenTelemetry Python SDK uses a BatchSpanProcessor with these limits:
max_queue_size = 2048
max_export_batch_size = 512
This means that if you generate more than 2048 spans quickly, spans will be silently dropped.
Even if Tempo and Grafana can handle 10,000+ spans, you’ll only see 2560 or fewer, because the rest never made it out of your script.
Update your BatchSpanProcessor like this:
from opentelemetry.sdk.trace.export import BatchSpanProcessor
span_processor = BatchSpanProcessor(
otlp_exporter,
max_queue_size=100000,
schedule_delay_millis=1000,
max_export_batch_size=10000
)
2 Likes