Trace duration, shown in table from traceQL result, is way longer(minutes) than longest span in that trace when clicking into the traceid to inspect the spans time length(seconds). It only happens to a few from thousands of traces a day, but messes up sorting by longest traces.
Hi, Tempo computes the trace duration as the difference between the earliest span start, and the latest span end. It does have some limitations in accurately knowing for long-running traces, due to how Tempo works internally. It flushes blocks to the backend every several minutes, and this means long-running traces get spread across the backend. We search 1 block at a time, so the trace duration shown is the longest start/end difference that was found in that block. In this case the 40 min duration means that most likely the last 40 minutes of spans are in a block together, and what was covered by search time range. The earliest spans (15h earlier) are in a separate block.
I know it’s not a solution, but hope this explanation helps.
1 Like