Hi,
Wasn’t sure if this is a “Tempo” issue or a “Grafana” issue, but… not sure if we’re doing tracing “wrong” but when I “Explore” my Tempo datasources it limits to 20 traces by default, but it shows the first 20 traces, rather than the most recent 20 traces.
This is quite confusing, and worse, I can’t figure out how to change it?
Hi 
@colemickens, you can change the time range to search the recent data, and increase the default limits to get more then 20 results.
see the red highlighted areas in this screecap:
Hm, let me try to describe my problem again. I am selecting a recent time window, that’s not quite the problem.
The problem is that when Grafana shows N results, it shows the first N results, rather than the last N results.
These screenshots demonstrate the issue. Note, no new traces were added between screenshots.
Notice how when I increase the limit, I see newer results. This is contrary to every other system I’ve used that shows most-recent first, and limits results. It was very confusing to be expecting to see the 10 most recent rows and “not see new traces”. They were there, just not being shown to me.
This feels unintuitive to the point that I’m wondering if I’ve misconfigured something. I don’t see anything that allows me to control sort.
we return the first N (N=limit) results that match the query in the selected time range, search results are not sorted.
you can use narrow down the search using traceql to filter down to sepefic spans, for example: { span.http.status_code >= 200 && span.http.status_code < 300 }
search results are not sorted.
would it hard to sort them, implementation-wise? or return the last N results?
It feels like in order for me to deliver this to other folks, I need to write an explanation of how to use this:
Because otherwise as soon as they go over 20 traces they just stop seeing new traces with no indication as to why.
would it hard to sort them, implementation-wise? or return the last N results?
yes, because to sort them we need to fetch all the results (i.e do the exhaustive search) that match your query, and for a broad query like { } will match all the traces…
for any big cluster(assuming terabytes of data in the query time range), fetch all the data, sorting and then retuning 20 results is not practical.
our recommendations:
That’s pretty annoying
@surajsidh
What does the first N (N=limit) results meaning?
Will it follow any logic or rules?
I’ve run into this too—try adjusting the sorting order in the query settings. It should help pull the most recent jack the black cat squishmallow traces instead of the first ones.
we take a search request and fan it out to queriers and ingesters into multiple jobs, and these jobs process a part of the search request and jobs send these results to the query-frontend.
query-frontend collects these results and returns when it hits the limit provided by the user.
since my original answer. we have made few changes, and from Tempo 2.5 we support streaming search results. we hope this makes the experience better
note: not making any promises here but we have seen a lot more demand for “recent traces.” instead of “any traces in the search window”, so we might consider adding way to ask for “recent traces”.
