Tempo in Grafana - Search limited to N rows returns first N rows, rather than last N rows


Wasn’t sure if this is a “Tempo” issue or a “Grafana” issue, but… not sure if we’re doing tracing “wrong” but when I “Explore” my Tempo datasources it limits to 20 traces by default, but it shows the first 20 traces, rather than the most recent 20 traces.

This is quite confusing, and worse, I can’t figure out how to change it?

Hi :wave: @colemickens, you can change the time range to search the recent data, and increase the default limits to get more then 20 results.

see the red highlighted areas in this screecap:

Hm, let me try to describe my problem again. I am selecting a recent time window, that’s not quite the problem.

The problem is that when Grafana shows N results, it shows the first N results, rather than the last N results.

These screenshots demonstrate the issue. Note, no new traces were added between screenshots.

Notice how when I increase the limit, I see newer results. This is contrary to every other system I’ve used that shows most-recent first, and limits results. It was very confusing to be expecting to see the 10 most recent rows and “not see new traces”. They were there, just not being shown to me.

This feels unintuitive to the point that I’m wondering if I’ve misconfigured something. I don’t see anything that allows me to control sort.

we return the first N (N=limit) results that match the query in the selected time range, search results are not sorted.

you can use narrow down the search using traceql to filter down to sepefic spans, for example: { span.http.status_code >= 200 && span.http.status_code < 300 }

search results are not sorted.

would it hard to sort them, implementation-wise? or return the last N results?

It feels like in order for me to deliver this to other folks, I need to write an explanation of how to use this:

  • make sure to set the limit to 10000 so you actually see the most recent traces
  • this shows first matching rows in timespan, not most recent ones

Because otherwise as soon as they go over 20 traces they just stop seeing new traces with no indication as to why.

would it hard to sort them, implementation-wise? or return the last N results?

yes, because to sort them we need to fetch all the results (i.e do the exhaustive search) that match your query, and for a broad query like { } will match all the traces…

for any big cluster(assuming terabytes of data in the query time range), fetch all the data, sorting and then retuning 20 results is not practical.

our recommendations:

  • narrow down the query using filters
  • narrow down the timerange

That’s pretty annoying

What does the first N (N=limit) results meaning?
Will it follow any logic or rules?