Problem Grafana Tempo Time out on long time range (Dashboard Mode Only)

Grafana Tempo
1

Hi.
I have Grafana Tempo running correctly.

The problem:
In a Grafana Dashboard, I have multiple filters, but they are not always used at the same time.

When I run a query over a long time range (7 days or more) with multiple filters enabled, my Tempo server reaches 99.9% CPU usage, and the query times out with an error.

However, when I run the “same” query in Explore mode, using only the specific filters I need, it works quickly and returns results within seconds.

This suggests that the dashboard query might be generating an inefficient or overly complex TraceQL request, leading to high CPU usage and timeouts.

Can anyone help me understand this ?
Thanks.

2

Hello, up.
Does anyone have an idea ?

3

We recommend following search tuning guide to tune your tempo install, and using enabling caching to improve the query performance.

I also recommend following the sizing guide to properly size your cluster for the data volume you have, undersized clusters will not be able to serve the big queries.

you can also control the max time range for which users are allowed to search by setting query_frontend.search.max_duration and query_frontend.metrics.max_duration values in the tempo config.