Hello. I’ve been racking my brain over this for a couple of weeks. Traces are coming into the system, and I can use the Search view in Grafana to view them. But, if I go back somewhere between 30m-60m and further, Traces stop returning.
I’m running the local, single binary docker version. Restarting the docker container for Tempo is enough to fix it for the first few queries. But, after querying a few times, changing the timeframe to -1h, the moving through a few different 1h windows, traces stop coming except for the last 30-60m, which will ALWAYS work.
I’ve attached a screenshot showing a search result, restarting the docker container, then hitting Run Query again. You can see that the results are now returning, with no changes to the query, the timestamp, or even leaving the screen. Just a quick docker container restart.
Also, the logs say that nothing is getting searched. This log shows up every time I run a query and get empty results:
level=info ts=2024-06-25T06:27:10.982441481Z caller=searchsharding.go:254 msg=“sharded search query request stats and SearchMetrics” tenant=single-tenant query=“end=1719291890&limit=5&q={}&spss=3&start=1719282059” duration_seconds=47.859µs request_throughput=0 total_requests=0 started_requests=0 cancelled_requests=0 finished_requests=0 totalBlocks=0 inspectedBytes=0 inspectedTraces=0 totalBlockBytes=0
Finally, I’m not sure if it’s related, but the Trace Limit is only respected most of the time, not all of the time. So, setting the Limit to 5 Traces might still mean I get 50 traces back. But, I’m not sure if this is related nor is it high priority.
Thoughts?
