My Tempo version is 2.8.1, deployed in scalable-single-binary mode with two nodes forming a memberlist cluster. Tempo is integrated with Grafana and can perform queries normally. However, when querying, if the time range includes the most recent 30 minutes, the query button in the upper right corner has a high probability of getting stuck in a loading state—that is, the spinning icon in front of the “Run query” button keeps spinning. This issue is especially noticeable when switching between the “Search” and “TraceQL” tabs. If querying data from more than 30 minutes ago, this problem does not occur.
This is the log when TraceQL is switched to Search， appear normal.
level=info ts=2025-12-15T03:02:49.997891549Z caller=tag_handlers.go:479 msg="search tag values request" tenant=single-tenant handler=SearchTagValuesV2 tag=resource.service.name query= range_seconds=0
level=info ts=2025-12-15T03:02:49.998069675Z caller=tag_handlers.go:479 msg="search tag values request" tenant=single-tenant handler=SearchTagValuesV2 tag=status query= range_seconds=0
level=debug ts=2025-12-15T03:02:49.998593167Z caller=server.go:2294 msg="GET /querier/api/v2/search/tag/status/values?start=0&end=0&q= (200) 62.035µs"
level=info ts=2025-12-15T03:02:49.99922612Z caller=tag_handlers.go:489 msg="search tag values response" tenant=single-tenant handler=SearchTagValuesV2 tag=status query= range_seconds=0 duration_seconds=0.001151255 inspected_bytes=0 request_throughput=0 error=null
level=info ts=2025-12-15T03:02:49.999291615Z caller=handler.go:135 msg="query stats" tenant=single-tenant method=GET traceID= url="/api/v2/search/tag/status/values?limit=1000" duration=1.233218ms response_size=130 status=200
level=debug ts=2025-12-15T03:02:49.99933539Z caller=server.go:2294 msg="GET /api/v2/search/tag/status/values?limit=1000 (200) 1.344595ms"
level=info ts=2025-12-15T03:02:50.001483985Z caller=tag_handlers.go:479 msg="search tag values request" tenant=single-tenant handler=SearchTagValuesV2 tag=name query= range_seconds=0
level=debug ts=2025-12-15T03:02:50.006500605Z caller=server.go:1208 method=/tempopb.Querier/SearchTagValuesV2 duration=7.663045ms msg="gRPC (success)"
level=debug ts=2025-12-15T03:02:50.006764342Z caller=server.go:2294 msg="GET /querier/api/v2/search/tag/resource.service.name/values?start=0&end=0&q= (200) 8.343445ms"
level=info ts=2025-12-15T03:02:50.007062283Z caller=tag_handlers.go:489 msg="search tag values response" tenant=single-tenant handler=SearchTagValuesV2 tag=resource.service.name query= range_seconds=0 duration_seconds=0.009168261 inspected_bytes=338621 request_throughput=3.693404888887871e+07 error=null
level=info ts=2025-12-15T03:02:50.007109937Z caller=handler.go:135 msg="query stats" tenant=single-tenant method=GET traceID= url="/api/v2/search/tag/resource.service.name/values?limit=1000" duration=9.23309ms response_size=170 status=200
level=debug ts=2025-12-15T03:02:50.007145364Z caller=server.go:2294 msg="GET /api/v2/search/tag/resource.service.name/values?limit=1000 (200) 9.325131ms"
level=debug ts=2025-12-15T03:02:50.009084888Z caller=server.go:1208 method=/tempopb.Querier/SearchTagValuesV2 duration=6.89187ms msg="gRPC (success)"
level=debug ts=2025-12-15T03:02:50.009338049Z caller=server.go:2294 msg="GET /querier/api/v2/search/tag/name/values?start=0&end=0&q= (200) 7.540522ms"
level=info ts=2025-12-15T03:02:50.009766697Z caller=tag_handlers.go:489 msg="search tag values response" tenant=single-tenant handler=SearchTagValuesV2 tag=name query= range_seconds=0 duration_seconds=0.008279539 inspected_bytes=330464 request_throughput=3.9913333338969715e+07 error=null
level=info ts=2025-12-15T03:02:50.010497494Z caller=handler.go:135 msg="query stats" tenant=single-tenant method=GET traceID= url="/api/v2/search/tag/name/values?limit=1000" duration=8.336773ms response_size=1143 status=200
level=debug ts=2025-12-15T03:02:50.010649168Z caller=server.go:2294 msg="GET /api/v2/search/tag/name/values?limit=1000 (200) 9.215241ms"
level=info ts=2025-12-15T03:02:50.048191993Z caller=tag_handlers.go:479 msg="search tag values request" tenant=single-tenant handler=SearchTagValuesV2 tag=name query={} range_seconds=0
level=debug ts=2025-12-15T03:02:50.059459425Z caller=server.go:1208 method=/tempopb.Querier/SearchTagValuesV2 duration=10.580503ms msg="gRPC (success)"
level=debug ts=2025-12-15T03:02:50.059783282Z caller=server.go:2294 msg="GET /querier/api/v2/search/tag/name/values?start=0&end=0&q=%7B%7D (200) 11.198041ms"
level=info ts=2025-12-15T03:02:50.060306411Z caller=tag_handlers.go:489 msg="search tag values response" tenant=single-tenant handler=SearchTagValuesV2 tag=name query={} range_seconds=0 duration_seconds=0.012108926 inspected_bytes=330464 request_throughput=2.7290942235504616e+07 error=null
level=info ts=2025-12-15T03:02:50.060372962Z caller=handler.go:135 msg="query stats" tenant=single-tenant method=GET traceID= url="/api/v2/search/tag/name/values?q=%7B%7D&limit=1000" duration=12.167819ms response_size=1143 status=200
level=debug ts=2025-12-15T03:02:50.060491413Z caller=server.go:2294 msg="GET /api/v2/search/tag/name/values?q=%7B%7D&limit=1000 (200) 12.353971ms"
level=info ts=2025-12-15T03:02:50.145885468Z caller=search_handlers.go:212 msg="search request" tenant=single-tenant query={} range_seconds=3600 limit=20 spans_per_spanset=3
level=debug ts=2025-12-15T03:02:50.149104758Z caller=server.go:1208 method=/tempopb.Querier/SearchRecent duration=1.848062ms msg="gRPC (success)"
level=debug ts=2025-12-15T03:02:50.149359917Z caller=server.go:2294 msg="GET /querier/api/search?start=1765765970&end=1765766570&limit=20&spss=3&q=%7B%7D (200) 2.422583ms"
level=debug ts=2025-12-15T03:02:50.15015103Z caller=server.go:1208 method=/tempopb.Querier/SearchRecent duration=2.788911ms msg="gRPC (success)"
level=debug ts=2025-12-15T03:02:50.152601924Z caller=server.go:2294 msg="GET /querier/api/search?start=1765766570&end=1765767170&limit=20&spss=3&q=%7B%7D (200) 5.454989ms"
level=debug ts=2025-12-15T03:02:50.155416483Z caller=server.go:1208 method=/tempopb.Querier/SearchRecent duration=7.806219ms msg="gRPC (success)"
level=debug ts=2025-12-15T03:02:50.155802033Z caller=server.go:2294 msg="GET /querier/api/search?start=1765767170&end=1765767770&limit=20&spss=3&q=%7B%7D (200) 8.931401ms"
level=debug ts=2025-12-15T03:02:50.311741525Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=47f31e62-2b0f-4dee-aae5-1d6d101f4491&pagesToSearch=1&size=58944&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13741 (200) 163.874593ms"
level=debug ts=2025-12-15T03:02:50.313994017Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=92c0b3a8-ecb6-44ac-82fd-cfd1d97a6eb3&pagesToSearch=1&size=44205&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13765 (200) 167.100061ms"
level=debug ts=2025-12-15T03:02:50.316432696Z caller=server.go:1539 method=/frontend.Frontend/Process duration=23m11.809154301s msg=gRPC err="context canceled"
level=debug ts=2025-12-15T03:02:50.316465677Z caller=server.go:1539 method=/frontend.Frontend/Process duration=23m11.809132864s msg=gRPC err="4 errors: context canceled; context canceled; context canceled; context canceled"
level=info ts=2025-12-15T03:02:50.316466731Z caller=search_handlers.go:193 msg="search response" tenant=single-tenant query={} range_seconds=3600 duration_seconds=0.170604714 request_throughput=2.2865136071210788e+06 total_requests=17 total_blockBytes=754165 total_blocks=14 completed_requests=2 inspected_bytes=390090 inspected_traces=0 inspected_spans=0 status_code=200 error=null
level=debug ts=2025-12-15T03:02:50.316437284Z caller=server.go:1539 method=/frontend.Frontend/Process duration=1m4.235257657s msg=gRPC err="7 errors: context canceled; context canceled; context canceled; context canceled; context canceled; context canceled; context canceled"
level=debug ts=2025-12-15T03:02:50.31652486Z caller=server.go:1539 method=/frontend.Frontend/Process duration=23m11.809428624s msg=gRPC err="context canceled"
level=info ts=2025-12-15T03:02:50.316555772Z caller=handler.go:135 msg="query stats" tenant=single-tenant method=GET traceID= url="/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770" duration=170.668575ms response_size=9802 status=200
level=debug ts=2025-12-15T03:02:50.316952177Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=f348a830-1c3c-40be-813f-4b49d8a51fa0&pagesToSearch=1&size=61096&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13741 (200) 169.432781ms"
level=debug ts=2025-12-15T03:02:50.316978869Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=38736228-f672-4869-a460-d77be1545539&pagesToSearch=1&size=47702&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13827 (200) 169.220584ms"
level=debug ts=2025-12-15T03:02:50.317018108Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=3fac42af-5536-4da2-b5a9-e2ea09f909b8&pagesToSearch=1&size=69261&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13839 (200) 169.510254ms"
level=debug ts=2025-12-15T03:02:50.317084964Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=8652e81c-66dc-4379-afa4-563dba79835e&pagesToSearch=1&size=70566&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13851 (200) 169.503048ms"
level=debug ts=2025-12-15T03:02:50.31707526Z caller=server.go:2294 msg="GET /api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770 (200) 171.268151ms"
level=debug ts=2025-12-15T03:02:50.317125609Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=89ba62be-3dce-4fc2-a185-bb4acaa5073e&pagesToSearch=1&size=45839&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13730 (200) 169.889824ms"
level=debug ts=2025-12-15T03:02:50.31737117Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=50e6fac7-cb6d-4ae5-ae7f-d548836fd5e1&pagesToSearch=1&size=41278&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13687 (200) 170.009867ms"
level=debug ts=2025-12-15T03:02:50.317432157Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=2ad1f323-9e20-4b85-8d9c-3365a75c3c59&pagesToSearch=1&size=42616&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13729 (200) 169.683512ms"
level=debug ts=2025-12-15T03:02:50.317451771Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=c232c327-9c82-4951-bf04-2c2db6049303&pagesToSearch=1&size=45655&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13840 (200) 169.551153ms"
level=debug ts=2025-12-15T03:02:50.317469725Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=64ab8a1e-efcc-44b2-b7b6-cdc311dd1114&pagesToSearch=1&size=63902&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13742 (200) 169.557917ms"
level=debug ts=2025-12-15T03:02:50.317488064Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=ca3d67db-537d-443b-974e-3f165cb417fb&pagesToSearch=1&size=68447&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13850 (200) 170.148645ms"
level=debug ts=2025-12-15T03:02:50.317509511Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=e849356e-2afc-4778-a98f-7cd56a55d278&pagesToSearch=1&size=47402&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13731 (200) 169.680478ms"
level=debug ts=2025-12-15T03:02:50.317767279Z caller=server.go:2294 msg="GET /querier/api/search?q=%7B%7D&limit=20&spss=3&start=1765764170&end=1765767770&blockID=e2178e4b-88fd-4d28-a30b-c5609b0e1d3e&pagesToSearch=1&size=47252&startPage=0&encoding=none&indexPageSize=0&totalRecords=1&dataEncoding=&version=vParquet3&footerSize=13823 (200) 170.237471ms"
level=info ts=2025-12-15T03:02:50.477744791Z caller=tag_handlers.go:479 msg="search tag values request" tenant=single-tenant handler=SearchTagValuesV2 tag=resource.service.name query={} range_seconds=0
level=info ts=2025-12-15T03:02:50.481296639Z caller=tag_handlers.go:479 msg="search tag values request" tenant=single-tenant handler=SearchTagValuesV2 tag=status query={} range_seconds=0
level=debug ts=2025-12-15T03:02:50.481758371Z caller=server.go:2294 msg="GET /querier/api/v2/search/tag/status/values?start=0&end=0&q=%7B%7D (200) 73.747µs"
level=info ts=2025-12-15T03:02:50.482044837Z caller=tag_handlers.go:489 msg="search tag values response" tenant=single-tenant handler=SearchTagValuesV2 tag=status query={} range_seconds=0 duration_seconds=0.000744258 inspected_bytes=0 request_throughput=0 error=null
level=info ts=2025-12-15T03:02:50.482097621Z caller=handler.go:135 msg="query stats" tenant=single-tenant method=GET traceID= url="/api/v2/search/tag/status/values?q=%7B%7D&limit=1000" duration=796.171µs response_size=130 status=200
level=debug ts=2025-12-15T03:02:50.482131967Z caller=server.go:2294 msg="GET /api/v2/search/tag/status/values?q=%7B%7D&limit=1000 (200) 885.577µs"
level=debug ts=2025-12-15T03:02:50.486121398Z caller=server.go:1208 method=/tempopb.Querier/SearchTagValuesV2 duration=7.569001ms msg="gRPC (success)"
level=debug ts=2025-12-15T03:02:50.486359042Z caller=server.go:2294 msg="GET /querier/api/v2/search/tag/resource.service.name/values?start=0&end=0&q=%7B%7D (200) 8.151872ms"
level=info ts=2025-12-15T03:02:50.486643531Z caller=tag_handlers.go:489 msg="search tag values response" tenant=single-tenant handler=SearchTagValuesV2 tag=resource.service.name query={} range_seconds=0 duration_seconds=0.008895658 inspected_bytes=338621 request_throughput=3.806587438500895e+07 error=null
level=info ts=2025-12-15T03:02:50.486706863Z caller=handler.go:135 msg="query stats" tenant=single-tenant method=GET traceID= url="/api/v2/search/tag/resource.service.name/values?q=%7B%7D&limit=1000" duration=8.95971ms response_size=170 status=200
level=debug ts=2025-12-15T03:02:50.486760831Z caller=server.go:2294 msg="GET /api/v2/search/tag/resource.service.name/values?q=%7B%7D&limit=1000 (200) 9.110633ms"
My configuration file is as follows:
target: scalable-single-binary
server:
http_listen_port: 3200
grpc_listen_port: 9095
grpc_server_max_recv_msg_size: 107374182400
grpc_server_max_send_msg_size: 107374182400
http_server_read_timeout: 777s
http_server_write_timeout: 600s
log_level: debug
distributor:
ring:
kvstore:
store: memberlist
instance_addr: ${HOSTNAME}
instance_port: 9095
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
retry_after_on_resource_exhausted: 5s
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 2
heartbeat_timeout: 1m
address: ${HOSTNAME}
port: 9095
heartbeat_period: 2s
heartbeat_timeout: 3s
observe_period: 5s
join_after: 5s
final_sleep: 5s
min_ready_duration: 3s
flush_all_on_shutdown: true
flush_check_period: 60s
complete_block_timeout: 600s
max_block_bytes: 5242880
trace_idle_period: 60s
max_block_duration: 600s
query_frontend:
search:
default_result_limit: 100
max_duration: 720h
duration_slo: 5s
throughput_bytes_slo: 1073741824
metadata_slo:
duration_slo: 5s
throughput_bytes_slo: 1073741824
trace_by_id:
duration_slo: 5s
metrics:
max_duration: 720h
concurrent_jobs: 100
multi_tenant_queries_enabled: false
metrics_generator:
ring:
kvstore:
store: memberlist
instance_addr: ${HOSTNAME}
instance_port: 9095
processor:
service_graphs:
dimensions: [net.host.name, net.host.ip, net.peer.name, net.peer.ip, resource.container.id, http.method, http.route, host.name]
enable_client_server_prefix: true
enable_messaging_system_latency_histogram: true
peer_attributes: []
enable_virtual_node_label: true
histogram_buckets: [1，2，3]
span_metrics:
dimensions: [server.address, client.address, network.peer.address, ip.address, protocol, host.name]
intrinsic_dimensions:
status_message: true
enable_target_info: true
histogram_buckets: [1，2，3]
local_blocks:
block:
version: vParquet3
filter_server_spans: true
flush_to_storage: true
storage:
path: /export/tempo_data/metrics_storage
remote_write:
- url: http://<IP>:9090/api/v1/write
send_exemplars: true
traces_storage:
path: /export/tempo_data/metrics_traces_storage
version: vParquet3
querier:
frontend_worker:
frontend_address: ${HOSTNAME}:9095
grpc_client_config:
max_recv_msg_size: 10737418240
max_send_msg_size: 10737418240
search:
query_timeout: 600s
max_concurrent_queries: 30
compactor:
ring:
kvstore:
store: memberlist
instance_addr: ${HOSTNAME}
instance_port: 9095
compaction:
compaction_cycle: 120s
compaction_window: 300s
block_retention: 72h
compacted_block_retention: 1h
retention_concurrency: 10
v2_out_buffer_bytes: 20971520
max_block_bytes: 5242880
storage:
trace:
backend: azure
azure:
storage_account_name: <ACCOUNT>
container_name: tempo
blocklist_poll: 5m
blocklist_poll_tolerate_consecutive_errors: 10
blocklist_poll_tolerate_tenant_failures: 10
wal:
path: /export/tempo_data/storage_wal
version: vParquet3
block:
version: vParquet3
memberlist:
randomize_node_name: false
retransmit_factor: 2
gossip_nodes: 2
message_history_buffer_bytes: 1000
gossip_to_dead_nodes_time: 3s
pull_push_interval: 3s
leave_timeout: 3s
bind_addr: ["0.0.0.0"]
bind_port: 7946
abort_if_cluster_join_fails: false
join_members:
- <IP1>:7946
- <IP2>:7946
overrides:
defaults:
ingestion:
rate_strategy: global
max_traces_per_user: 1000000000000
metrics_generator:
processors:
- local-blocks
- service-graphs
- span-metrics
No matter how these parameters are adjusted, the result remains the same.
query_frontend:
search:
default_result_limit: 100
max_result_limit: 0
max_duration: 720h0m0s
query_backend_after: 15m0s
query_ingesters_until: 30m0s
How to resolve the issue of the query button being stuck in a loading state?