Hello everyone, sorry for the bad english.
I’m trying to configure Tempo as a Grafana data source.
While it says the configuration is successful when I try to query something in “explore” menu it gives me this error.
failed to get trace with id: Status: 404 Not Found Body: 404 page not found
this is my tempo-local.yaml file
server:
http_listen_port: 3200
# Distributor config block
distributor:
# receiver configuration for different protocols
# config is passed down to opentelemetry receivers
# for a production deployment you should only enable the receivers you need!
receivers:
otlp:
protocols:
grpc:
http:
jaeger:
protocols:
thrift_http:
grpc:
thrift_binary:
thrift_compact:
ingester:
trace_idle_period: 10s # the length of time after a trace has not received spans to consider it complete and flush it
max_block_bytes: 1_000_000 # cut the head block when it hits this size or ...
max_block_duration: 5m # this much time passes
compactor:
compaction:
compaction_window: 1h # blocks in this time window will be compacted together
max_block_bytes: 100_000_000 # maximum size of compacted blocks
block_retention: 1h
compacted_block_retention: 10m
storage:
trace:
backend: local # backend configuration to use
block:
bloom_filter_false_positive: .05 # bloom filter false positive rate. lower values create larger filters but fewer false positives
index_downsample_bytes: 1000 # number of bytes per index record
encoding: zstd # block encoding/compression. options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd, s2
wal:
path: /tmp/tempo/wal # where to store the the wal locally
encoding: snappy # wal encoding/compression. options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd, s2
local:
path: /tmp/tempo/blocks
pool:
max_workers: 100 # worker pool determines the number of parallel requests to the object store backend
queue_depth: 10000
when I access the page http://my-tempo-address:3200 it gives me this error:
404 page not found
so… what am I doing wrong?
Hi @dvmorais!
Can you share the logs from Tempo? We may find something there that points us to the problem.
Does search work? Does it show any traces?
level=info ts=2022-09-28T15:16:09.89874147Z caller=main.go:200 msg="initialising OpenTracing tracer"
level=info ts=2022-09-28T15:16:09.901877356Z caller=main.go:115 msg="Starting Tempo" version="(version=, branch=main, revision=752d9a90)"
level=info ts=2022-09-28T15:16:09.902855612Z caller=server.go:306 http=[::]:3200 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-09-28T15:16:09.907157297Z caller=frontend.go:43 msg="creating middleware in query frontend"
level=warn ts=2022-09-28T15:16:09.908181748Z caller=modules.go:181 msg="Worker address is empty in single binary mode. Attempting automatic worker configuration. If queries are unresponsive consider configuring the worker explicitly." address=127.0.0.1:9095
level=info ts=2022-09-28T15:16:09.908439014Z caller=worker.go:103 msg="Starting querier worker connected to query-frontend" frontend=127.0.0.1:9095
ts=2022-09-28T15:16:09Z level=info msg="OTel Shim Logger Initialized" component=tempo
level=info ts=2022-09-28T15:16:09.911900643Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-09-28T15:16:09.912086176Z caller=module_service.go:82 msg=initialising module=store
level=info ts=2022-09-28T15:16:09.912516908Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2022-09-28T15:16:09.912625102Z caller=module_service.go:82 msg=initialising module=usage-report
level=info ts=2022-09-28T15:16:09.912669757Z caller=module_service.go:82 msg=initialising module=ring
level=info ts=2022-09-28T15:16:09.912865341Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"
level=info ts=2022-09-28T15:16:09.913086245Z caller=client.go:255 msg="value is nil" key=collectors/ring index=1
level=info ts=2022-09-28T15:16:09.912612235Z caller=module_service.go:82 msg=initialising module=overrides
level=info ts=2022-09-28T15:16:09.914476236Z caller=module_service.go:82 msg=initialising module=compactor
level=info ts=2022-09-28T15:16:09.91458421Z caller=tempodb.go:417 msg="polling enabled" interval=5m0s concurrency=50
level=info ts=2022-09-28T15:16:09.914865633Z caller=compactor.go:160 msg="enabling compaction"
level=info ts=2022-09-28T15:16:09.914943809Z caller=tempodb.go:391 msg="compaction and retention enabled."
level=info ts=2022-09-28T15:16:09.91505372Z caller=module_service.go:82 msg=initialising module=querier
level=info ts=2022-09-28T15:16:09.915270974Z caller=module_service.go:82 msg=initialising module=query-frontend
level=info ts=2022-09-28T15:16:09.915889764Z caller=module_service.go:82 msg=initialising module=distributor
ts=2022-09-28T15:16:09Z level=info msg="No sampling strategies provided or URL is unavailable, using defaults" component=tempo
level=info ts=2022-09-28T15:16:09.917257406Z caller=module_service.go:82 msg=initialising module=ingester
level=info ts=2022-09-28T15:16:09.917520569Z caller=ingester.go:328 msg="beginning wal replay"
ts=2022-09-28T15:16:09Z level=info msg="Starting GRPC server on endpoint 0.0.0.0:4317" component=tempo
ts=2022-09-28T15:16:09Z level=info msg="Starting HTTP server on endpoint 0.0.0.0:4318" component=tempo
level=warn ts=2022-09-28T15:16:09.917889783Z caller=rescan_blocks.go:23 msg="failed to open search wal directory" err="open /tmp/tempo/wal/search: no such file or directory"
level=info ts=2022-09-28T15:16:09.917966242Z caller=ingester.go:413 msg="wal replay complete"
level=info ts=2022-09-28T15:16:09.918241699Z caller=ingester.go:427 msg="reloading local blocks" tenants=0
level=info ts=2022-09-28T15:16:09.918493024Z caller=worker.go:179 msg="adding connection" addr=127.0.0.1:9095
level=info ts=2022-09-28T15:16:09.918702962Z caller=lifecycler.go:547 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2022-09-28T15:16:09.918891672Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=ingester
level=info ts=2022-09-28T15:16:09.919361569Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
level=info ts=2022-09-28T15:16:09.922098175Z caller=app.go:195 msg="Tempo started"
Seems like everything is working but the errors described above keep showing.
search in…? Grafana is not showing me any traces but if I access the /metrics it shows the tempo metrics as usual.
A 404 when accessing http://tempo:3200/ is expected. Have you been able to send traces? With a trace ID in hand, you might try accessing the API directly using that trace to confirm. /api/traces/<traceID> from the the docs might be useful. The logs look sane to me.
I… discovered what I was doing wrong and I feel a bit dumb for that…
I was pointing to the wrong port (9095, not 4317 that is the grpc default port).
And I discovered that I needed to enable search in grafana using this flag on tempo.yaml:
# Enable search functionality
search_enabled: true
anyway… thanks for the help guys.
