Installed Tempo with S3 backend, S3 is empty


I installed Tempo in K8S via

helm upgrade --install --values grafana-tempo-values.yaml tempo grafana/tempo

with grafana-tempo-values.yaml containing:

backend: s3
bucket: …
endpoint: …
access_key: …
secret_key: …
insecure: false

Its log ends with:

level=info ts=2021-09-24T12:22:28.305099313Z caller=app.go:251 msg=“Tempo started”
level=info ts=2021-09-24T12:23:28.268449016Z caller=compactor.go:103 msg=“enabling compaction”
level=info ts=2021-09-24T12:23:28.268725758Z caller=tempodb.go:359 msg=“compaction and retention enabled.”

Now when I update working Jaeger setup so that agent does not connect to Jaeger collector but to this Tempo service, the Jaeger agent log shows:

{“level”:“info”,“ts”:1632487767.9997396,“caller”:“grpc/builder.go:120”,“msg”:“Agent collector connection state change”,“dialTarget”:“dns:///tempo.jztest:14250”,“status”:“READY”}

But nothing appears in the S3 bucket. How do I verify that the setup is correct? There is nothing regarding S3 in the Tempo log…

Are you able to query those traces in Tempo?

And how long has Tempo been running when you checked S3? We keep traces in the ingesters for a while before we flush them to the backend. This is because the ingesters will batch multiple traces into a single block. How quickly blocks are flushed to the backend depends on the amount of traces you are ingesting, but with the default config this can take up to 1h.

I don’t know how without configuring other things. That requires several components configured and interconnected properly.

So I want to go step by step and setup Tempo first before trying to connect Grafana to it (possibly even without trying to connect Jaeger agent to it but since Jaeger agent seems happy I don’t have to worry now). It is always easier to debug the setup of one software than trying to get 2 or 3 programs correct at once.

It is running for several days now and the last thing in the Tempo’s log is still

level=info ts=2021-09-24T12:23:28.268725758Z caller=tempodb.go:359 msg=“compaction and retention enabled.”

S3 still empty (I would expect that it would create some directory structure or at least some empty file to verify the setup).

Yeah, if you’ve been ingesting traces for multiple days now we should see some data in the bucket.

I’d recommend following the steps in this doc to troubleshoot components one-by-one starting from the write path: Unable to Find Traces | Grafana Labs

tempo_distributor_spans_received_total nor tempo_ingester_traces_created_total exist.
I added

    log_received_traces: true

under tempo: section in grafana-tempo-values.yaml but I don’t know if it was picked up. Should it be visible in some env variable?

There is still nothing in Tempo’s log… (I only update option of the Jaeger agent and when reset to its default value (jaeger-collector) I can see spans in Jaeger UI, so I don’t think the problem is on Jaeger’s side).

If these metrics don’t exist and you are scraping your pods correctly then you are not receiving traces. Check your jaeger metrics on the agent to see if it thinks it’s successfully sending spans.

1 Like

BINGO! Shared development environment is the way to get the blood pressure higher :slight_smile:

Thank you!