Grafana Tempo Traces Setup Help

Grafana Tempo
1

Hi Team,

I did a setup of grafana tempo as per the documentation using helm chart.

What is the issue then?
Unable to get traces. Loki does not show traces for tempo

What has been done?

  1. Loki has been configured
  2. Tempo has been configured
  3. Loki has derived fileds added
  4. Tempo has loki search integrated.
  5. Logs are being sent to loki using promtail

Logs are also coming fine on loki. However the traces/traceID is not showing up and keeps on throwing

failed to get trace with id: 1654748158356114788 Status: 404 Not Found Body: trace not found

Regex has also been added on loki as per documentation (?:traceID|trace_id|tid)=(\w+)

2

Hi!

You will also need to instrument your applications with tracing libraries. You can take a look at OpenTelemetry for those.

If your applications are already instrumented, can you check basic Tempo metrics such as tempo_distributor_bytes_received_total to see if traces are reaching it? Does the instrumentation throw any errors when forwarding the traces?

3

HI mario.
So, until and unless the application is instrumented, there is no way I will get the traceID is it?
Also, will promtail work for that or I will need grafana agent for that?

4

Hi!

Yes, that’s correct. You will need your applications instrumented with a tracing library to create traces. Otherwise, you won’t have data to visualise, even if you have a trace ID in your logs.

Promtail is a tool for log collection. It does not work with trace data. To forward traces to Tempo you have two options: 1) Send traces directly to Tempo 2) Use a tracing pipeline (e.g. Grafana Agent or OTel collector) to offload traces from your application and forward them to Tempo (Recommended).

You have some resources on how to get started with distributed traces here.

5

@mariorodriguez thank you for all the clarification.

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.