Grafana Tempo is not showing traces

Grafana Tempo
, , , ,
1

Hi Grafana community,

I’m Harsh and I need help setting up traces at a central location. I’ve deployed Grafana and Tempo at a central cluster and I’m trying to send trace data from different clusters where instrumentation with OpenTelemetry has been done. I’ve tried using Grafana Agent, but my central Tempo instance is not receiving the traces. Interestingly, I am able to send logs to a central Loki instance from the same clusters. The error I’m encountering is “404 trace body not found.”

Error message from tempo

image
image1338×444 73.9 KB

Any help would be greatly appreciated.

Thanks,
Harsh

2

@harshkakra

The error I’m encountering is “404 trace body not found.
This is from the Grafana UI to Tempo, that is different then the ingestion of all trace data to Tempo.

Can you check the logging of Grafana and can you check the logging Tempo to see if you see messages / errors?

For example if you do a query for data from Grafana UI to Tempo, you will see a request coming in on Tempo. If that is not the case, then the datasource configuration in Grafana is not correct.

3

Hi @cbos,

Thank you for your response. I’m currently deploying Tempo in a distributed setup, and it’s successfully attached to Grafana. However, I’m encountering issues with the agent. I’m trying to send trace data from a remote cluster to Tempo, but I’m unable to do so. Could you please assist me with the configuration?

Thanks
Harsh

4

Hi Harsh,

I do have experience with the setup of OpenTelemetry Collector or just applications for sending data to Tempo (see GitHub - cbos/observability-toolkit: Observability stack for local development and troubleshooting toolkit). I have done some things with Grafana Agent in the past, but not related to Traces.
But what I see in the configuration is that is similar to OpenTelemetry Collector (traces_config | Grafana Agent documentation).

But as you mentioned ‘distributed Tempo’ (Tempo architecture | Grafana Tempo documentation), can it be that the frontend query or querier is not correctly configured and not able to load the data?

5

Hi cbos,

I’m facing an issue with sending traces from the OpenTelemetry Collector to the Tempo instance. The collector is unable to connect to Tempo, and I’m not receiving any traces. I tried exposing the IP to test the connection, but it’s not working. However, Tempo seems to be working fine.

Could you please share the configuration for both Tempo and the OpenTelemetry Collector? I’m deploying them using Helm charts, so please also suggest the chart.

Your help would be greatly appreciated.

Thanks,
Harsh

6

Hi cbos,

Thanks for the help. Now i’m able to see the traces in tempo.
I configured it with OTLP collector and it’s working for me now.

Thanks!