How to create a Grafana Dashboard for my OpenTelemetry instruments > Collector > Tempo setup

Grafana
, ,
1

I am new to all this architecture, and I am a bit lost.

I have integrated all these instruments in my Rails application:

gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-mysql2"
gem "opentelemetry-instrumentation-net_http"
gem "opentelemetry-instrumentation-rack"
gem "opentelemetry-instrumentation-rails"

I am sending the metrics to an instance of OpenTelemetryCollector, and from there to a Tempo instance.

In Grafana, I have configured the Tempo datasource.

I can see the Traces in the Grafana > Explorer:

Screenshot 2024-05-27 at 20.22.31
Screenshot 2024-05-27 at 20.22.311699×1150 209 KB

But I don’t know how to build a proper dashboard with it. I have tried to import a dashboard like this. But the data source is Prometheus, not Loki. The are only 2 dashboards for Tempo on this database. And they don’t look like the one I want.

I would like to see the requests, the statuses, the request_time, … as in the OpenTelemetry APM Dashboard

Any help?

If my architecture is wrong, I am also willing to change it.

2

See doc for OpenTelemetry APM Dashboard

  1. OpenTelemetry Collector: Receive traces and process them to metrics with Span Metrics Connector
  2. Prometheus: Scrape metrics from OpenTelemetry Collector

You are missing these steps (older collector versions have had spanmetricsprocessor, not connector) - this connector (processor) aggregates Request, Error and Duration (R.E.D) metrics from span data - so you will generate METRICS from TRACES and you will save these generated METRICS into metric storage - Prometheus.

3

Oh, that has sense. Very bad on me, I did not read the documentation of the dashboard. I will do it from now on.

@jangaraj does it means that I don’t need Tempo in my setup?

4

If you need only RED metrics, then you don’t need Tempo (traces).

But what will you do when your RED metrics show increased error rate or latency?
That’s exactly a showtime for traces: you will filter traces with error, increased latency, you’ll inspect them and you will get a root cause (probably).

5

Thanks, I definetely need both. I’ll configure the Collector to export metrics to Prometheus and at the same time keeping sending them to Tempo.