Working with Grafana Tempo locally with dockerimages

Grafana Tempo
#1

Hi,
I have spent days now trying to get a proof of concept with Grafana Tempo and OpenTelemetry running with my .net core application and I have tried everything now but with no progress.

I have followed this guide (Getting started | Grafana Labs) and my docker images is up and running as they should. However it seems that my problem is that no traces whatsoever are exported by my Jaeger exporter to my tempo docker image running on port 6831 as in your guide. As you can see below is my setup of opentelemetry and the AddJaegerExporter() part is using localhost and 6831 as default values so I expected to be all set with that. However, the only thing my tempo docker image logs are showing is startup info and no incoming traces of any kind.

services.AddOpenTelemetryTracing(builder => builder
.SetResourceBuilder(ResourceBuilder.CreateDefault()
.AddService(“Authentication API”))
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddJaegerExporter()

If I try a search in the querycontainer for a traceId I obviously get nothing since nothing seems to be transferred to the tempo container. However, I get an error in the logs saying:
{“level”:“error”,“ts”:1611087947.883616,“caller”:“zap/logger.go:33”,“msg”:“failed to flush Jaeger spans to server: write udp 127.0.0.1:37973->127.0.0.1:6831: write: connection refused”,“stacktrace”:"github.com/uber/jaeger-client-go/log/zap.

Tempo isn’t flushing traces when I search is it? Where 127.0.0.1:37973 comes from I have no Idea. My application is configured to send traces to tempo:6831. What have I missed?

#2

Where 127.0.0.1:37973 comes from I have no Idea.

This is the local port your application is using to try and send data to Tempo.

failed to flush Jaeger spans to server: write udp 127.0.0.1:37973->127.0.0.1:6831: write: connection refused

This seems to indicate that your application is not properly configured to send its trace data to Tempo. it is trying to connect to 127.0.0.1 (its own IP). I do not know a lot about configuring the OTEL .net client, but I’d review those documents.

1 Like
#3

Hi @andreashellquist, thanks for reaching out, and sorry you are having so much trouble with setting up a PoC with Tempo.

Some thoughts:

  • Is your application also running in docker? What is the address of tempo being used at the application?
  • By default most SDKs use a sampler which does probabilistic sampling. Can you try setting sampling to a constant of 1 and try again?

We also have a troubleshooting guide at Troubleshooting | Grafana Labs if that helps.

{“level”:“error”,“ts”:1611087947.883616,“caller”:“zap/logger.go:33”,“msg”:“failed to flush Jaeger spans to server: write udp 127.0.0.1:37973->127.0.0.1:6831: write: connection

This log line from the query container is OK. The Query container traces itself and this is reporting that it is unable to send spans anywhere… it should not affect the querying of your application traces.

1 Like
#4

Hi Annanayagarwal,

Yes my application is also running in docker. The address of tempo being used is localhost:6831 as default. Setting samling to 1? Not really sure what you mean there.

Thanks for your reply.

image001.png

1 Like
#5

If your application is running inside docker the correct address would be tempo:3100. (localhost would refer to the same container which would be wrong).

Regarding sampling, the OpenTelemetry .NET SDK has the following sampler options - opentelemetry-dotnet/README.md at 97ed5f54d492be5ac5146fd8bd3679a08136c635 · open-telemetry/opentelemetry-dotnet · GitHub. It could be useful to check which one is the default, and trying the AlwaysOnSampler to set up the PoC.

1 Like
#6

Hmm tempo:3100 is not valid to use for jaeger exporter configuration in application though. And default port is 6831. But I guess this is where I get it all wrong somehow. Do I need a collector/agent container as well besides the tempo container?

image001.png

#7

3100 is the HTTP port in tempo. You are right it should be tempo:6831 which is jaeger compact thrift.

Do I need a collector/agent container as well besides the tempo container?

You can push directly to Tempo. Just make sure you have the right receiver enabled. Here is an example enabling all receivers: tempo/tempo-local.yaml at master · grafana/tempo · GitHub