Inconsistent link between traces and logs

Hello support,

I’m doing some tests with Tempo and Loki and I noticed that I have logs that are associated with a particular span (when you click on “logs” on a span). But some of them are not associated at all. I’m using the opentelemetry implementation in Golang for the traces, and slog for the logs, so there is no context correlation and I’m not adding any trace id. It’s just “vanilla” configuration.
How link between traces and logs is done in Grafana and how can I make it better?

Thank you,

This sounds like the answer is very likely “it depends”. Is there an example you can give of where it’s not associated, or any other details about how you’ve instrumented the go app?

Of course I can share an example! Here are the full logs I got during a particular timeframe (using explore with the correct datasource)


I have my trace here:

As you can see, the timeframe is similar. So we should get all the logs I showed in my first picture.
If I click to see the logs of the first span:

If I click to see the logs of the second span:

The last log in my first picture is nowhere. If I look closer, where this log is emitted, it’s in an http handler:


func TraceRequest(next http.Handler) http.Handler {
	return http.HandlerFunc(
		func(w http.ResponseWriter, r *http.Request) {
			span := trace.SpanFromContext(r.Context())
			defer span.End()
			var bodyBytes []byte
			if r.Body != nil {
				bodyBytes, _ = io.ReadAll(r.Body)
			}

			// Restore the body so it can be read again
			r.Body = io.NopCloser(bytes.NewBuffer(bodyBytes))
			span.AddEvent("", trace.WithAttributes(attribute.String("request", string(bodyBytes))))

			crw := &customResponseWrite{
				writer: w,
				body:   &bytes.Buffer{},
			}
			next.ServeHTTP(crw, r)
			//Get response body
			span.AddEvent("", trace.WithAttributes(attribute.String("response", crw.body.String())))
			span.SetAttributes(attribute.Int("http.response_code", crw.responseCode))
			slog.Info("", slog.String("request", r.URL.String()), slog.String("method", r.Method), slog.Int("response.code", crw.responseCode))
		})

And used like this:

handler := otelhttp.WithRouteTag(pattern, TraceRequest(http.HandlerFunc(controller)))

I’m able to use the span in the http handler, so I think my span get closed before I emit the log.

Are you expecting correlation based on timeframe?

I don’t know where I should expect correlation, since I didn’t find any documentation page stating how the correlation is done and here the correlation cannot be done inside the code (slog is not part of opentelemetry SDK).

Hello?

Yes sure, I will check myself the documentation and find a way to adapt it to my issue. I will try to keep this thread up to date to say if this solution was able to help.