Inconsistent link between traces and logs

Grafana Tempo
1

Hello support,

I’m doing some tests with Tempo and Loki and I noticed that I have logs that are associated with a particular span (when you click on “logs” on a span). But some of them are not associated at all. I’m using the opentelemetry implementation in Golang for the traces, and slog for the logs, so there is no context correlation and I’m not adding any trace id. It’s just “vanilla” configuration.
How link between traces and logs is done in Grafana and how can I make it better?

Thank you,

2

This sounds like the answer is very likely “it depends”. Is there an example you can give of where it’s not associated, or any other details about how you’ve instrumented the go app?

3

Of course I can share an example! Here are the full logs I got during a particular timeframe (using explore with the correct datasource)

image
image1138×110 7.54 KB

I have my trace here:
image
image1883×610 66.6 KB

As you can see, the timeframe is similar. So we should get all the logs I showed in my first picture.
If I click to see the logs of the first span:
image
image954×177 27.5 KB

If I click to see the logs of the second span:
image
image989×140 17.5 KB

The last log in my first picture is nowhere. If I look closer, where this log is emitted, it’s in an http handler:


func TraceRequest(next http.Handler) http.Handler {
	return http.HandlerFunc(
		func(w http.ResponseWriter, r *http.Request) {
			span := trace.SpanFromContext(r.Context())
			defer span.End()
			var bodyBytes []byte
			if r.Body != nil {
				bodyBytes, _ = io.ReadAll(r.Body)
			}

			// Restore the body so it can be read again
			r.Body = io.NopCloser(bytes.NewBuffer(bodyBytes))
			span.AddEvent("", trace.WithAttributes(attribute.String("request", string(bodyBytes))))

			crw := &customResponseWrite{
				writer: w,
				body:   &bytes.Buffer{},
			}
			next.ServeHTTP(crw, r)
			//Get response body
			span.AddEvent("", trace.WithAttributes(attribute.String("response", crw.body.String())))
			span.SetAttributes(attribute.Int("http.response_code", crw.responseCode))
			slog.Info("", slog.String("request", r.URL.String()), slog.String("method", r.Method), slog.Int("response.code", crw.responseCode))
		})

And used like this:

handler := otelhttp.WithRouteTag(pattern, TraceRequest(http.HandlerFunc(controller)))

I’m able to use the span in the http handler, so I think my span get closed before I emit the log.

4

Are you expecting correlation based on timeframe?

5

I don’t know where I should expect correlation, since I didn’t find any documentation page stating how the correlation is done and here the correlation cannot be done inside the code (slog is not part of opentelemetry SDK).

6
7

Hello?

Yes sure, I will check myself the documentation and find a way to adapt it to my issue. I will try to keep this thread up to date to say if this solution was able to help.

8

This was the solution after I had to manage myself other issues related to Grafana Cloud not addressed in this topic nor in the documentation.

9

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.