Using Loki log lables to link to Tempo traceids

[brentdreyer]

Hey,

I am looking to link my Loki logs to my Tempo trace from opentelemetry.

From my understanding the only way to do that is use the log message text and the Derived fields regex to create a detected label.

However the Logging Instrumentor (opentelemetry.instrumentation.logging) adds the fields as extra content to the log with out using the message field. Which is detected and correctly displayed by Loki, with out the need of extra configuration.

Is a way to use these already detected log labels to link the log to the corresponding trace?

[maxenceb]

Hi,

I’m looking to do the same thing as you, which is creating a link from a Loki field to a Tempo trace.

In my case, I do have the traceId in my logs which are formatted in logfmt, but I’m trying to create a nice looking dashboard for my application logs, in which I display only the essential informations which are the type of log and the actual log message.

My dashboard panels look like this, so I only display the info and msg fields. And you can see the other fields by opening a specific log line.

And my logs are parsed using the Logfmt operation as you can this in the next screenshot

The problem I’m facing is that by doing things like this, I didn’t find a way to link my traceID to a Tempo trace. Apart from displaying trace_id=xxxx in my panels, which I don’t want to do as this info is detrimental to my panels readability.

Does anyone know if there is a way a way to achieve what I’m trying to do, and to create a link from a Loki label instead of from the actual displayed message ? Or to create Derived Fields before the Logfmt operation ?

Thanks

[b0b]

Maybe this old thread will help Indexing trace_id as a label in Loki

[maxenceb]

I am already able to index my trace_id as a label by using the Logfmt operation, as you can see in my screenshot. The problem is that I then want to only display the info and msg fields, and not the trace_id one, because it makes the panel harder to read

In your thread the link to Tempo that is in the Derived Field is created because the trace_id is displayed, and I don’t want to display it.
So I’m looking for a way to create a link to Tempo directly from a label, and not from the message that is displayed.

And to prove my point here is a screenshot of how it would look if I did that, I would have to display the trace_id

[b0b]

Right. Sorry, misunderstood…

So you want to index your trace_id? That would probably lead to poor performance. You want to keep index cardinality (number of indexed labels) as low as practical.

Best practices | Grafana Loki documentation

[maxenceb]

No this is not what I meant.

My trace_id are not indexed, they are parsed from my logs which are formatted in logfmt. My logs look like this originally →

I then use the Logfmt operation to parse my logs into fields, including a trace_id field, and I display only the level and msg field →

And I just want to know if there is a way to create a link from these fields

[sylvainol]

Hello,
I’ve got the same question, is it possible?

[sylvainol]

well I just added the datasource field stuff and it works actually:

  jsonData:
    derivedFields:
    - datasourceName: Tempo
      datasourceUid: tempo
      matcherRegex: trace_id=(\w+)
      name: traceID
      url: $${__value.raw}

[samuelreay]

Looks like this might be possible in Grafana 10.3 or 10.4 when its released due to Loki: Option to add derived fields based on labels by 5cat · Pull Request #76162 · grafana/grafana · GitHub