I am looking to link my Loki logs to my Tempo trace from opentelemetry.
From my understanding the only way to do that is use the log message text and the Derived fields regex to create a detected label.
However the Logging Instrumentor (opentelemetry.instrumentation.logging) adds the fields as extra content to the log with out using the message field. Which is detected and correctly displayed by Loki, with out the need of extra configuration.
Is a way to use these already detected log labels to link the log to the corresponding trace?
I’m looking to do the same thing as you, which is creating a link from a Loki field to a Tempo trace.
In my case, I do have the traceId in my logs which are formatted in logfmt, but I’m trying to create a nice looking dashboard for my application logs, in which I display only the essential informations which are the type of log and the actual log message.
My dashboard panels look like this, so I only display the info and msg fields. And you can see the other fields by opening a specific log line.
And my logs are parsed using the Logfmt
operation as you can this in the next screenshot
The problem I’m facing is that by doing things like this, I didn’t find a way to link my traceID to a Tempo trace. Apart from displaying
trace_id=xxxx in my panels, which I don’t want to do as this info is detrimental to my panels readability.
Does anyone know if there is a way a way to achieve what I’m trying to do, and to create a link from a Loki label instead of from the actual displayed message ? Or to create Derived Fields before the Logfmt operation ?
I am already able to index my trace_id as a label by using the Logfmt operation, as you can see in my screenshot. The problem is that I then want to only display the info and msg fields, and not the trace_id one, because it makes the panel harder to read
In your thread the link to Tempo that is in the Derived Field is created because the trace_id is displayed, and I don’t want to display it.
So I’m looking for a way to create a link to Tempo directly from a label, and not from the message that is displayed.
And to prove my point here is a screenshot of how it would look if I did that, I would have to display the trace_id
Right. Sorry, misunderstood…
So you want to index your
trace_id? That would probably lead to poor performance. You want to keep index cardinality (number of indexed labels) as low as practical.
No this is not what I meant.
My trace_id are not indexed, they are parsed from my logs which are formatted in logfmt. My logs look like this originally →
I then use the Logfmt operation to parse my logs into fields, including a trace_id field, and I display only the level and msg field →
And I just want to know if there is a way to create a link from these fields
I’ve got the same question, is it possible?
well I just added the datasource field stuff and it works actually:
- datasourceName: Tempo