Hi @tzaspelfinatix ,
you probably do not want to index trace_id
. Dynamic labels are bad for Loki performance as is and with trace_id
being a unique value per log message will blow up the Loki index (I would assume).
I have spent some time getting the trace_id
lookup from logs work to link directly to Tempo traces.
In your Loki data source settings you need to create some Derived fields
.
Take this log line as an example
2021-12-03T08:18:53.594Z INFO frontend/best_eta.go:106 Dispatch successful {"service": "frontend", "trace_id": "15129c0846deed11", "span_id": "15129c0846deed11", "driver": "T768375C", "eta": "2m0s"}
This is not a very Loki friendly log format but you can still find the value of trace_id
with a pretty simple regex
In Exlore
the raw log looks like this
Again, not Loki friendly but not to worry. You need to apply some parsing so the trace_id
becomes a Detected field
In the above case this means the query ends up looking like this
Which can then link to Tempo
Hope that helps