Indexing trace_id as a label in Loki

Hi @tzaspelfinatix ,

you probably do not want to index trace_id. Dynamic labels are bad for Loki performance as is and with trace_id being a unique value per log message will blow up the Loki index (I would assume).

I have spent some time getting the trace_id lookup from logs work to link directly to Tempo traces.

In your Loki data source settings you need to create some Derived fields.

Take this log line as an example

2021-12-03T08:18:53.594Z	INFO	frontend/best_eta.go:106	Dispatch successful	{"service": "frontend", "trace_id": "15129c0846deed11", "span_id": "15129c0846deed11", "driver": "T768375C", "eta": "2m0s"}

This is not a very Loki friendly log format but you can still find the value of trace_id with a pretty simple regex

In Exlore the raw log looks like this

Again, not Loki friendly but not to worry. You need to apply some parsing so the trace_id becomes a Detected field

In the above case this means the query ends up looking like this

Which can then link to Tempo

Hope that helps :slight_smile:

1 Like