OpenTelemetry collector with Loki

I have Otel Collector using Loki exporter and the logs are showing up fine in Loki. I see the log in Loki with several labels and Detected fields. I can query the logs on the labels but not the Detected fields

I am interested in querying the logs based on TraceID or traceID but that doesnt work. What am I doing wrong ?

Hi @rupinder10 ,

I the field is not an indexed label, then you will have to apply some parsing first. Even then, the result will basically be a text filter (but you can just click a button instead of typing it in)

Here is an example

Only a few Log labels and a couple of Detected fields

Some parsing applied

Now you can see more Log labels, still the same logs.

Here is the full query as text

{container="hotrod"} | pattern "<time>\t<level>\t<component>\t<status>\t<json_log>\n" | line_format "{{.json_log}}" | json

I can now use the filters, like indexed labels. The query just adds a filter at the end resulting in

{container="hotrod"} | pattern "<time>\t<level>\t<component>\t<status>\t<json_log>\n" | line_format "{{.json_log}}" | json | trace_id="584397635b19c4b6"

Hope that helps. I can elaborate if needed :slight_smile:

Thanks. My intention is to configure Trace to Logs. I configured it but the query to jump to the logs is {TraceID=""}. But since that doesnt work the Trace to Logs link doesnt yeild anything.

So how do I add parsing logic ? And where do I add it ? I am only 3 days into using Loki.

This post might help. It took me a few weeks to get my head around this :slight_smile:

@b0b Thanks for that link. But that works for going from logs to Traces. So Loki to Tempo. I already have that working. My question is from Temp to Loki. So I want to get that little button (highlighted in yellow) to show up the logs associated with that trace.

I didn’t even know about that function :grimacing:

Looks like the spans and logs need shared tags/labels to work. I had a look at my traces to see what tags they have

If I set one of those tags in the Tempo datasource then the Loki query is pre-filled the way I would expect but my logs do not share the tags so nothing is found.

I tried that. I have a tag named rootContextId on my traces. When I set that up I get an error and it doesnt let me add to the tempo datasource. See below:

I get that error as well. I just ignore it… My changes get saved anyway.

I think some initial data was lost before I got everything to work, so the trace with id: 0 was sent into a black hole. This is just an explanation I have made up for myself. Probably not what happened.

Did you ever work this out? I’m currently in the same position; I can go from Loki to Tempo but not from Tempo to Loki. Would really appreciate any guidance!

Hi @rupinder10,

currently I try to setup OTel collector using a Loki exporter too. How did you get the logs into Loki? According to the collector log output the exporter has been started, but my pod’s logs were not forwarded to Loki.
My setup is:

[Pod + OTel instrumentation] → [OTel collector w/ Loki exporter] → [Loki] → [Grafana w/ datasource Loki]

Thanks for your help!