Grafana version 8.2.4
I am not able to customize trace logs option to “Filter by Trace ID” and “Loki Search” from {} |= “6ddb5d310cfaf4beb4793dd1688a3e5c”
to something like this: {filename=~"/var/log/pods/spring-petclinic.+"} |= “6ddb5d310cfaf4beb4793dd1688a3e5c”
The default query fails when opening the trace logs from tempo as the image shows.
Where is the file that this could be modified?
You can configure what tags Grafana should use in the Tempo datasource settings, under trace to logs. I think it uses cluster, namespace and pod by default (if no tags are configured).
Note that if the configured tags are not present in your span, the Loki query will be empty. That’s probably what is happening in your case.
Thanks for your reply, I added the filename tag, saved and test, explore, tired and still the same default Loki query,
{} |=“f50ae1b8aec19eaf9bf8c92af14e512”
Is there any file that we can update with a default query or maybe even passing parameters?
Are you sure there is a tag named filename on your span? Grafana will build a LogQL query using the tags configured in the datasource but it only uses tags that exist on the span.
So if you, for example, configure the tags foo and bar and your span only has the tag foo=myValue. Grafana will create the query {foo="myValue"} |= <traceID>. Note that the tag bar is omitted because the span does not have it.
Btw, if you don’t want to add |= <traceID> to the query, you can disable “Filter by Trace ID”. This will only work if your logs always log the trace ID as well.
That tag should also be present on the span that you click “Logs for this span” on. For this feature to work your traces and your logs should have the tag, otherwise you can’t correlate them with each other.
For example, in my Grafana instance I have configured the tags namespace, cluster and pod (since I’m running on Kubernetes). If I open a span and click “Logs for this span” it will build a query using these tags:
If my span was missing these tags, the query would be empty because Grafana can not find any values.
So I suspect your Loki queries are empty because the filename tag is missing from your spans (and not your logs).
Ok, I know what I was missing thanks for the hint.
Unfortunately, the “filename” field cannot be easily added to the Opentelemetry instrumented with the Java app, and we are streaming logs to Loki with promtail and Tempo with Opentelemetry Collector, so both backends will only have trace_id as the intercept.
Any suggestion on how to link the Tempo trace_id with Loki trace_id?
On the other hand, Loki easily links back the trace_id to tempo and Jaeger with the derived fields. (Last image)
Also, the image shows that LokiQL is not friendly with label names concatenated with a period.
Does filename refer to log files being created by the apps? In that case yeah, you can’t add these to your traces because the code creating and exporting the spans isn’t aware of the logfiles being created.
If you only want to see logs related to this specific trace you should add the trace ID to every logline and set the “Filter by TraceID” toggle. This will create LogQL query like
{...} |= "<traceID>"
{...} to have valid LogQL queryOuch, I’ll ask the Grafana team about this. I think we’ll need to do some extra processing to get this working.
What label do your Loki logs have? Does it have a label service or service_name?
There is a already a GitHub issue for this: Rename tags in Traces to Logs feature. · Issue #31778 · grafana/grafana · GitHub
Feel free to share your use-case/issue. This will help prioritise this work 
Hey I am using otel java agent to for tracing and tempo data source to show them. Currently my spans are missing the kubernetes specific data(like namespace, container etc) How can I add that data to spans?
The OpenTelemetry Collector or Grafana Agent both have the ability to decorate span data with additional information. See “scrape_configs” in the Grafana Agent link.
Also, a simpler approach, is to use environment variables to add key value pairs to all spans produced by a process. See “OTEL_RESOURCE_ATTRIBUTES”.
Thanks. I have a follow up question. I am trying to add the trace to log feature config instead of configuring manually using this documentation.
apiVersion: 1
datasources:
- name: Jaeger
type: jaeger
uid: jaeger-spectra
access: proxy
url: http://localhost:16686/
basicAuth: true
basicAuthUser: my_user
editable: true
isDefault: false
jsonData:
tracesToLogs:
# Field with internal link pointing to a logs data source in Grafana.
# datasourceUid value must match the `datasourceUid` value of the logs data source.
datasourceUid: 'loki'
tags: ['job', 'instance', 'pod', 'namespace']
mappedTags: [{ key: 'service.name', value: 'service' }]
mapTagNamesEnabled: false
spanStartTimeShift: '1h'
spanEndTimeShift: '1h'
filterByTraceID: false
filterBySpanID: false
secureJsonData:
basicAuthPassword: my_password
I don’t know if the datasourceUid to loki exists. If you view the config for loki data source the key to set uid is missing from the yaml.
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://localhost:3100
basicAuth: true
basicAuthUser: my_user
basicAuthPassword: test_password
jsonData:
maxLines: 1000
Where can I find uid of loki if I haven’t set it in config???
You have to set the uid manually, otherwise you will get a randomly generated uid.
Config should look something like this:
datasources:
- name: Loki
type: loki
uid: loki
# ...
Following up on this, you can now map span attribute names (service.name) to log label names (service) by selecting map tag names in the trace to logs config.
