Grafana version 8.2.4
I am not able to customize trace logs option to “Filter by Trace ID” and “Loki Search” from {} |= “6ddb5d310cfaf4beb4793dd1688a3e5c”
to something like this: {filename=~“/var/log/pods/spring-petclinic.+”} |= “6ddb5d310cfaf4beb4793dd1688a3e5c”
The default query fails when opening the trace logs from tempo as the image shows.
Where is the file that this could be modified?
You can configure what tags Grafana should use in the Tempo datasource settings, under trace to logs. I think it uses cluster, namespace and pod by default (if no tags are configured).
Note that if the configured tags are not present in your span, the Loki query will be empty. That’s probably what is happening in your case.
Thanks for your reply, I added the filename tag, saved and test, explore, tired and still the same default Loki query,
{} |=“f50ae1b8aec19eaf9bf8c92af14e512”
Is there any file that we can update with a default query or maybe even passing parameters?
Are you sure there is a tag named filename on your span? Grafana will build a LogQL query using the tags configured in the datasource but it only uses tags that exist on the span.
So if you, for example, configure the tags foo and bar and your span only has the tag foo=myValue. Grafana will create the query {foo="myValue"} |= <traceID>. Note that the tag bar is omitted because the span does not have it.
Btw, if you don’t want to add |= <traceID> to the query, you can disable “Filter by Trace ID”. This will only work if your logs always log the trace ID as well.
That tag should also be present on the span that you click “Logs for this span” on. For this feature to work your traces and your logs should have the tag, otherwise you can’t correlate them with each other.
For example, in my Grafana instance I have configured the tags namespace, cluster and pod (since I’m running on Kubernetes). If I open a span and click “Logs for this span” it will build a query using these tags:
If my span was missing these tags, the query would be empty because Grafana can not find any values.
So I suspect your Loki queries are empty because the filename tag is missing from your spans (and not your logs).
Ok, I know what I was missing thanks for the hint.
Unfortunately, the “filename” field cannot be easily added to the Opentelemetry instrumented with the Java app, and we are streaming logs to Loki with promtail and Tempo with Opentelemetry Collector, so both backends will only have trace_id as the intercept.
Any suggestion on how to link the Tempo trace_id with Loki trace_id?
On the other hand, Loki easily links back the trace_id to tempo and Jaeger with the derived fields. (Last image)
Also, the image shows that LokiQL is not friendly with label names concatenated with a period.
Does filename refer to log files being created by the apps? In that case yeah, you can’t add these to your traces because the code creating and exporting the spans isn’t aware of the logfiles being created.
If you only want to see logs related to this specific trace you should add the trace ID to every logline and set the “Filter by TraceID” toggle. This will create LogQL query like
{...} |= "<traceID>"
{...} to have valid LogQL queryOuch, I’ll ask the Grafana team about this. I think we’ll need to do some extra processing to get this working.
What label do your Loki logs have? Does it have a label service or service_name?
There is a already a GitHub issue for this: Rename tags in Traces to Logs feature. · Issue #31778 · grafana/grafana · GitHub
Feel free to share your use-case/issue. This will help prioritise this work 
Hey I am using otel java agent to for tracing and tempo data source to show them. Currently my spans are missing the kubernetes specific data(like namespace, container etc) How can I add that data to spans?
The OpenTelemetry Collector or Grafana Agent both have the ability to decorate span data with additional information. See “scrape_configs” in the Grafana Agent link.
Also, a simpler approach, is to use environment variables to add key value pairs to all spans produced by a process. See “OTEL_RESOURCE_ATTRIBUTES”.
Thanks. I have a follow up question. I am trying to add the trace to log feature config instead of configuring manually using this documentation.
apiVersion: 1
datasources:
- name: Jaeger
type: jaeger
uid: jaeger-spectra
access: proxy
url: http://localhost:16686/
basicAuth: true
basicAuthUser: my_user
editable: true
isDefault: false
jsonData:
tracesToLogs:
# Field with internal link pointing to a logs data source in Grafana.
# datasourceUid value must match the `datasourceUid` value of the logs data source.
datasourceUid: 'loki'
tags: ['job', 'instance', 'pod', 'namespace']
mappedTags: [{ key: 'service.name', value: 'service' }]
mapTagNamesEnabled: false
spanStartTimeShift: '1h'
spanEndTimeShift: '1h'
filterByTraceID: false
filterBySpanID: false
secureJsonData:
basicAuthPassword: my_password
I don’t know if the datasourceUid to loki exists. If you view the config for loki data source the key to set uid is missing from the yaml.
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://localhost:3100
basicAuth: true
basicAuthUser: my_user
basicAuthPassword: test_password
jsonData:
maxLines: 1000
Where can I find uid of loki if I haven’t set it in config???
You have to set the uid manually, otherwise you will get a randomly generated uid.
Config should look something like this:
datasources:
- name: Loki
type: loki
uid: loki
# ...
Following up on this, you can now map span attribute names (service.name) to log label names (service) by selecting map tag names in the trace to logs config.
Hello All,
I was trying the same approach has you doing. Instrumenting petclinic Springboot Demo application. with Opentelemetry java agent to export trace to tempo.
I am getting ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export metrics. The request could not be executed. Full error message: Failed to connect to /...:4317.
I have not used docker to build the application and the grafana stack. It is installed manually on centos machine.
It will be very helpful if anyone can figure it out with this complication
By default the OTel client is trying to send metrics, traces and possibly logs to the configured GRPC endpoint. Tempo will only consume traces so you are getting this error about metrics.
You can
OTEL_EXPORTER=otlp_span but I’m not seeing that in current docs. Maybe it’s been removed? Try OTEL_METRICS_EXPORTER=none and OTEL_LOGS_EXPORTER=noneThanks Joe, It worked the error has been resolved. but now in the logs Before & After request is printed which indicated the traces has be generated. but in Grafana under explore window no records is shown. also in logs trace id is not generated can you please help me out from this complication.
Please find the attachment for more details.
Sorry for the trouble Again.
I see so many hardcoded solutions for TracesToLogs for different datasources , yet the simplest one is a plain old templated url like http://mycustomlogs.com/search?var1={traceid} ? mentioned here : Tempo logs button/link with custom url? .
It could open up a lot of use cases like opening dashboards with prefilled variable values.
