Can we generate trace data for Tempo from log data in Loki?

Grafana Tempo
, ,
1

Hi, everyone. My first topic here.

I am considering generating trace data for Tempo from the log data in Loki directly for the following 3 reasons:

  • First of all, the default sampling ratio is so confusing. It could cause losing important data while I wanna trace the chain from log. However, 100% sampling is absolutely unavailable to me. I believe that generate the Trace data from Loki data is a good way to fix the problem.

  • Secondly, the applications in my environment are not easy to use agent. Maybe the engineers are not willing to make any change in their work excluding the log format, especially in the projects in C/C++. So I will try to make them believe that the change in log could be useful in Tracing.

  • Last but not least, Tempo is really easy to use. I could save my time from building ELK/EFK and operating so many VM.

Just wanna get some information. Is there any open-source project which could transfer Loki data to Tempo data? If not, I will try to build one and share it. Hope it helpful in the future.

Could anyone give me any suggestion or idea?

Thank you so much. slight_smile:

2

It sounds crazy, but IMHO it is possible (at least as a proof of concept).
You need to consider a trace as a log in very well defined format (json with defined fields). You will need to code it on your own, because only you know your log format (and you may discover that you don’t have enough data in logs to generate valid traces).

1 Like
3

The Loki Promtail plugin can be configured to extract trace IDs from Loki logs and send them to Tempo. This method is relatively simple to implement, but it can only generate trace data for logs that explicitly contain trace IDs.

4

But trace is not only traceid. You need to build whole structure (spans, attributes,…)., which may be a challenge if you heterogeneous logs format.