Suppose you want to use Loki in conjunction with Promtail to collect logs of an application that records log messages whenever a state transition occurs. The log messages are encoded in logfmt format and always contain the two properties request_id and new_state.
LogQL allows me to extract dynamic labels out of the log message, which I can then use in further expressions in the query. In order to visualize the state transitions, Grafana’s built-in “State timeline” visualization seemed to be the obvious choice.
I use the following query to access the data that I need for the visualization:
{job="varlogs"} |= "request_id=" | logfmt request_id, state
To my surprise, the table preview that is provided in the “Edit panel” view remained largely unchanged compared to a query containing only the stream selector. The table still has the following layout:
| labels | Time | Line | tsNs | id |
|---|
Therefore, I came to the conclusion that (dynamic) labels are not directly mapped to fields. I then tried to use the Labels to fields transformation which I expected to do precisely this, but Grafana doesn’t show me any labels to select from.
Does anyone know why dynamic fields of a Loki data source cannot be converted to fields or - if that functionality exists - what I might have done wrong in the process?
As a workaround, I opted for the “extract fields” transformation that takes an already existing field as a source and extracts additional fields out of it. In a certain way, this transformation provides the functionality that I expected logfmt (in conjunction with the labels to fields transformation) to provide in the first place, but just in a less explicit manner.
Grafana v9.5.1 (bc353e4b2d)
Loki v2.8.0
Promtail v2.8.0
Using the default Loki and Promtail demo configuration as provided in the documentation.