Why are there additional "parsed fields" in Grafana Drilldown that I do not specified explicitly?

Grafana Drilldown-logs
,
1

I am using Alloy to send Kubernetes log to Loki and Explore/Drilldown to visualize the log. Here is the Alloy config snippet that I use to specify the labels and structured metadata:

discovery.relabel "default_pod_logs" {

 	<omitted>

	rule {
		source_labels = ["__meta_kubernetes_namespace"]
		action        = "replace"
		target_label  = "namespace"
	}
	rule {
		source_labels = ["__meta_kubernetes_pod_name"]
		action        = "replace"
		target_label  = "pod"
	}
	rule {
		source_labels = ["__meta_kubernetes_pod_container_name"]
		action        = "replace"
		target_label  = "container"
	}
	rule {
		source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
		action        = "replace"
		target_label  = "app"
	}

 	<omitted>

}

loki.process "java_pod_logs" {

 	<omitted>

	stage.structured_metadata {
		values = {
			java_service_name = "",
			level = "",
			primary_context = "",
			secondary_context = "",
			retention = "",
		}
	}

	<omitted>

	stage.output {
		source = "message"
	}

	<omitted>

}

and here is the details of the log as seen in Explore:

image
image2252Ă—1124 284 KB

which is as I expected: the labels and structured metadata fields are what I specified with several auto-added label from Alloy’s discovery component.

But when seen in Drilldown, there are additional “parsed labels”:

image
image2716Ă—1066 293 KB

image
image2744Ă—1430 252 KB

I am guessing that there is a mechanism in somewhere that automatically detect the log format and parse it, maybe mistakenly as Logfmt.

My questions are

  1. Why is this happening?
  2. I don’t want the parsed labels to be shown at all in Drilldown, including in the Fields tab. Is it possible?
2

I am not aware of Loki doing this except for log level detection. What is your actual logql query?

3

LogQL in Explore:

{service_name="paycoreservice", pod="paycoreservice-f8dbd95cd-vbzzh"} |= `c66f7b891d444277a760de99e6ec702d` |= `statement_timeout`

Also here is the filters in Drilldown:

image
image2770Ă—1582 459 KB

4

Sorry, I forget to include the version info. I am using Grafana v12.0.0, simple scalable deployment Loki v3.5.0 from Helm chart v6.3.0, and Alloy v1.9.0 as Deployment from Helm chart v1.1.0

5

I haven’t upgraded to Grafana 12, so I can’t say for sure, but I think that’s what the log drill down feature does. See What's new in Grafana v12.0 | Grafana documentation

6

@hz00 Logs Drilldown automatically adds parsers (logfmt, json) to your queries so the application can visualize/filter on fields parsed in unstructured log lines. If you add | logfmt to the end of your query in Explore, you’ll see the same parsed fields that are showing in Logs Drilldown.

1 Like
7

Is it possible to turn it off?

8

@hz00 parsers are always added to the logs panel queries in Logs Drilldown, but will only be added to time series queries (i.e. in the labels/fields breakdown tabs) if the user has added a (non metadata) filter, or the field being displayed is not a metadata field.

It is not something that can be turned off as that would make it impossible for users to filter on fields. If you need this level of control over your queries I would suggest sticking to dashboards and classic explore, and not Logs Drilldown.

1 Like
9

Thank you. My plan is to let non-engineer teams access logs from Grafana, I am a bit hesitant to tell them to use Explore.