JSON Not Being Parsed. No Error Thrown But No New Labels

Hey Grafana Community,

I’m new to PLG stack so please excuse any ignorance. I am running k8 and I have “Loki-Stack” (all defaults) helm chart deployed in my EKS cluster.

My logs are already in JSON format, however none of the underlying fields are being parsed as “Log Labels”. When I try to pass the logs through the json pipeline, I still don’t see the labels being extracted. I do not see any errors in the UI saying my JSON is malformed or anything like that.

Here is an example:

Log line as it appears in Query Explorer

“{“log”:{“level”:“warn”,“logger”:“pino”},”@timestamp":“2021-02-15T14:56:51.487Z”,“process”:{“pid”:17},“host”:{“hostname”:“graphql-api-54589bd4cf-frhvd”},“error”:{“name”:“UnauthorizedError”,“message”:“No authorization token was found”,“code”:“credentials_required”,“status”:401,“inner”:{“message”:“No authorization token was found”}},“ecs”:{“version”:“1.5.0”},“message”:“Auth Error”} "

Expanded Log line as it appears in Explorer

I would like the underlying fields in my JSON to be defined as “Log Labels” so I can query against them more effectively.

After doing extensive research it seems the expected behavior is for this JSON object to be flattened and it’s fields parsed out in the log labels section.

hey @lwooden!

You should be able to add the json parser to your query to parse the json and turn the contents into labels:

{app="foo"} | json

check out the docs for some more info!

Hey Mr. Welch! I promise I have tried this time and time again, but it seems like no labels are being extracted and my json is not being flattened as it says in the docs. In fact, nothing is happening. See attached snippets of when i add that in my query and execute and what the log line looks like afterward. Nothing changes

It sounds like maybe you are running an older version of Loki (and/or maybe Grafana)? You would need at least 2.0 or newer?

Interesting…

I’m using the Loki-Stack as defined here:

https://github.com/grafana/helm-charts/tree/main/charts

Should I not be using this?

kubernetes output from describe commands on the loki and grafana pods that are created using the loki-stack i mentioned previously

Containers:
loki:
Container ID: docker://00dd581eb4b45f7108763849061644257cf0e3e98ce18a91b84f0df0ddbcdda0
Image: grafana/loki:2.0.0
Image ID: docker-pullable://grafana/loki@sha256:77e138f81a8e253f1d0ea5d2dc329a02212ecab3247c87f85f1f2182a0160ccd
Port: 3100/TCP
Host Port: 0/TCP
Args:
-config.file=/etc/loki/loki.yaml

Containers:
grafana:
Container ID: docker://41ac185db83c991ba5b9dc177699ac1333e47dbe6f4551a89935a22843da31c9
Image: grafana/grafana:6.7.0
Image ID: docker-pullable://grafana/grafana@sha256:f01763cb0f2ae680e63ba7ad1605e91ec15eaeb419f7c1e31e81ed48c372546a
Ports: 80/TCP, 3000/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Tue, 23 Feb 2021 08:42:04 -0500

We can close this.

Solution for anyone else who may run into this issue:

The loki-stack helm chart uses version/image tag 6.7.0 by default. This image does not contain the features detailed in many of the new docs and videos.

Use the following command to deploy the loki-stack with the latest version of Grafana (per docker)

helm upgrade --install loki grafana/loki-stack --namespace=loki --set grafana.enabled=true --set grafana.image.tag=7.4.3 (or whatever version you would like)

Hi
I’m facing the same issue as described with Grafana 7.4.3 & Loki/Promtail 2.2.0