Why is the `json` parser failing?

[shivjm]

I’m using a default installation of the Loki Stack on Kubernetes (no added processing). I get logs like this from a pod:

{"level":"debug","msg":"listing project pipelines","project-name":"my-group/project-a","ref":"master","time":"2021-06-03T08:45:59Z"}
{"level":"debug","msg":"could not find any pipeline for the ref","project-name":"my-group/project-a","ref":"master","ref-kind":"branch","time":"2021-06-03T08:45:59Z"}
{"level":"debug","msg":"listing project pipelines","project-name":"my-group/project-b","ref":"master","time":"2021-06-03T08:45:59Z"}
{"level":"debug","msg":"could not find any pipeline for the ref","project-name":"my-group/project-b","ref":"master","ref-kind":"branch","time":"2021-06-03T08:45:59Z"}

I can follow these live without problems in Loki, but I can’t parse them as JSON:

All of them show JSONParserErr under __error__. Am I missing something?

[dannykopping]

Hey @shivjm

If you look closely, all your logs are starting with stdout F - this is why the JSON parsing is failing.
You should enable the CRI pipeline stage so these lines can be correctly parsed from docker: Configuration | Grafana Labs

[shivjm]

Ah, I didn’t realize that was part of the message! I thought it was something being added for display. Thank you, I’ll try adding the cri stage.

[benjeau]

@dannykopping @shivjm Hey, I’m new to k8s/helm/grafana in general and was wondering how you fixed the parser? I can’t seem to be able to extract any field, either the timestamp, stream or output, as specified here cri | Grafana Loki documentation

Checking my helm config (using helm get values loki --all), it is using the cri pipeline-stage:

promtail:
  ...
  pipelineStages:
  - cri: {}
  ...

Is there something that I’m missing? Thanks in advance for anyone replying

[benjeau]

grafana was looking at old logs and not the newest ones. Everything is working…

[shivjm]

You were right on the money. I ended up bypassing the problem by installing the newest Loki & Promtail Helm charts separately. Now it works fine. Thanks again!

(Sorry for taking months to reply. Long story short: I had to completely rebuild my infrastructure from scratch, but that was preempted by health issues until recently.)