Why is the `json` parser failing?

I’m using a default installation of the Loki Stack on Kubernetes (no added processing). I get logs like this from a pod:

{"level":"debug","msg":"listing project pipelines","project-name":"my-group/project-a","ref":"master","time":"2021-06-03T08:45:59Z"}
{"level":"debug","msg":"could not find any pipeline for the ref","project-name":"my-group/project-a","ref":"master","ref-kind":"branch","time":"2021-06-03T08:45:59Z"}
{"level":"debug","msg":"listing project pipelines","project-name":"my-group/project-b","ref":"master","time":"2021-06-03T08:45:59Z"}
{"level":"debug","msg":"could not find any pipeline for the ref","project-name":"my-group/project-b","ref":"master","ref-kind":"branch","time":"2021-06-03T08:45:59Z"}

I can follow these live without problems in Loki, but I can’t parse them as JSON:

All of them show JSONParserErr under __error__. Am I missing something?

Hey @shivjm

If you look closely, all your logs are starting with stdout F - this is why the JSON parsing is failing.
You should enable the CRI pipeline stage so these lines can be correctly parsed from docker: Configuration | Grafana Labs

1 Like

Ah, I didn’t realize that was part of the message! I thought it was something being added for display. Thank you, I’ll try adding the cri stage.

@dannykopping @shivjm Hey, I’m new to k8s/helm/grafana in general and was wondering how you fixed the parser? I can’t seem to be able to extract any field, either the timestamp, stream or output, as specified here cri | Grafana Labs

Checking my helm config (using helm get values loki --all), it is using the cri pipeline-stage:

promtail:
  ...
  pipelineStages:
  - cri: {}
  ...

Is there something that I’m missing? Thanks in advance for anyone replying :slight_smile:

:man_facepalming: grafana was looking at old logs and not the newest ones. Everything is working…

You were right on the money. I ended up bypassing the problem by installing the newest Loki & Promtail Helm charts separately. Now it works fine. Thanks again!

(Sorry for taking months to reply. Long story short: I had to completely rebuild my infrastructure from scratch, but that was preempted by health issues until recently.)

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.