# JSONParserError on json formatter

**URL:** <https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201>\
**Category:** Grafana Loki\
**Created:** [February 7, 2023, 2:17pm UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201 "2023-02-07T14:17:58Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![sarasensible](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sarasensible](https://community.grafana.com/u/sarasensible)\
**Post date:** [February 7, 2023, 2:17pm UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201/1 "2023-02-07T14:17:58Z")

</div>

We recently upgraded to Kubernetes 1.24 and Promtail is no longer gathering logs in JSON format  
I’m guessing this is due to Dockershim being deprecated in 1.24 such that Docker may no longer be outputting the JSON file format it used to.

The symptom is that outputting logs to the json formatter results in an error:

```auto
{namespace=~".+"} | json

```

produces

```auto
pipeline error: 'JSONParserErr' for series: '{ __error__ ="JSONParserErr", <<REDACTED>>, stream="stderr"}'. Use a label filter to intentionally skip this error. (e.g | __error__!="JSONParserErr"). To skip all potential errors you can match empty errors.(e.g __error__ ="") The label filter can also be specified after unwrap. (e.g | unwrap latency | __error__ ="" )

```

I tried using template to output JSON but it seems that template only outputs strings, even though it might look like JSON.

```auto
      - cri: {}
      - regex:
          expression: "^(?P<content>.*)$"
      - template:
          source: content
          template: '{ "log": "{{ .Value }}" }'
      - output:
          source: content

```

How can I get the json formatter working again?

---

<div class="post-metadata">

**Author:** ![sarasensible](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sarasensible](https://community.grafana.com/u/sarasensible)\
**Post date:** [February 7, 2023, 9:53pm UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201/2 "2023-02-07T21:53:01Z")

</div>

Per @dylanguedes1 promtail does not support the feature of converting non-JSON logs to JSON. Since we are using EKS configuring our Kubernetes to use dockershim is not an option in 1.24 - see [Amazon EKS ended support for Dockershim - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/dockershim-deprecation.html) . I’m going to have to drop using the json filter in favor of pattern in my dashboards and hope that in time promtail might add this feature. I will link back a feature request once I create it.

---

<div class="post-metadata">

**Author:** ![sarasensible](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sarasensible](https://community.grafana.com/u/sarasensible)\
**Post date:** [February 8, 2023, 4:08pm UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201/3 "2023-02-08T16:08:31Z")

</div>

Feature request [Promtail - Pipeline stage to convert non-JSON formatted logs to JSON formatted · Issue #8465 · grafana/loki · GitHub](https://github.com/grafana/loki/issues/8465)

---

<div class="post-metadata">

**Author:** ![b0b](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/b0b/32/8761_2.png) [@b0b](https://community.grafana.com/u/b0b)\
**Post date:** [February 9, 2023, 1:16pm UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201/4 "2023-02-09T13:16:54Z")

</div>

What is your current Promtail `pipeline_stages` config?

We are not using Kubernetes v1.24 yet so I have not faced this issue but I would still expect that container logs would still keep their standard Docker log format

> **[docker | Grafana Loki documentation](https://grafana.com/docs/loki/latest/clients/promtail/stages/docker/#docker-stage)**
>
> docker stage The docker stage is a parsing stage that reads log lines in the standard format of Docker log files.
> Schema docker: {} Unlike most stages, the docker stage provides no configuration options and only supports the specific Docker log...

And as far as I know, Loki always just stores plain strings. If your logs are in JSON or `logfmt` format, then you can parse them using the built-in parsers at query time. Of course Promtail can parse logs to extract fields to assign values to labels etc. but the logs stored in Loki are plain strings (except for the labels in the index).

Based on the error message, it looks like error logs (the `stderr` stream) is not structured JSON. You should be able to see that with a query like `{namespace=~".+", stream="stderr"}` without adding the `| json` parser.

---

<div class="post-metadata">

**Author:** ![sarasensible](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sarasensible](https://community.grafana.com/u/sarasensible)\
**Post date:** [February 9, 2023, 3:03pm UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201/5 "2023-02-09T15:03:30Z")

</div>

@b0b Thanks so much for the response!

The pipeline stages config I landed on (from the Helm values file) just uses cri

```auto
    pipelineStages:
      - cri: {}

```

however I’m not outputting anything from it so I don’t think this really changes much. Based on tailing the logs from the containers they are not outputting in JSON format at all. I was hoping that Promtail would be able to capture the groups based on a regex or similar and output them in a true JSON format for Loki to parse with the json filter. I did try using `template` to output to a structured JSON formatted string, however Loki still failed to parse this with the json filter.

The pipeline stages I used to try outputting JSON formatted strings:

```auto
      - cri: {}
      - regex:
          expression: "^(?P<content>.*)$"
      - template:
          source: content
          template: '{ "log": "{{ .Value }}" }'
      - output:
          source: content

```

however this seemed to make no difference. I assumed it was because it was still a string rather than JSON under the hood, but there may be more to it, such as the timestamp not being included.

Any guidance would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![b0b](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/b0b/32/8761_2.png) [@b0b](https://community.grafana.com/u/b0b)\
**Post date:** [February 9, 2023, 3:51pm UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201/6 "2023-02-09T15:51:10Z")

</div>

As these are Kubernetes logs I would definitely try using `docker` instead of `cri` as your first _stage_. That will not magically turn your logs into JSON but it gives you the basic `log`, `time` and `stream` fields parsed.

From years of working in the team looking after the centralised logging platform we use, I am a firm believer in having the application code format the logs into whatever format they should be. It might be technically possible to do in the agent or some later ingestion stage, depending on what system you use, but that is so much less efficient and prone to parsing errors than instrumenting logging in the application code.

---

<div class="post-metadata">

**Author:** ![sarasensible](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sarasensible](https://community.grafana.com/u/sarasensible)\
**Post date:** [February 9, 2023, 5:49pm UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201/7 "2023-02-09T17:49:12Z")

</div>

Hmmm, I tried using the docker stage and the logs look less cleanly formatted than with the cri stage.

With docker stage:

```auto
2023-02-09T17:36:12.693584297Z stdout F e[32mINFOe[0m: 2023-02-09 17:36:12,693 :: 10.0.44.236:44434 - "e[1mGET /docs HTTP/1.1e[0m" e[32m200 OKe[0m

```

With cri stage:

```auto
e[32mINFOe[0m: 2023-02-09 17:36:24,373 :: 10.1.34.41:42946 - "e[1mGET /docs HTTP/1.1e[0m" e[32m200 OKe[0m

```

Maybe I’m misunderstanding how cri and docker are used with Kubernetes. For example, I was under the impression that the docker stage was specifically for use with docker-formatted logs which I assumed would go away once dockershim was deprecated. Whereas with CRI this is a log format that is standardized for use with any container runtime - sourced from [Logging Architecture | Kubernetes](https://kubernetes.io/docs/concepts/cluster-administration/logging/) - such that no matter whether I’m using docker or something else it should all funnel into CRI with Kubernetes.

I admit that having promtail generate JSON from raw logs is a workaround for what is ultimately a design decision on the part of the Kubernetes authors. Maybe the question becomes whether this use case of preserving stability of dashboards and monitoring built on JSON outweighs the risk of inefficiency and possible parsing errors. In our case it might not - we only have a couple of dashboards and alerts that use the json filter. However considering that more and more people will be upgrading to K8s 1.24 there may be others who have more entrenched usage of this filter.

---

<div class="post-metadata">

**Author:** ![b0b](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/b0b/32/8761_2.png) [@b0b](https://community.grafana.com/u/b0b)\
**Post date:** [February 10, 2023, 8:30am UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201/8 "2023-02-10T08:30:01Z")

</div>

I think you are right regarding the `cri` vs. `docker` stage. As I mentioned, this is not something I have encountered yet. Good to know what to prepare for.

Now, after reading up a bit on CRI, the output differences make perfect sense. The `cri` output is the same as the `docker` output but without `time` and `stream`. So looks like `cri` is the one to use.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/grafana/original/2X/e/ef5308cc58c9e5a13187b5012eb99f406928462f.png) [@system](https://community.grafana.com/u/system)\
**Post date:** [February 10, 2024, 8:30am UTC](https://community.grafana.com/t/jsonparsererror-on-json-formatter/81201/9 "2024-02-10T08:30:44Z")

</div>

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.