Panel log doesn't show parsed JSON

[otalv]

Hello,

I’m using Promtail (same version to send logs from two K8s clusters (QA and PROD), both installations use the same helm chart values for Promtail, and both applications log messages in JSON format as follows:

Prod environment:

❯ echo '{"timestamp":"2024-03-27T07:18:05.574Z","level":"WARN","thread":"grpc-executor-3","logger":"REDACTED","message":"REDACTED","context":"default"}' | jq
{
  "timestamp": "2024-03-27T07:18:05.574Z",
  "level": "WARN",
  "thread": "grpc-executor-3",
  "logger": "REDACTED",
  "message": "REDACTED",
  "context": "default"
}

QA environment:

 ❯ echo '{"timestamp":"2024-03-26T19:53:47.425Z","level":"INFO","thread":"http-nio-8024-exec-5","logger":"AUDIT.io.axoniq.axonserver.rest.BackupInfoRestCon
troller","message":"REDACTED","context":"default"}' | jq
{
  "timestamp": "2024-03-26T19:53:47.425Z",
  "level": "INFO",
  "thread": "http-nio-8024-exec-5",
  "logger": "REDACTED",
  "message": "REDACTED",
  "context": "default"
}

As you guys can see, both log messages are valid JSON values.

The issue is: the panel log doesn’t show the logs from the Prod environment the same way it shows the logs from the QA environment.

Here is how the logs from QA appears:

Here is how the logs from Prod appear:

Both screenshots are from the same Dashboard, I just changed the cluster variable on the top.

Here are the values I’m using in the helm chart:

snippets:
    pipelineStages:
      - cri: {}
      - static_labels:
          cluster: ${cluster_id}
      - json:
          expressions:
            timestamp: timestamp
            level: level
            thread: thread
            class: logger
            message: message
            context: context
      - labels:
          level:
          class:
          context:
          thread:
          project_code:
      - timestamp:
          format: RFC3339
          source: timestamp
      - output:
          source: message

What am I doing wrong?

[tonyswumac]

Comparing your production log screenshot to your promtail configuration, you are definitely missing one layer.

Your production logs are: {"log": "{<ACTUAL_LOGS>}"}, which is what i would expect from Docker. With that your promtail pipelines should be:

    - json:
          expressions:
            log:
    - json:
          source: log
          expressions:
            timestamp: timestamp
            level: level
            thread: thread
            class: logger
            message: message
            context: context

As to why your logs are in different format, I don’t know. I’d double check your Kubernetes / docker log configurations and promtail configurations and see if there is any mismatch somewhere.

[otalv]

Hi @tonyswumac, thanks for your answer.

The container run time in QA is containerd and on PROD is still docker, this is the only difference I can see.

I’ve deployed Promtail helm chart adding your suggestion, and the parser worked.