How to display structured JSON logs in Grafana Loki

[dariamimi]

I have a logging config in my project:

import logging
import json
from copy import deepcopy
from logging import config
from typing import Any, Dict


class JsonBodyFilter(logging.Filter):
    def filter(self, record):
        if hasattr(record, 'body'):
            try:
                record.body = json.loads(record.body)
            except (ValueError, TypeError):
                pass
        return True


LOGGING_DICT: Dict[str, Any] = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "json": {
            "()": "pythonjsonlogger.orjson.OrjsonFormatter",
            "format": "%(levelname)s %(asctime)s %(name)s %(message)s",
            "datefmt": "%Y-%m-%dT%H:%M:%S.%03d",
            "json_indent": False,
        },
    },
    "filters": {
        "json_body": {
            "()": JsonBodyFilter,
        },
    },
    "handlers": {
        "console": {
            "formatter": "json",
            "filters": ["json_body"],
            "class": "logging.StreamHandler",
            "stream": "ext://sys.stdout",
        },
    },
    "loggers": {},
}


def configure_logging(
    logging_dict: dict[str, Any] | None = None,
    app_log_level_map: dict[str, str] | None = None,
) -> None:
    logging_dict = logging_dict or deepcopy(LOGGING_DICT)
    logging_config = logging_dict.copy()
    if app_log_level_map:
        for app_name, log_level in app_log_level_map.items():
            if app_name not in logging_config["loggers"]:
                logging_config["loggers"][app_name] = {}
            logging_config["loggers"][app_name]["level"] = log_level
            logging_config["loggers"][app_name]["handlers"] = ["console"]

        logging.config.dictConfig(logging_config)

There is a Kubernetes cluster with nodes where my applications are deployed. Each node has a Promtail agent that scrapes logs from endpoints and sends them to the Loki cluster (which consists of multiple services responsible for validation, storage, etc.).

For Promtail, which collects logs and pushes them to Loki, I have the following config:

scrape_configs:
- pipeline_stages:
- docker: {}
- multiline:
firstline: ‘^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}’
max_lines: 128
max_wait_time: 3s
job_name: kubernetes-pods-name
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: host
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- action: labelmap
regex: _meta_kubernetes_pod_label(.+)
- replacement: /var/log/pods/$1/.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: path

Grafana is connected to Loki as a data source, and it fetches logs by querying the loki-storage service.

If I write this query in Grafana:

{app=“$app”, namespace=“hmi-$stand”} | json

I get the error JSONParserErr.

I also tried using “Prettify JSON” in the panel settings, but nothing worked.

Currently my logs look like this:

but I want them to look like this — expanded and pretty-printed JSON:

[galenkistler]

@dariamimi if you are getting a JSONParserErr when running the | json parser that’s because your log lines are not valid JSON, you’ll either need to only ingest the JSON portion of the logs, or you’ll need to use template functions before the | json parser to strip out the non-json portion of your log lines.

Once you are working with JSON then pretty print should work as expected.

[dariamimi]

In my project, I’m outputting JSON logs to stdout. Where exactly is the place where these logs eventually get collected as strings?

I need to receive the entire log (the same way I’m getting it right now), but I also want to make it pretty printed (formatted nicely for readability).

[galenkistler]

Unfortunately if your logs are not valid json pretty print will not work, and since your logs are not logfmt or json format everything you try to do with these logs is going to be more difficult, I highly recommend moving any relevant information inside the json and only ingesting valid json.

But if you can’t fix the logs at ingest, you can always strip out the non-json bits at query time:

{service_name="MY_SERVICE"} | pattern "stdout F <json>" | line_format "{{.json}}" | json

This query will add the stuff after the stdout F to a new field called json using the pattern parser, and then line_format will replace the current log line with the content of that field:

[dariamimi]

I tried applying this query, but I still get the JSONParserErr error.
My service outputs raw logs in JSON format, but before they reach Loki they go through a transformation via Promtail (which I don’t have access to). Most likely, the logs arrive in Grafana with metadata attached — specifically this part: <datetime> stdout F.

When I run a query with a regular expression, I get an error.

Do you write in Python?
Do you have a logging configuration?
Here’s my logging config:

import logging
from copy import deepcopy
from logging import config
from typing import Any, Dict


class IgnoreExcludedPathsFilter(logging.Filter):
    excluded_log_paths = {
        "GET /metrics",
        "GET /healthz",
    }

    def filter(self, record):
        message = record.getMessage()
        return not any(path in message for path in self.excluded_log_paths)


LOGGING_DICT: Dict[str, Any] = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "json": {
            "()": "pythonjsonlogger.orjson.OrjsonFormatter",
            "format": "%(path)s %(levelname)s %(asctime)s %(name)s %(message)s",
            "datefmt": "%Y-%m-%dT%H:%M:%S.%03d",
            "json_indent": False,
        },
    },
    "filters": {
        "ignore_excluded_paths": {
            "()": IgnoreExcludedPathsFilter,
        },
    },
    "handlers": {
        "console": {
            "formatter": "json",
            "filters": ["ignore_excluded_paths"],
            "class": "logging.StreamHandler",
            "stream": "ext://sys.stdout",
        },
    },
    "loggers": {},
}


def configure_logging(
    logging_dict: dict[str, Any] | None = None,
    app_log_level_map: dict[str, str] | None = None,
) -> None:
    logging_dict = logging_dict or deepcopy(LOGGING_DICT)
    logging_config = logging_dict.copy()
    if app_log_level_map:
        for app_name, log_level in app_log_level_map.items():
            if app_name not in logging_config["loggers"]:
                logging_config["loggers"][app_name] = {}
            logging_config["loggers"][app_name]["level"] = log_level
            logging_config["loggers"][app_name]["handlers"] = ["console"]

        logging.config.dictConfig(logging_config)

As you can see, I’m using JSON output in my project.

[galenkistler]

Assuming the json portion of the log line is valid JSON, you should just need to tweak the pattern to separate the json from the extra stuff, I didn’t see that the date isn’t part of your log line as the logs panel can be configured to show the date, try something like this:

{service_name="MY_SERVICE"} | pattern "<date> stdout F <json>" | line_format "{{.json}}" | json

You might need to tweak the pattern a bit if that <json> field is not extracting.

If you can’t get that working send me the full text of one of your logs and I’ll try to find some time to help