Journald fields in loki

Grafana Loki
1

I have multiple services and journald is used and each log entry has different fields, I would like each log entry to be displayed in loki/grafana.

Right now only “msg” field is being displayed.

How can I remove all the system fields that are prefixed by “_” and show the logs in grafana as:

For example (passing journalctl -o json-pretty):

    {
            "ROLE" : "manager",
            "_UID" : "1001",
            "MESSAGE" : "Some short message",
            "DUTYSLOT" : "123",
            "SLOTINEPOCH" : "10",
            "_PID" : "34779",
            "_GID" : "1001",
            "_SELINUX_CONTEXT" : "unconfined\n",
            "__REALTIME_TIMESTAMP" : "1608325979002140",
            "_COMM" : "run-v1.0.",
            "CURRENTSLOT" : "125163",
            "__CURSOR" : "1",
            "_HOSTNAME" : "server",
            "_CMDLINE" : "/home/mo/run.sh --config-file=/etc/run/run.yaml",
            "__MONOTONIC_TIMESTAMP" : "1",
            "_SOURCE_REALTIME_TIMESTAMP" : "1",
            "_MACHINE_ID" : "1",
            "_SYSTEMD_CGROUP" : "/system.slice/run.service",
            "_SYSTEMD_INVOCATION_ID" : "1",
            "PREFIX" : "run",
            "PRIORITY" : "6",
            "_CAP_EFFECTIVE" : "0",
            "_TRANSPORT" : "journal",
            "_EXE" : "/home/mo/run/run-v1.0.5-linux-amd64",
            "_BOOT_ID" : "1",
            "SECONDSLEFT" : "48",
            "_SYSTEMD_UNIT" : "run.service",
            "_SYSTEMD_SLICE" : "system.slice"
    }

It looks like this only in grafana:

Some short message
Some short message
Some short message
Some short message

Instead I want to show the fields as well, removing everything with _ and all system fields:

{
        "MESSAGE" : "Some short message",
        "DUTYSLOT" : "123",
        "SLOTINEPOCH" : "10",
        "CURRENTSLOT" : "125163",
        "SECONDSLEFT" : "48",
}

And Grafana can show them in one single line:

Some short message, DUTYSLOT=123, SLOTINEPOCH=10, CURRENTSLOT=125163, SECONDSLEFT=48
Some short message, DUTYSLOT=123, SLOTINEPOCH=10, CURRENTSLOT=125163, SECONDSLEFT=60
Some short message, DUTYSLOT=123, SLOTINEPOCH=10, CURRENTSLOT=125163, SECONDSLEFT=80

How can I do something like that? Right now my grafana loki is missing a lot of fields. I have hundreds of fields I want to expose cause every service has their own fields.

2

I think one option which would get you closer to what you want is to set json: true in your promtails journal config section. This will marshal all the fields to json and put them in the log line.

This isn’t exactly what you want however as it will include every field, however you can use the Loki 2.0 parsing features to work with this on the query side.

As a rough example:

{job="systemd/journal"} | json | line_format "{{ .message }} {{ .dutyslot }} {{ .currentslot}}"
3

Unfortunately this will not work because I have multiple types of logs and each type has different fields. The solution you recommended will only work for one of the types.

The feature request that would be nice to see is the ability to add an additional output format to loki which would format it as a comma separated string like in this order(msg then user fields then system fields):

msg=foo, fieldA=bar, fieldB=ok, __system=internal

Another feature request would be to filter out system fields by supplying another param.

By including these two feature requests we can have a clean linear log viewing if needed.

msg=foo, fieldA=bar, fieldB=ok

4

I think the JSON config in the journal doesn’t care what the fields are, it would turn the entire message and all the fields into JSON