Extract log level via regex and set it as a label

Grafana Alloy
1

Hi there,

I used to extract my logs level through logfmt with the code below:

loki.process "add_dynamic_label" {
    stage.logfmt {
        mapping = { "extracted_level" = "level" }
    }

    stage.labels {
        values = { "level" = "extracted_level" }
    }

    forward_to = [loki.write.local_loki.receiver]
}

This works but unfortunately, somehow Alloy reports lots of logfmt errors:

msg=“failed to decode logfmt” component_path=/ component_id=loki.process.add_dynamic_label component=stage type=logfmt err=“logfmt syntax error at pos 117 on line 1: unexpected ‘=’”
msg=“failed to decode logfmt” component_path=/ component_id=loki.process.add_dynamic_label component=stage type=logfmt err=“logfmt syntax error at pos 46 on line 1: unexpected ‘"’”

After doing some researches, those errors do not appear to be fixable for the time being.
Nonetheless, I have decided to go down the regex path to extract my log levels but I am failing. Here is my config:

loki.process "add_dynamic_label" {
    stage.regex {
        expression = `(?i)\b(info|debug|error|warn|trace|fatal)\b`
    }

    stage.labels {
        values = { "level" = "" }
    }

    forward_to = [loki.write.local_loki.receiver]
}

Would any body know how I can extract the log level and assign it to the label “level” with Alloy?

2

Can you provide some examples of your logs, please?

3

Hi Tony,

Thank you for helping.
My logs are simply coming from Journald:

> journalctl
Oct 14 05:14:34 Lxde zeitgeist-fts[2693]: Error releasing name org.gnome.zeitgeist.SimpleIndexer: The connection is closed
Oct 14 05:14:34 Lxde zeitgeist-fts[2693]: zeitgeist-fts.vala:252: The connection is closed
Oct 14 05:14:34 Lxde org.gnome.zeitgeist.Engine[2678]: [16:14:34.416492 WARNING] zeitgeist-daemon.vala:453: The connection is closed
-- Boot 2c84a3183511457babcdbe528a957ef7 --
Oct 14 09:22:02 Lxde xrdp-sesman[1464]: [INFO ] Starting X server on display 10: /usr/lib/xorg/Xorg :10 -auth .Xauthority -config xrdp/xorg.conf -noreset -nolisten tcp -logfile .xorgxrdp.%s.log
Oct 14 09:22:03 Lxde xrdp-sesman[1463]: [INFO ] Found X server running at /tmp/.X11-unix/X10
Oct 14 09:22:03 Lxde xrdp-sesman[1463]: [INFO ] Starting the default window manager on display 10: /etc/xrdp/startwm.sh
Oct 14 09:22:03 Lxde xrdp-chansrv[1472]: [INFO ] Socket 12: AF_UNIX connection received
Oct 14 09:22:03 Lxde dbus-daemon[1494]: [session uid=10000 pid=1491] AppArmor D-Bus mediation is enabled
Oct 14 09:22:03 Lxde xrdp-chansrv[1472]: [INFO ] sound_process_training: round trip time 100
Oct 14 09:22:03 Lxde dbus-daemon[1552]: [session uid=10000 pid=1550] AppArmor D-Bus mediation is enabled
Oct 14 09:22:03 Lxde dbus-daemon[1552]: [session uid=10000 pid=1550] Activating service name='org.gtk.vfs.Daemon' requested by ':1.2' (uid=10000 pid=1581 comm="/usr/bin/ibus-daemon --daemonize --xim " label="unconfined")
Oct 14 09:22:03 Lxde dbus-daemon[1552]: [session uid=10000 pid=1550] Successfully activated service 'org.gtk.vfs.Daemon'
Oct 14 09:22:03 Lxde dbus-daemon[1552]: [session uid=10000 pid=1550] Activating service name='org.freedesktop.portal.IBus' requested by ':1.2' (uid=10000 pid=1581 comm="/usr/bin/ibus-daemon --daemonize --xim " label="unconfined")
Oct 14 09:22:03 Lxde dbus-daemon[1552]: [session uid=10000 pid=1550] Successfully activated service 'org.freedesktop.portal.IBus'

I can clearly see now that some do not have a log level. Would it be possible to set a default label like “unknown” when there are not any identifiable level?

Cheers,

4

You are not doing a group capture, so when you reference level in your label stage it’s not created anywhere. You can also add a template stage to insert a default value. Try this (not tested):

stage.regex {
  expression = `(?P<level>(?i)\b(info|debug|error|warn|trace|fatal)\b)`
}

stage.template {
    source   = "level_cleansed"
    template = `{{ default "INFO" .level }}`
}

stage.labels {
  values = { "level" = "level_cleansed" }
}
5

Hi Tony,

It works. However, labels are created for both upper and lower case.

image
image523×206 31.9 KB

Would you know if there is a way to have everything lower case?

Cheers,

6

I think the best approach would probably be to change the label during processing to make everything lower case. Maybe something like this (not tested):

stage.regex {
  expression = `(?P<level>(?i)\b(info|debug|error|warn|trace|fatal)\b)`
}

stage.template {
    source   = "level_cleansed"
    template = `{{ default "INFO" .level | ToLower }}`
}

stage.labels {
  values = { "level" = "level_cleansed" }
}
7

Hi Tony,

Thank you, it works perfectly :slight_smile:

8

hi @tonyswumac isn’t there a free online tool that one can use to vet out alloy? do you remember the url for it?

9

There is a config generator, but I don’t know if it actually verifies configuration. It does show the flow graph though, that’s somewhat useful.

10

also this is a very nice tool