Can anyone help with merge regex&match block?

Grafana Alloy
1

Hi team, i have a little problem with merge templating my loki.process block

So.. For first time i grab more logs from my test k8s and i need to parse some words

      loki.source.kubernetes "pods" {
        targets    = discovery.relabel.pod_logs.output
        forward_to = [loki.process.cleanless.receiver]
      }

      loki.process "cleanless" {
        forward_to = [loki.process.process_error.receiver]

        stage.drop {
          older_than          = "1h"
          drop_counter_reason = "too old"
        }
        
        stage.label_drop {
          values = [ "job", "service_name", "pod_label_pod_template_generation", "pod_label_controller_revision_hash"  ]
        }
      }

So then i checked for words like “FAILED/failed/FATAL/fatal” and mark them as “level=error” (for Grafana view)
And after i need to find words like “[INFO]/[DEBUG]” and mark them as “level=info”
Template for strings, marks as level=info:

[INFO] 10.10.15.123:32885 - 6242 "AAAA IN test.vz.ru. udp 47 false 1232" NOERROR qr,aa,rd,ra 124 0.000039094s

Template for strings, marks as level=error:

failed to create fsnotify watcher: too many open files
E0317 12:54:57.171829       1 cacher.go:470] cacher (vcdmachines.infrastructure.cluster.x-k8s.io): unexpected ListAndWatch error: failed to list infrastructure.cluster.x-k8s.io/v1alpha4, Kind=VCDMachine: conversion webhook for infrastructure.cluster.x-k8s.io/v1beta2, Kind=VCDMachine failed: Post "https://capvcd-webhook-service.capvcd-system.svc:443/convert?timeout=30s": service "capvcd-webhook-service" not found; reinitializing...

I’m trying to write something like

      loki.process "process_error" {
        forward_to = [loki.process.process_info.receiver]

        stage.regex {
          expression = ".*\\s+(?P<msglevel>failed|fatal).*"
        }

        stage.labels {
          values = {
            msglevel = "",
          }
        }

        stage.match {
          selector = "{msglevel=~\"(failed|fatal)\"}"

          stage.static_labels {
            values = {
              level = "error",
              levelcatch = "true",
            }
          }
        }
      }

But it’s not work. I’m checked this on regex101, and it seems like everything should work - but not. Maybe problem with brackets, maybe with \\s+, idk
But if i use regex with .* on start/end and use a word that i need, all lines start catching
And if i listing all words in stage.match with query like selector = "{msglevel=\"fatal\"}", this stage is work. The regular expression does not work for me =(

I use simple design like

      loki.process "process_error" {
        forward_to = [loki.process.process_info.receiver]

        stage.regex {
          expression = ".*(?P<msglevel>failed|fatal).*"
        }

        stage.labels {
          values = {
            msglevel = "",
          }
        }

        stage.match {
          selector = "{msglevel=\"failed\"}"

          stage.static_labels {
            values = {
              level = "error",
              levelcatch = "true",
            }
          }
        }
        stage.match {
          selector = "{msglevel=\"fatal\"}"

          stage.static_labels {
            values = {
              level = "error",
              levelcatch = "true",
            }
          }
        }
      }

but it’s not comfortable. Can anyone help me with explain and help configure this regex please ?

2

If you only care about a couple of keywords, why not match the logline for the keywords instead of trying to match each one of them individually?

For example:

loki.process "process_error" {
  forward_to = [loki.process.process_info.receiver]

  // This is just an arbitrary label for the match block so we can filter through all logs
  stage.labels {
    values = {
      level_match  = "true",
    }
  }

  stage.match {
    selector = "{level_match=\"true\"} |~ \"(?i)failed|fatal\""

    stage.static_labels {
      values = {
        level = "error",
        levelcatch = "true",
      }
    }
  }

  stage.match {
    selector = "{level_match=\"true\"} |~ \"(?i)info|debug\""

    <...>
  }

  stage.label_drop {
    values = ["level_match"]
  }
}

Also please share an example of your logs.

1 Like
3

I just want to match, but together the lines don’t seem to be processed

Here is example of logs:
For error:

E0319 10:46:23.652506       1 cacher.go:470] cacher (kubeadmconfigs.bootstrap.cluster.x-k8s.io): unexpected ListAndWatch error: failed to list bootstrap.cluster.x-k8s.io/v1alpha3, Kind=KubeadmConfig: conversion webhook for bootstrap.cluster.x-k8s.io/v1beta1, Kind=KubeadmConfig failed: Post "https://capi-kubeadm-bootstrap-webhook-service.capi-kubeadm-bootstrap-system.svc:443/convert?timeout=30s": service "capi-kubeadm-bootstrap-webhook-service" not found; reinitializing...
W0319 10:46:23.652493       1 reflector.go:535] storage/cacher.go:/bootstrap.cluster.x-k8s.io/kubeadmconfigs: failed to list bootstrap.cluster.x-k8s.io/v1alpha3, Kind=KubeadmConfig: conversion webhook for bootstrap.cluster.x-k8s.io/v1beta1, Kind=KubeadmConfig failed: Post "https://capi-kubeadm-bootstrap-webhook-service.capi-kubeadm-bootstrap-system.svc:443/convert?timeout=30s": service "capi-kubeadm-bootstrap-webhook-service" not found

failed to create fsnotify watcher: too many open files

error - no org id

For info:

W0319 13:46:22.994956       1 connection.go:172] Still connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock

[{"date":1742381180.481097,"log":"10.10.1.115 - - [19/Mar/2025:13:46:20 +0300] \"GET / HTTP/1.1\" 200 391 \"-\" \"kube-probe/1.28\""}]

Trace[2127454126]: ---"Writing http response done" 3599685ms (10:46:10.845)

2025-03-19T10:46:09.171247Z  INFO stackable_operator::logging::controller: Reconciled object controller.name="statefulset.restarter.commons.stackable.tech" object=StatefulSet.v1.apps/simple-superset-node-default.superset-v2

I know that format of logs is strange, but it’s my second headache

4

For the 4 info logs you provided only one of them has the word “info” in the log stream.

5

Sorry for the long reply,
I know about log stream, but in some cases grafana notes some lines as error (but it shouldn’t be like this).

I went back to configure alloy and you helped a lot - thank you very much!
This block works exactly as I need it.

      loki.process "process_error" {
        forward_to = [loki.relabel.default.receiver]

        stage.static_labels {
          values = {
            level_match  = "true",
          }
        }

        stage.match {
          selector = "{level_match=\"true\"} |~ \"(?i)failed|fatal\""

          stage.static_labels {
            values = {
              level = "error",
              levelcatch = "true",
            }
          }
        }

        stage.match {
          selector = "{level_match=\"true\"} |~ \"(?i)info|debug\""

          stage.static_labels {
            values = {
              level = "info",
              levelcatch = "true",
            }
          }
        }
        stage.label_drop {
          values = ["level_match"]
        }
      }