Issue Parsing Component and Level Fields in Loki + VictoriaLogs Pipeline

Grafana Alloy
, , ,
1

Hi all,

I’m trying to parse pod logs using Loki and VictoriaLogs, extracting the component, date, time, level, and message. Here’s what I’ve set up so far:

Sample log:

<COMPONENT>-<HHMMSS.MS-YYMMDD-SEQ-LEVEL: msg
APP-POOL-170357.020-250826-7-DBG1: reInit() level 1

Pipeline config:

loki.process “pod_logs” {stage.cri {}

stage.regex {expression = “^(?P[A-Z]+(?:-[A-Z]+))-(?P\d{6}\.\d{3})-(?P\d{6})-(?P\d+)-(?P[A-Z0-9]+): (?P.)$”}

stage.timestamp {format = “RFC3339”source = “2025-08-26T17:03:57.022Z”}

stage.labels {values = {“component” = “component”,“level”     = “level”,}}

stage.output {source = “message”}

forward_to = [loki.write.victorialogs.receiver,]}

Actual output in VictoriaLogs:

{
“_msg”: “app-105642.321-250908-1-DBG1: took 16158.545 seconds\n”,
“_stream”: “{app=“app”,container=“app”,instance=“test/app-6c8b9989b6-jcmxd:app”,job=“test/app”,namespace=“test”,node_name=“crc-j5m2n-master-0”,pod=“app-6c8b9989b6-jcmxd”}”,
“_stream_id”: “0000000000000000a705063a76c004ee5c5dbc25b557babf”,
“_time”: “2025-09-08T10:56:42.321935372Z”,
“app”: “app”,
“container”: “app”,
“instance”: “test/test-6c8b9989b6-jcmxd:app”,
“job”: “test/app”,
“namespace”: “test”,
“node_name”: “”,
“pod”: “app-6c8b9989b6-jcmxd”
}

Issue:

  • The component and level fields are not showing in VictoriaLogs.

  • I also see debug logs like:

{“ts”:“2025-09-08T11:15:18.259530955Z”,“level”:“debug”,“msg”:“extracted data did not contain a timestamp”,“component_path”:“/”,“component_id”:“loki.process.pod_logs”}
{“ts”:“2025-09-08T11:15:18.259582911Z”,“level”:“debug”,“msg”:“extracted data did not contain output source”,“component_path”:“/”,“component_id”:“loki.process.pod_logs”}

I’m not sure why my regex or label extraction isn’t working correctly. How can I fix this so that component and level are properly parsed and sent to VictoriaLogs?

config: https://grafana.github.io/alloy-configurator/?c=Ly8gV2VsY29tZSB0byB0aGUgR3JhZmFuYSBBbGxveSBDb25maWd1cmF0b3IhCi8vIFlvdSBjYW4gcGFzdGUgeW91ciBjb25maWd1cmF0aW9uIGhlcmUgb3Igc3RhcnQgYnkgdXNpbmcgdGhlIGNvbmZpZ3VyYXRpb24gd2l6YXJkIG9yIGJ5IGxvYWRpbmcgYW4gZXhhbXBsZSBmcm9tIHRoZSBjYXRhbG9nLgoKbG9nZ2luZyB7CiAgbGV2ZWwgID0gImRlYnVnIgp9CgpkaXNjb3Zlcnkua3ViZXJuZXRlcyAicG9kcyIgewogIHJvbGUgPSAicG9kIgp9CgpkaXNjb3ZlcnkucmVsYWJlbCAicG9kX2xvZ3MiIHsKICB0YXJnZXRzID0gZGlzY292ZXJ5Lmt1YmVybmV0ZXMucG9kcy50YXJnZXRzCgogIHJ1bGUgewogICAgc291cmNlX2xhYmVscyA9IFsiX19tZXRhX2t1YmVybmV0ZXNfcG9kX25hbWUiXQogICAgcmVnZXggICAgICAgICA9ICIuKmRlbGV0ZWQuKiIKICAgIGFjdGlvbiAgICAgICAgPSAia2VlcCIKICB9CgogIHJ1bGUgewogICAgc291cmNlX2xhYmVscyA9IFsiX19tZXRhX2t1YmVybmV0ZXNfcG9kX2NvbnRyb2xsZXJfbmFtZSJdCiAgICByZWdleCAgICAgICAgID0gIihbMC05YS16LS5dKz8pKC1bMC05YS1mXXs4LDEwfSk%2FIgogICAgdGFyZ2V0X2xhYmVsICA9ICJfX3RtcF9jb250cm9sbGVyX25hbWUiCiAgfQoKICBydWxlIHsKICAgIHNvdXJjZV9sYWJlbHMgPSBbCiAgICAgICJfX21ldGFfa3ViZXJuZXRlc19wb2RfbGFiZWxfYXBwX2t1YmVybmV0ZXNfaW9fbmFtZSIsCiAgICAgICJfX21ldGFfa3ViZXJuZXRlc19wb2RfbGFiZWxfYXBwIiwKICAgICAgIl9fdG1wX2NvbnRyb2xsZXJfbmFtZSIsCiAgICAgICJfX21ldGFfa3ViZXJuZXRlc19wb2RfbmFtZSIsCiAgICBdCiAgICByZWdleCAgICAgICAgPSAiXjsqKFteO10rKSg7LiopPyQiCiAgICB0YXJnZXRfbGFiZWwgPSAiYXBwIgogIH0KCiAgcnVsZSB7CiAgICBzb3VyY2VfbGFiZWxzID0gWwogICAgICAiX19tZXRhX2t1YmVybmV0ZXNfcG9kX2xhYmVsX2FwcF9rdWJlcm5ldGVzX2lvX2luc3RhbmNlIiwKICAgICAgIl9fbWV0YV9rdWJlcm5ldGVzX3BvZF9sYWJlbF9pbnN0YW5jZSIsCiAgICBdCiAgICByZWdleCAgICAgICAgPSAiXjsqKFteO10rKSg7LiopPyQiCiAgICB0YXJnZXRfbGFiZWwgPSAiaW5zdGFuY2UiCiAgfQoKICBydWxlIHsKICAgIHNvdXJjZV9sYWJlbHMgPSBbCiAgICAgICJfX21ldGFfa3ViZXJuZXRlc19wb2RfbGFiZWxfYXBwX2t1YmVybmV0ZXNfaW9fY29tcG9uZW50IiwKICAgICAgIl9fbWV0YV9rdWJlcm5ldGVzX3BvZF9sYWJlbF9jb21wb25lbnQiLAogICAgXQogICAgcmVnZXggICAgICAgID0gIl47KihbXjtdKykoOy4qKT8kIgogICAgdGFyZ2V0X2xhYmVsID0gImNvbXBvbmVudCIKICB9CgogIHJ1bGUgewogICAgc291cmNlX2xhYmVscyA9IFsiX19tZXRhX2t1YmVybmV0ZXNfcG9kX25vZGVfbmFtZSJdCiAgICB0YXJnZXRfbGFiZWwgID0gIm5vZGVfbmFtZSIKICB9CgogIHJ1bGUgewogICAgc291cmNlX2xhYmVscyA9IFsiX19tZXRhX2t1YmVybmV0ZXNfbmFtZXNwYWNlIl0KICAgIHRhcmdldF9sYWJlbCAgPSAibmFtZXNwYWNlIgogIH0KCiAgcnVsZSB7CiAgICBzb3VyY2VfbGFiZWxzID0gWyJuYW1lc3BhY2UiLCAiYXBwIl0KICAgIHNlcGFyYXRvciAgICAgPSAiLyIKICAgIHRhcmdldF9sYWJlbCAgPSAiam9iIgogIH0KCiAgcnVsZSB7CiAgICBzb3VyY2VfbGFiZWxzID0gWyJfX21ldGFfa3ViZXJuZXRlc19wb2RfbmFtZSJdCiAgICB0YXJnZXRfbGFiZWwgID0gInBvZCIKICB9CgogIHJ1bGUgewogICAgc291cmNlX2xhYmVscyA9IFsiX19tZXRhX2t1YmVybmV0ZXNfcG9kX2NvbnRhaW5lcl9uYW1lIl0KICAgIHRhcmdldF9sYWJlbCAgPSAiY29udGFpbmVyIgogIH0KCiAgcnVsZSB7CiAgICBzb3VyY2VfbGFiZWxzID0gWyJfX21ldGFfa3ViZXJuZXRlc19wb2RfbmFtZSIsICJfX21ldGFfa3ViZXJuZXRlc19uYW1lc3BhY2UiLCAiX19tZXRhX2t1YmVybmV0ZXNfcG9kX2NvbnRhaW5lcl9uYW1lIl0KICAgIHNlcGFyYXRvciAgICAgPSAiXyIKICAgIHRhcmdldF9sYWJlbCAgPSAiX19wYXRoX18iCiAgICByZXBsYWNlbWVudCAgID0gIi92YXIvbG9nL2NvbnRhaW5lcnMvJHsxfV8kezJ9XyR7M30tKi5sb2ciCiAgfQp9Cgpsb2tpLnNvdXJjZS5rdWJlcm5ldGVzICJwb2RfbG9ncyIgewogIHRhcmdldHMgICAgPSBkaXNjb3ZlcnkucmVsYWJlbC5wb2RfbG9ncy5vdXRwdXQKICBmb3J3YXJkX3RvID0gW2xva2kucHJvY2Vzcy5wb2RfbG9ncy5yZWNlaXZlcl0KfQoKbG9raS5wcm9jZXNzICJwb2RfbG9ncyIgewogICBzdGFnZS5jcmkge30KCiAgc3RhZ2UucmVnZXggewogICAgc291cmNlICAgICA9ICJsb2ciCiAgICBleHByZXNzaW9uID0gYF4oP1A8YXBwPltBLVpdKyktKD9QPHRpbWU%2BXGR7Nn0pXC4oP1A8bWlsbGlzPlxkezN9KS0oP1A8ZGF0ZT5cZHs2fSktKD9QPHNlcXVlbmNlPlxkKyktKD9QPGxvZ2xldmVsPlx3Kyk6ICg%2FUDxtZXNzYWdlPi4rKSRgCiAgICBsYWJlbHNfZnJvbV9ncm91cHMgPSAgdHJ1ZQogIH0KCiAgc3RhZ2UudGVtcGxhdGUgewogICAgc291cmNlICAgPSAiY29tcG9uZW50IgogICAgdGVtcGxhdGUgPSAie3sgZGVmYXVsdCBcInVua25vd25fY29tcG9uZW50XCIgLmNvbXBvbmVudCB9fSIKICB9CgogIHN0YWdlLnRlbXBsYXRlIHsKICAgIHNvdXJjZSAgID0gImxldmVsIgogICAgdGVtcGxhdGUgPSAie3sgZGVmYXVsdCBcIklORk9cIiAubGV2ZWwgfX0iCiAgfQoKICBzdGFnZS5sYWJlbHMgewogICAgdmFsdWVzID0gewogICAgICBjb21wb25lbnQgPSAie3sgLmNvbXBvbmVudCB9fSIsCiAgICAgIGxldmVsICAgICA9ICJ7eyAubGV2ZWwgfX0iLAogICAgfQogIH0KCiAgc3RhZ2Uub3V0cHV0IHsKICAgIHNvdXJjZSAgID0gIm1lc3NhZ2UiCiAgfQoKICBmb3J3YXJkX3RvID0gWwogICAgbG9raS53cml0ZS52aWN0b3JpYWxvZ3MucmVjZWl2ZXIsCiAgXQp9CgoKbG9raS53cml0ZSAidmljdG9yaWFsb2dzIiB7CiAgZW5kcG9pbnQgewogICAgdXJsICAgICAgID0gImh0dHA6Ly92bHMtdmljdG9yaWEtbG9ncy1zaW5nbGUtc2VydmVyOjk0MjgvaW5zZXJ0L2xva2kvYXBpL3YxL3B1c2giCiAgICB0ZW5hbnRfaWQgPSAiMDowIgogIH0KICBleHRlcm5hbF9sYWJlbHMgPSB7fQp9Cgo%3D

2
  1. Since you have cri stage, can you share an actual original log with and without the Docker part?
  2. In your regex, try changing double quote to `
1 Like
3

Sample log with and without the Docker part:

2025-08-26T15:03:57.022256111+00:00 stderr F APP-POOL-170357.020-250826-7-DBG1: Initialization level 1
2025-08-26T15:03:57.022256111+00:00 stderr F APP-170357.020-250826-7-DBG1: Initialization started
2025-08-26T01:57:02.561000000+00:00 stdout F APP-0357022.561-250808-1-INFO: Connection string:
2025-08-26T01:57:02.561000000+00:00 stdout F failover:(tcp://testsvc.apps.testing:8080,tcp://testsvc-1-svc-rte-test.apps.testing:8080)

APP-POOL-170357.020-250826-7-DBG1: Initialization level 1
APP-170357.020-250826-7-DBG1: Initialization started
APP-0357022.561-250808-1-INFO: Connection string:
failover:(tcp://testsvc.apps.testing:8080,tcp://testsvc-1-svc-rte-test.apps.testing:8080)

4

updated to single quotes as well. but its the same , not showing the extracted fields.

5

Consider this line of log:

Which part is supposed to be component and which part is level?

1 Like
6

component: APP-POOL
time : 170357.020 ==> HHMMSS.millseconds
date : 250826 ==> YYMMDD
sequnce no: 7
level: DBG1
msg: Initialization level 1

7

here is one way you might tackle it is by using pattern to parse out your log post ingest, simplifies things instead of complicated regex on the config level

image
image1389×117 13.6 KB

image
image830×586 49.5 KB

8

Try something like this:

stage.cri {}

  stage.regex {
    expression = `^(?P<parsed_component>[^\d]+)-(?P<parsed_time>\d{6}\.\d{3}-\d{6})-\d+-(?P<parsed_level>[^\:]+)`
  }

  stage.labels {
    values = {
      "component" = "parsed_component",
      "level" = "parsed_level",
    }
  }

Things to note:

  1. Your regex group capture had no name, you need a name so they can be referenced later.
  2. stage.cri already sets the timestamp and log output for you, you don’t need to do it separately unless you wish to overwrite it.
1 Like