Delete field from extracted data map in regex stage

Grafana Alloy
,
1

Hello,

I have this config in Alloy:

logging {
	level  = "debug"
	format = "logfmt"
}

loki.source.syslog "syslog" {

  listener {
    address  = "0.0.0.0:5514"
    protocol = "tcp"
    use_rfc5424_message = true    
    labels   = { component = "loki.source.syslog", protocol = "tcp"}
  }
  forward_to = [loki.process.decode_syslog_labels.receiver]
}

//loki.source.syslog
loki.process "decode_syslog_labels" {

  stage.regex {
    expression = `^<(?P<pri>\d{1,3})>\d\s(?P<timestamp>\S+)\s(?P<hostname>\S+)\s(?P<appname>\S+)\s(?P<procid>\S+)\s(?P<msgid>\S+)\s(?P<structureddata>(-|\[.*\]))\s(?P<message>.*)`
  }
    
  forward_to = [loki.echo.example.receiver]
}

loki.echo "example" { }

After regex stage process one map with values above are created:

ts=2024-08-19T20:22:21.914957118Z level=debug msg="extracted data debug in regex stage" component_path=/ component_id=loki.process.decode_syslog_labels component=stage type=regex "extracted data"="map[appname:TMNX component:loki.source.syslog hostname:my-hostname message:497967 management USER-MINOR-cli_user_logout-2002 [backaggr]: User from 10.200.35.220 logged out msgid:- pri:171 procid:- protocol:tcp structureddata:- timestamp:2024-08-19T20:22:21Z]"

I want drop fields with value is “-”,

I tried this, but not works:

  stage.drop {
    source = "msgid"
    value  = "-"
  }

How to drop fields if value is “-”?

Tks,

Paulo

2

First, to answer your question, to drop a label you’d want to use stage.label_drop, see loki.process | Grafana Alloy documentation.

Below is my personal opinion. I’d advise you to not do it, because in my opinion it makes your log streams fundamentally different even though they should be the same.

Consider the logs below:

{"label1":"value1","label2":"value2"} this is log 1
{"label1":"value1","label2":"-"} this is log 2

and

{"label1":"value1","label2":"value2"} this is log 1
{"label1":"value1"} this is log 2

The second set is fundamentally different, and you may run into unanticipated issue when crafting queries. For example, how do you specifically find logs where label2 is empty?

I’d recommend keeping the label as -, and if you want to exclude those from query you can simply do label2!="-". Keeping the label gives you options, and personally I like having options.

3

Thanks for the answer.

What you’re saying really makes sense. Sorry, but I’m just starting out with Alloy and Loki.

Basically, what I want is to process the syslog messages from my devices, which currently come out in rfc3164 format, which I send to a syslog-ng that parses them and sends them to Alloy in rfc5424 format. Because of this, these parts of the message appear with the “-” character.

In the end, what I need to save in Loki is the message and some fields that I want to enrich the log.

With that in mind, it might be interesting not to save these fields that appear with the “-” character as labels, but rather to add them to the message label when they are not empty. What do you think?

4

I am not sure I quite understand. Can you provide an example or two, please?

5

Yes my friend.

I just put output:

stage.template {
source = "message"
template = "{{ .timestamp }} {{ .hostname }} {{ .appname }} {{ .procid }} {{ .msgid }} {{ .structureddata }} {{ .messageTmp }}"
}

Tks,

6

I see, then yes I would keep - in your message.

In general I’d say it would be a good idea to keep the format of both labels and the general message the same within the same log stream, so it’s easier to parse down the line.

7

Tks Tony