Grafana Alloy | Filter log lines and transforming to JSON

Grafana Alloy
1

Hi Community,

We have a use case to filter a specific log from all log messages which like which looks like this “Critical Message – [Service:ServiceName, Title:Service Started, ErrorCode:CH0000, Severity:INFO, Date:2024-07-19-12-36+0000] Service Version Service Instance restarted” and transform it to json to extact values of key like “Service”,“Title” etc.

i am trying to solve using loki.process, but no luck as of now.
loki.process “alert_logs” {
forward_to = null
stage.pack {
labels = [“filename”,“hostname”,“type”]
}
stage.json {
expressions = {logline = “_entry”}
}

    stage.json {
      source      = "logline"
      expressions = {message = ""}
    }
    stage.labels {
      values = { "message" = "" }
    }
    stage.match{
      selector  = "{message=\"Critical Message --\"}"
      action = "keep"
      drop_counter_reason = "non-alert log"
      stage.output {
          source = "message"
      }
    }
   stage.regex {
       expression = `Critical Message -- \[Service:ServiceName, Title:(?P<title>.*), ErrorCode:(?P<errorCode>.*), Severity:(?P<severity>.*), Date:(?P<datetime>.*?)\] (?P<msg>.*)`
    }
}

Also is there any way to debug loki.process stages?

2

Your log is not in JSON format, so you cannot use json filter here. Just use regex with group capture.

3

Thanks for suggesting. I am looking for logs filtering solution where I can keep only selected logs and drop all remaining. I have checked sub component under loki.process, stage.drop take expression to filter logs to drop, I guess it do not support negate expression to drop all other. Second option may be stage.match but still I see it passes all log line to forward list of Loki.process. can you suggest what to use in this case or am I missing some bacis concept.

4

Can you provide some example log and what exactly you are trying to achieve?

One more thing to keep in mind is that the regex filter is RE2 under Golang (see Syntax · google/re2 Wiki · GitHub), with its own caveats.