Extract dynamic structured metadata from json log message

Grafana Alloy
1

I have a .net application inside a docker container which produces json formatted log messages. In addition to the regular fields, the json formatted messages may include a list of “scopes”. Each scope is an object that includes one “Message” property, and N other properties with names and values that we do not know in advance.

”EverthingElse”: {},
”Scopes”: [
{
"Message": "SpanId:a7bf44b686481585, TraceId:8c6a47b50a450da605f5c7371899b1b6, ParentId:0000000000000000",
"SpanId": "a7bf44b686481585",
"TraceId": "8c6a47b50a450da605f5c7371899b1b6",
"ParentId": "0000000000000000"
},
{
"Message": "ConnectionId:0HNHCGOH5DK7J",
"ConnectionId": "0HNHCGOH5DK7J"
},
{
"Message": "RequestPath:/metrics RequestId:0HNHCGOH5DK7J:00000001",
"RequestId": "0HNHCGOH5DK7J:00000001",
"RequestPath": "/metrics"
}
]

I would like to process each of the key value pairs in each of the scopes and add them to the structured metadata produced by the alloy loki pipeline, ie, have each of the following KV pairs:

{
"RequestId": "0HNHCGOH5DK7J:00000001",
"RequestPath": "/metrics"
"ConnectionId": "0HNHCGOH5DK7J"
"SpanId": "a7bf44b686481585",
"TraceId": "8c6a47b50a450da605f5c7371899b1b6",
"ParentId": "0000000000000000"
}

I can extract the scopes easily enough using a loki.process block with stage.json and stage.regex

stage.json {
expressions = {
"level" = "LogLevel",
"category" = "Category",
"scopes" = "Scopes",
}
}

stage.regex {
source = "scopes"
expression = "(?P<key>\"(?!Message)\\w+\"): (?P<value>\".+\")"
}

Now how do I pack each of the capture groups into the structured metadata as key value pairs?

2

This calls for a programmatic, and I don’t think you can do this with Alloy. The only log agent I know that supports running inline script is logstash.

I would approach this with a script, to be honest.