Getting labels from syslog msg field

Grafana Loki
1

Hi,

I’m relative new to setting up a syslog stream and have managed to fetch syslog messages from a device into Loki (3.0) via:

  • rsyslog using a template making it rfc5424 compliant
  • rsyslog forwarding it to Grafana Alloy agent
  • Grafana Alloy sending it to Loki

The stream works fine and I am able to read all the lines in Grafana using the Loki datasource.

Next challenge I face is to split up the syslog “msg” field in usable labels as values do not come as structured-data. Syslog msg field I get looks according the following format:

msg: ‘variable1: value1, variable2: value2, variable3: value3’ (and so on)

I have been going through a lot of documentation from Alloy, Loki and Rsyslog but I am unsure where I should be fixing this and especially how.

Any help to point me in the right direction would be very appreciated!

2

I would recommend you to do all your parsing in Alloy.

Do you have an example log and what the end result you want looks like?

3

You are already using rsyslog and Grafana Alloy to forward syslog messages to Loki, you could try modifying the syslog message format using rsyslog’s templates. If that doesn’t work, explore if Grafana Alloy offers any features for parsing messages. If both options fail, consider writing a custom parser to preprocess the messages before they reach Loki. Good luck!

4

Hi,

Here is a debug-output from rsyslog with the full (anonymized) message. Fields I’d like to be able to use as labels are for example IngressZone, EgressZone, AccessControlRuleName.

Debug line with all properties:
FROMHOST: ‘a.a.a.a’, fromhost-ip: ‘a.a.a.a’, HOSTNAME: ‘a.a.a.a’, PRI: 166,
syslogtag ‘’, programname: ‘’, APP-NAME: ‘-’, PROCID: ‘-’, MSGID: ‘-’,
TIMESTAMP: ‘Jun 11 11:29:21’, STRUCTURED-DATA: ‘-’,

msg: ’ : %ABC-1-123456: EventPriority: Low, DeviceUUID: abcd123-a00a-b11b-c22c-12abcd4567, InstanceID: 3, FirstPacketSecond: 2024-06-11T11:29:21Z, ConnectionID: 10111, AccessControlRuleAction: Block, SrcIP: x.x.x.x, DstIP: y.y.y.y, SrcPort: 137, DstPort: 137, Protocol: udp, IngressInterface: INSIDE, EgressInterface: OUTSIDE, IngressZone: Inside-Zone, EgressZone: Internet-Zone, SourceSecurityGroup: 4, SourceSecurityGroupTag: 4, SourceSecurityGroupType: Session Directory, IngressVRF: Global, EgressVRF: Global, ACPolicy: AA-BBB-ABC-Policy, AccessControlRuleName: block-netbios-port, Prefilter Policy: Prefilter-AAAA-Fix, Client: NetBIOS-ns, ApplicationProtocol: NetBIOS-ns, InitiatorPackets: 1, ResponderPackets: 0, InitiatorBytes: 96, ResponderBytes: 0, NAPPolicy: Balanced Security and Connectivity, NAT_InitiatorPort: 49515, NAT_ResponderPort: 137, NAT_InitiatorIP: z.z.z.z, NAT_ResponderIP: y.y.y.y, ClientAppDetector: AppID’

escaped msg: ’ : %ABC-1-123456: EventPriority: Low, DeviceUUID: abcd123-a00a-b11b-c22c-12abcd4567, InstanceID: 3, FirstPacketSecond: 2024-06-11T11:29:21Z, ConnectionID: 10111, AccessControlRuleAction: Block, SrcIP: x.x.x.x, DstIP: y.y.y.y, SrcPort: 137, DstPort: 137, Protocol: udp, IngressInterface: INSIDE, EgressInterface: OUTSIDE, IngressZone: Inside-Zone, EgressZone: Internet-Zone, SourceSecurityGroup: 4, SourceSecurityGroupTag: 4, SourceSecurityGroupType: Session Directory, IngressVRF: Global, EgressVRF: Global, ACPolicy: AA-BBB-ABC-Policy, AccessControlRuleName: block-netbios-port, Prefilter Policy: Prefilter-AAAA-Fix, Client: NetBIOS-ns, ApplicationProtocol: NetBIOS-ns, InitiatorPackets: 1, ResponderPackets: 0, InitiatorBytes: 96, ResponderBytes: 0, NAPPolicy: Balanced Security and Connectivity, NAT_InitiatorPort: 49515, NAT_ResponderPort: 137, NAT_InitiatorIP: z.z.z.z, NAT_ResponderIP: y.y.y.y, ClientAppDetector: AppID’

inputname: imudp rawmsg: ‘<166>2024-06-11T11:29:21Z : %ABC-1-123456: EventPriority: Low, DeviceUUID: abcd123-a00a-b11b-c22c-12abcd4567, InstanceID: 3, FirstPacketSecond: 2024-06-11T11:29:21Z, ConnectionID: 10111, AccessControlRuleAction: Block, SrcIP: x.x.x.x, DstIP: y.y.y.y, SrcPort: 137, DstPort: 137, Protocol: udp, IngressInterface: INSIDE, EgressInterface: OUTSIDE, IngressZone: Inside-Zone, EgressZone: Internet-Zone, SourceSecurityGroup: 4, SourceSecurityGroupTag: 4, SourceSecurityGroupType: Session Directory, IngressVRF: Global, EgressVRF: Global, ACPolicy: AA-BBB-ABC-Policy, AccessControlRuleName: block-netbios-port, Prefilter Policy: Prefilter-AAAA-Fix, Client: NetBIOS-ns, ApplicationProtocol: NetBIOS-ns, InitiatorPackets: 1, ResponderPackets: 0, InitiatorBytes: 96, ResponderBytes: 0, NAPPolicy: Balanced Security and Connectivity, NAT_InitiatorPort: 49515, NAT_ResponderPort: 137, NAT_InitiatorIP: z.z.z.z, NAT_ResponderIP: y.y.y.y, ClientAppDetector: AppID’

Since I also do some relabeling in Alloy it might make sense to do the parsing there too if possible.

5

Thanks, will look further into parsing via rsyslog templates too.

6

Managed to fix it for now (not sure yet if it is optimal) via loki.process in Alloy configuration. Here follows an example with just one field:

loki.process “syslog” {
forward_to = [
loki.write.syslog.receiver,
]
stage.regex {
expression = “.*AccessControlRuleAction: (?P<action>([^,]+)?)”
}
stage.labels {
values = {
action = “action”,
}
}
}

Have also tried adjusting rsyslog configuration but was unable to translate it to proper input for the alloy job.

Thanks for the suggestions!

7

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.