Grafana Agent Flow, Syslogs and Loki

Grafana Grafana Agent
1

Hi folks,

I’m deploying the agent in flow mode with the intent of collecting syslog messages and forwarding them to a Loki instance.

I’m new to flow mode, but from my understanding of the config I have (see below), everything appears to be configured correctly:

Agent flow configuration 
logging {
        level = "debug"
}

loki.source.syslog "syslog" {
        listener {
                address = "0.0.0.0:1514"
                labels  = {component = "loki.source.syslog", protocol = "tcp", customer = "CUSTOMER"}
        }

        listener {
                address  = "0.0.0.0:1514"
                protocol = "udp"
                labels   = {component = "loki.source.syslog", protocol = "udp", customer = "CUSTOMER"}
        }

        forward_to = [loki.write.default.receiver]
}

loki.write "default" {
        endpoint {
                url = "http://IPADDR:3100/api/v1/push"
        }
}

Looking in the Grafana Agent UI, all the components are showing as healthy, and the logs don’t appear to be showing any errors that would lead me to think that the agent can’t reach the Loki endpoint.

I’ve confirmed that I’ve got logs going into the listeners by doing a pcap and can see I’ve got plenty coming in on :1514/UDP. But there’s no outgoing traffic to the loki endpoint on :3100/tcp.

I’m at a bit of a loss as to why the agent doesn’t seem to be forwarding the logs and. I’m assuming that I’m missing something super simple :confused:

2

Bumping for visibility. Would love if someone could lend a bit of insight as to what I’m doing wrong.

3

Hello! There are a few things you could try:

4

“syslog” standard is weak and vague. I would say that each vendor may have own “syslog” implementation, which may be different from the RFC standard. So I guess your logs are not RFC5424 compliant, which is mandatory for Loki syslog source. They can be in older RFC 3164 format.

I would use OTEL collector, where you can accept RFC 3164 or RFC 5424 standard. Also you can prepare logs before RFC 3164/RFC 5424 parser. For example Grafana doesn’t produce syslog logs with correct time format, so you can “fix” vendor opinionated formatting and have RFC compliant logs: feature request: use rfc3164-compliant timestamps for syslog messages · Issue #72703 · grafana/grafana · GitHub

5

Thanks a ton, and you as well @paulintodev . Digging into it, I’ve got a bunch of Fortinet devices and am checking with our network team to see if they’ve configured the devices to send in RFC 5424. I’m assuming that that’s the problem. Thanks for pointing me in the right direction :smiley: