Rsyslog to Alloy loki.source.syslog with TLS: Lose some logs (and "omfwd: TCPSendBuf error -2078, destruct TCP Connection")

Grafana Alloy
1

I have an Alloy server listening for syslog logs:

loki.source.syslog "remote_syslog" {
  forward_to    = [loki.write.grafana_loki.receiver]
  relabel_rules = loki.relabel.remote_syslog.rules
  listener {
    address = "0.0.0.0:1514"
    tls_config {
      cert_file = "/etc/pki/realms/domain/default.crt"
      key_file = "/etc/pki/realms/domain/default.key"
    }
  }
}

And my servers have Rsyslog configured to send logs to Alloy:

cat /etc/rsyslog.d/00forward-logs-loki.conf
*.* action(
  type="omfwd"
  target="loki.example.com"
  port="1514"
  protocol="tcp"
  Template="RSYSLOG_SyslogProtocol23Format"
  TCP_Framing="octet-counted"
  queue.type="linkedList"
  queue.size="10000"
  action.resumeRetryCount="100"
  streamDriver="gtls"
  streamDriverMode="1"
  streamDriverAuthMode="x509/name"
  streamDriverPermittedPeers="*.example.com"
)

It kinda works, but quite often it’s losing logs, and Rsyslog complains:

rsyslogd: omfwd: TCPSendBuf error -2078, destruct TCP Connection to loki.example.com:1514 [v8.2312.0 try https://www.rsyslog.com/e/2078 ]
omfwd: TCPSendBuf error -2078, destruct TCP Connection to loki.example.com:1514 [v8.2312.0 try https://www.rsyslog.com/e/2078 ]
action 'action-0-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2312.0 try https://www.rsyslog.com/e/2007 ]
action 'action-0-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2312.0 try https://www.rsyslog.com/e/2359 ]

Servers are, at the same time, also sending their logs to a central Rsyslog server syslog.example.com (which is actually only a CNAME to the same host than loki.example.com), and this flow is not losing any log, so I feel the problem is coming from Alloy.

Am I doing something wrong?

2

I don’t use rsyslog with alloy much, but was there any error message from Alloy when this was happening?

3

Thanks for your reply @tonyswumac. AFAICT, only Rsyslog is logging the problem, nothing on Alloy side, with level = "info". I tried with debug, but then it’s logging massive amount of logs, and even Loki cannot show them (message net/http: timeout awaiting response headers (Client.Timeout exceeded while awaiting headers))