Syslog from Juniper router to Loki

Grafana Loki
, ,
1

I am in a closed environment so I will do my best to type everything in here. I have a Juniper MX204 at work that I am trying to get syslog sent to a Linux server I have Grafana, Loki, and Promtail installed and running with no errors as services. My main error is that I get 404 error not I have a promtail config as follows:

server:
http_listen_port: 9080
grpce_listen_port: 0

positions:
filename: /tmp/positions.yaml

clients:
-url: http://localhost:3100/loki/api/v1/push

scrap_configs:

  • job_name: junos_syslog
    syslog:
    listen_address: 0.0.0.0:514
    relabel_configs:
    • source_labels: [ā€˜__syslog_hostā€™]
      target_label: ā€˜hostā€™
    • source_labels: [ā€˜__syslog_programā€™]
      target_label: ā€˜appā€™

-Ran tcpdump on linux host -i any ā€˜src <junos_ip> and port 514ā€™ on linux server output is as follows:
timestamp ip junos_rtr.domain.syslog > linux_host.domain.sylog: syslog user.debug
timestamp ip junos_rtr.domain.syslog > linux_host.domain.sylog:
syslog local3.info
timestamp ip junos_rtr.domain.syslog > linux_host.domain.sylog:
syslocal7.info

Ran 'monitor traffic interface <xe-0/1/5 no-resolve matchin ā€œudp and port 514ā€ on Juniper router, output is as follows:
timestamp out IP truncated-ip 169 bytes missing: <junos_ip>.514 > <linux_ip>.514: SYSLOG local3.dinfo, length: 191
timestamp out IP truncated-ip 169 bytes missing: <junos_ip>.514 > <linux_ip>.514: SYSLOG local3.dinfo, length: 191
timestamp out IP truncated-ip 169 bytes missing: <junos_ip>.514 > <linux_ip>.514: SYSLOG local3.dinfo, length: 191
etc.

-Have the linux server as the syslog host on juniper with correct port configured
-Ran 'journalctl -u promtail.service -xe no errors
-No errors when running ā€˜promtailā€™
-verified connectivity to and from the junos router to the linux vm
-no firewall in place
-Am successfully ingesting other logs from Prometheus and Loki
-Tried using rsyslog, promtail, and vector

What am I doing wrong??!?!?

2

Couple of thoughts:

  1. What does lsof -i :514 say?
  2. Is promtail running as root user?
1 Like
3

When I am running promtail as a service, that port is being used by the service. Itā€™s not duplicated anywhere, I have troubleshot this command. It says promtail. Iā€™ve killed it and restarted promtail manually and with the service. No joy.

4

However, I wonder if that junos monitor traffic interface output is a problem? I think itā€™s mtu fragments? Again, this is on my work environment, I will play around with it tonight on my home lab. Any other insight? I appreciate your help!

5

Yes I modified promtail config for user:root and group: root. By the way, is this recommended? I donā€™t mind currently for testing. Just wondering if itā€™s a security issue. Any other recommendations?

6

I was just wondering because Iā€™ve seen promtail without root privileges not listening to traffic correctly before, but if you are running as root thatā€™s probably not your issue.

1 Like
7

ā€œsyslogā€ is very vague protocol/format with many options. So I guess you donā€™t have exactly promtail supported syslog format. See doc for recommendations:

The recommended deployment is to have a dedicated syslog forwarder like syslog-ng or rsyslog in front of Promtail. The forwarder can take care of the various specifications and transports that exist (UDP, BSD syslog, ā€¦)

Or use better tool than Promtail, where you can configure all options exactly for your syslog format/transport.

1 Like
8

I have looked at rsyslog, and havenā€™t had success using it. I will reinvestigate and go through the errors I was getting. In the meantime, do you any documentation or anything you can point me to about getting syslog from a Juniper router into rsyslog or passing it along to loki? Iā€™m finding out I am an extreme noob when it comes to syslog. To clarify: Iā€™ve only seen videos of people configuring syslog ingestion for linux-based machines, but never anything for external routers/switches.

9

I would use OTEL collector:

syslog receiver - opentelemetry-collector-contrib/receiver/syslogreceiver at main Ā· open-telemetry/opentelemetry-collector-contrib Ā· GitHub

But be prepared that syslog is mess and vendors donā€™t follow those weaks syslog standards. Grafana included, so then hackish parsing may be needed, e. g. feature request: use rfc3164-compliant timestamps for syslog messages Ā· Issue #72703 Ā· grafana/grafana Ā· GitHub

1 Like
10

I have successfully started rsyslog and promtail, and configured them with rsyslog pointing to promtail. However, the /var/log directory gets filled up within seconds and Grafana and Prometheus crash. (Grafana, Loki, and Prometheus live on the same server.) Once, I stop both rsyslog and promtail, everything resumes as normal.

11

To anyone following this thread I ended up going with vector by datadog and this successfully worked.

12

do you mind to share your vector config? I am in the same boat. so your traffic flow should like this right junos ā†’ vector (source: syslog. transform??) ā†’ loki
millions of thanks in advance