I am in a closed environment so I will do my best to type everything in here. I have a Juniper MX204 at work that I am trying to get syslog sent to a Linux server I have Grafana, Loki, and Promtail installed and running with no errors as services. My main error is that I get 404 error not I have a promtail config as follows:
server:
http_listen_port: 9080
grpce_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
-url: http://localhost:3100/loki/api/v1/push
scrap_configs:
- job_name: junos_syslog
syslog:
listen_address: 0.0.0.0:514
relabel_configs:
- source_labels: [‘__syslog_host’]
target_label: ‘host’
- source_labels: [‘__syslog_program’]
target_label: ‘app’
-Ran tcpdump on linux host -i any ‘src <junos_ip> and port 514’ on linux server output is as follows:
timestamp ip junos_rtr.domain.syslog > linux_host.domain.sylog: syslog user.debug
timestamp ip junos_rtr.domain.syslog > linux_host.domain.sylog:
syslog local3.info
timestamp ip junos_rtr.domain.syslog > linux_host.domain.sylog:
syslocal7.info
Ran 'monitor traffic interface <xe-0/1/5 no-resolve matchin “udp and port 514” on Juniper router, output is as follows:
timestamp out IP truncated-ip 169 bytes missing: <junos_ip>.514 > <linux_ip>.514: SYSLOG local3.dinfo, length: 191
timestamp out IP truncated-ip 169 bytes missing: <junos_ip>.514 > <linux_ip>.514: SYSLOG local3.dinfo, length: 191
timestamp out IP truncated-ip 169 bytes missing: <junos_ip>.514 > <linux_ip>.514: SYSLOG local3.dinfo, length: 191
etc.
-Have the linux server as the syslog host on juniper with correct port configured
-Ran 'journalctl -u promtail.service -xe no errors
-No errors when running ‘promtail’
-verified connectivity to and from the junos router to the linux vm
-no firewall in place
-Am successfully ingesting other logs from Prometheus and Loki
-Tried using rsyslog, promtail, and vector
What am I doing wrong??!?!?
When I am running promtail as a service, that port is being used by the service. It’s not duplicated anywhere, I have troubleshot this command. It says promtail. I’ve killed it and restarted promtail manually and with the service. No joy.
However, I wonder if that junos monitor traffic interface output is a problem? I think it’s mtu fragments? Again, this is on my work environment, I will play around with it tonight on my home lab. Any other insight? I appreciate your help!
Yes I modified promtail config for user:root and group: root. By the way, is this recommended? I don’t mind currently for testing. Just wondering if it’s a security issue. Any other recommendations?
I was just wondering because I’ve seen promtail without root privileges not listening to traffic correctly before, but if you are running as root that’s probably not your issue.
“syslog” is very vague protocol/format with many options. So I guess you don’t have exactly promtail supported syslog format. See doc for recommendations:
The recommended deployment is to have a dedicated syslog forwarder like syslog-ng or rsyslog in front of Promtail. The forwarder can take care of the various specifications and transports that exist (UDP, BSD syslog, …)
Or use better tool than Promtail, where you can configure all options exactly for your syslog format/transport.
I have looked at rsyslog, and haven’t had success using it. I will reinvestigate and go through the errors I was getting. In the meantime, do you any documentation or anything you can point me to about getting syslog from a Juniper router into rsyslog or passing it along to loki? I’m finding out I am an extreme noob when it comes to syslog. To clarify: I’ve only seen videos of people configuring syslog ingestion for linux-based machines, but never anything for external routers/switches.
I have successfully started rsyslog and promtail, and configured them with rsyslog pointing to promtail. However, the /var/log directory gets filled up within seconds and Grafana and Prometheus crash. (Grafana, Loki, and Prometheus live on the same server.) Once, I stop both rsyslog and promtail, everything resumes as normal.
To anyone following this thread I ended up going with vector by datadog and this successfully worked.
