Promtail syslog job is not showing any error, but can't see syslog logs in queries

Grafana Loki
1

I use the docker image grafana/promtail:2.9.2 to deploy my promtail. The varlog jobs are working well.

I use this configuration to have a syslog receiver:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: syslog-receiver
  syslog:
    listen_address: 0.0.0.0:40514
    idle_timeout: 12h
    use_incoming_timestamp: true
    labels:
      job: syslog-receiver

  relabel_configs:
    - source_labels: ['__syslog_message_hostname']
      target_label: host
    - source_labels: ['__syslog_message_severity']
      target_label: level
    - source_labels: ['__syslog_message_facility']
      target_label: syslog_facility
    - source_labels: ['__syslog_message_app_name']
      target_label: syslog_identifier

I can see varlogs log entries, working well.

I use the following Python app to test syslogs:

import logging
import logging.handlers

my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.DEBUG)

handler = logging.handlers.SysLogHandler(address = ('v145',40514))

my_logger.addHandler(handler)

my_logger.debug('this is debug')
my_logger.critical('this is critical')
It sends logs to my promtail container. No error. I checked docker logs, no error.

However, this query does not return any logs:

{job="syslog-receiver"} |= ``

How can I diagnose this issue?

2

In case someone else come across this issue,

The root cause was TCP and UDP protocol mismatch

Syslog uses UDP by default. Docker uses TCP by default. When there is UDP:TCP protocol mismatch, there won’t be any error - the packets will be dropped silently :frowning:

So port mapping in the docker compose should be like this:

    ports:
      - "40514:40514/tcp"
      - "40514:40514/udp"