Rsyslog + Promtail + loki - spooling timestamp issues

Grafana Loki
1

Hey,

my setup is the following:

  • Rsyslog listening on port 514 listening for relayed messages with spooling, transforms the log into the right format and relays them to port 1514
  • promtail (as container) listening on port 1514 processing the logdata and sending it to loki
  • loki (also as container)

My problem with this setup is that promtail doesn’t seem to preserve the timestamps. For testing I stopped promtail, waited for 2 minutes and then start it again. I’ve also set up a tcpdump with a filter on the minute in the timestamps where promtail is down.
the dump shows that the timestamps get send correctly towards promtail, but if I check loki afterwards, I see the 2 minute gap timestamp wise with a big peak of log messages afterwards.

So it seems for me like promtail (or loki) takes the timestamps where the data gets ingested in stead of the timestamps offered by the logs themself. Any Ideas what I need to change in order to get the real timestamps?

Here is my promtail config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      idle_timeout: 60s
      label_structured_data: yes
      labels:
        job: "syslog"
        message:
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'
    pipeline_stages:
      - drop:
          expression: 'type=\w+ msg='
      - regex:
          expression: '^(?P<type>(messenger|request|app|security))\.(?P<classification>[\w]+): (?P<message>[^{]+) ((?P<json>{.+})|(\[\])) \[\]$'
      - labels:
          type:
          classification:
2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.