Loki Timestamps behind timestamps within log entry

Grafana Loki
1

image

Timestamps within the log messages are sometimes several minute behind the timestamp of Loki ingestion, usually under peak load conditions. Not sure where the bottleneck could be. I have syslog-ng forwarding BSD syslog formatted messages to promtail, promtail sending to singlebinary Loki setup with gateway enabled ( 3 replicas). Syslog-forwarder, promtail and Loki all on same kuberntes cluster.

2

I’d say tcpdump at each stop as a troubleshooting measure.

I would also recommend you to parse logs for timestamp in promtail. Delay of a couple of minutes is a bit too much and you should definitely look into it, but some delay is expected and if you want your log’s timestamp to match your logs you should always parse for timestamp in your log pipeline.