Identical log lines missing

Grafana Loki
1

Running Loki v2.9.0 and pushing nginx logs with Promtail. Having two indentical request right after each other seems to make one of them go missing. I some cases if there are more than 5 lines sent, 2-3 lines are in Loki but the rest are not.

For example the nginx logs below.

10.20.4.212 - - [12/Jan/2024:16:01:02 +0000] "GET /api/live/ws2 HTTP/1.1" 404 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
10.20.4.212 - - [12/Jan/2024:16:01:02 +0000] "GET /api/live/ws2 HTTP/1.1" 404 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
10.20.4.212 - - [12/Jan/2024:16:01:02 +0000] "GET /api/live/ws2 HTTP/1.1" 404 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
10.20.4.212 - - [12/Jan/2024:16:01:02 +0000] "GET /api/live/ws2 HTTP/1.1" 404 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
10.20.4.212 - - [12/Jan/2024:16:01:02 +0000] "GET /api/live/ws2 HTTP/1.1" 404 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
10.20.4.212 - - [12/Jan/2024:16:01:02 +0000] "GET /api/live/ws2 HTTP/1.1" 404 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"

These end up as the following in Grafana

image
image1927×31 8.47 KB

The Promtail config is simply just parsing the timestamp from Nginx.


server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /etc/promtail/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push
    external_labels:
      hostname: tools

scrape_configs:
- job_name: nginx
  pipeline_stages:
  - regex:
      expression: "^.* - - \\[(?P<time>\\w{2}\\/\\w{3}\\/\\w{4}:\\w{2}:\\w{2}:\\w{2} [+-]\\w{4})\\].*"
  - timestamp:
      source: time
      format: "02/Jan/2006:15:04:05 -0700"
  static_configs:
  - targets:
      - localhost
    labels:
      job: nginx
      __path__: /var/log/nginx/*.log

I do understand that one solution to this is to append a more exact timestamp to the nginx log lines but I feel like this should not need to be a problem. I have also tried to set increment_duplicate_timestamp: true in the Loki config.

Any help much appreciated.