How to prevent duplicate log lines from file_sd_configs and rename-and-create rotation of config?

Grafana Loki
1

I have written a python script that creates a JSON file that lists files to watch:

 {
  "targets": [
   "localhost"
  ],
  "labels": {
   "__path__": "/home/sysavtcme/cvs/amm/locallog/20240903/QM2CMELDN1.log-jdk-avt.01"
  }
 },
 {
  "targets": [
   "localhost"
  ],
  "labels": {
   "__path__": "/home/sysavtcme/cvs/amm/locallog/20240903/QM2CMELDN1.log-jdk-avt.00"
  }
 }

  - job_name: "files"
    file_sd_configs:
      - files:
          # Generated by promtail_gen_files.py script.
          - ${NOMAD_ALLOC_DIR}/data/promtail_gen_files.json
        refresh_interval: 30s # 1m  # default 5m
    pipeline_stages:
       ....

The application uses rename-and-create method of rotating the logs.

However I am observing duplicated log lines in loki with different filenames:

image
image1077×287 21 KB

Is there anything wrong with the setup?

Because the application uses rename-and-create, when the rotation happen does
promtail then re-read the whole …01 log file again and send it? Should the logfile picker/selector be aware and tell promtail to watch only for original …00 files and ignore rotated files?

Thank you!

2

When you rename and create a file, promtail will read through the entire file again because the file name is now different, and promtail doesn’t have a record of the file being read.

I think the easiest solution is to just watch for the first file (assuming the file name for the first file stays the same).