Scrape logs from different file everyday


New guy here. Currently facing one problem that I don’t know how to solve best, so maybe I’ll get some help here.

Current situation: I have an API running, which generates a new log file everyday, starting from midnight. File name pattern - apiLogs-YYYY-MM-dd.log. I managed to setup grafana windows agent to provide me metrics, scrape logs and push those to prometheus so that it can be displayed on Grafana dahsboard. Windows agent configuration:

    - name: appLogsServer1
        filename: /tmp/positions.yaml
        - job_name: system
            - targets: [localhost]
                job: logsAppServer1
                __path__: C:\xxx\xxx\xxx\apiLogs-2022-04-28.log

Problem: Everyday when a new log file is created, loki does not scrape it, unless I manually change the config and restart grafana agent, which is a really bad approach for many reasons. What I’d like to do is to tell loki to take the new file everyday. Searched through loki documentation and didn’t managed to find a solution.

Closest thing to what worked was that I created a bash script to update the config file and wanted to create a windows background job, which runs this script and restarts Grafana Agent, but the machine doesn’t have a proper user with required policies and I can’t set it up myself.

All help will be greatly appreciated!

Can this help?

Use of promtail to read logs and ship to loki?