Grafana Agent Logs - Collecting Logs with DATE format file names

Grafana Loki
1

Hey all, have a odd question and wasnt sure where to ask so ill likely cross post this in a few locations.

I have a grafana agent that is looking for log files in a few locations, however the directories contain YEARS AND YEARS of historical logs in which we do not care about.
Essentially we want to only grab logs that are from today-forward rather then try to ingest thousands and thousands of log files here is an example config

    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash2
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/103-cash2/*.LOG

Essentially i want it to look for *.LOG but * containing todays date like so 20230823.LOG each day a new file is created with the YEARMODAY format

2

You can do something whacky like this (haven’t tested):

    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash2
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/103-cash2/2023*.LOG
        - targets: [localhost]
          labels:
            POS: 103-cash2
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/103-cash2/2024*.LOG
        - targets: [localhost]
          labels:
            POS: 103-cash2
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/103-cash2/2025*.LOG
<...>

Let the early months of 2023 naturally process through, and add enough targets to cover yourself for the next 50 or so years.

3

Hmmmm that would definitely work for the interim…

to bad there isnt a way to match the path using regex

I guess another option is i could have it analyze the log files and if it has a time stamp older then 2023 reject it possibly however currently its also not really reading the time stamp from the log file either

4

in my loki config.yml
If i set the -from to 2023-01-01 it should actually ignore the older files right?

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
5

No, that’s schema configuration. If you want to reject old logs on Loki try reject_old_samples and reject_old_samples_max_age (Grafana Loki configuration parameters | Grafana Loki documentation)

6

limits_config:
enforce_metric_name: false
max_cache_freshness_per_query: ‘10m’
reject_old_samples: true
reject_old_samples_max_age: 168h
split_queries_by_interval: 15m

yet it still ingests the logs that are as old as 2015

7

Are you parsing the timestamp in the log file?

8

i dont believe this is all i am doing with the files

    scrape_configs:
    - job_name: integrations/agent
      journal:
        max_age: 24h
        labels:
          instance: Collector01
          job: integrations/agent
      pipeline_stages:
        - match:
            selector: '{unit!="grafana-agent.service"}'
            action: drop
            drop_counter_reason: only_keep_grafana_agent_logs
        - regex:
            expression: "(level=(?P<log_level>[\\s]*debug|warn|info|error))"
        - labels:
            level: log_level
      relabel_configs:
        - source_labels: ['__journal__systemd_unit']
          target_label: 'unit'
    - job_name: integrations/node_exporter_journal_scrape
      journal:
        max_age: 24h
        labels:
          instance: Collector01
          job: integrations/node_exporter
      relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal__boot_id']
        target_label: 'boot_id'
      - source_labels: ['__journal__transport']
        target_label: 'transport'
      - source_labels: ['__journal_priority_keyword']
        target_label: 'level'
    - job_name: integrations/node_exporter_direct_scrape
      static_configs:
        - targets:
          - localhost
          labels:
            instance: Collector01
            __path__: /var/log/{syslog,messages,*.log}
            job: integrations/node_exporter
    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 102-cash1
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/102-cash1/*.LOG
    - job_name: DPALM-NAXML
      static_configs:
        - targets: [localhost]
          labels:
            POS: 102-cash1
            job: DPALM-NAXML
            __path__: /mnt/DPALM-LOGS/102-cash1/*.NAXML
    - job_name: DPALM-Controller
      static_configs:
        - targets: [localhost]
          labels:
            POS: 102-cash1
            job: DPALM-Controller
            __path__: /mnt/DPALM-LOGS/102-cash1/*.controller
    - job_name: DPALM-PX1
      static_configs:
        - targets: [localhost]
          labels:
            POS: 102-cash1
            job: DPALM-PX1
            __path__: /mnt/DPALM-LOGS/102-cash1/*.PX1
    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash1
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/101-cash1/*.LOG
    - job_name: DPALM-NAXML
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash1
            job: DPALM-NAXML
            __path__: /mnt/DPALM-LOGS/101-cash1/*.NAXML
    - job_name: DPALM-Controller
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash1
            job: DPALM-Controller
            __path__: /mnt/DPALM-LOGS/101-cash1/*.controller
    - job_name: DPALM-PX1
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash1
            job: DPALM-PX1
            __path__: /mnt/DPALM-LOGS/101-cash1/*.PX1
    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash2
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/101-cash2/*.LOG
    - job_name: DPALM-NAXML
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash2
            job: DPALM-NAXML
            __path__: /mnt/DPALM-LOGS/101-cash2/*.NAXML
    - job_name: DPALM-Controller
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash2
            job: DPALM-Controller
            __path__: /mnt/DPALM-LOGS/101-cash2/*.controller
    - job_name: DPALM-PX1
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash2
            job: DPALM-PX1
            __path__: /mnt/DPALM-LOGS/101-cash2/*.PX1
    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash4
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/101-cash4/*.LOG
    - job_name: DPALM-NAXM
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash4
            job: DPALM-NAXML
            __path__: /mnt/DPALM-LOGS/101-cash4/*.NAXML
    - job_name: DPALM-Controller
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash4
            job: DPALM-Controller
            __path__: /mnt/DPALM-LOGS/101-cash4/*.controller
    - job_name: DPALM-PX1
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash4
            job: DPALM-PX1
            __path__: /mnt/DPALM-LOGS/101-cash4/*.PX1
    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash5
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/101-cash5/*.LOG
    - job_name: DPALM-NAXML
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash5
            job: DPALM-NAXML
            __path__: /mnt/DPALM-LOGS/101-cash5/*.NAXML
    - job_name: DPALM-Controller
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash5
            job: DPALM-Controller
            __path__: /mnt/DPALM-LOGS/101-cash5/*.controller
    - job_name: DPALM-PX1
      static_configs:
        - targets: [localhost]
          labels:
            POS: 101-cash5
            job: DPALM-PX1
            __path__: /mnt/DPALM-LOGS/101-cash5/*.PX1
    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash1
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/103-cash1/*.LOG
    - job_name: DPALM-NAXML
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash1
            job: DPALM-NAXML
            __path__: /mnt/DPALM-LOGS/103-cash1/*.NAXML
    - job_name: DPALM-Controller
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash1
            job: DPALM-Controller
            __path__: /mnt/DPALM-LOGS/103-cash1/*.controller
    - job_name: DPALM-PX1
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash1
            job: DPALM-PX1
            __path__: /mnt/DPALM-LOGS/103-cash1/*.PX1
    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash2
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/103-cash2/*.LOG
    - job_name: DPALM-NAXML
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash2
            job: DPALM-NAXML
            __path__: /mnt/DPALM-LOGS/103-cash2/*.NAXML
    - job_name: DPALM-Controller
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash2
            job: DPALM-Controller
            __path__: /mnt/DPALM-LOGS/103-cash2/*.controller
    - job_name: DPALM-PX1
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash2
            job: DPALM-PX1
            __path__: /mnt/DPALM-LOGS/103-cash2/*.PX1
    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash3
            job: DPALM-LOG
            __path__: /mnt/DPALM-LOGS/103-cash3/*.LOG
    - job_name: DPALM-NAXML
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash3
            job: DPALM-NAXML
            __path__: /mnt/DPALM-LOGS/103-cash3/*.NAXML
    - job_name: DPALM-Controller
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash3
            job: DPALM-Controller
            __path__: /mnt/DPALM-LOGS/103-cash3/*.controller
    - job_name: DPALM-PX1
      static_configs:
        - targets: [localhost]
          labels:
            POS: 103-cash3
            job: DPALM-PX1
            __path__: /mnt/DPALM-LOGS/103-cash3/*.PX1```
9

If you aren’t parsing timestamp in your log pipeline, then doesn’t matter what the log says the timestamp will be the time it gets sent to Loki. You either have to parse it properly, or you’ll have to make sure you somehow only collect logs from the most resent file, which judging by your naming convention is probably not very easy to do.

10

how could i parse the time stamps from the logs?

time stamps are formatted like this 06/13/2022 01:00:00.265

11

Please see Promtail configuration | Grafana Loki documentation, there is a timestamp section in there.

12

image
image963×423 18.6 KB

found a comment on another thread from you pertaining to timestamps also

13

So in thinking i am going to try this i think

- job_name: DPALM-LOG
  static_configs:
    - targets: [localhost]
      labels:
        POS: 103-cash3
        job: DPALM-LOG
        __path__: /mnt/DPALM-LOGS/103-cash3/*.LOG
  relabel_configs:
    - action: replace
      target_label: __timestamp__
      regex: '(\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}\.\d{3})'
14

I’d recommend you to grab a couple of lines of code and try it out.

Timestamp is not a label. What you need to do is parse parse the timestamp part from the log line (with filter such as regex), then match that timestamp string to a format.

15

ahhh that makes sense i think i got it now

    - job_name: DPALM-LOG
      static_configs:
        - targets: [localhost]
          labels:
            POS: 102-cash1
            job: TEST
            __path__: /mnt/DPALM-LOGS/103-cash3/*.LOG
      pipeline_stages:
        - regex:
            expression: '^(?P<timestamp>\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}\.\d{3})'
        - timestamp:
            source: timestamp
            format: "01/02/2006 15:04:05.000"

image
image1870×124 13.8 KB

16

did some more refining and this works the best as the config above was giving some 429 errors


    - job_name: DPALM-LOG
      pipeline_stages:
        - match:
            selector: '{job="DPALM-LOG"}'
            stages:
            - regex:
                expression: '^(?P<timestamp>\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}\.\d{3}) \[\w+\] (?P<message>.*)'
            - timestamp:
                source: timestamp
                format: "01/02/2006 15:04:05.000"
      static_configs:
      - targets:
          - localhost
        labels:
          job: DPALM-LOG
          POS: 103-cash1
          __path__: /mnt/DPALM-LOGS/103-cash1/*.LOG
17

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.