Promtail ignore unit in journal logs

Grafana Loki
1

We have promtail on our Linux hosts and the are using the journal scraper to ingest all linux logs. I have seen that we are creating thousands of labels because we have units with the name session-xxxx.scope. Is there a way to ignore/drop these events when ingesting?

1 Like
2

This IS the fix and that works great!

  - job_name: journal
    pipeline_stages:
      - drop:
          source: systemd_unit
          expression: session-\d+\.scope
    journal:
      json: false
    relabel_configs:
    - source_labels:
      - __journal__systemd_unit
      target_label: systemd_unit
1 Like
4

Thanks for sharing your fix! We have similar setup.

We also encountered thousands of units with names like someservice@29759-192.50.1.182:6556-192.50.1.222:50892.service.

They are created by systemd when using a systemd.socket.

We removed them with a promtail configuration like this:

  - job_name: journal
    journal:
      labels:
        host: somehostname.com
        job: systemd-journal
    pipeline_stages:
    - match:
        selector: '{unit=~"session-\\d+\\.scope"}'
        stages:
        - drop:
            expression: .*
    - match:
        selector: '{unit=~"someservice@.*service"}'
        stages:
        - drop:
            expression: .*
    relabel_configs:
    - source_labels:
      - __journal__systemd_unit
      target_label: unit

I like your solution more than ours and will change it accordingly.

5

Hi,
I think the best way is to drop entries as early as possible.That can be done with a drop in relabel_config:

    - job_name: journal
      relabel_configs:
        - action: drop
          source_labels: [__journal__systemd_unit]
          regex: session-.*

        - source_labels: [__journal__systemd_unit]
          target_label: systemd_unit

In my case, I am interested in a single unit so I use keep instead:

    - job_name: journal
      relabel_configs:
        - action: keep
          source_labels: [__journal__systemd_unit]
          regex: my-server.service

        - source_labels: [__journal__systemd_unit]
          target_label: systemd_unit
1 Like
6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.