Promtail never scrapes `journal` logs (using README suggestions in official helm chart)

Grafana Loki
1

Describe the bug
Hi thanks for Loki, but I fail to use it to get journald logs.

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki (SHA or version): 2.8.2
  2. Started Promtail (SHA or version): 2.8.2
  3. it does not handle journal logs at all

Expected behavior
A clear and concise description of what you expected to happen.

Environment:

  • Infrastructure: [e.g., Kubernetes, bare-metal, laptop] k8s
  • Deployment tool: [e.g., helm, jsonnet] helm

Screenshots, Promtail config, or terminal output

I use the official promtail helm chart, and deliberately disabled the original section and only remain the journalctl section that I added, in order to show it more clearly.

helm values:

config:
  snippets:
    # #10154, https://github.com/grafana/helm-charts/blob/main/charts/promtail/README.md#journald-support
    extraScrapeConfigs: |
      # Add an additional scrape config for syslog
      - job_name: journal
        journal:
          path: /var/log/journal
          max_age: 12h
          labels:
            job: systemd-journal
        relabel_configs:
          - source_labels:
              - __journal__hostname
            target_label: hostname

          # example label values: kubelet.service, containerd.service
          - source_labels:
              - __journal__systemd_unit
            target_label: unit

          # example label values: debug, notice, info, warning, error
          - source_labels:
              - __journal_priority_keyword
            target_label: level

# Mount journal directory and machine-id file into promtail pods
extraVolumes:
  # #10154, https://github.com/grafana/helm-charts/blob/main/charts/promtail/README.md#journald-support
  - name: journal
    hostPath:
      path: /var/log/journal
  - name: machine-id
    hostPath:
      path: /etc/machine-id

extraVolumeMounts:
  # #10154, https://github.com/grafana/helm-charts/blob/main/charts/promtail/README.md#journald-support
  - name: journal
    mountPath: /var/log/journal
    readOnly: true
  - name: machine-id
    mountPath: /etc/machine-id
    readOnly: true

config when cat inside the container (DaemonSet pod):

root@promtail-7mtbh:/# cat /etc/promtail/promtail.yaml
server:
  log_level: info
  http_listen_port: 3101
  

clients:
  - url: http://loki:3100/loki/api/v1/push

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  
  # Add an additional scrape config for syslog
  - job_name: journal
    journal:
      path: /var/log/journal
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels:
          - __journal__hostname
        target_label: hostname
  
      # example label values: kubelet.service, containerd.service
      - source_labels:
          - __journal__systemd_unit
        target_label: unit
  
      # example label values: debug, notice, info, warning, error
      - source_labels:
          - __journal_priority_keyword
        target_label: level
  

limits_config:
  

tracing:
  enabled: false
root@promtail-7mtbh:/#

promtail logs, when level=debug

level=debug ts=2023-07-12T14:05:32.163270151Z caller=promtail.go:125 msg="Reloading configuration file"
level=info ts=2023-07-12T14:05:32.164971302Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=979bfe7c7e37b5cb2aaee0a6ada862ed
level=info ts=2023-07-12T14:05:32.172243698Z caller=server.go:323 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2023-07-12T14:05:32.172409725Z caller=main.go:174 msg="Starting Promtail" version="(version=2.8.2, branch=HEAD, revision=9f809eda7)"
level=warn ts=2023-07-12T14:05:32.172441948Z caller=promtail.go:265 msg="enable watchConfig"
level=debug ts=2023-07-12T14:05:41.417964184Z caller=logging.go:76 msg="GET /ready (200) 52.01µs"
level=debug ts=2023-07-12T14:05:41.418061331Z caller=logging.go:76 msg="GET /ready (200) 142.741µs"
level=debug ts=2023-07-12T14:05:42.385697851Z caller=logging.go:76 msg="GET /metrics (200) 1.833019ms"

systemctl version

systemctl --version
systemd 249 (249.11-0ubuntu3.6)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.