Journal logs not scraped

Grafana Loki
1

Hi all,

I am trying to pass journal logs via Promtail on server A to Loki on server B. Now it seems none of the journal logs ever gets read by Promtail.

screenshot_2023-10-27_19-42-02
screenshot_2023-10-27_19-42-021158×691 46 KB

Promtail config

global:
  file_watch_config:
    min_poll_frequency: 250ms
    max_poll_frequency: 250ms
server:
  http_listen_network: tcp
  http_listen_address: ""
  http_listen_port: 9080
  http_listen_conn_limit: 0
  grpc_listen_network: tcp
  grpc_listen_address: ""
  grpc_listen_port: 0
  grpc_listen_conn_limit: 0
  tls_cipher_suites: ""
  tls_min_version: ""
  http_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  grpc_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  register_instrumentation: true
  graceful_shutdown_timeout: 30s
  http_server_read_timeout: 30s
  http_server_write_timeout: 30s
  http_server_idle_timeout: 2m0s
  grpc_server_max_recv_msg_size: 4194304
  grpc_server_max_send_msg_size: 4194304
  grpc_server_max_concurrent_streams: 100
  grpc_server_max_connection_idle: 2562047h47m16.854775807s
  grpc_server_max_connection_age: 2562047h47m16.854775807s
  grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
  grpc_server_keepalive_time: 2h0m0s
  grpc_server_keepalive_timeout: 20s
  grpc_server_min_time_between_pings: 5m0s
  grpc_server_ping_without_stream_allowed: false
  log_format: logfmt
  log_level: info
  log_source_ips_enabled: false
  log_source_ips_header: ""
  log_source_ips_regex: ""
  log_request_headers: false
  log_request_at_info_level_enabled: false
  log_request_exclude_headers_list: ""
  http_path_prefix: ""
  external_url: ""
  health_check_target: null
  disable: false
  profiling_enabled: false
  enable_runtime_reload: false
client:
  url: ""
  batchwait: 1s
  batchsize: 1048576
  follow_redirects: false
  enable_http2: false
  backoff_config:
    min_period: 500ms
    max_period: 5m0s
    max_retries: 10
  timeout: 10s
  tenant_id: ""
  drop_rate_limited_batches: false
  stream_lag_labels: ""
clients:
- url: http://serverb:3100/loki/api/v1/push
  batchwait: 1s
  batchsize: 1048576
  follow_redirects: true
  enable_http2: true
  backoff_config:
    min_period: 500ms
    max_period: 5m0s
    max_retries: 10
  timeout: 10s
  tenant_id: ""
  drop_rate_limited_batches: false
  stream_lag_labels: ""
positions:
  sync_period: 10s
  filename: /tmp/positions.yaml
  ignore_invalid_yaml: false
scrape_configs:
- job_name: syslog
  syslog:
    listen_address: 0.0.0.0:10514
    listen_protocol: ""
    idle_timeout: 0s
    label_structured_data: false
    labels:
      job: syslog
    use_incoming_timestamp: false
    use_rfc5424_message: false
    max_message_length: 0
  relabel_configs:
  - source_labels: [__syslog_message_hostname]
    separator: ;
    regex: (.*)
    target_label: hostname
    replacement: $1
    action: replace
  - source_labels: [__syslog_message_severity]
    separator: ;
    regex: (.*)
    target_label: level
    replacement: $1
    action: replace
  - source_labels: [__syslog_message_app_name]
    separator: ;
    regex: (.*)
    target_label: application
    replacement: $1
    action: replace
  - source_labels: [__syslog_message_facility]
    separator: ;
    regex: (.*)
    target_label: facility
    replacement: $1
    action: replace
  static_configs: []
- job_name: system
  static_configs:
  - targets:
    - localhost
    labels:
      __path__: /var/log/*log
      __path_exclude__: /var/log/lastlog
      job: varlogs
- job_name: journal
  journal:
    max_age: 12h
    json: false
    labels:
      job: systemd-journal
    path: /var/log/journal
    matches: _TRANSPORT=kernel
  relabel_configs:
  - source_labels: [__journal__systemd_unit]
    separator: ;
    regex: (.*)
    target_label: unit
    replacement: $1
    action: replace
  static_configs: []
target_config:
  sync_period: 10s
  stdin: false
limits_config:
  readline_rate: 1000
  readline_burst: 2000
  readline_rate_enabled: true
  readline_rate_drop: true
  max_streams: 0
  max_line_size: 0B
  max_line_size_truncate: false
tracing:
  enabled: true
wal:
  enabled: false
  dir: ""
  cleanSegmentsOlderThan: 0s
  watchConfig:
    minReadFrequency: 0s
    maxReadFrequency: 0s

I created /var/log/journal by following systemd-journald.service and executing the following commands:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

Systemd

systemctl --version
systemd 252 (252-14.el9_2.3.0.1)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

The Promtail process is running as user promtail and I explicitly verified it can read all the logs (accomplished by using file ACLs).

getfacl /var/log/
getfacl: Removing leading '/' from absolute path names
# file: var/log/
# owner: root
# group: root
user::rwx
user:promtail:r-x
group::r-x
mask::r-x
other::r-x

The logs from the jobs syslog and system do get passed to Loki. The promtail log (in journald) does not show any signs of errors.
Am I missing something in the config for it to work?

2

Sorry for bumping but I can’t get it to work. Are there any ideas?

3

Try to remove journal filter matches, maybe you don’t have any logs, which match that filter

4

@jangaraj Thank you for your answer but unfortunately still nothing coming through.

5

Bumping again as I cannot find the solution for my problem.

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.