Promtail - cannot allocate memory for thread-local data: ABORT

Grafana Loki
1

I’m trying to get Promtail 2.5 running on Cumulus 4.1 (Debian 4.9.2) Linux version 4.1.0-cl-7-amd64.

If I start Promtail with this config:

server:
  disable: true

clients:
  - url: http://loki:3100/loki/api/v1/push

positions:
  filename: /opt/promtail/positions.yaml

scrape_configs:
  - job_name: journal
    journal:
      json: false
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ["__journal__systemd_unit"]
        target_label: "unit"
      - source_labels: ["__journal__hostname"]
        target_label: "hostname"

I get the following error: cannot allocate memory for thread-local data: ABORT

free -h
             total       used       free     shared    buffers     cached
Mem:          1.9G       1.1G       817M        40M       876K       727M
-/+ buffers/cache:       366M       1.5G
Swap:           0B         0B         0B

Swap is disabled, does promtail need the swap to be enabled?

If I remove the scrape config or point it to an empty file, promtail will start without error. It seems this is only happening if it’s trying to scrape the systemd journal.

Initially I got this error:
./promtail-linux-amd64 -config.file=./promtail_config.yaml
level=info ts=2022-07-05T20:20:38.109078844Z caller=main.go:119 msg=“Starting Promtail” version=“(version=2.5.0, branch=HEAD, revision=2d9d0ee23)”
runtime/cgo: out of memory in thread_start

After trying to run it a few times it would only show the error I mention above.

Any ideas? Thoughts on how I can dig into this more?

2

I narrowed it down to the journal being used in the scrape_config. I started to scrape individual log files instead.

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.