Promtail - cannot allocate memory for thread-local data: ABORT

I’m trying to get Promtail 2.5 running on Cumulus 4.1 (Debian 4.9.2) Linux version 4.1.0-cl-7-amd64.

If I start Promtail with this config:

  disable: true

  - url: http://loki:3100/loki/api/v1/push

  filename: /opt/promtail/positions.yaml

  - job_name: journal
      json: false
      max_age: 12h
        job: systemd-journal
      - source_labels: ["__journal__systemd_unit"]
        target_label: "unit"
      - source_labels: ["__journal__hostname"]
        target_label: "hostname"

I get the following error: cannot allocate memory for thread-local data: ABORT

free -h
             total       used       free     shared    buffers     cached
Mem:          1.9G       1.1G       817M        40M       876K       727M
-/+ buffers/cache:       366M       1.5G
Swap:           0B         0B         0B

Swap is disabled, does promtail need the swap to be enabled?

If I remove the scrape config or point it to an empty file, promtail will start without error. It seems this is only happening if it’s trying to scrape the systemd journal.

Initially I got this error:
./promtail-linux-amd64 -config.file=./promtail_config.yaml
level=info ts=2022-07-05T20:20:38.109078844Z caller=main.go:119 msg=“Starting Promtail” version=“(version=2.5.0, branch=HEAD, revision=2d9d0ee23)”
runtime/cgo: out of memory in thread_start

After trying to run it a few times it would only show the error I mention above.

Any ideas? Thoughts on how I can dig into this more?

I narrowed it down to the journal being used in the scrape_config. I started to scrape individual log files instead.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.