Promtail: memory consumption, possible leak?

[migrandjean]

Hi everyone!

We are using the Promtail Helm Chart (Chart version 6.7.4, with Promtail 2.7.0) on EKS to send all Pod/Docker logs to Loki. The Promtail Pods on the EKS nodes show a slow but steady increase in memory usage over time, e.g.:

IMHO this looks very typical for a memory leak. Not every EKS node gets old enough for this to become a real problem, but some nodes do and they experience OOM kills for Promtail. However, I wanted to make sure if this is expected/unexpected behavior before opening an issue on GitHub - maybe I’m missing some configuration option?

We are using the default values of the Helm Chart (with exception of the Loki URL and some podAnnotations and tolerations).

I found promtail memory leak · Issue #6968 · grafana/loki · GitHub and Promtail: Add support for max stream limit by liguozhong · Pull Request #7101 · grafana/loki · GitHub which let’s you limit max_streams, but I’m not sure how to determine how many streams I need or even how many I’m currently using.

Any hints or tips?

[intelliturtle]

It really frustrates me that the Grafana Loki/Promtail team doesn’t respond to well-presented, likely important issues like this. I’ve never gotten a response on this forum, even when I found an error in an official tutorial of theirs and wanted to know how to fix it.