We are using the Promtail Helm Chart (Chart version 6.7.4, with Promtail 2.7.0) on EKS to send all Pod/Docker logs to Loki. The Promtail Pods on the EKS nodes show a slow but steady increase in memory usage over time, e.g.:
IMHO this looks very typical for a memory leak. Not every EKS node gets old enough for this to become a real problem, but some nodes do and they experience OOM kills for Promtail. However, I wanted to make sure if this is expected/unexpected behavior before opening an issue on GitHub - maybe I’m missing some configuration option?
We are using the default values of the Helm Chart (with exception of the Loki URL and some podAnnotations and tolerations).
I found promtail memory leak · Issue #6968 · grafana/loki · GitHub and Promtail: Add support for max stream limit by liguozhong · Pull Request #7101 · grafana/loki · GitHub which let’s you limit
max_streams, but I’m not sure how to determine how many streams I need or even how many I’m currently using.
Any hints or tips?