We are deploying Grafana Loki 3.0.0 (2.9.8) has the same issue with the Helm chart, using the distributed deployment mode.
We have recently tried to ingest logs with structured metadata and noticed that this seems to cause the ingesters to eventually run out of memory. See attached graph. We have also been able to capture pprof heap information, also attached below.
The memory seems to be allocated inside the gRPC server and referenced somewhere in Loki after the requests finishes. Note that removing structured metadata from the logs make the problem disappear.
We are ingesting about 3-4 k lines/s, with < 5000 streams. Some of the structured metadata is of very high cardinality (traceids - which I assume will not be an issue)
