Hi, I’m operation grafana loki on k8s env.
But Ingester keeps getting killed because of OOM.
I want Ingester to flush all the chunks he has in his memory before he uses up all the memory limits in the Pod, save them as Object storage, and empty the memory.
Can anyone help me with this?
- loki version: 3.0.0 (distributed mode)
- ingester cpu: 7000m
- ingester memory: 14Gi
- distributor received line: about 1.5K/s
- distributor reveived bytes: about 1.5~2.0MB/s
limits_config: # Configuring the retention period
query_timeout: 30m
volume_max_series: 1000000 # The maximum number of aggregated series in a log-volume response
reject_old_samples: true
reject_old_samples_max_age: 1w
split_queries_by_interval: 15m # default: 1h
max_global_streams_per_user: 0 # default: 5000
max_streams_per_user: 0 # default: 0
retention_period: 30d
per_stream_rate_limit: 128MB # default: 3MB
per_stream_rate_limit_burst: 256MB # default 15MB
max_query_parallelism: 64 # default 32
max_query_series: 100000
max_query_length: 0 # default 30d1h
volume_enabled: true
ingestion_rate_mb: 128 # default 4
ingestion_burst_size_mb: 256 # default 8
discover_log_levels: false # default true
distributor:
rate_store:
max_request_parallelism: 100 # default 200
debug: true
ingester:
wal:
enabled: false
autoforget_unhealthy: true
#chunk_retain_period: 1s
concurrent_flushes: 1024 # How many flushes can happen concurrently from each stream.
chunk_block_size: 786432 # 768KB default 262144
chunk_target_size: 6164480 # 5MB default 1572864
chunk_idle_period: 30m # default 30m
# The validity window for unordered writes is the highest timestamp present minus 1/2 * max-chunk-age.
max_chunk_age: 2h # default 2h.
chunk_encoding: snappy # The algorithm to use for compressing chunk. (none, gzip, lz4-64k, snappy,lz4-256k, lz4-1M, lz4, flate, zstd)