Optimizing RAM for small installation

Grafana Loki
1

I’m using Loki in a small kubernetes cluster. I don’t have precise numbers right now, but we’re talking about few gigabytes of compressed logs per day. Log queries are very infrequent.

One particular issue is that one service sometimes logs extraordinary-long strings. It’ll be reworked, but that’s the state of things for now.

I configured and installed Loki without operator to better understand it. I’m using it in a monolythic mode.

The issue that I’m having is that loki seems to consume an extra-ordinate amount of memory. It’s the most heavy service by far in the entire cluster. So I’m trying to understand ways to reduce RAM consumption to a more reasonable levels, while keeping it operational. I feel that Loki could be much more lightweight. I guess, the default config values are optimized for some heavy-duty installations.

I made adjustments to the config mostly to deal with various errors, which occured during queries. It seems to work with current config. I tried to keep most values to their default values.

We’re using OpenStack S3, which might be not as fast as AWS, unfortunately there’s no option to use AWS for us.

Right now loki consumes around 500 Mi on average, which is acceptable, I guess, but for executing queries, its consumption might peak at 2644 Mi which seems very high.

Current configuration is as follows:

auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 0
  http_server_write_timeout: 310s
  http_server_read_timeout: 310s
  grpc_server_max_recv_msg_size: 8388608
  grpc_server_max_send_msg_size: 8388608
querier:
  max_concurrent: 4
query_scheduler:
  max_outstanding_requests_per_tenant: 32768
frontend:
  compress_responses: true
ingester:
  chunk_encoding: snappy
storage_config:
  aws:
    bucketnames: ${S3_BUCKET}
    endpoint: ${S3_ENDPOINT}
    region: ${S3_REGION}
    access_key_id: ${S3_ACCESS_KEY_ID}
    secret_access_key: ${S3_SECRET_ACCESS_KEY}
  tsdb_shipper:
    active_index_directory: /loki/index
    shared_store: s3
    cache_location: /loki/cache
schema_config:
  configs:
    - from: 2024-02-23
      store: tsdb
      object_store: aws
      schema: v13
      index:
        prefix: index_
        period: 24h
compactor:
  working_directory: /loki/compactor
  shared_store: s3
limits_config:
  max_query_lookback: 400d
  query_timeout: 2m
  allow_structured_metadata: true
table_manager:
  retention_deletes_enabled: true
  retention_period: 400d
common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /loki

I’d appreciate suggestions about which particular configuration options I shall look into, because configuration file is vast and I struggle to understand all of those performance knobs.

2

If you are running a single monolithic Loki container, try disabling query splitting and disabling concurrency.