Loki retention - chunks count&space usage too high

Grafana Loki
1

Hi
I’m new with loki and trying to understand how to correctly set retention for it. I currently have loki in docker setup with such config:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

table_manager:
  retention_deletes_enabled: true
  retention_period: 24h

ruler:
  alertmanager_url: http://localhost:9093

After a month of working on host, loki consumed nearly 300 Gb of space and created ~200 000 chunks. So I wanted to reduces this number of chunks and used space somehow. I tried to add Compactor with such config.

compactor:
  working_directory: /loki/retention
  shared_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

And I see in log that it already run 3 times, but it always compacts the same tables - starting with 19420 and ending with 19450

level=info ts=2023-04-04T10:42:30.779132767Z caller=compactor.go:495 msg="finished compacting table" table-name=index_19420
....
level=info ts=2023-04-04T10:42:30.784918253Z caller=compactor.go:495 msg="finished compacting table" table-name=index_19450

So, not really understand why it’s not looping over all data and tries to optimize it?
I added also limits_config: retention_period: 744h to main config but gave no effect.
Will be glad to see some basic explanation for a newbie like me.
Thanks in advance!

2

  1. Retention period is configured under limits_config, see Retention | Grafana Loki documentation.

  2. Table manager is no longer needed if you run compactor, you can disable it like so:

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
3

Seems that it did the trick, thanks a lot.

1 Like
4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.