Loki configuration for logs retention

Grafana Loki
,
1

Hello,

i’m a french developper, that I’m trying to use the open-source stack of grafana. I have grafana, mimir, tempo and loki.
Grafana and Tempo are deployed on one server. Mimir on a other server. Loki on a third server.
Each stack had their own configuration, and all run almost perfectly.
But with Loki, I had issues. I have 20Go on the server to store logs, and I tried to configure to have 1h of retention logs.
But in 8 days, it’s runing out stockage.

I have an idea of the process. If I have 1h of logs retention, and I get the logs of all my backends in 1h, either I have enough storage during this hour to collect and delete logs (and that’s good).
Either, I do not have enough storage to collect and delete the logs, and in one hour, my 20 Go are full ?

Tell me if I’m wrong ?

But in my case, i have the figure : (in x-axis this days of the month)

I start my loki with docker compose :
services:
loki:
image: grafana/loki:latest #3.1.2 # https://hub.docker.com/r/grafana/loki/tags and Releases · grafana/loki · GitHub
network_mode: bridge
command: [‘-config.file=/etc/loki/local-config.yaml’]
volumes:
- ./loki.yml:/etc/loki.yml:ro
ports:
- “3100:3100”

This is my configuration :
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 1h
storage_config:
boltdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
shared_store: filesystem
filesystem:
directory: /loki/chunks

Configuration cruciale pour la rétention avec boltdb-shipper et filesystem

table_manager:
retention_deletes_enabled: true
retention_period: 1h # Exemple pour 7 jours
# working_directory: /loki/table-manager # Chemin pour les opérations du table_manager

  #limits_config:
  #retention_periods:
  #chunk: 1h
  #index: 1h

max_outstanding_per_tenant: 4096
compactor:
retention_enabled: true
working_directory: /loki/compactor # Ou un autre chemin pour les opérations du compacteur
shared_store: filesystem
compaction_interval: 10m # Exemple
retention_enabled: true
retention_delete_delay: 1h # Délai avant la suppression effective
retention_delete_worker_count: 150 # Nombre de workers
apply_retention_period: 1h # Ex: 7d, 168h

Tell me if my configuration is wrong and how to improve it ?

Thank you for your help,
David Lhullier

2
  1. Because Loki’s index period is 24h, I am pretty sure your minimum retention policy has to be at least 24h, but I’d make it 48h just to be safe.
  2. You want to disable table manager and use compactor instead.

For more information please see Log retention | Grafana Loki documentation

3

I put 1h. I can’t override the retention policy ?
I put my schema_config.configs.index.period to 24h.
And I comment the table_manager.

For future, I’d like to have 30d at least. But to start, I just want to manage well with 1 hour.
But I guess that impossible ^^
I will give you back information, if that’s work or not.
Thank you for your help !