Loki does not delete logs after retention delete delay

Grafana Loki
1

I have loki on my local windows pc i run loki fine and i can store and see logs with grafana, but i am not able to delete logs after a specific amount of time: Is this even a feature of Loki?
For reference this is my config.yaml
`auth_enabled: false

server:
http_listen_port: 3100
grpc_listen_port: 9096

common:
instance_addr: 127.0.0.1
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory

query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100

compactor:
working_directory: /tmp/loki/compactor
retention_enabled: true
retention_delete_delay: 5h

  schema_config:

configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h

ruler:
alertmanager_url: http://localhost:9093

2

Doesn’t look like you specify a retention period. Take a look at Configuration | Grafana Loki documentation, specifically retention_period.

1 Like
3

Oh i see, i thought retention_period was under the compactor configuration. I changed my config to this

compactor:
  working_directory: /tmp/loki/compactor
  retention_enabled: true
  retention_delete_delay: 5m
  retention_delete_worker_count: 150

limits_config:
  retention_period: 5m

And then i did this:

  1. Started Loki
  2. Sending some logs to loki
  3. Starting a timer of 5 minutes
  4. Checking on Grafana after five minutes but the logs are still there.

I maybe is the retention_delete_delay that should not be set to 5 minutes?
Thanks again for the reply!

4

I am new with Grafana Loki but maybe I can help. Maybe try looking at this discussion:

I see you have everything as suggested under compactor except “compaction_interval” maybe that will help your issue. As far as I know compactor is the way forward as table manager is obsolete, so you should be on the right track. I have also read that Loki might remove your logs from Grafana view but it might take 24h before the chunks are deleted too, but at the moment I cannot find the link to this information. Perhaps try to see if anything has changed tomorrow?

5

retention_delete_delay is the amount of time between a chunk is marked for deletion and a chunk is actually deleted. For example, with your configuration, if it were working, would have chunks be deleted 10 minutes after injection.

I am not entirely sure why it’s not working for you, might help if you post your entire loki configuration. Also check for logs from compactor and see if there is anything obvious there.

6

I understand that now, thanks. This is my configuration:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

compactor:
  working_directory: /tmp/loki/compactor
  retention_enabled: true
  retention_delete_delay: 5m
  retention_delete_worker_count: 150
  deletion_mode: filter-and-delete # this was added after the edit
  delete_request_cancel_period: 5m # this was added after the edit

limits_config:
  retention_period: 5m

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Checking the Loki cmd i see this but i don’t know specifically what they mean:

Loki_Logs
Loki_Logs1925Ă—1030 56.4 KB

Clearly you know something more than i do, but i’m really trying to work this out. I appreciate you helping me!

EDIT:
After a while i think i have part of the solution here:
Under compactor section in my YAML i added deletion_mode: filter-and-delete and this is able to delete the logs but only after 2h. By setting: retention_delete_delay: 1m this does not work. I used a timer to check and the logs will be deleted after aproximately 2h. I don’t know what is causing this tho. But now i know that my logs are being deleted after 2h…

7

Anything new for this ?