Log retention in Loki is not working / Query rollback is not changeing as well

Grafana Loki
1

I am trying to set a log retention policy for loki so that the logs will be deleted after 28 days. I also want to set the query roll back to 28 days but no matter what I do , the limit stays : 30d1h

global:
  dnsService: "coredns"
loki:
  enabled: true
  isDefault: true
  auth_enabled: false
  url: http://{{(include "loki.serviceName" .)}}:{{ .Values.loki.service.port }}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  livenessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  singleBinary:
    replicas: 1
  commonConfig:
    replication_factor: 1

backend:
  replicas: 1
read:
  replicas: 1
write:
  replicas: 1

minio:
  enabled: true

compactor:
  working_directory: /data/retention
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 1m
  retention_delete_worker_count: 150
  delete_request_store: aws
schema_config:
    configs:
      - from: "2020-07-31"
        index:
            period: 24h
            prefix: index_
        object_store: aws
        schema: v13
        store: tsdb
storage_config:
  tsdb_shipper:
    active_index_directory: /data/index
    cache_location: /data/index_cache
  aws:
    s3:
      bucket_name: loki
limits_config:
  max_query_lookback: 672h # 28 days
  retention_period: 672h   # 28 days

I apply loki with command: helm upgrade --install loki grafana/loki --values loki-values.yaml --set loki.useTestSchema=true

loki queries in grafana roll back to 28 days and retention policy set to 28 days

2

Your configuration looks good, how are you determining that retention is still set to 30 days?

Also by roll back i presume you meant look back?

3

Sorry, I meant that the query time range is set to 30d1h, but shouldn’t it be 28 days since I set the lookback to 28 days?

Regarding the retention policy, I am not sure if my configuration is working correctly. I tried setting it for 24 hours or 48 hours, but the logs still persist beyond those times. I can retrieve logs older than that.

My goal for the retention policy is to delete logs older than 28 days to free up some space in my storage.

4

Let’s focus on your retention issue first. The compactor has a bit of delay, but if you are able to query much further back past your retention policy then something definitely is not working right. Your configuration seems correct to me. Double check your chunk and index files are stored correct on Minio, and consider setting compactor_address configuration in the common configuration block.

Also check the backend container log and see if there is anything interesting there.