Loki Retention Not Working (docker)

florian · May 19, 2026, 9:36am

I’m using Loki version 3.5.1 (image grafana/loki:3.5.1) ‘on-prem’ (single instance)

i’m using tsdb filesystem storage
i’m not using table manager
i turn on retention_enabled flag
my default retention is 360h (i reduce it to 48h for testing)
no errors in logs

The retention policy did not remove old chunks, how to delete them ?

Can you explain when chunks will be removed after there creation ? which one can i remove safetly ?

my current config :

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  grpc_server_max_concurrent_streams: 1000
  log_format: json

common:
  instance_addr: 127.0.0.1
  path_prefix: /mnt/loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

limits_config:
  allow_structured_metadata: true
  volume_enabled: true
  discover_log_levels: true
  log_level_fields:
    - level
    - LEVEL
    - Level
    - Severity
    - severity
    - SEVERITY
    - lvl
    - LVL
    - Lvl
    - severity_text
    - Severity_Text
    - SEVERITY_TEXT
    - LogLevel
  metric_aggregation_enabled: true
  retention_period: 360h
  max_global_streams_per_user: 0
  per_stream_rate_limit: 5Mb
  per_stream_rate_limit_burst: 20Mb

ingester_client:
  grpc_client_config:
    grpc_compression: 'gzip'

pattern_ingester:
  enabled: true
  metric_aggregation:
    loki_address: localhost:3100

ruler:
  alertmanager_url: http://localhost:9093

frontend:
  encoding: protobuf

analytics:
  reporting_enabled: false
compactor:
  #working_directory: /mnt/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 30m
  retention_delete_worker_count: 150
  delete_request_store: filesystem

Partial logs (ask for more info) :

2026-05-18T12:32:33.616835359Z {"caller":"main.go:126","level":"info","msg":"Starting Loki","ts":"2026-05-18T12:32:33.518887825Z","version":"(version=3.5.1, branch=release-3.5.x, revision=d4e637ce)"}
2026-05-18T12:32:33.616934725Z {"caller":"main.go:127","filename":"/etc/loki/config.yml","level":"info","msg":"Loading configuration file","ts":"2026-05-18T12:32:33.519792798Z"}
2026-05-18T12:32:33.616949791Z {"caller":"modules.go:1420","level":"info","msg":"Ruler storage is not configured; ruler will not be started.","ts":"2026-05-18T12:32:33.521213731Z"}
2026-05-18T12:32:33.616986595Z {"caller":"server.go:368","grpc":"[::]:9096","http":"[::]:3100","level":"info","msg":"server listening on addresses","ts":"2026-05-18T12:32:33.52315728Z"}
2026-05-18T12:32:33.616998258Z {"caller":"table_manager.go:448","err":"invalid table name","index-store":"tsdb-2020-10-24","level":"error","msg":"failed to load table","table-name":"index","ts":"2026-05-18T12:32:33.531837453Z"}
2026-05-18T12:32:33.617009419Z {"caller":"table_manager.go:300","distinct_users":"","distinct_users_len":0,"duration":"6.856µs","index-store":"tsdb-2020-10-24","level":"info","msg":"query readiness setup completed","ts":"2026-05-18T12:32:33.531885999Z"}
2026-05-18T12:32:33.617022541Z {"caller":"shipper.go:165","index-store":"tsdb-2020-10-24","level":"info","msg":"starting index shipper in RW mode","ts":"2026-05-18T12:32:33.531931095Z"}
2026-05-18T12:32:33.617032871Z {"caller":"table_manager.go:136","index-store":"tsdb-2020-10-24","level":"info","msg":"uploading tables","ts":"2026-05-18T12:32:33.531963782Z"}
2026-05-18T12:32:33.617043061Z {"caller":"head_manager.go:308","component":"tsdb-head-manager","groups":0,"index-store":"tsdb-2020-10-24","level":"info","msg":"loaded wals by period","ts":"2026-05-18T12:32:33.532714975Z"}
2026-05-18T12:32:33.617053998Z {"caller":"manager.go:110","component":"tsdb-manager","index-store":"tsdb-2020-10-24","level":"info","msg":"skip loading, table not in range: index","reason":"invalid table name","ts":"2026-05-18T12:32:33.532825866Z"}
2026-05-18T12:32:33.617065152Z {"buckets":0,"caller":"manager.go:86","component":"tsdb-manager","err":null,"failures":0,"index-store":"tsdb-2020-10-24","indices":0,"level":"info","msg":"loaded leftover local indices","successful":true,"ts":"2026-05-18T12:32:33.532852381Z"}
2026-05-18T12:32:33.617079312Z {"caller":"head_manager.go:308","component":"tsdb-head-manager","groups":0,"index-store":"tsdb-2020-10-24","level":"info","msg":"loaded wals by period","ts":"2026-05-18T12:32:33.53292981Z"}
2026-05-18T12:32:33.617090134Z {"caller":"worker.go:131","component":"querier","level":"info","msg":"Starting querier worker using query-scheduler and scheduler ring for addresses","ts":"2026-05-18T12:32:33.537976156Z"}
2026-05-18T12:32:33.617100963Z {"caller":"modules.go:1443","level":"warn","msg":"RulerStorage is nil. Not starting the ruler.","ts":"2026-05-18T12:32:33.542553454Z"}
2026-05-18T12:32:33.617111228Z {"caller":"module_service.go:82","level":"info","module":"cache-generation-loader","msg":"starting","ts":"2026-05-18T12:32:33.581613054Z"}
2026-05-18T12:32:33.617121619Z {"caller":"module_service.go:82","level":"info","module":"server","msg":"starting","ts":"2026-05-18T12:32:33.581656994Z"}
2026-05-18T12:32:33.617132048Z {"caller":"module_service.go:82","level":"info","module":"query-frontend-tripperware","msg":"starting","ts":"2026-05-18T12:32:33.581841556Z"}
2026-05-18T12:32:33.617142114Z {"caller":"module_service.go:82","level":"info","module":"memberlist-kv","msg":"starting","ts":"2026-05-18T12:32:33.582586261Z"}
2026-05-18T12:32:33.617152134Z {"caller":"module_service.go:82","level":"info","module":"query-scheduler-ring","msg":"starting","ts":"2026-05-18T12:32:33.582708252Z"}
2026-05-18T12:32:33.617167904Z {"caller":"module_service.go:82","level":"info","module":"store","msg":"starting","ts":"2026-05-18T12:32:33.582767069Z"}
2026-05-18T12:32:33.617177954Z {"caller":"ring.go:361","level":"info","msg":"ring doesn't exist in KV store yet","ts":"2026-05-18T12:32:33.582906063Z"}
2026-05-18T12:32:33.617192372Z {"caller":"module_service.go:82","level":"info","module":"ring","msg":"starting","ts":"2026-05-18T12:32:33.583193036Z"}
2026-05-18T12:32:33.617207050Z {"caller":"ring.go:361","level":"info","msg":"ring doesn't exist in KV store yet","ts":"2026-05-18T12:32:33.583374065Z"}
2026-05-18T12:32:33.617221870Z {"caller":"basic_lifecycler.go:301","instance":"e269dc9a5213","level":"info","msg":"instance not found in the ring","ring":"scheduler","ts":"2026-05-18T12:32:33.583263551Z"}
2026-05-18T12:32:33.617239618Z {"caller":"module_service.go:82","level":"info","module":"ingester","msg":"starting","ts":"2026-05-18T12:32:33.583455128Z"}
2026-05-18T12:32:33.617255382Z {"caller":"ingester.go:564","component":"ingester","level":"info","msg":"recovering from checkpoint","ts":"2026-05-18T12:32:33.583498654Z"}
2026-05-18T12:32:33.617268168Z {"caller":"module_service.go:82","level":"info","module":"ingester-querier","msg":"starting","ts":"2026-05-18T12:32:33.583545638Z"}
2026-05-18T12:32:33.617278260Z {"caller":"module_service.go:82","level":"info","module":"compactor","msg":"starting","ts":"2026-05-18T12:32:33.58356755Z"}
2026-05-18T12:32:33.617288362Z {"caller":"module_service.go:82","level":"info","module":"pattern-ring-client","msg":"starting","ts":"2026-05-18T12:32:33.583562482Z"}
2026-05-18T12:32:33.617302304Z {"caller":"basic_lifecycler_delegates.go:63","level":"info","msg":"not loading tokens from file, tokens file path is empty","ts":"2026-05-18T12:32:33.583478381Z"}
2026-05-18T12:32:33.617313027Z {"caller":"module_service.go:82","level":"info","module":"rule-evaluator","msg":"starting","ts":"2026-05-18T12:32:33.583647082Z"}
2026-05-18T12:32:33.617323151Z {"caller":"ring.go:361","level":"info","msg":"ring doesn't exist in KV store yet","ts":"2026-05-18T12:32:33.583673671Z"}
2026-05-18T12:32:33.617333142Z {"caller":"basic_lifecycler.go:301","instance":"e269dc9a5213","level":"info","msg":"instance not found in the ring","ring":"compactor","ts":"2026-05-18T12:32:33.583939554Z"}
2026-05-18T12:32:33.617349180Z {"caller":"ringmanager.go:186","level":"info","msg":"waiting until scheduler is JOINING in the ring","ts":"2026-05-18T12:32:33.584013497Z"}
2026-05-18T12:32:33.617366407Z {"caller":"basic_lifecycler_delegates.go:63","level":"info","msg":"not loading tokens from file, tokens file path is empty","ts":"2026-05-18T12:32:33.584020333Z"}
2026-05-18T12:32:33.617382416Z {"caller":"ring.go:361","component":"pattern-ring-client","level":"info","msg":"ring doesn't exist in KV store yet","ts":"2026-05-18T12:32:33.584361104Z"}
2026-05-18T12:32:33.617397427Z {"caller":"compactor.go:443","level":"info","msg":"waiting until compactor is JOINING in the ring","ts":"2026-05-18T12:32:33.584421803Z"}
2026-05-18T12:32:33.617435263Z {"caller":"compactor.go:447","level":"info","msg":"compactor is JOINING in the ring","ts":"2026-05-18T12:32:33.584445074Z"}
2026-05-18T12:32:33.617451492Z {"caller":"module_service.go:82","level":"info","module":"pattern-ingester-tee","msg":"starting","ts":"2026-05-18T12:32:33.584443189Z"}
2026-05-18T12:32:33.617467487Z {"caller":"compactor.go:457","level":"info","msg":"waiting until compactor is ACTIVE in the ring","ts":"2026-05-18T12:32:33.584501618Z"}
2026-05-18T12:32:33.617485521Z {"caller":"module_service.go:82","level":"info","module":"distributor","msg":"starting","ts":"2026-05-18T12:32:33.584916948Z"}
2026-05-18T12:32:33.617500996Z {"caller":"module_service.go:82","level":"info","module":"pattern-ingester","msg":"starting","ts":"2026-05-18T12:32:33.584925118Z"}
2026-05-18T12:32:33.617516456Z {"caller":"ratestore.go:109","err":"empty ring","level":"error","msg":"error getting ingester clients","ts":"2026-05-18T12:32:33.585000745Z"}
2026-05-18T12:32:33.617531997Z {"caller":"basic_lifecycler.go:301","component":"distributor","instance":"e269dc9a5213","level":"info","msg":"instance not found in the ring","ring":"distributor","ts":"2026-05-18T12:32:33.58514558Z"}
2026-05-18T12:32:33.617549537Z {"caller":"lifecycler.go:677","component":"pattern-ingester","level":"info","msg":"not loading tokens from file, tokens file path is empty","ts":"2026-05-18T12:32:33.586387873Z"}
2026-05-18T12:32:33.617567442Z {"caller":"lifecycler.go:704","component":"pattern-ingester","level":"info","msg":"instance not found in ring, adding with no tokens","ring":"pattern-ingester","ts":"2026-05-18T12:32:33.586452298Z"}
2026-05-18T12:32:33.617586668Z {"caller":"lifecycler.go:546","component":"pattern-ingester","level":"info","msg":"auto-joining cluster after timeout","ring":"pattern-ingester","ts":"2026-05-18T12:32:33.586547489Z"}
2026-05-18T12:32:33.617605416Z {"caller":"ingester.go:282","component":"pattern-ingester","delay":"36.729103219s","level":"info","msg":"sleeping for initial delay before starting periodic flushing","ts":"2026-05-18T12:32:33.58658155Z"}
2026-05-18T12:32:33.617624230Z {"caller":"ingester.go:580","component":"ingester","elapsed":"12.038109ms","errors":false,"level":"info","msg":"recovered WAL checkpoint recovery finished","ts":"2026-05-18T12:32:33.595535496Z"}
2026-05-18T12:32:33.617643148Z {"caller":"ingester.go:586","component":"ingester","level":"info","msg":"recovering from WAL","ts":"2026-05-18T12:32:33.595584366Z"}
2026-05-18T12:32:33.617660631Z {"caller":"ingester.go:602","component":"ingester","elapsed":"13.122093ms","errors":false,"level":"info","msg":"WAL segment recovery finished","ts":"2026-05-18T12:32:33.59661964Z"}
2026-05-18T12:32:33.617679391Z {"caller":"ingester.go:550","component":"ingester","level":"info","msg":"closing recoverer","ts":"2026-05-18T12:32:33.596662592Z"}
2026-05-18T12:32:33.617696851Z {"caller":"ingester.go:558","component":"ingester","level":"info","msg":"WAL recovery finished","time":"13.197855ms","ts":"2026-05-18T12:32:33.596693847Z"}
2026-05-18T12:32:33.617723847Z {"caller":"wal.go:157","component":"wal","level":"info","msg":"started","ts":"2026-05-18T12:32:33.596745222Z"}
2026-05-18T12:32:33.617738341Z {"caller":"ingester.go:771","component":"ingester","delay":"21.832185716s","level":"info","msg":"sleeping for initial delay before starting periodic flushing","ts":"2026-05-18T12:32:33.596810811Z"}
2026-05-18T12:32:33.617754949Z {"caller":"lifecycler.go:677","component":"ingester","level":"info","msg":"not loading tokens from file, tokens file path is empty","ts":"2026-05-18T12:32:33.59680403Z"}
2026-05-18T12:32:33.617770501Z {"caller":"lifecycler.go:704","component":"ingester","level":"info","msg":"instance not found in ring, adding with no tokens","ring":"ingester","ts":"2026-05-18T12:32:33.596938979Z"}
2026-05-18T12:32:33.617841985Z {"caller":"lifecycler.go:546","component":"ingester","level":"info","msg":"auto-joining cluster after timeout","ring":"ingester","ts":"2026-05-18T12:32:33.597042478Z"}
2026-05-18T12:32:33.717351728Z {"caller":"ringmanager.go:190","level":"info","msg":"scheduler is JOINING in the ring","ts":"2026-05-18T12:32:33.709703617Z"}
2026-05-18T12:32:33.717406399Z {"caller":"ringmanager.go:199","level":"info","msg":"waiting until scheduler is ACTIVE in the ring","ts":"2026-05-18T12:32:33.710080827Z"}
2026-05-18T12:32:33.817305186Z {"caller":"compactor.go:461","level":"info","msg":"compactor is ACTIVE in the ring","ts":"2026-05-18T12:32:33.77675698Z"}
2026-05-18T12:32:33.879095633Z {"caller":"ringmanager.go:203","level":"info","msg":"scheduler is ACTIVE in the ring","ts":"2026-05-18T12:32:33.877869344Z"}
2026-05-18T12:32:33.879147462Z {"caller":"module_service.go:82","level":"info","module":"query-scheduler","msg":"starting","ts":"2026-05-18T12:32:33.877972807Z"}
2026-05-18T12:32:33.879154221Z {"caller":"module_service.go:82","level":"info","module":"querier","msg":"starting","ts":"2026-05-18T12:32:33.878234741Z"}
2026-05-18T12:32:33.879159505Z {"caller":"module_service.go:82","level":"info","module":"query-frontend","msg":"starting","ts":"2026-05-18T12:32:33.878353504Z"}
2026-05-18T12:32:33.879164360Z {"caller":"loki.go:581","level":"info","msg":"Loki started","startup_time":"514.262901ms","ts":"2026-05-18T12:32:33.878498365Z"}
2026-05-18T12:32:36.917156152Z {"addr":"127.0.0.1:9096","caller":"worker.go:232","component":"querier","level":"info","msg":"adding connection","ts":"2026-05-18T12:32:36.878495683Z"}
2026-05-18T12:32:36.917201465Z {"caller":"scheduler.go:653","level":"info","msg":"this scheduler is in the ReplicationSet, will now accept requests.","ts":"2026-05-18T12:32:36.878544766Z"}
2026-05-18T12:32:38.817503865Z {"caller":"compactor.go:522","level":"info","msg":"this instance has been chosen to run the compactor, starting compactor","ts":"2026-05-18T12:32:38.777943808Z"}
2026-05-18T12:32:38.817568005Z {"caller":"compactor.go:551","level":"info","msg":"waiting 10m0s for ring to stay stable and previous compactions to finish before starting compactor","ts":"2026-05-18T12:32:38.778077242Z"}
2026-05-18T12:32:43.916805059Z {"addr":"127.0.0.1:9096","caller":"frontend_scheduler_worker.go:107","level":"info","msg":"adding connection to scheduler","ts":"2026-05-18T12:32:43.879209232Z"}
2026-05-18T12:33:03.616703027Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:33:03.597834688Z"}
2026-05-18T12:33:03.616765618Z {"caller":"recalculate_owned_streams.go:63","level":"info","msg":"detected ring changes, re-evaluating streams ownership","ts":"2026-05-18T12:33:03.597912582Z"}
2026-05-18T12:33:03.616770849Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:33:03.597926176Z"}
2026-05-18T12:33:33.616718885Z {"caller":"table_manager.go:136","index-store":"tsdb-2020-10-24","level":"info","msg":"uploading tables","ts":"2026-05-18T12:33:33.533072235Z"}
2026-05-18T12:33:33.616830498Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:33:33.59766032Z"}
2026-05-18T12:33:33.616847168Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:33:33.597802586Z"}
2026-05-18T12:34:03.616537372Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:34:03.597014387Z"}
2026-05-18T12:34:03.616628275Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:34:03.597089251Z"}
2026-05-18T12:34:33.617466007Z {"caller":"table_manager.go:136","index-store":"tsdb-2020-10-24","level":"info","msg":"uploading tables","ts":"2026-05-18T12:34:33.532578991Z"}
2026-05-18T12:34:33.617564649Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:34:33.597840422Z"}
2026-05-18T12:34:33.617591762Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:34:33.597907849Z"}
2026-05-18T12:35:03.616541126Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:35:03.597643671Z"}
2026-05-18T12:35:03.616618229Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:35:03.597792851Z"}
2026-05-18T12:35:33.616926723Z {"caller":"table_manager.go:136","index-store":"tsdb-2020-10-24","level":"info","msg":"uploading tables","ts":"2026-05-18T12:35:33.5324071Z"}
2026-05-18T12:35:33.617012353Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:35:33.597309972Z"}
2026-05-18T12:35:33.617020638Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:35:33.597409232Z"}
2026-05-18T12:36:03.617203060Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:36:03.597561696Z"}
2026-05-18T12:36:03.617278775Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:36:03.597669817Z"}
2026-05-18T12:36:12.416801572Z {"avg_comp":"145 B","avg_uncomp":"3.4 kB","caller":"flush.go:304","component":"ingester","fp":"1e20b7abfb0b36b5","immediate":false,"labels":"{container_name=\"grafana\", service_name=\"grafana\"}","level":"info","max_age":1,"msg":"flushing stream","num_chunks":1,"total_comp":"145 B","total_uncomp":"3.4 kB","ts":"2026-05-18T12:36:12.407493002Z","user":"fake"}
2026-05-18T12:36:33.616826916Z {"caller":"table_manager.go:136","index-store":"tsdb-2020-10-24","level":"info","msg":"uploading tables","ts":"2026-05-18T12:36:33.532126386Z"}
2026-05-18T12:36:33.616908870Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:36:33.59731318Z"}
2026-05-18T12:36:33.616916848Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:36:33.597371857Z"}
2026-05-18T12:37:03.616619525Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:37:03.596937445Z"}
2026-05-18T12:37:03.616718313Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:37:03.597012576Z"}
2026-05-18T12:37:33.616803773Z {"caller":"table_manager.go:239","index-store":"tsdb-2020-10-24","level":"info","msg":"syncing tables","ts":"2026-05-18T12:37:33.532645344Z"}
2026-05-18T12:37:33.616904924Z {"caller":"table_manager.go:136","index-store":"tsdb-2020-10-24","level":"info","msg":"uploading tables","ts":"2026-05-18T12:37:33.532684441Z"}
2026-05-18T12:37:33.616912120Z {"caller":"table_manager.go:300","distinct_users":"","distinct_users_len":0,"duration":"3.354µs","index-store":"tsdb-2020-10-24","level":"info","msg":"query readiness setup completed","ts":"2026-05-18T12:37:33.532765482Z"}
2026-05-18T12:37:33.616919062Z {"caller":"checkpoint.go:611","level":"info","msg":"starting checkpoint","ts":"2026-05-18T12:37:33.596829411Z"}
2026-05-18T12:37:33.616922929Z {"caller":"checkpoint.go:336","dir":"/mnt/loki/wal/checkpoint.080965","level":"info","msg":"attempting checkpoint for","ts":"2026-05-18T12:37:33.597187695Z"}
2026-05-18T12:37:33.616926585Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:37:33.597222063Z"}
2026-05-18T12:37:33.616930782Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:37:33.597286958Z"}
2026-05-18T12:38:03.616856421Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:38:03.597330275Z"}
2026-05-18T12:38:03.616938440Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:38:03.597459029Z"}
2026-05-18T12:38:33.617131083Z {"caller":"table_manager.go:136","index-store":"tsdb-2020-10-24","level":"info","msg":"uploading tables","ts":"2026-05-18T12:38:33.532108751Z"}
2026-05-18T12:38:33.617241054Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:38:33.597544429Z"}
2026-05-18T12:38:33.617250598Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:38:33.597621759Z"}
2026-05-18T12:39:03.616839000Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:39:03.597318104Z"}
2026-05-18T12:39:03.616930602Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:39:03.59747835Z"}
2026-05-18T12:39:33.616635485Z {"caller":"table_manager.go:136","index-store":"tsdb-2020-10-24","level":"info","msg":"uploading tables","ts":"2026-05-18T12:39:33.532429732Z"}
2026-05-18T12:39:33.616732017Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-18T12:39:33.597469846Z"}
2026-05-18T12:39:33.616742801Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-18T12:39:33.597554356Z"}
...
2026-05-19T09:02:53.396338936Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20577","ts":"2026-05-19T09:02:53.29508414Z"}
2026-05-19T09:02:53.396373857Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20577","ts":"2026-05-19T09:02:53.295092694Z"}
2026-05-19T09:02:53.396500557Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20577","ts":"2026-05-19T09:02:53.295101139Z"}
2026-05-19T09:02:53.396534025Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20591","ts":"2026-05-19T09:02:53.295110224Z"}
2026-05-19T09:02:53.396567926Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20591","ts":"2026-05-19T09:02:53.295119554Z"}
2026-05-19T09:02:53.396608901Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20591","ts":"2026-05-19T09:02:53.295129729Z"}
2026-05-19T09:02:53.581447015Z {"caller":"util.go:77","file-name":"1779180353-4aa9c080e461-1750753121782741574.tsdb.gz","level":"info","msg":"downloaded file","table-name":"index_20592","total_time":"267.229µs","ts":"2026-05-19T09:02:53.528699325Z"}
2026-05-19T09:02:53.581525929Z {"caller":"util.go:124","download time":"267.229µs","extract time":"2.738558ms","file-name":"1779180353-4aa9c080e461-1750753121782741574.tsdb.gz","level":"info","msg":"downloaded and extracted file","size":"820 B","table-name":"index_20592","ts":"2026-05-19T09:02:53.531474666Z"}
2026-05-19T09:02:53.681591012Z {"caller":"checkpoint.go:611","level":"info","msg":"starting checkpoint","ts":"2026-05-19T09:02:53.596949631Z"}
2026-05-19T09:02:53.681637699Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-19T09:02:53.59712076Z"}
2026-05-19T09:02:53.681642967Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-19T09:02:53.597180404Z"}
2026-05-19T09:02:53.681646662Z {"caller":"checkpoint.go:336","dir":"/mnt/loki/wal/checkpoint.079543","level":"info","msg":"attempting checkpoint for","ts":"2026-05-19T09:02:53.597537455Z"}
2026-05-19T09:02:53.782533392Z {"caller":"table_manager.go:300","distinct_users":"","distinct_users_len":0,"duration":"2.322µs","index-store":"tsdb-2020-10-24","level":"info","msg":"query readiness setup completed","ts":"2026-05-19T09:02:53.722637777Z"}
2026-05-19T09:03:02.982222874Z {"caller":"marker.go:202","level":"info","msg":"no marks file found","ts":"2026-05-19T09:03:02.930259056Z"}
2026-05-19T09:03:05.282279476Z {"caller":"expiration.go:81","level":"info","msg":"overall smallest retention period 1779008585.193, default smallest retention period 1779008585.193","ts":"2026-05-19T09:03:05.193125526Z"}
2026-05-19T09:03:05.282348884Z {"caller":"compactor.go:798","level":"info","msg":"compacting table","table-name":"index_20592","ts":"2026-05-19T09:03:05.193934009Z"}
2026-05-19T09:03:05.282355132Z {"caller":"table.go:132","count":1,"level":"info","msg":"listed files","table-name":"index_20592","ts":"2026-05-19T09:03:05.19440599Z"}
2026-05-19T09:03:05.282359584Z {"caller":"util.go:77","file-name":"1779180353-4aa9c080e461-1750753121782741574.tsdb.gz","level":"info","msg":"downloaded file","table-name":"index_20592","total_time":"195.356µs","ts":"2026-05-19T09:03:05.194854308Z"}
2026-05-19T09:03:05.282365145Z {"caller":"util.go:124","download time":"195.356µs","extract time":"5.710783ms","file-name":"1779180353-4aa9c080e461-1750753121782741574.tsdb.gz","level":"info","msg":"downloaded and extracted file","size":"820 B","table-name":"index_20592","ts":"2026-05-19T09:03:05.200595947Z"}
2026-05-19T09:03:05.282369561Z {"caller":"util.go:77","file-name":"1779180482960797699-compactor-1778578725881-1779178259740-49240e8d.tsdb.gz","level":"info","msg":"downloaded file","table-name":"index_20592","total_time":"282.3µs","ts":"2026-05-19T09:03:05.201282821Z","user-id":"fake"}
2026-05-19T09:03:05.282373697Z {"caller":"util.go:124","download time":"282.3µs","extract time":"482.105µs","file-name":"1779180482960797699-compactor-1778578725881-1779178259740-49240e8d.tsdb.gz","level":"info","msg":"downloaded and extracted file","size":"18 kB","table-name":"index_20592","ts":"2026-05-19T09:03:05.20180198Z","user-id":"fake"}
2026-05-19T09:03:05.282378062Z {"caller":"index_set.go:270","count":1,"level":"info","msg":"removing source db files from storage","table-name":"index_20592","ts":"2026-05-19T09:03:05.234801145Z","user-id":"fake"}
2026-05-19T09:03:05.282381830Z {"caller":"index_set.go:270","count":1,"level":"info","msg":"removing source db files from storage","table-name":"index_20592","ts":"2026-05-19T09:03:05.235156289Z"}
2026-05-19T09:03:05.282399517Z {"caller":"compactor.go:803","level":"info","msg":"finished compacting table","table-name":"index_20592","ts":"2026-05-19T09:03:05.235476818Z"}
2026-05-19T09:03:05.282404480Z {"caller":"compactor.go:798","level":"info","msg":"compacting table","table-name":"index_20591","ts":"2026-05-19T09:03:05.235514713Z"}
2026-05-19T09:03:05.282408067Z {"caller":"table.go:132","count":0,"level":"info","msg":"listed files","table-name":"index_20591","ts":"2026-05-19T09:03:05.235837311Z"}
2026-05-19T09:03:05.282411709Z {"caller":"compactor.go:803","level":"info","msg":"finished compacting table","table-name":"index_20591","ts":"2026-05-19T09:03:05.236148783Z"}
...

Thanks a lot for helping

Florian

Cass · May 19, 2026, 3:07pm

Hi Florian,

From the logs, the compactor seems to be running, but I would check a few things:

Retention in Loki is not an immediate physical deletion of chunk files. The compactor first rewrites the index, removes references to expired chunks, writes marker files, and then the sweeper deletes the chunks asynchronously after retention_delete_delay.
Your logs show “no marks file found”, which may indicate that the compactor has not marked any chunks for deletion yet.
Also, retention changes are not retroactive. If you reduced retention_period from 360h to 48h for testing, previously ingested data may not be affected the way you expect.
I would explicitly configure and persist the compactor working_directory. In your config it is commented:

compactor:
working_directory: /mnt/loki/compactor
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 30m
retention_delete_worker_count: 150
delete_request_store: filesystem

Make sure /mnt/loki/compactor is writable and persisted across restarts.

Since you are using filesystem storage, do not delete random chunk files manually unless you are deleting the entire Loki storage and accept data loss. Loki stores chunks under the tenant folder, usually “fake” when auth_enabled: false. Manual deletion can leave index references pointing to missing chunks.

I would check:

Are there marker files under the compactor working directory?
Are the chunks actually older than the configured retention based on log timestamps, not filesystem mtime?
Can you still query those old logs in Grafana Explore?
Is the mounted filesystem persistent and writable by the Loki container?
What does Loki print with -print-config-stderr to confirm the effective config?

Hope it helps!

florian · May 19, 2026, 4:14pm

Thanks for the reply.

1 => yes i understand that

2 => ok

3 => i already read that from the loki documentation, thanks to specify it. i recently change the retention but i got chunks older than one year. BUT, my index prefix was index/v13/, this index seems not supported in whole loki’s modules. Then i rewrite all files with index/V13/ to index_ (btw it must not change writed chunks…). Can i remove manualy oldest chunks safetly ? if needed i can read encoded file name to know when remove the chunk.

4 => ok (sub folder are correctly created under compactor folder without any explicit configuration - this is the default value shown in loki:3100/config uri)

Sure, /mnt/loki/compactor is writable and persisted across restarts, this folder /mnt/loki is binded to the host

About marker files :
i got some marker created sometime, but one at same time in the folder (i didn’t see more than one marker)
I guess some related logs :

2026-05-19T09:32:53.682593341Z {"caller":"util.go:77","file-name":"1779182153-4aa9c080e461-1750753121782741574.tsdb.gz","level":"info","msg":"downloaded file","table-name":"index_20592","total_time":"216.282µs","ts":"2026-05-19T09:32:53.641459894Z"}
2026-05-19T09:32:53.682607714Z {"caller":"util.go:124","download time":"216.282µs","extract time":"3.62267ms","file-name":"1779182153-4aa9c080e461-1750753121782741574.tsdb.gz","level":"info","msg":"downloaded and extracted file","size":"1.6 kB","table-name":"index_20592","ts":"2026-05-19T09:32:53.645117348Z"}
2026-05-19T09:32:53.781272713Z {"caller":"table_manager.go:300","distinct_users":"","distinct_users_len":0,"duration":"2.317µs","index-store":"tsdb-2020-10-24","level":"info","msg":"query readiness setup completed","ts":"2026-05-19T09:32:53.764792913Z"}
2026-05-19T09:33:02.981481011Z {"caller":"marker.go:202","level":"info","msg":"no marks file found","ts":"2026-05-19T09:33:02.929713714Z"}
2026-05-19T09:33:05.281795034Z {"caller":"expiration.go:81","level":"info","msg":"overall smallest retention period 1779010385.192, default smallest retention period 1779010385.192","ts":"2026-05-19T09:33:05.192524757Z"}
2026-05-19T09:33:05.281880658Z {"caller":"compactor.go:798","level":"info","msg":"compacting table","table-name":"index_20592","ts":"2026-05-19T09:33:05.193093246Z"}
2026-05-19T09:33:05.281888555Z {"caller":"table.go:132","count":1,"level":"info","msg":"listed files","table-name":"index_20592","ts":"2026-05-19T09:33:05.193403752Z"}
2026-05-19T09:33:05.281893810Z {"caller":"util.go:77","file-name":"1779182153-4aa9c080e461-1750753121782741574.tsdb.gz","level":"info","msg":"downloaded file","table-name":"index_20592","total_time":"263.228µs","ts":"2026-05-19T09:33:05.193789683Z"}
2026-05-19T09:33:05.281900255Z {"caller":"util.go:124","download time":"263.228µs","extract time":"1.382519ms","file-name":"1779182153-4aa9c080e461-1750753121782741574.tsdb.gz","level":"info","msg":"downloaded and extracted file","size":"1.6 kB","table-name":"index_20592","ts":"2026-05-19T09:33:05.195202844Z"}
2026-05-19T09:33:05.281906306Z {"caller":"util.go:77","file-name":"1779182282951670049-compactor-1778578725881-1779181862982-6c4ee932.tsdb.gz","level":"info","msg":"downloaded file","table-name":"index_20592","total_time":"176.22µs","ts":"2026-05-19T09:33:05.195656889Z","user-id":"fake"}
2026-05-19T09:33:05.281911885Z {"caller":"util.go:124","download time":"176.22µs","extract time":"408.116µs","file-name":"1779182282951670049-compactor-1778578725881-1779181862982-6c4ee932.tsdb.gz","level":"info","msg":"downloaded and extracted file","size":"19 kB","table-name":"index_20592","ts":"2026-05-19T09:33:05.196091613Z","user-id":"fake"}
2026-05-19T09:33:05.281917523Z {"caller":"index_set.go:270","count":1,"level":"info","msg":"removing source db files from storage","table-name":"index_20592","ts":"2026-05-19T09:33:05.230978796Z","user-id":"fake"}
2026-05-19T09:33:05.281933414Z {"caller":"index_set.go:270","count":1,"level":"info","msg":"removing source db files from storage","table-name":"index_20592","ts":"2026-05-19T09:33:05.231583851Z"}
2026-05-19T09:33:05.281938495Z {"caller":"compactor.go:803","level":"info","msg":"finished compacting table","table-name":"index_20592","ts":"2026-05-19T09:33:05.232390324Z"}
2026-05-19T09:33:05.281944307Z {"caller":"compactor.go:798","level":"info","msg":"compacting table","table-name":"index_20591","ts":"2026-05-19T09:33:05.232482635Z"}
2026-05-19T09:33:05.281949051Z {"caller":"table.go:132","count":0,"level":"info","msg":"listed files","table-name":"index_20591","ts":"2026-05-19T09:33:05.233032349Z"}
2026-05-19T09:33:05.281954078Z {"caller":"compactor.go:803","level":"info","msg":"finished compacting table","table-name":"index_20591","ts":"2026-05-19T09:33:05.234016819Z"}
2026-05-19T09:33:05.281958674Z {"caller":"compactor.go:798","level":"info","msg":"compacting table","table-name":"index_20590","ts":"2026-05-19T09:33:05.234118326Z"}
2026-05-19T09:33:05.281963190Z {"caller":"table.go:132","count":0,"level":"info","msg":"listed files","table-name":"index_20590","ts":"2026-05-19T09:33:05.234624153Z"}
2026-05-19T09:33:05.281971404Z {"caller":"util.go:77","file-name":"1779178678826313819-compactor-1778578725881-1779178119725-2de71813.tsdb.gz","level":"info","msg":"downloaded file","table-name":"index_20590","total_time":"419.95µs","ts":"2026-05-19T09:33:05.235281176Z","user-id":"fake"}
2026-05-19T09:33:05.282020045Z {"caller":"util.go:124","download time":"419.95µs","extract time":"5.013283ms","file-name":"1779178678826313819-compactor-1778578725881-1779178119725-2de71813.tsdb.gz","level":"info","msg":"downloaded and extracted file","size":"2.7 kB","table-name":"index_20590","ts":"2026-05-19T09:33:05.240351627Z","user-id":"fake"}
2026-05-19T09:33:05.282026439Z {"caller":"marker.go:77","file":"/mnt/loki/compactor/retention/filesystem_2020-10-24/markers/1779183185240712507","level":"info","msg":"mark file created","ts":"2026-05-19T09:33:05.242617579Z"}
2026-05-19T09:33:05.282031542Z {"caller":"compactor.go:803","level":"info","msg":"finished compacting table","table-name":"index_20590","ts":"2026-05-19T09:33:05.24702846Z"}
2026-05-19T09:33:23.682294740Z {"caller":"recalculate_owned_streams.go:49","level":"info","msg":"starting recalculate owned streams job","ts":"2026-05-19T09:33:23.597319536Z"}
2026-05-19T09:33:23.682351064Z {"caller":"recalculate_owned_streams.go:52","level":"info","msg":"completed recalculate owned streams job","ts":"2026-05-19T09:33:23.597442467Z"}

About Older chunks
Yes, i got files with more than a year, same date between filesystem and timestamp file name

About Old logs :
I got no logs older than the retention date

About persistant filesystem:
Yes, folder is persisted and writable

About the configuration :
Sorry i don’t know what did you says about, is it a command flag used on container start ? is it an internal command to type inside my container ?

I really appreciate your time

Florian

Cass · May 19, 2026, 5:31pm

Dear florian, thanks for the new information.

About the configuration: yes, I meant the Loki startup/configuration, not a command to type manually inside Loki.

It can be defined in the Loki YAML config file, in the command/args used to start the container, or in Helm values if you use Kubernetes.

The simplest way to check the effective configuration is:

http://:3100/config

or from inside the container:

wget -qO- http://localhost:3100/config

This shows the real config Loki is currently using, including defaults and flags. I was mainly interested in the compactor, retention, storage and schema sections.

About your new information: if you cannot query logs older than the retention period anymore, then retention is probably working correctly at the index/query level.

The remaining old chunk files may be orphaned chunks, especially because you mentioned that the previous index prefix was `index/v13/` and you later rewrote it to `index_`. If the current TSDB index no longer references those chunks, Loki will not return them in queries, even if the chunk files are still physically present on disk.

I would still be careful with manual deletion. Loki retention normally works like this: the compactor updates the index, writes marker files for chunks to delete, and then the sweeper deletes the chunks asynchronously after `retention_delete_delay`.

Your log confirms that a marker file is being created:

/mnt/loki/compactor/retention/filesystem_2020-10-24/markers/1779183185240712507

So I would first wait for the delete delay and check whether the marked chunks are removed.

Before manually deleting old chunks, I would try to confirm whether they are still referenced by the current index. If they are not queryable anymore and they belong to the old broken prefix/layout, then they are probably orphaned files. In that case, deleting them may be safe from a Loki query perspective, but I would still recommend making a backup or moving them to a quarantine folder first instead of deleting them directly.

For example:

mkdir -p /mnt/loki/quarantine-old-chunks
mv /mnt/loki/quarantine-old-chunks/

Then restart Loki or wait a bit and test queries again. If everything works and no query errors appear, you can delete them later.

So, in short:

If the logs are still queryable, do not delete the chunks manually.
If the logs are not queryable anymore, and the chunks are from the old index layout, they may be orphaned.
Prefer moving them aside first instead of deleting them permanently.
Since marker files are being created, I would also check if the sweeper eventually removes them automatically.

florian · May 20, 2026, 9:22am

Dear Cass,

I cannot upload a file, then i extract mentions section only

storage_config:
  boltdb:
    directory: ""
  filesystem:
    directory: ""
  hedging:
    at: 0s
    up_to: 2
    max_per_second: 5
  named_stores:
    aws: {}
    azure: {}
    bos: {}
    filesystem: {}
    gcs: {}
    alibabacloud: {}
    swift: {}
    cos: {}
  index_cache_validity: 5m0s
  congestion_control:
    enabled: false
    controller:
      strategy: ""
      aimd:
        start: 2000
        upper_bound: 10000
        backoff_factor: 0.5
    retry:
      strategy: ""
      limit: 2
    hedging:
      config:
        at: 0s
        up_to: 0
        max_per_second: 0
      strategy: ""
  object_prefix: ""
  max_parallel_get_chunk: 150
  use_thanos_objstore: false
  object_store:
    filesystem:
      dir: /mnt/loki
    storage_prefix: ""
    named_stores:
      azure: {}
      filesystem: {}
      gcs: {}
      s3: {}
      swift: {}
  max_chunk_batch_size: 50
  boltdb_shipper:
    active_index_directory: ""
    cache_location: ""
    cache_ttl: 24h0m0s
    resync_interval: 5m0s
    query_ready_num_days: 0
    index_gateway_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: ""
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
          min_period: 100ms
          max_period: 10s
          max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
        cluster_validation:
          label: ""
      log_gateway_requests: false
    ingestername: e269dc9a5213
    mode: RW
    ingesterdbretainperiod: 20m0s
    build_per_tenant_index: false
  tsdb_shipper:
    active_index_directory: /mnt/loki/tsdb-shipper-active
    cache_location: /mnt/loki/tsdb-shipper-cache
    cache_ttl: 1h0m0s
    resync_interval: 5m0s
    query_ready_num_days: 0
    index_gateway_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: ""
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
          min_period: 100ms
          max_period: 10s
          max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
        cluster_validation:
          label: ""
      log_gateway_requests: false
    ingestername: e269dc9a5213
    mode: RW
    ingesterdbretainperiod: 20m0s
chunk_store_config:
  chunk_cache_config:
    default_validity: 1h0m0s
    background:
      writeback_goroutines: 1
      writeback_buffer: 500000
      writeback_size_limit: 500MB
    memcached:
      expiration: 0s
      batch_size: 4
      parallelism: 5
    memcached_client:
      host: ""
      service: memcached
      addresses: ""
      timeout: 100ms
      max_idle_conns: 16
      max_item_size: 0
      update_interval: 1m0s
      consistent_hash: true
      circuit_breaker_consecutive_failures: 10
      circuit_breaker_timeout: 10s
      circuit_breaker_interval: 10s
      tls_enabled: false
      tls_cert_path: ""
      tls_key_path: ""
      tls_ca_path: ""
      tls_server_name: ""
      tls_insecure_skip_verify: false
      tls_cipher_suites: ""
      tls_min_version: ""
    redis:
      endpoint: ""
      master_name: ""
      timeout: 500ms
      expiration: 0s
      db: 0
      pool_size: 0
      username: ""
      password: ""
      tls_enabled: false
      tls_insecure_skip_verify: false
      idle_timeout: 0s
      max_connection_age: 0s
      route_randomly: false
    embedded_cache:
      enabled: true
      max_size_mb: 100
      max_size_items: 0
      ttl: 1h0m0s
    prefix: store.chunks-cache.
  chunk_cache_config_l2:
    default_validity: 1h0m0s
    background:
      writeback_goroutines: 1
      writeback_buffer: 500000
      writeback_size_limit: 500MB
    memcached:
      expiration: 0s
      batch_size: 4
      parallelism: 5
    memcached_client:
      host: ""
      service: memcached
      addresses: ""
      timeout: 100ms
      max_idle_conns: 16
      max_item_size: 0
      update_interval: 1m0s
      consistent_hash: true
      circuit_breaker_consecutive_failures: 10
      circuit_breaker_timeout: 10s
      circuit_breaker_interval: 10s
      tls_enabled: false
      tls_cert_path: ""
      tls_key_path: ""
      tls_ca_path: ""
      tls_server_name: ""
      tls_insecure_skip_verify: false
      tls_cipher_suites: ""
      tls_min_version: ""
    redis:
      endpoint: ""
      master_name: ""
      timeout: 500ms
      expiration: 0s
      db: 0
      pool_size: 0
      username: ""
      password: ""
      tls_enabled: false
      tls_insecure_skip_verify: false
      idle_timeout: 0s
      max_connection_age: 0s
      route_randomly: false
    embedded_cache:
      max_size_mb: 100
      max_size_items: 0
      ttl: 1h0m0s
    prefix: store.chunks-cache-l2.
  write_dedupe_cache_config:
    default_validity: 1h0m0s
    background:
      writeback_goroutines: 1
      writeback_buffer: 500000
      writeback_size_limit: 500MB
    memcached:
      expiration: 0s
      batch_size: 4
      parallelism: 5
    memcached_client:
      host: ""
      service: memcached
      addresses: ""
      timeout: 100ms
      max_idle_conns: 16
      max_item_size: 0
      update_interval: 1m0s
      consistent_hash: true
      circuit_breaker_consecutive_failures: 10
      circuit_breaker_timeout: 10s
      circuit_breaker_interval: 10s
      tls_enabled: false
      tls_cert_path: ""
      tls_key_path: ""
      tls_ca_path: ""
      tls_server_name: ""
      tls_insecure_skip_verify: false
      tls_cipher_suites: ""
      tls_min_version: ""
    redis:
      endpoint: ""
      master_name: ""
      timeout: 500ms
      expiration: 0s
      db: 0
      pool_size: 0
      username: ""
      password: ""
      tls_enabled: false
      tls_insecure_skip_verify: false
      idle_timeout: 0s
      max_connection_age: 0s
      route_randomly: false
    embedded_cache:
      max_size_mb: 100
      max_size_items: 0
      ttl: 1h0m0s
    prefix: store.index-cache-write.
  skip_query_writeback_cache_older_than: 0s
  l2_chunk_cache_handoff: 0s
  cache_lookups_older_than: 0s
schema_config:
  configs:
  - from: "2020-10-24"
    store: tsdb
    object_store: filesystem
    schema: v13
    index:
      path_prefix: index/
      prefix: index_
      period: 1d
      tags: {}
    chunks:
      prefix: ""
      period: 0s
      tags: {}
    row_shards: 16
compactor:
  working_directory: /mnt/loki/compactor
  compaction_interval: 10m0s
  apply_retention_interval: 15m0s
  retention_enabled: true
  retention_delete_delay: 30m0s
  retention_delete_worker_count: 150
  retention_table_timeout: 0s
  retention_backoff_config:
    min_period: 100ms
    max_period: 10s
    max_retries: 10
  delete_request_store: filesystem
  delete_request_store_key_prefix: index/
  delete_request_store_db_type: boltdb
  backup_delete_request_store_db_type: ""
  delete_batch_size: 70
  delete_request_cancel_period: 24h0m0s
  delete_max_interval: 24h0m0s
  max_compaction_parallelism: 1
  upload_parallelism: 10
  compactor_ring:
    kvstore:
      store: inmemory
      prefix: collectors/
      consul:
        host: localhost:8500
        acl_token: ""
        http_client_timeout: 20s
        consistent_reads: false
        watch_rate_limit: 1
        watch_burst_size: 1
        cas_retry_delay: 1s
      etcd:
        endpoints: []
        dial_timeout: 10s
        max_retries: 10
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        username: ""
        password: ""
      multi:
        primary: ""
        secondary: ""
        mirror_enabled: false
        mirror_timeout: 2s
    heartbeat_period: 15s
    heartbeat_timeout: 1m0s
    tokens_file_path: ""
    zone_awareness_enabled: false
    num_tokens: 1
    replication_factor: 1
    instance_id: e269dc9a5213
    instance_interface_names:
    - eth0
    - lo
    instance_port: 0
    instance_addr: 127.0.0.1
    instance_availability_zone: ""
    instance_enable_ipv6: false
  _: false
  tables_to_compact: 0
  skip_latest_n_tables: 0
limits_config:
  ingestion_rate_strategy: global
  ingestion_rate_mb: 4
  ingestion_burst_size_mb: 6
  max_label_name_length: 1024
  max_label_value_length: 2048
  max_label_names_per_series: 15
  reject_old_samples: true
  reject_old_samples_max_age: 1w
  creation_grace_period: 10m
  max_line_size: 256KB
  max_line_size_truncate: false
  increment_duplicate_timestamp: false
  simulated_push_latency: 0s
  enable_multi_variant_queries: false
  discover_generic_fields: {}
  discover_service_name:
  - service
  - app
  - application
  - app_name
  - name
  - app_kubernetes_io_name
  - container
  - container_name
  - k8s_container_name
  - component
  - workload
  - job
  - k8s_job_name
  discover_log_levels: true
  log_level_fields:
  - level
  - LEVEL
  - Level
  - Severity
  - severity
  - SEVERITY
  - lvl
  - LVL
  - Lvl
  - severity_text
  - Severity_Text
  - SEVERITY_TEXT
  - LogLevel
  log_level_from_json_max_depth: 2
  use_owned_stream_count: false
  max_streams_per_user: 0
  max_global_streams_per_user: 0
  unordered_writes: true
  per_stream_rate_limit: 5MB
  per_stream_rate_limit_burst: 20MB
  max_chunks_per_query: 2000000
  max_query_series: 500
  max_query_lookback: 0s
  max_query_length: 30d1h
  max_query_range: 0s
  max_query_parallelism: 32
  tsdb_max_query_parallelism: 128
  tsdb_max_bytes_per_shard: 600MB
  tsdb_sharding_strategy: power_of_two
  tsdb_precompute_chunks: false
  cardinality_limit: 100000
  max_streams_matchers_per_query: 1000
  max_concurrent_tail_requests: 10
  max_entries_limit_per_query: 5000
  max_cache_freshness_per_query: 10m
  max_metadata_cache_freshness: 1d
  max_stats_cache_freshness: 10m
  max_queriers_per_tenant: 0
  max_query_capacity: 0
  query_ready_index_num_days: 0
  query_timeout: 1m
  split_queries_by_interval: 1h
  split_metadata_queries_by_interval: 1d
  split_recent_metadata_queries_by_interval: 1h
  recent_metadata_query_window: 0s
  split_instant_metric_queries_by_interval: 1h
  split_ingester_queries_by_interval: 0s
  min_sharding_lookback: 0s
  max_query_bytes_read: 0B
  max_querier_bytes_read: 150GB
  volume_enabled: true
  volume_max_series: 1000
  ruler_max_rules_per_rule_group: 0
  ruler_max_rule_groups_per_tenant: 0
  ruler_alertmanager_config: null
  ruler_tenant_shard_size: 0
  ruler_enable_wal_replay: true
  ruler_remote_write_disabled: false
  ruler_remote_write_url: ""
  ruler_remote_write_timeout: 0s
  ruler_remote_write_headers: {}
  ruler_remote_write_queue_capacity: 0
  ruler_remote_write_queue_min_shards: 0
  ruler_remote_write_queue_max_shards: 0
  ruler_remote_write_queue_max_samples_per_send: 0
  ruler_remote_write_queue_batch_send_deadline: 0s
  ruler_remote_write_queue_min_backoff: 0s
  ruler_remote_write_queue_max_backoff: 0s
  ruler_remote_write_queue_retry_on_ratelimit: false
  ruler_remote_write_sigv4_config: null
  ruler_remote_evaluation_timeout: 0s
  ruler_remote_evaluation_max_response_size: 0
  deletion_mode: filter-and-delete
  retention_period: 2d
  per_tenant_override_config: ""
  per_tenant_override_period: 10s
  allow_deletes: false
  shard_streams:
    enabled: true
    time_sharding_enabled: false
    time_sharding_ignore_recent: 40m0s
    logging_enabled: false
    desired_rate: 1536KB
  index_gateway_shard_size: 1
  bloom_gateway_enable_filtering: false
  bloom_build_max_builders: 0
  bloom_build_task_max_retries: 3
  bloom_build_builder_response_timeout: 0s
  bloom_creation_enabled: false
  bloom_planning_strategy: split_keyspace_by_factor
  bloom_split_series_keyspace_by: 256
  bloom_task_target_series_chunk_size: 20GB
  bloom_block_encoding: none
  bloom_prefetch_blocks: false
  bloom_max_block_size: 200MB
  bloom_max_bloom_size: 128MB
  allow_structured_metadata: true
  max_structured_metadata_size: 64KB
  max_structured_metadata_entries_count: 128
  otlp_config:
    resource_attributes:
      attributes_config:
      - action: index_label
        attributes:
        - service.name
        - service.namespace
        - service.instance.id
        - deployment.environment
        - deployment.environment.name
        - cloud.region
        - cloud.availability_zone
        - k8s.cluster.name
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.container.name
        - container.name
        - k8s.replicaset.name
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.job.name
        regex: null
  block_ingestion_policy_until: {}
  block_ingestion_until: "0"
  block_ingestion_status_code: 260
  enforced_labels: []
  policy_enforced_labels: {}
  policy_stream_mapping: {}
  ingestion_partitions_tenant_shard_size: 0
  pattern_ingester_tokenizable_json_fields_default: log,message,msg,msg_,_msg,content
  pattern_ingester_tokenizable_json_fields_append: ""
  pattern_ingester_tokenizable_json_fields_delete: ""
  metric_aggregation_enabled: true
  s3_sse_type: ""
  s3_sse_kms_key_id: ""
  s3_sse_kms_encryption_context: ""
common:
  path_prefix: /mnt/loki
  storage:
    filesystem:
      chunks_directory: ""
      rules_directory: ""
    hedging:
      at: 0s
      up_to: 2
      max_per_second: 5
    congestion_control:
      enabled: false
      controller:
        strategy: ""
        aimd:
          start: 2000
          upper_bound: 10000
          backoff_factor: 0.5
      retry:
        strategy: ""
        limit: 2
      hedging:
        config:
          at: 0s
          up_to: 0
          max_per_second: 0
        strategy: ""
    object_store:
      filesystem:
        dir: /mnt/loki
      storage_prefix: ""
  replication_factor: 1

I do not have logs older than 24h since i rename indexes files

in logs i’ve got many “cleaning up unwanted indexes from table” for the same table and on multiple tables
over 10m it look like for index_20265 :

2026-05-20T07:37:53.382951443Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:37:53.304491153Z"}
2026-05-20T07:37:53.382958250Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:37:53.304516383Z"}
2026-05-20T07:37:53.382965160Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20265","ts":"2026-05-20T07:37:53.304525988Z"}
2026-05-20T07:38:53.381920305Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:38:53.292725133Z"}
2026-05-20T07:38:53.381927793Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:38:53.29274437Z"}
2026-05-20T07:38:53.381931643Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20265","ts":"2026-05-20T07:38:53.292753999Z"}
2026-05-20T07:39:53.383806646Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:39:53.295905396Z"}
2026-05-20T07:39:53.383815166Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:39:53.29591638Z"}
2026-05-20T07:39:53.383823810Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20265","ts":"2026-05-20T07:39:53.295927198Z"}
2026-05-20T07:40:53.385148993Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:40:53.293169317Z"}
2026-05-20T07:40:53.385159493Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:40:53.293178695Z"}
2026-05-20T07:40:53.385169041Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20265","ts":"2026-05-20T07:40:53.293188844Z"}
2026-05-20T07:41:53.383931345Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:41:53.296208931Z"}
2026-05-20T07:41:53.383942005Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:41:53.296219258Z"}
2026-05-20T07:41:53.383953012Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20265","ts":"2026-05-20T07:41:53.296230896Z"}
2026-05-20T07:42:53.382976095Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:42:53.294574112Z"}
2026-05-20T07:42:53.382980492Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:42:53.294580773Z"}
2026-05-20T07:42:53.382985053Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20265","ts":"2026-05-20T07:42:53.294586848Z"}
2026-05-20T07:43:53.382871122Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:43:53.294646699Z"}
2026-05-20T07:43:53.382874428Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:43:53.294658917Z"}
2026-05-20T07:43:53.382877796Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20265","ts":"2026-05-20T07:43:53.294673068Z"}
2026-05-20T07:44:53.385034534Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:44:53.295851558Z"}
2026-05-20T07:44:53.385063280Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:44:53.295861854Z"}
2026-05-20T07:44:53.385118287Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20265","ts":"2026-05-20T07:44:53.295872644Z"}
2026-05-20T07:45:53.383076354Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:45:53.293285734Z"}
2026-05-20T07:45:53.383103677Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:45:53.293297859Z"}
2026-05-20T07:45:53.383113594Z {"caller":"index_set.go:186","level":"info","msg":"cleaning up unwanted indexes from table index_20265","ts":"2026-05-20T07:45:53.293309974Z"}
2026-05-20T07:46:53.386786679Z {"caller":"index_set.go:86","level":"info","msg":"uploading table index_20265","ts":"2026-05-20T07:46:53.299161951Z"}
2026-05-20T07:46:53.386792399Z {"caller":"index_set.go:107","level":"info","msg":"finished uploading table index_20265","ts":"2026-05-20T07:46:53.29917257Z"}

how can i find sweeper logs ? there is nothing with “sweeper” in logs

florian · June 3, 2026, 9:29am

i have added to my configuration

storage_config:
  filesystem:
    directory: "/mnt/loki"

and rename indexes from

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index/v13/
        period: 24h

to

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

here my script to migrate indexes

#!/bin/bash

if find /mnt/loki -type f -print | grep "index/v13" 1> /dev/null; then
  echo "starting renaming loki indexes"
  echo "renaming loki/index *.tsdb.gz"
  find /mnt/loki/index/index/v13 -type f -name '*.tsdb.gz' | while read -r p; do
    f=$(basename "$p")
    d=$(basename "$(dirname "$p")")
    dest="/mnt/loki/index/index_$d"
    mkdir -p "$dest"
    mv "$p" "$dest/$f"
  done
  echo "renaming loki/tsdb-shipper-cache *.tsdb"
  find /mnt/loki/tsdb-shipper-cache/index/v13 -type f -name '*.tsdb' | while read -r p; do
    f=$(basename "$p")
    d=$(basename "$(dirname "$p")")
    dest="/mnt/loki/tsdb-shipper-cache/index_$d"
    mkdir -p "$dest"
    mv "$p" "$dest/$f"
  done
  echo "renaming loki/tsdb-shipper-active *.tsdb"
  find /mnt/loki/tsdb-shipper-active/multitenant/index/v13 -type f -name '*.tsdb' | while read -r p; do
    f=$(basename "$p")
    d=$(basename "$(dirname "$p")")
    dest="/mnt/loki/tsdb-shipper-active/multitenant/index_$d"
    mkdir -p "$dest"
    mv "$p" "$dest/$f"
  done
  echo "removing older folders"
  rm -rf /mnt/loki/index/index
  rm -rf /mnt/loki/tsdb-shipper-cache/index
  rm -rf /mnt/loki/tsdb-shipper-active/multitenant/index
endif

and script used to removing older files

#!/bin/bash

REF_TS=$(date --date="-30 days" +%s)

echo "removing older files than @$REF_TS"

find /mnt/loki/fake -type f -print0 |
xargs -0 -n1 -P$(nproc) bash -c '
f="$2"
REF_TS="$1"

#echo "DEBUG f=$f, inner date $REF_TS"

fname=${f##*/}

decoded=$(printf "%s" "$fname" | base64 -d 2>/dev/null) || {
    echo "base64 decode failed for $fname from $f"
    exit 0
}

if [[ ! "$decoded" =~ ^[0-9a-fA-F]+:[0-9a-fA-F]+: ]]; then
    echo "filename not expected $decoded from $f"
    exit 0
fi

hex_end=${decoded#*:}
hex_end=${hex_end%%:*}

(( end_ts = 16#$hex_end ))
(( end_ts /= 1000 ))

#echo "$hex_end $end_ts ref: $REF_TS"
(( end_ts < REF_TS )) && rm -f "$f"
' _ "$REF_TS"

echo "cleaning empty folders"
find /mnt/loki/fake -type d -empty -delete

echo "loki cleaning done"

Topic		Replies	Views
Loki chunks and index not keeping forever Grafana Loki	8	208	June 2, 2025
Loki disk space inflation Grafana Loki	10	3043	June 19, 2024
Logs deletion problem Grafana Loki	9	333	January 21, 2025
Filesystem retention not working Grafana Loki	5	3875	April 9, 2024
Loki log retention policy and period Grafana Loki loki , retention	5	284	December 3, 2025

Loki Retention Not Working (docker)

Related topics