Store TSDB cant delete chunks from s3?

Grafana Loki
1

Hi,
based on this comment nad information that only boltdb-shipper supports deleting logs i understand it as follows:

TSDB:

  • Indexes are deleted by compactor
  • Chunks needs to be deleted by TTL policy on S3
  • retention_stream: only deletes indexes but not chunks

boltdb-shipper:

  • Indexes are deleted by compactor
  • Chunks are deleted by Loki
  • retention_stream: deletes both indexes and chunks

Am i Correct?

PS: Reason why i ask is that i cant get retention policy working. Here is my (simplified for forum) configuration:

limits_config:
  retention_period: 720h
  retention_stream:
    - selector: '{app="myapp"}'
      priority: 1
      period: 24h
schema_config:
  configs:
    - from: "2023-01-01"
      index:
        period: 24h
        prefix: loki_index_
      object_store: s3
      schema: v12
      store: tsdb
compactor:
  working_directory: /srv/loki/boltdb-shipper-compactor
  shared_store: s3
  retention_enabled: true
  delete_request_cancel_period: 1h
  retention_delete_delay: 1h
storage_config:
  tsdb_shipper:
    active_index_directory: /srv/loki/tsdb-index
    cache_location: /srv/loki/tsdb-index-cache
    shared_store: s3
2

I am pretty sure compactor still deletes chunks for TSDB. The difference between TSDB and boltdb from compactor’s standpoint is that compactor combines all boltdb index files from all ingesters into one table per file, but it doesn’t do it for TSDB. I read on a blog (can’t find the link at the moment) that TSDB is already optimized enough that doing the same merging / deduplication for TSDB doesn’t yield much performance gain.

Do you have any logs from compactor?

3

Chunks are beeing deleted as expected but there are few left undeleted probably from time when we switched from boltdb to TSDB. Closing.