Loki not deleting logs from s3

Grafana Loki
1

We are running Loki since 4 months now. We figured that there was about 4.5TB of S3 storage used till now.
So we wanted to delete the logs.
I enabled retention settings on the compactor and it has been enabled since 10 days. Still I don’t see any reduction in the S3 storage.

Here is my config for Loki

distributor:
      ring:
        kvstore:
          store: memberlist

    ingester:
      # Disable chunk transfer which is not possible with statefulsets
      # and unnecessary for boltdb-shipper
      max_transfer_retries: 0
      chunk_idle_period: 1h
      chunk_target_size: 1536000
      max_chunk_age: 1h
      wal:
        enabled: true
        dir: /var/loki/wal
        flush_on_shutdown: true
        replay_memory_ceiling: 2GB
      lifecycler:
        join_after: 0s
        ring:
          kvstore:
            store: memberlist

    memberlist:
      join_members:
        - {{ include "loki.fullname" . }}-memberlist

    limits_config:
      ingestion_rate_mb: 10
      ingestion_burst_size_mb: 20
      max_concurrent_tail_requests: 20
      max_cache_freshness_per_query: 10m
      retention_period: 744h

    schema_config:
      configs:
        - from: 2020-09-07
          store: boltdb-shipper
          object_store: aws
          schema: v11
          index:
            prefix: loki_index_
            period: 24h

    compactor:
      working_directory: /loki/compactor
      shared_store: aws
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 15000

    storage_config:
      aws:
        s3: s3://ap-southeast-1
        bucketnames: loki-prod
      boltdb_shipper:
        active_index_directory: /var/loki/index
        shared_store: s3
        cache_location: /var/loki/cache
        cache_ttl: 168h
        {{- if .Values.indexGateway.enabled }}
        index_gateway_client:
          server_address: dns:///{{ include "loki.indexGatewayFullname" . }}:9095
        {{- end }}

    query_range:
      align_queries_with_step: true
      max_retries: 5
      split_queries_by_interval: 15m
      cache_results: true

      results_cache:
        cache:
          enable_fifocache: true
          fifocache:
            max_size_items: 1024
            validity: 24h

I also tried the delete api where I set the start date as 0 and end date of a month back. Even after that I don’t see any deletion

  • Are the settings correct. Am I missing anything?
  • Does Loki actually delete the chunks or should I do it manually?
  • Is it ok to delete chunks manually?

I am running loki 2.4 (helm - loki-distributed)

2

I’m no expert and still trying to figure this thing out myself but I think this is supposed to be handled by the Compactor. Are you running a Compactor instance and do you have retention configured as described here?: Retention | Grafana Labs

3

Responding again as I think I finally understand this…

Loki will not delete old log data – it only compacts it with the Compactor. The Storage | Grafana Loki documentation documentation mentions this:

With the exception of the filesystem chunk store, Loki will not delete old chunk stores. This is generally handled instead by configuring TTLs (time to live) in the chunk store of your choice (bucket lifecycles in S3/GCS, and TTLs in Cassandra). Neither will Loki currently delete old data when your local disk fills when using the filesystem chunk store – deletion is only determined by retention duration.
We’re interested in adding targeted deletion in future Loki releases (think tenant or stream level granularity) and may include other strategies as well.
For more information, see the retention configuration documentation.

4

I think this part of the documentation is not updated.

The compactor supports granular retention. We can have some streams for a longer duration while removing others.
Deleting the chunks from s3 lifecycles would delete all the chunks that are older than a certain period.

According to the documentation for the compactor retention

The compactor’s algorithm to update the index:

  • For each table within each day:
    • Compact the table into a single index file.
    • Traverse the entire index. Use the tenant configuration to identify and mark chunks that need to be removed.
    • Remove marked chunks from the index and save their reference in a file on disk.
    • Upload the new modified index files.

The retention algorithm is applied to the index. Chunks are not deleted while applying the retention algorithm. The chunks will be deleted by the compactor asynchronously when swept.

Here the documentation says that the compactor will delete the chunks. But it does not specify when or how fast.

5

Ah you may be right. That documentation certainly implies deletion support. I’ve definitely been confused by seemingly contradictory documentation on this stuff… I know it isn’t easy to keep things well documented but hopefully someone from Grafana can clarify.

6

I’m interested in deleting chunks from s3 storage too.
Now I’m using TTL storage.

7

Hi seems like the logs are getting deleted. But a bit too slowly. Its been more than a month since I made the changes to delete the old logs. The size of the S3 bucket is gradually decreasing and after a month it is 3.5TB.

Expecting it to be close to 1.5TB. Lets see how much time that takes

8

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.