How to make loki deletes logs when using SimpleScalable deployment

Grafana Loki
,
1

Compator is not allowed to be used within SimpleScalable deployment model.
We have set retention_streams and global retention. Yet, with streams, it lived for indefinite and never got deleted

Retention docs

Our configs

  deploymentMode: SimpleScalable
  loki:
    auth_enabled: false
    compactor:
      # working_directory: /logs/retention
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150
      delete_request_store: s3
    limits_config:
      retention_period: 72h
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 30m
      split_queries_by_interval: 15m
      query_timeout: 300s
      ingestion_rate_mb: 24
      ingestion_burst_size_mb: 24
      per_stream_rate_limit: "6MB"
      per_stream_rate_limit_burst: "15MB"
      volume_enabled: true
      retention_stream:
        - selector: '{retention="high"}'
          priority: 1
          period: 48h
        - selector: '{retention="low"}'
          priority: 1
          period: 24h
...
  compactor:
    replicas: 0

Promtail

promtail:
  config:
    enabled: true
    logLevel: info
    logFormat: logfmt
    serverPort: 3101
    clients:
      - url: http://loki-gateway:80/loki/api/v1/push
    snippets:
      pipelineStages:
        - cri: {}
        - match:
            selector: '{app="APP1"} |~ ".*HealthChecker/2.0.*"'
            action: drop
      extraRelabelConfigs:
        - source_labels: [__meta_kubernetes_pod_label_retention]
          target_label: retention
        - source_labels: [retention]
          target_label: retention
          action: replace
          regex: ^$
          replacement: normal
        - source_labels: [retention]
          regex: drop
          action: drop

We’re relying on promtail to add custom label to log stream and its working fine.
Then in Loki, we would like to retain logs based on this custom label.
Dropping is working fine as promtail won’t push logs to loki. Yet, for streams that should live for 24h, 48h or more. Loki never deleted it from S3 Bucket

2
  1. Are logs being removed after your default retention of 72h?
  2. Any error log you see from compactor?
  3. Can you share your entire Loki configuration?
  4. Any reason compactor replica count is 0?
3

Having the same issue right now.

Compactor replica count seems to be 0 because that’s how it is recommended as per the Simple Scalable Deployment mode (see: Install the simple scalable Helm chart | Grafana Loki documentation for example), and compactor is already included in the backend in this deployment mode.

If we try to set compactor replica count to anything other than 0 now, this warning would pop up:

So, what is the correct approach to take here? Say if we want to have retention applied per stream so that we can keep some important logs, and discard the rest.

4

Same issue here as well. Our config:

  loki:
    auth_enabled: false
    schemaConfig:
      configs:
        - from: 2024-04-17
          store: tsdb
          object_store: s3
          schema: v13
          index:
            prefix: loki_index_
            period: 24h
    compactor:
      working_directory: /var/loki/compactor
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150
      delete_request_store: s3
    ingester:
      chunk_encoding: snappy
      chunk_target_size: 1536000
    limits_config:
      max_global_streams_per_user: 0
      max_query_lookback: 30d
      retention_period: 30d
    storage:
      type: s3
      bucketNames:
        chunks: chunks
        ruler: ruler
        admin: admin
      s3:
        endpoint: XXX
        accessKeyId: XXX
        secretAccessKey: XXX
        insecure: false
        s3ForcePathStyle: true
        http_config:
          insecure_skip_verify: true
  deploymentMode: SimpleScalable
  minio:
    enabled: false

  # Zero out replica counts of other deployment modes
  singleBinary:
    replicas: 0
  ingester:
    replicas: 0
  querier:
    replicas: 0
  queryFrontend:
    replicas: 0
  queryScheduler:
    replicas: 0
  distributor:
    replicas: 0
  compactor:
    replicas: 0
  indexGateway:
    replicas: 0
  bloomCompactor:
    replicas: 0
  bloomGateway:
    replicas: 0
5

The documentation here could be improved to include relevant helm values and how simple scalable mode interacts with the compactor. Log retention | Grafana Loki documentation . From an infrastructure side this can be managed via a lifecycle rule in S3, however it is unclear what happens to one time created files such as seed.json, since this file will be deleted after lifecycle expiry.