Loki retention overrides in helm not working

Grafana Loki
1

Hi, im trying to get Loki retention overrides to work

Global retention timings seems to be working, but not the override.
I´m running loki in fully distributed mode, and with helm version 6.19.0, with this configuration
The tenant i´m trying to override the retention for is named “team_x_logs”

From what i can see, the configmap is created and mounted to the correct path on the compactor pod, but the logs are stored even after the 24h that i have defined.

Any help would be appreciated

---
loki:
  compactor:
    working_directory: /var/loki/data/compactor
    compaction_interval: 10m
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150
    delete_request_store: s3
  limits_config:
    retention_period: 200h
    per_tenant_override_config: /etc/loki/config/override/loki-tenant-override-rules.yaml
  schemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  ingester:
    chunk_encoding: snappy
  tracing:
    enabled: true
  annotations: 
    kyverno.io/inject-cacerts: enabled
  querier:
    max_concurrent: 3
  storage:
    filesystem: null
    type: s3
    s3:
      endpoint: "${LOKI_S3_ENDPOINT}"
      accessKeyId: "${LOKI_S3_ACCESS_KEY_ID}"
      secretAccessKey: "${LOKI_S3_SECRET_ACCESS_KEY}"
      s3ForcePathStyle: false
    bucketNames:
      chunks: "${BUCKET_CHUNK}"
      ruler: "${BUCKET_RULER}"
  auth_enabled: true
  analytics:
    reporting_enabled: false
    usage_stats_url: ""


usage_stats:
  enabled: false

gateway:
  replicas: 1
  basicAuth:
    enabled: true
    existingSecret: basic-auth

deploymentMode: Distributed



ingester:
  replicas: 3
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: LOKI_S3_ENDPOINT
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_ENDPOINT
    - name: LOKI_S3_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_ACCESS_KEY_ID
    - name: LOKI_S3_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_SECRET_ACCESS_KEY
    - name: BUCKET_CHUNK
      valueFrom:
        secretKeyRef:
          key: BUCKET_CHUNK
          name: loki-block-storage
    - name: BUCKET_RULER
      valueFrom:
        secretKeyRef:
          key: BUCKET_RULER
          name: loki-block-storage


querier:
  replicas: 3
  maxUnavailable: 2
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: LOKI_S3_ENDPOINT
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_ENDPOINT
    - name: LOKI_S3_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_ACCESS_KEY_ID
    - name: LOKI_S3_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_SECRET_ACCESS_KEY
    - name: BUCKET_CHUNK
      valueFrom:
        secretKeyRef:
          key: BUCKET_CHUNK
          name: loki-block-storage
    - name: BUCKET_RULER
      valueFrom:
        secretKeyRef:
          key: BUCKET_RULER
          name: loki-block-storage

queryFrontend:
  replicas: 2
  maxUnavailable: 1


queryScheduler:
  replicas: 2

distributor:
  replicas: 3
  maxUnavailable: 2

compactor:
  replicas: 1
    - "-config.expand-env=true"
  extraEnv:
    - name: LOKI_S3_ENDPOINT
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_ENDPOINT
    - name: LOKI_S3_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_ACCESS_KEY_ID
    - name: LOKI_S3_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_SECRET_ACCESS_KEY
    - name: BUCKET_CHUNK
      valueFrom:
        secretKeyRef:
          key: BUCKET_CHUNK
          name: loki-block-storage
    - name: BUCKET_RULER
      valueFrom:
        secretKeyRef:
          key: BUCKET_RULER
          name: loki-block-storage
  extraVolumes:
    - name: override-rules
      configMap:
        name: loki-tenant-override-rules
  extraVolumeMounts:
    - name: override-rules
      mountPath: /etc/loki/config/override/

indexGateway:
  replicas: 2
  maxUnavailable: 1
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: LOKI_S3_ENDPOINT
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_ENDPOINT
    - name: LOKI_S3_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_ACCESS_KEY_ID
    - name: LOKI_S3_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: loki-block-storage
          key: LOKI_S3_SECRET_ACCESS_KEY
    - name: BUCKET_CHUNK
      valueFrom:
        secretKeyRef:
          key: BUCKET_CHUNK
          name: loki-block-storage
    - name: BUCKET_RULER
      valueFrom:
        secretKeyRef:
          key: BUCKET_RULER
          name: loki-block-storage

chunksCache:
  enabled: true
resultsCache:
  enabled: true


# optional experimental components
bloomPlanner:
  replicas: 0
bloomBuilder:
  replicas: 0
bloomGateway:
  replicas: 0



lokiCanary:
  enabled: false
test:
  enabled: false

# optional experimental components
bloomPlanner:
  replicas: 0
bloomBuilder:
  replicas: 0
bloomGateway:
  replicas: 0


# Zero out replica counts of other deployment modes
backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0
singleBinary:
  replicas: 0
minio:
  enabled: false


memcachedExporter:
  enabled: true
  resources:
    requests: {}
    limits: {}


extraObjects:
- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: loki-tenant-override-rules
  data:
    loki-tenant-override-rules.yaml: |-
      overrides:
        "team_x_logs":
          retention_period: 24h
          - selector: '{os="linux"}'
            priority: 1
            period: 24h
2

What does /etc/loki/config/override/loki-tenant-override-rules.yaml look like?

3

It looks like my whole configuration was cut in the middle
Anyway, it looks like this


overrides:
  "team_x_logs":
    retention_period: 24h
    - selector: '{os="linux"}'
      priority: 1
      period: 24h
4

That looks wrong to me. Assuming:

  • team_x_logs is the tenant that you are trying to override configurations in.
  • os="linux" is the label filter for the logs.

Try this (all logs under tenant team_x_logs are set to 24h retention):

overrides:
  "team_x_logs":
    retention_period: 24h

If you just want os=linux logs to be overridden, try:

overrides:
  "team_x_logs":
    retention_stream:
      - selector: '{os="linux"}'
        priority: 1
        period: 24h
5

Thanks, i will try removing the retention_stream part from the configuration, and see if the retention overrides works as intended after that