Loki (v3.1.1) SimpleScalable Setup with Helm - Retention

Grafana Loki
,
1

I’ve implemented Loki (v3.1.1) using the SimpleScalable deployment mode via Helm. Here’s a snippet of my custom values.yaml file:

loki:
  auth_enabled: false

  schemaConfig:
    configs:
    - from: 2024-04-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h
  ingester:
    chunk_encoding: snappy
  tracing:
    enabled: true
  querier:
    # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
    max_concurrent: 4

  storage:
    bucketNames:
      chunks: loki-bucket-xxxxx
      ruler: loki-bucket-xxxxx
      admin: loki-bucket-xxxxx
    s3:
      # s3: null
      endpoint: https://s3.amazonaws.com
      region: us-east-1
      secretAccessKey: eZIBinNroYAXnE9nxm93x1NXXXXXXX
      accessKeyId: AKIAYYYXXXXXXXXX
      # s3ForcePathStyle: true

  compactor:
    # working_directory: /data/retention
    working_directory: /var/loki/compactor
    compaction_interval: 10m
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150
    delete_request_store: s3

#gateway:
#  ingress:
#    enabled: true
#    hosts:
#      - host: FIXME
#        paths:
#          - path: /
#            pathType: Prefix

deploymentMode: SimpleScalable

backend:
  replicas: 1
read:
  replicas: 1
write:
  replicas: 2 # minimum 2 replicas required
  # affinity: {}
  # place the pods on the same node
  affinity:
    podAntiAffinity: null 
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app.kubernetes.io/component: write
          topologyKey: kubernetes.io/hostname


# Enable minio for storage
# minio:
#   enabled: true

# Zero out replica counts of other deployment modes
singleBinary:
  replicas: 0

ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0

chunksCache:
  allocatedMemory: 2048

I have few questions:

  1. Persistent Volumes: The deployment created three PVCs: data-loki-backend-0, data-loki-write-0, and data-loki-write-1. What exactly is stored in these volumes, and is persistence absolutely required for Loki to function properly? If it is required, how do I configure proper retention policies for these volumes?

  2. S3 Bucket Content: I’ve configured AWS S3 as the object store, and now I see files such as loki_cluster_seed.json, a fake/ directory, and an index/ directory in my S3 bucket. Could someone explain what these files represent? How do they correspond to the configured buckets (admin, chunks, ruler)?

  3. Retention Configuration: I’ve found it somewhat confusing to configure retention. The documentation seems to mix up Loki’s config.yaml and Helm’s values.yaml. I attempted to enable the compactor and set the working directory to /var/loki/compactor (the default /data/retention was not working). How do I verify that the compactor is working correctly? Are there any clear guidelines for configuring retention when using Helm?

Any guidance or insights would be greatly appreciated!

2

The persistent volume is for storing marker files, see Log retention | Grafana Loki documentation. It’s not required, but if you don’t have persistent marker file it is possible for compactor to miss deleting chunks. This can be worked around by also having a S3 bucket lifecycle policy. You do not need retention policies for the volumes.

I don’t run Loki on K8s myself, so I can’t comment on helm chart changes. But you can see the general configuration guidelines for retention here: Log retention | Grafana Loki documentation. You can verify if compactor is running by doing the following:

  • When you have multiple ingesters, each ingester will create one index file in S3. Compactor attempts to periodically combines the individual index file into one for better performance. If you see individual index files being combined then compactor is working.
  • You can also set retention to be extremely low (like 24h) and check next day and see if chunks are being removed. Do this in lower environment, of course.
1 Like