Loki won't persist logs to Minio between container recreations

Grafana Loki
,
1

Im trying to set up a (very) minimal Loki/Grafana docker installation with MinIO as storage backend.
What’s unusual about my set up is that logs aren’t collected with promtail/alloy but pushed directly to Loki api using /loki/api/v1/push endpoint with json body.
I discovered that most of the time (but not always) after rebooting the containers with docker-compose down; docker-compose up -d I would lose the pushed logs. Inspecting MinIO web console let me to discover no objects would be created in the loki-data bucket aside from loki_cluster_seed.json.
What is puzzling me is that sometimes logs would get persisted, but I cant find any documentation on what is the trigger for flusing them to s3 backend. Is it log age, size? How do I adjust these settings?

My docker-compose.yaml:

networks:
  loki:

services:
  loki:
    image: grafana/loki:3.4.1
    restart: unless-stopped
    command: "-config.file=/etc/loki/config.yaml"
    ports:
      - "3100:3100"
    volumes:
      - "./loki-config.yaml:/etc/loki/config.yaml"
    depends_on:
      - minio
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - loki

  minio:
    image: minio/minio
    restart: unless-stopped
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /data/loki-data && \
        mkdir -p /data/loki-ruler && \
        minio server /data --console-address ":9001"
    environment:
      - MINIO_ROOT_USER=loki
      - MINIO_ROOT_PASSWORD=supersecret
      - MINIO_PROMETHEUS_AUTH_TYPE=public
      - MINIO_UPDATE=off
    ports:
      - 9000
      - "9001:9001"
    volumes:
      - "./data/minio:/data"
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
      interval: 15s
      timeout: 20s
      retries: 5
    networks:
      - loki

  grafana:
    image: grafana/grafana-oss
    restart: unless-stopped
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
          - name: Loki
            type: loki
            access: proxy
            url: http://loki:3100
        EOF
        /run.sh
    user: "root"
    volumes:
      - "./data/grafana:/var/lib/grafana"
    ports:
      - "3000:3000"
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - loki

loki-config.yaml

auth_enabled: false

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 3100

schema_config:
  configs:
    - from: 2023-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  discover_log_levels: false

common:
  path_prefix: /loki
  replication_factor: 1
  storage:
    s3:
      endpoint: minio:9000
      insecure: true
      bucketnames: loki-data
      access_key_id: loki
      secret_access_key: supersecret
      s3forcepathstyle: true
  ring:
    kvstore:
      store: memberlist
ruler:
  storage:
    s3:
      bucketnames: loki-ruler
2

Loki flush is controlled by three configurations, chunk_idle_period, chunk_target_size, and max_chunk_age. But that’s not your problem, doesn’t matter how you tune those settings if you don’t shutdown your Loki instance gracefully you’ll always lose some logs.

What you should do instead is enable write-ahead logs, and either create a Docker volume or mount a directory from host into the Loki container to store the WAL data. That way when your Loki instance restarts, as long as you didn’t replace the host, you’ll have the WAL logs still present. See Write Ahead Log | Grafana Loki documentation

3

Indeed bind mounting /loki/wal helped, closing the thread.
Edit: I guess manually closing it isnt possible? Oh well will just wait until it expires itself.