Loki does not ship logs to external S3 storage (garage)

seduardlungu · November 28, 2025, 10:24am

Hi team,

I need guidance regarding the following issue:

I want to deploy the Grafana+Loki (with S3 compatible storage garage) + Alloy for logs visualization of my 2 application (ui+backend). My setup is a 3VMs k8s cluster hosted on a vcenter with another vm used for garage s3 storage (all in the same network, rechable).

So far I configured garage, and after I managed to deploy Loki, it looks like loki just sends some GET requests to S3 storage but does not ship logs at all. Even tho it looks like I have some logs:

tree
.
├── compactor
├── tsdb-shipper-active
│ ├── multitenant
│ ├── per_tenant
│ ├── scratch
│ ├── uploader
│ │ └── name
│ └── wal
│ └── s3_2020-05-15
│ └── 1764324052
│ └── 00000000
├── tsdb-shipper-cache
└── wal
├── 00000005
└── checkpoint.000004
└── 00000000

Logs from loki-0 pod:

level=info ts=2025-11-28T10:15:52.37764283Z caller=table_manager.go:136 index-store=tsdb-2020-05-15 msg=“uploading tables”
level=info ts=2025-11-28T10:15:52.450348933Z caller=recalculate_owned_streams.go:49 msg=“starting recalculate owned streams job”
level=info ts=2025-11-28T10:15:52.450381144Z caller=recalculate_owned_streams.go:52 msg=“completed recalculate owned streams job”
level=info ts=2025-11-28T10:16:04.767798089Z caller=memberlist_client.go:552 msg=“initiating cleanup of obsolete entries”
level=info ts=2025-11-28T10:16:22.449813983Z caller=recalculate_owned_streams.go:49 msg=“starting recalculate owned streams job”
level=info ts=2025-11-28T10:16:22.449840425Z caller=recalculate_owned_streams.go:52 msg=“completed recalculate owned streams job”
level=info ts=2025-11-28T10:16:34.768306431Z caller=memberlist_client.go:552 msg=“initiating cleanup of obsolete entries”
level=info ts=2025-11-28T10:16:52.377556072Z caller=table_manager.go:136 index-store=tsdb-2020-05-15 msg=“uploading tables”
level=info ts=2025-11-28T10:16:52.381707604Z caller=table_manager.go:239 index-store=tsdb-2020-05-15 msg=“syncing tables”
level=info ts=2025-11-28T10:16:52.381736433Z caller=table_manager.go:300 index-store=tsdb-2020-05-15 msg=“query readiness setup completed” duration=4.177µs distinct_users_len=0 distinct_users=
level=info ts=2025-11-28T10:16:52.448689797Z caller=checkpoint.go:611 msg=“starting checkpoint”
level=info ts=2025-11-28T10:16:52.449761996Z caller=recalculate_owned_streams.go:49 msg=“starting recalculate owned streams job”
level=info ts=2025-11-28T10:16:52.449786838Z caller=recalculate_owned_streams.go:52 msg=“completed recalculate owned streams job”
level=info ts=2025-11-28T10:16:52.45080956Z caller=checkpoint.go:336 msg=“attempting checkpoint for” dir=/var/loki/wal/checkpoint.000006
level=info ts=2025-11-28T10:16:52.454506983Z caller=checkpoint.go:498 msg=“atomic checkpoint finished” old=/var/loki/wal/checkpoint.000006.tmp new=/var/loki/wal/checkpoint.000006
level=info ts=2025-11-28T10:17:04.767585763Z caller=memberlist_client.go:552 msg=“initiating cleanup of obsolete entries”
level=info ts=2025-11-28T10:17:22.450004695Z caller=recalculate_owned_streams.go:49 msg=“starting recalculate owned streams job”
level=info ts=2025-11-28T10:17:22.450038394Z caller=recalculate_owned_streams.go:52 msg=“completed recalculate owned streams job”
level=info ts=2025-11-28T10:17:34.768060332Z caller=memberlist_client.go:552 msg=“initiating cleanup of obsolete entries”
level=info ts=2025-11-28T10:17:52.376992716Z caller=table_manager.go:136 index-store=tsdb-2020-05-15 msg=“uploading tables”
level=info ts=2025-11-28T10:17:52.449972564Z caller=recalculate_owned_streams.go:49 msg=“starting recalculate owned streams job”
level=info ts=2025-11-28T10:17:52.449992888Z caller=recalculate_owned_streams.go:52 msg=“completed recalculate owned streams job”
level=info ts=2025-11-28T10:18:04.767701235Z caller=memberlist_client.go:552 msg=“initiating cleanup of obsolete entries”

Here is my values.yaml file I used:

loki:

auth_enabled: false

server:
http_listen_port: 3100
common:
ring:

  instance_addr: 127.0.0.1

  kvstore:

    store: inmemory

replication_factor: 1

path_prefix: /loki
schemaConfig:
configs:

  - from: 2020-05-15

    store: tsdb

    object_store: s3

    schema: v13

    index:

      prefix: loki_index\_

      period: 24h
storage_config:
tsdb_shipper:

  active_index_directory: /var/loki/index

  cache_location: /var/loki/index_cache

aws:

  s3: http://my_key:my_key_secret@garage_vm_ip:3900/chunksforloki

  s3forcepathstyle: true

  region: garage

  insecure: true
storage:
bucketNames:

  chunks: chunksforloki

  \# ruler: rulerforloki

  \# admin: adminforloki

type: s3

s3:

  endpoint: http://garage_vm_ip:3900

  access_key_id: my_key_id

  secret_access_key: my_key_secret

  region: garage

  s3forcepathstyle: true

  insecure: true
minio:

enabled: false

ruler:

enabled: false

deploymentMode: SingleBinary

singleBinary:

persistence:
enabled: false
extraVolumes:
- name: loki-data

  persistentVolumeClaim:

    claimName: loki-pvc  
extraVolumeMounts:
- name: loki-data

  mountPath: /var/loki  
backend:

replicas: 0

read:

replicas: 0

write:

replicas: 0

ingester:

replicas: 0

querier:

replicas: 0

queryFrontend:

replicas: 0

queryScheduler:

replicas: 0

distributor:

replicas: 0

compactor:

replicas: 0

indexGateway:

replicas: 0

bloomCompactor:

replicas: 0

bloomGateway:

replicas: 0

test:

enabled: false # ← required so you can disable canary

gateway:

enabled: false

lokiCanary:

enabled: false

chunksCache:

enabled: false

resultsCache:

enabled: false

Garage service logs:

nov 28 11:42:32 my-virtual-machine garage[41972]: 2025-11-28T09:42:32.448813Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:32435 (key GKb99f25f376d2637a701432de) GET /chunksforloki/loki_cluster_seed.jsonnov 28 11:42:52 my-virtual-machine garage[41972]: 2025-11-28T09:42:52.446459Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:43:52 my-virtual-machine garage[41972]: 2025-11-28T09:43:52.446044Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:44:52 my-virtual-machine garage[41972]: 2025-11-28T09:44:52.445979Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:45:52 my-virtual-machine garage[41972]: 2025-11-28T09:45:52.446152Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:46:52 my-virtual-machine garage[41972]: 2025-11-28T09:46:52.446013Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:47:52 my-virtual-machine garage[41972]: 2025-11-28T09:47:52.445677Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:48:52 my-virtual-machine garage[41972]: 2025-11-28T09:48:52.446543Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:49:52 my-virtual-machine garage[41972]: 2025-11-28T09:49:52.446605Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:50:52 my-virtual-machine garage[41972]: 2025-11-28T09:50:52.446140Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:51:52 my-virtual-machine garage[41972]: 2025-11-28T09:51:52.446439Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:51:58 my-virtual-machine garage[41972]: 2025-11-28T09:51:58.617401Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:11810 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=%2F&list-type=2&prefix=index%2Fnov 28 11:52:52 my-virtual-machine garage[41972]: 2025-11-28T09:52:52.446202Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:53:52 my-virtual-machine garage[41972]: 2025-11-28T09:53:52.445776Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:54:52 my-virtual-machine garage[41972]: 2025-11-28T09:54:52.446243Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:55:52 my-virtual-machine garage[41972]: 2025-11-28T09:55:52.446278Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:56:52 my-virtual-machine garage[41972]: 2025-11-28T09:56:52.445476Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:57:52 my-virtual-machine garage[41972]: 2025-11-28T09:57:52.445448Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2Fnov 28 11:58:52 my-virtual-machine garage[41972]: 2025-11-28T09:58:52.446186Z INFO garage_api_common::generic_server: [::ffff:10.241.13.31]:51655 (key GKb99f25f376d2637a701432de) GET /chunksforloki?delimiter=&list-type=2&prefix=rules%2F

Notes:

The PVC is fine.

The bucket exists (I tried with ruler and admin enabled as well and same result)

The s3 key-secret is ok (I tested it using s3cmd)

The GET methods are fine, but there is nothing shipped.

I also tried using chatgpt, here is the values.yaml suggested, with the same result:

loki:

auth_enabled: false

server:
http_listen_port: 3100
common:
ring:

  instance_addr: 127.0.0.1

  kvstore:

    store: inmemory

replication_factor: 1

path_prefix: /loki
# ---------- SCHEMA: TSDB v13 ----------

schemaConfig:
configs:

  - from: 2020-05-15

    store: tsdb

    object_store: s3

    schema: v13

    index:

      prefix: loki_index\_

      period: 24h
# ---------- TSDB STORAGE CONFIG ----------

storage:
tsdb:

  dir: /var/loki/tsdb

  block_ranges_period: 15m   # upload frequency: smaller = more often



tsdb_shipper:

  active_index_directory: /var/loki/tsdb-index

  cache_location: /var/loki/tsdb-cache



bucket_store:

  shared_store: s3



bucketNames:

  chunks: chunksforloki



\# ---------- S3 (Garage) backend ----------

s3:

  endpoint: http://garage_vm_ip:3900

  access_key_id: my_key_id

  secret_access_key: my_secret

  region: garage

  insecure: true

  s3forcepathstyle: true
# Disable MinIO from chart

minio:

enabled: false

# Disable ruler

ruler:

enabled: false

# ----------- DEPLOYMENT MODE (single binary) ----------

deploymentMode: SingleBinary

singleBinary:

persistence:
enabled: false
extraVolumes:
- name: loki-data

  persistentVolumeClaim:

    claimName: loki-pvc
extraVolumeMounts:
- name: loki-data

  mountPath: /var/loki
backend:

replicas: 0

read:

replicas: 0

write:

replicas: 0

ingester:

replicas: 0

querier:

replicas: 0

queryFrontend:

replicas: 0

queryScheduler:

replicas: 0

distributor:

replicas: 0

compactor:

replicas: 0

indexGateway:

replicas: 0

bloomCompactor:

replicas: 0

bloomGateway:

replicas: 0

test:

enabled: false # ← required so you can disable canary

gateway:

enabled: false

lokiCanary:

enabled: false

chunksCache:

enabled: false

resultsCache:

enabled: false

Can anyone provide some guidance into how can I implement my setup please?

tonyswumac · December 2, 2025, 1:38am

Try changing this:

Also, I don’t know what garage is, and i am not sure if it’s officially supported or not. I’d check and see if there are any errors on your storage side.

Topic		Replies	Views
Loki not pushing logs to s3 Grafana Loki loki	2	611	March 12, 2025
How to store loki logs into s3 please provide loki-config.yaml Grafana Loki loki	8	1785	September 25, 2024
How to install Loki on (AWS) EKS using Terraform with S3 Grafana Loki loki , terraform	6	1064	November 20, 2025
Grafana + Loki-Distributed + Promtail on EKS Configuration loki	0	780	November 9, 2023
Logs no longer returned once in s3 Grafana Loki aws	1	653	March 29, 2023

Loki does not ship logs to external S3 storage (garage)

Related topics