Loki 3.6.10 Ruler: "stored rulegroup" logged but rules not persisted to S3

gunjanvmirchandani · September 17, 2026, 4:48pm

Hi all,

We’re experiencing an issue where the Loki ruler logs “stored rulegroup” but rules are not actually written to S3. Interestingly, log chunks work fine on the same S3 endpoint.

Environment

Loki version: 3.6.10
Deployment: Kubernetes (Simple Scalable mode)
Storage: S3-compatible (Dell ECS)
Config: use_thanos_objstore: false (legacy ruler.storage)

Symptoms

Rules pushed via API → Loki logs msg="stored rulegroup"
S3 rules bucket remains empty
Ruler polls S3 → msg="failed to get rule group"
Rules disappear from Grafana

What works

Log chunks write to S3 successfully (same endpoint, same credentials)
Logs are queryable
AWS CLI can write to the rules bucket manually

Related issues we found

#21848 - Added ChecksumAlgorithm: SHA256 triggering aws-chunked
#24214 - Fix merged in 3.6.16: “Pre-compute SHA-256 to avoid aws-chunked”
#24206 - Reports silent corruption on S3-compatible backends

Questions

Does the aws-chunked issue affect ruler storage specifically? Why would logs work but not rules on the same endpoint?
Will upgrading to 3.6.16+ fix this for ruler storage?
Has anyone else experienced this with S3-compatible storage?

Thanks for any insights!

tonyswumac · September 24, 2026, 6:58pm

Can you share your config, please?

gunjanvmirchandani · September 29, 2026, 2:45pm

Thanks for the quick response! Here’s our configuration and additional findings.

### Loki Configuration (sanitized)

```yaml

auth_enabled: false

common:

compactor_grpc_address: ‘loki-backend.logging.svc.cluster.local:9095’

path_prefix: /var/loki

replication_factor: 3

storage:

s3:

access_key_id: ${LOKI_S3_ACCESS_KEY}

bucketnames: ${LOKI_S3_BUCKET_NAME}

endpoint: ${LOKI_S3_ENDPOINT}

http_config:

response_header_timeout: 15s

insecure: false

s3forcepathstyle: true

secret_access_key: ${LOKI_S3_SECRET_KEY}

ruler:

alertmanager_url: http://prometheus-kube-prometheus-alertmanager.prometheus:9093

enable_api: true

storage:

s3:

access_key_id: ${LOKI_S3_ACCESS_KEY}

bucketnames: ${LOKI_S3_BUCKET_NAME}

endpoint: ${LOKI_S3_ENDPOINT}

http_config:

response_header_timeout: 15s

insecure: false

s3forcepathstyle: true

secret_access_key: ${LOKI_S3_SECRET_KEY}

type: s3

wal:

dir: /var/loki/ruler-wal

storage_config:

hedging:

at: 250ms

max_per_second: 20

up_to: 3

use_thanos_objstore: false

schema_config:

configs:

from: “2020-09-07”

index:

period: 24h

prefix: loki_index_

object_store: s3

schema: v11

store: boltdb-shipper

from: “2026-06-01”

index:

period: 24h

prefix: loki_tsdb_index_

object_store: s3

schema: v13

store: tsdb

# Other relevant settings

limits_config:

max_query_length: 2165h

max_query_parallelism: 50

query_timeout: 5m

```

### Additional Findings

After further investigation, we discovered the root cause:

**The rules ARE being persisted to S3** - we verified this by listing objects in the bucket:

```

rules/fake/YWxsb3ktazYtY29yZS10ZXN0LXNjOTAtd29ya2Zsb3dzLWxvZy1hbGVydHMt…/c2M5MC13b3JrZmxvd3MtbG9nLWFsZXJ0cw==

```

**The issue is with GetObject, not PutObject.** Our S3-compatible storage (Dell ECS) is returning HTTP 500 errors on ~20% of GetObject requests:

|-----------|---------------|---------------|------------|

| GetObject | 1,600,817 | 401,454 | **20%** |

| PutObject | 304,792 | 1 | ~0% |

This explains why:

1. `msg=“stored rulegroup”` appears (PutObject succeeds)

2. `msg=“failed to get rule group”` appears immediately after (GetObject fails)

3. Rules work intermittently (when GetObject succeeds)

### S3 Metrics from Loki

```

loki_s3_request_duration_seconds_count{operation=“S3.GetObject”,status_code=“200”} 1600817

loki_s3_request_duration_seconds_count{operation=“S3.GetObject”,status_code=“500”} 401454

loki_s3_request_duration_seconds_count{operation=“S3.PutObject”,status_code=“200”} 304792

loki_s3_request_duration_seconds_count{operation=“S3.PutObject”,status_code=“500”} 1

```

### Questions

1. Is there any known compatibility issue between Loki’s AWS SDK v2 client and S3-compatible storage like Dell ECS?

2. Would enabling `use_thanos_objstore: true` use a different S3 client that might behave differently?

3. Are there any recommended S3 client settings for non-AWS S3-compatible storage?

### Environment

- **Loki Version:** 3.6.10

- **Deployment:** Kubernetes (loki-distributed helm chart)

- **S3 Storage:** Dell ECS (S3-compatible)

- **Ruler Target:** `backend` component