Hi, I upgraded my Loki from version 3.4.2 to 3.5.7 and changed my object storage config to use Thanos store client. After upgrading, Loki compactor pod cannot start with init compactor: failed to init delete store: The ap-east-2 location constraint is incompatible for the region specific endpoint this request was sent to. error message (complete error message is provided below.
This is my loki configmap. I’ve tried adding ap-east-2 in both s3 and sts endpoints wherever possible, but it seems like the compactor is still not using the ap-east-2 endpoint. Can someone point me in the right direction?
I’m deploying loki inside EKS cluster using https://grafana.github.io/helm-charts helm chart
data:
config.yaml: |2
auth_enabled: false
bloom_build:
builder:
planner_address: loki-bloom-planner-headless.loki.svc.cluster.local:9095
enabled: false
bloom_gateway:
client:
addresses: dnssrvnoa+_grpc._tcp.loki-bloom-gateway-headless.loki.svc.cluster.local
enabled: false
common:
compactor_grpc_address: 'loki-compactor.loki.svc.cluster.local:9095'
path_prefix: /var/loki
replication_factor: 1
storage:
object_store:
s3:
endpoint: s3.ap-east-2.amazonaws.com
region: ap-east-2
sts_endpoint: https://sts.ap-east-2.amazonaws.com
s3:
bucketnames: bucket-name
insecure: false
region: ap-east-2
s3forcepathstyle: false
endpoint: s3.ap-east-2.amazonaws.com
compactor:
compaction_interval: 10m
delete_request_store: s3
retention_enabled: true
frontend:
scheduler_address: loki-query-scheduler.loki.svc.cluster.local:9095
tail_proxy_url: http://loki-querier.loki.svc.cluster.local:3100
frontend_worker:
scheduler_address: loki-query-scheduler.loki.svc.cluster.local:9095
index_gateway:
mode: simple
ingester:
chunk_encoding: snappy
chunk_target_size: 1572864
limits_config:
allow_structured_metadata: true
max_cache_freshness_per_query: 10m
query_timeout: 300s
reject_old_samples: true
reject_old_samples_max_age: 168h
retention_period: 90d
split_queries_by_interval: 15m
volume_enabled: true
memberlist:
join_members:
- loki-memberlist.loki.svc.cluster.local
pattern_ingester:
enabled: true
querier:
max_concurrent: 4
query_range:
align_queries_with_step: true
runtime_config:
file: /etc/loki/runtime-config/runtime-config.yaml
schema_config:
configs:
- from: "2025-04-01"
index:
period: 24h
prefix: index_
object_store: s3
schema: v13
store: tsdb
server:
grpc_listen_port: 9095
http_listen_port: 3100
http_server_read_timeout: 600s
http_server_write_timeout: 600s
storage_config:
aws:
endpoint: s3.ap-east-2.amazonaws.com
region: ap-east-2
bloom_shipper:
working_directory: /var/loki/data/bloomshipper
boltdb_shipper:
index_gateway_client:
server_address: dns+loki-index-gateway-headless.loki.svc.cluster.local:9095
hedging:
at: 250ms
max_per_second: 20
up_to: 3
object_store:
s3:
bucket_name: bucket-name
endpoint: s3.ap-east-2.amazonaws.com
region: ap-east-2
native_aws_auth_enabled: true
dualstack_enabled: false
sts_endpoint: https://sts.ap-east-2.amazonaws.com
bucket_lookup_type: path
storage_prefix: loki
tsdb_shipper:
index_gateway_client:
server_address: dns+loki-index-gateway-headless.loki.svc.cluster.local:9095
use_thanos_objstore: true
tracing:
enabled: false
Complete error message
level=info ts=2026-01-12T07:04:02.762793676Z caller=main.go:126 msg="Starting Loki" version="(version=3.5.7, branch=release-3.5.x, revision=d5b382b9)"
level=info ts=2026-01-12T07:04:02.762836103Z caller=main.go:127 msg="Loading configuration file" filename=/etc/loki/config/config.yaml
level=info ts=2026-01-12T07:04:02.763854702Z caller=server.go:368 msg="server listening on addresses" http=[::]:3100 grpc=[::]:9095
level=info ts=2026-01-12T07:04:02.76745454Z caller=memberlist_client.go:463 msg="Using memberlist cluster label and node name" cluster_label= node=loki-compactor-0-f5ef4c47
level=info ts=2026-01-12T07:04:02.767672495Z caller=memberlist_client.go:594 msg="memberlist fast-join starting" nodes_found=1 to_join=4
level=info ts=2026-01-12T07:04:02.780311986Z caller=memberlist_client.go:614 msg="memberlist fast-join finished" joined_nodes=3 elapsed_time=12.641969ms
level=info ts=2026-01-12T07:04:02.780341837Z caller=memberlist_client.go:626 phase=startup msg="joining memberlist cluster" join_members=loki-memberlist.loki.svc.cluster.local
level=info ts=2026-01-12T07:04:02.792663604Z caller=memberlist_client.go:633 phase=startup msg="joining memberlist cluster succeeded" reached_nodes=3 elapsed_time=12.315963ms
init compactor: failed to init delete store: The ap-east-2 location constraint is incompatible for the region specific endpoint this request was sent to.
error initialising module: compactor
github.com/grafana/dskit/modules.(*Manager).initModule
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:531
main.main
/src/loki/cmd/loki/main.go:129
runtime.main
/usr/local/go/src/runtime/proc.go:283
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1700
level=error ts=2026-01-12T07:04:03.36678577Z caller=log.go:223 msg="error running loki" err="init compactor: failed to init delete store: The ap-east-2 location constraint is incompatible for the region specific endpoint this request was sent to.\nerror initialising module: compactor\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:531\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"