Loki with Hetzner Object Storage S3 - error 403 with "UnknownError"

Grafana Loki
1

Hello,

I have a Talos-based Kubernetes cluster on Hetzner Cloud. I’m deploying Grafana Loki for logs in SingleBinary mode with S3 as storage backend.

Unfortunately, Loki is unable to connect to S3 successfully, I am getting the following errors in logs: Loki logs with Hetzner Object Storage - Pastebin.com

My Loki configuration of the Helm chart:

I’ve validated that the S3 credentials work with aws CLI and curl. THe buckets exists, etc. Everything is in correct Hetzner Cloud project, so no permissions issues, etc…

If anyone has an idea why is Loki unable to write to the S3, I would appreciate any help at this point.

Thanks

2

Can you share the schema part of your configuration, please?

3

Sure, here is my schemaConfig:

schemaConfig:
  configs:
    - from: 2025-12-15
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: loki_index_
        period: 24h
4

Seems to be ok.

I’ve never used Hetzner object storage before, does it have event logs or audit logs?

5

AFAIK it does not have either.

It’s built on top of Ceph, so I guess there are logs, just not exposed to the end user unfortunately.

I think Loki should work without issues on Ceph (with Object Gateway), so I’m wondering why is it not working.

6

I figured the issue, in single binary mode, the global.extraArgs does not get propagated to singleBinary.

You must set singleBinary.extraArgs explicitely.

After adding the following:

singleBinary:
  extraArgs:
    - -config.expand-env=true

The config got expanded and Loki successfully connected to the S3 backend, got authenticated and access the bucket(s).

Unfortunately, I am facing another issue.

ts=2025-12-30T11:35:42.84143361Z caller=spanlogger.go:152 table-name=loki_index_20452 user=fake level=error msg="failed to init user index set" err="operation error S3: ListObjectsV2, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed"

Loki is trying to use EC2 IMDS and to get identity credentials from AWS, but I haven’t figured out yet how to disable this since I’m not running on AWS, but on Hetzner instead.

TBH I haven’t looked into it properly yet since I’m enjoying the holidays and time off rather then diving into yet another bug.

7

Sigh

I’ve solved the issue.

Again, the single binary mode does not respect the global options.

Simply adding the extraEnvFrom to singleBinary solved it.

singleBinary:
  extraEnvFrom:
    - secretRef:
      name: loki-s3-credentials