Loki s3 error, can't send data to s3

Grafana Loki
1

Hi all,

I can’t send data to s3.
I took an example from the documentation, but I get an error in the logs. Without s3 everything works fine.

level=error ts=2022-11-12T10:30:51.610538387Z caller=flush.go:220 org_id=fake msg=“failed to flush user” err="AccessDenied: Access Denied\n\tstatus code: 403, request id: 6a290faf19228ec6, host id: "

What am I doing wrong?

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

ingester:
  wal:
    enabled: true
    dir: /tmp/wal
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h     
  max_chunk_age: 1h          
  chunk_target_size: 1048576 
  chunk_retain_period: 30s   
  max_transfer_retries: 0     

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h        
    shared_store: s3
  aws:
   s3: s3://KeyId:AccessKey@endpoint/bucket_name
   s3forcepathstyle: true
compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: s3

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

ruler:
  storage:
    type: local
    local:
      directory: /tmp/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
2

Error is quite clear, access denied. How are you deploying loki? How are you making sure the necessary permissions are granted?

3

I know it’s been a while, but I have to say, the error is anything BUT clear when Loki spews a 403.

Why?

Because Loki does not LOG THE URL IT ATTEMPTED in any way.

In my case I can setup access to the bucket from any application using the same keys and it works, because I can figure out how to get those applications to use the minio server URL pathing correctly - BUT NOT LOKI

I’m sitting here iterating through multiple incantations of configuring loki for over 24 hours now. All spew 403 - and I can’t tell what effect my changes are having, because again… Loki isn’t logging what URL it’s trying so I can’t tell if it’s doing something it shouldn’t (which it obivously is)

So no - 403 errors from S3 buckets with Loki are not clear. At all.

4

If you can share your configuration I’ll be happy to take a look at it for you.

5

Current config I am trying for the storage section:

loki:
  storage:
    type: s3
    s3:
      endpoint: https://s3.home.routh.io/minio/loki-logs
      secretAccessKey: ${secret_key}
      accessKeyId: ${access_key}
      s3ForcePathStyle: true
      insecure: false

I was trying to get this working with a 3 buckets (one for each admin, chunks and rules) but with all the problems I’ve been having I decided to try 1 bucket with path style.

https://s3.home.routh.io/minio/loki-logs is the correct full URL for the bucket that should be used by loki (verified with the s3 client) - the bucket name is loki-logs currently.

Other incarnations I’ve tried:

Was trying 3 buckets here.

---
loki:
  storage:
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin
    type: s3
    s3:
      endpoint: https://s3.home.routh.io/minio/
      secretAccessKey: ${secret_key}
      accessKeyId: ${access_key}
      s3ForcePathStyle: false
      insecure: false

Also trying using the s3.s3 option to explicitly supply the url to the bucket after seeing some issue threads.

loki:
  storage:
    type: s3
    s3:
      s3: s3://s3.home.routh.io/minio/loki-logs
      secretAccessKey: ${secret_key}
      accessKeyId: ${access_key}
      s3ForcePathStyle: true
      insecure: false

Ultimately with no logging anywhere from Loki about what URL these configurations wind up with it thinking it should use - troubleshooting this has been obtuse - and Minio doesn’t log this on the TrueNAS side either (nor does the traefik ingress log the request path) - so I’m about to need to start doing tcpdumps here to try and see what URL it’s actually trying to wind up with a 403.

6

Finally found this, multifaceted issue.

  1. The client I tested with used my admin creds, and it created a /minio/ bucket and put the stuff in a subfolder of that bucket.

  2. The terraform provider I used is ultimately returning the wrong URL - it should be https://s3.home.routh.io/<bucket> which is what is in the presigned URLs from the UI - but he provider is adding an extra /minio/ to the path for some reason.

7

Ultimately though, loki really needs more context on failures to write to S3, like the URL it was trying.

8

I agree more information on failure is usually better. I don’t have much experience using Loki with Minio, but glad you were able to track it down.

9

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.