Problem with multiple s3 buckets to resolve S3 Rate limit

Grafana Loki
1

Hello, I did an upgrade from Loki v2 to v3, and it started to throw a lot of SlowDown: Please reduce your request rate.
I tried to tweak the configs, read a lot in issues, docs, slack, community and even chatgpt, but nothing that I did reduced the SlowDown error.
Then tried multi tenancy, to avoid the S3 prefix rate limit, but multi tenancy doesn’t work for us because we need to search across all dynamic generated tenants.
And now I saw that Loki supports multiples buckets, from the config s3_storage_config :

# Comma separated list of bucket names to evenly distribute chunks over.
# Overrides any buckets specified in s3.url flag
# CLI flag: -<prefix>.storage.s3.buckets
[bucketnames: <string> | default = ""]

Knowing that I created another s3 bucket and now my Loki setup is using two buckets, apparently it’s working and I’ll see if the SlowDown will reduce tomorrow, BUT now the problem is that Loki doesn’t find any log prior to that config.
Is there something I need to do fix this?
I’ll be very grateful for any kind of help.

2

I don’t think you can just change bucket_names because of how Loki calculates which bucket to save objects to if you have more than one. I don’t use multiple S3 buckets myself, and I haven’t tried to migrate, but you might try this (mock config, not tested):

  1. Create one additional storage configuration, for example:
common:
  storage_config:
    s3: # this being your current / old config
      bucket_names: current_bucket
      region: aws_region
    s3_multi:
      bucket_names: current_bucket, new_bucket
      region: aws_region
  1. In your schema config, create a new schema with a future date with the new storage config:
schema_config:
  configs:
    - from: 2019-07-01
      store: tsdb # or whatever you use
      object_store: s3
      schema: <VERSION>
      index:
        prefix: index_
        period: 24h
    - from: <FUTURE_DATE>
      store: tsdb # or whatever you use
      object_store: s3_multi
      schema: <VERSION>
      index:
        prefix: index_
        period: 24h

And see if that works. Obviously, test on a test cluster, and hopefully someone who’s actually done this can comment as well.

1 Like
3

Hi @tonyswumac thanks for your response!
It makes sense, I’ll try it.

4

@tonyswumac It looks common.storage_config.s3_multi config does not work with error field s3_multi not found in type common.Storage

5

I don’t use multiple S3 buckets, so I couldn’t test for you. If it’s not supported under common, then define it under storage_config, and reference it in your index configuration.

6

I might be wrong but I think you need named_stores like this:

storage_config:
  aws:
    s3: https://${MINIO_ACCESS_KEY}:${MINIO_SECRET_KEY}@s3api.domain.com
    s3forcepathstyle: true
    bucketnames: loki
  # New tsdb-shipper configuration
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache
  named_stores:
    aws:
      amazon-s3:
        region: eu-central-1
        bucketnames: sfw-loki

Then you can use it like this:

schema_config:
  configs:
    - from: "2024-06-28"
      index:
        period: 24h
        prefix: index_
      object_store: aws
      schema: v13
      store: tsdb
    - from: "2024-12-16" # <---- A date in the future on wich the new db schema is used
      index:
        period: 24h
        prefix: index_
      object_store: amazon-s3
      schema: v13
      store: tsdb
1 Like