I have a Loki cluster deployed in microservice mode, currently writing about 10 tb of data per day, and expect to write several pb of data per day in the future, but my s3 limits the number of objects per bucket, will it negatively affect s3 if I use more than one bucket, e.g. 50 buckets.
That’s a lot of logs!
I think you are already a lot more experienced than I am when it comes to Loki, but still I’d say you are probably safe to use more S3 buckets to circumvent rate limits. I am not an expert in this, and I am sure there is some theoretical limit somewhere, but a company I worked for before were keeping close to 50PB of data in S3. It probably also wouldn’t hurt to speak with your AWS tech account rep as well just in case