We’re using logging-operator to send streams log logging data to various loki instances which all reside in the same namespace. There are three instances to be precise. These three loki instances which are comprised of the various pieces of the loki ecosystem seem to run fine.
- One instance is the common “fake” (common stuff) instance.
- One is one customer; we’ll say customer “A” that we want to make sure remains separate from common and customer “B”
- The last one is customer “B” let’s say, whom we want to make sure remains separate from common and customer “A”
We store the logs in S3 using IRSA to store said logs to respective S3 buckets: common, customer “A”, and customer “B”
While troubleshooting a problem with searching for log entries in grafana, we noticed a strange problem where we were receiving 404s from S3. We later noticed this was happening for all customers in this cluster.
While checking the common ring status using the API, I observed that all of the distributors were listed instead of just the common set of distributors. This surprised us as we expected only those distributors for their collective set to show up.
Why would this happen? And is this normal? My guess is that we’ve configured something wrong.
