I have 2 separate clusters both using Loki with a config like this:
config: auth_enabled: false storage_config: aws: s3: s3://us-east-1 bucketnames: a-shared-loki-bucket boltdb_shipper: active_index_directory: /data/loki/index shared_store: s3 cache_location: /data/loki/boltdb-cache
(Notice that auth_enabled is false, and both clusters would be sending the same or no tenant ID)
I have loki instances running in both a blue and a green Kubernetes cluster. Both have the same config as above and are using the same s3 bucket.
However, when I query Loki in the blue cluster, I only see logging from pods in the blue cluster. When I query Loki in the green cluster, I only see logging from pods in the green cluster.
How are the clusters sharing an s3 bucket for Loki, but somehow not actually sharing the logs?
I am deploying Loki via the official helm chart, it gets deployed with grafana and promtail and prometheus.