I have a multi-environment Kubernetes setup with separate environments:
dev
test
prod
Each environment contains multiple Kubernetes clusters.
On one cluster inside each environment I deployed a separate Loki instance, and on every cluster I deployed Grafana Alloy to collect and send logs to the Loki instance of that environment.
Example:
dev clusters → send logs to Loki in dev
test clusters → send logs to Loki in test
prod clusters → send logs to Loki in prod
Each Alloy instance has its own Loki endpoint configured correctly. However, I still see logs from other environments when querying Loki.
Important detail:
All Loki instances use the same S3 bucket, but I created separate folders/prefixes:
In my opinion you should just use dedicated S3 bucket for each of your Loki cluster, and save yourself some headache. It’s not only easier, it’s also more secure in terms of data separation between environments.
I would suggest have single Loki for all environments and have multi-tenancy enabled. I have similar setup with a central Grafana LGTM stack. All components are multi-tenancy enabled. Each environment has its own tenant id.
For Grafana UI I have three Grafana UI deployments per each environment and they only can see their tenant (configured in Datasource definition)
This is, in my humble opinion, cost effective and easy to maintain. Single cluster for Grafana stack, single Loki, single S3 bucket set and proper isolation of data between environments.
For your case setup, if not recently changed, you can only define S3 bucket name to Loki for storage not a prefix/path_prefix..
if you want to go with three different Loki setup, then what @tonyswumac said is the way to go forward…
What you’re seeing is usually caused by Loki instances sharing the same storage namespace/index, even if they point to different “folders” conceptually in S3.
A few important points:
Labels like environment=dev are only for querying/filtering. They do NOT provide storage isolation.
Real isolation in Loki comes from separate storage prefixes/buckets/tenants.
Using the same S3 bucket is fine, but only if each Loki instance is configured with a unique storage prefix for BOTH chunks and index data.
Just creating folders manually in S3 is not enough unless Loki is explicitly configured to use them.
For TSDB/Boltdb-shipper storage, you need separate prefixes such as:
observability-dev/
observability-test/
observability-prod/
configured in Loki itself.
For example, depending on your storage backend/version, look at settings like:
storage_config.aws.s3
storage_config.aws.bucketnames
storage_config.boltdb_shipper.shared_store
schema_config.configs[].index.prefix
common.path_prefix
The critical part is that each Loki instance must write/read from a distinct prefix namespace.
Right now, your Loki instances are most likely scanning the same index location, so queries return mixed data across environments.
Best practice is usually one of these:
Separate S3 buckets per environment (simplest + safest)
Same bucket but strict per-environment prefixes configured in Loki
Multi-tenant Loki using X-Scope-OrgID isolation
For most setups, separate buckets or strict prefixes are easier to manage than multi-tenancy.
Also make sure:
Index prefixes are unique per environment
Compactor is not sharing the same working directory/storage namespace
Retention configs are separated if needed
Adding an environment label is still a good idea for querying/debugging, but it should not be relied on for isolation.
