Recommended way to split data between environments in a single Grafana instance

Grafana
, , ,
1

We’re fairly new to Grafana and would love some guidance on how to organize a single self-hosted instance that fronts metrics and logs for multiple application environments.

Backend today: Mimir + Loki single-tenant, with everything tagged by a deployment_environment label.

We’re considering moving to one Mimir/Loki tenant per environment (collectors would set X-Scope-OrgID: ) and also using Grafana organizations to separate things in the UI. But we’re not sure how to structure it, or whether Grafana organizations are even recommended for this use case anymore.

Options we’re weighing:

  1. One Grafana org per environment. Each org would point to a single data source on Mimir and Loki. We like how this feels, because our team generally looks at a single environment at a time, so being able to switch context with a simple drop down is nice, and we don’t have to add an environment label to every query.
  2. One Grafana org per concern (e.g. “Applications” vs “Infrastructure Monitoring”), each with multiple data sources covering all envs. So all related logs and metrics live in the same org, and we could still use the data source as a way to switch between environments when looking at logs/metrics and running queries.
  3. A single Grafana org with one data source per env. So Mimir and Loki would still have multitenancy enabled for the different environments that send in data. This still gives us the ability to use the data source as the “environment picker”, but steers us away from organizations altogether.

A few questions:

  1. Are any of these common patterns for teams running a single grafana instance with data from multiple environments?
  2. Are Grafana organizations recommended for this kind of use case, or considered legacy / discouraged for new deployments in general?
  3. If Grafana organizations are discouraged, is per-environment multitenancy in Mimir/Loki still worth doing on its own, or does it lose most of its value without orgs to separate the data sources and if we already have an environment label?

We’re running Grafana OSS with under 10 environments, a small team, and we expect to manage dashboards as code.

2

I will say that single grafana org with one datasource per environment is best option
for small teams running a single grafana instance
Grafana organisations are not recommended as they add more tasks when managing dashboards as code
I have run this entire setup locally using Docker compose with Grafana, Mimir, Loki and Promtail

Screenshot (70)
Screenshot (70)1920×1080 339 KB

Screenshot (71)
Screenshot (71)1920×1080 308 KB

Multitenancy in Mimir and Loki gives real data isolation at storage level using X-Scope-OrgID headers .

Screenshot (72)
Screenshot (72)1920×1080 310 KB

Screenshot (74)
Screenshot (74)1920×1080 288 KB

On the Grafana side each environment gets its own datasource and you use the datasource picker as your environment switcher.

Screenshot (75)
Screenshot (75)1920×1080 282 KB

3

Hi @aotpseven

I have a similar setup as @infofcc3 well detailed above, with one variation. I have different Grafana UIs per environment and each UI has its own datasource. As Grafana OpenSoure does not have RBAC / LBAC kind of access controls, I can restrict access to specific Grafana UI and users can only query data from their allowed environments.

Wanted to share if you want such segregation at user level :wink:

-Suleyman Kutlu (a.k.a. SNK)