Multi cluster grafana OSS architecture

  • What Grafana version and what operating system are you using?
    Grafana v11.3.1

  • What are you trying to achieve?

  • Setup multi cluster and region grafana stack deployment for centralized logs and metrics

  • How are you trying to achieve it?
    Using the kubernetes helm charts

  • What happened?

  • Technical challenges in terms of authentication and loki/mimir configuration

  • What did you expect to happen?

  • Able to see logs and metrics in one place handling multi clusters located in multiple regions.

  • Using AWS as the cloud provider and loki-distributed and mimir-distributed helm charts for the deployment.

loki version - loki-distributed-0.79.3 app version: 2.9.8
mimir version - mimir-distributed-5.5.0 app version: 2.14.0

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

  • GitHub - grafana/helm-charts

In terms of architectural point of view., we want to achieve central observability for the kubernetes logs and metrics.

  1. we will have one central loki instance running in AWS with S3 as as tsdb storage
  2. All the kubernetes clusters will forward the pod logs to the central loki instance via grafana agent deployed in the clusters.
  3. we will have a central grafana UI that connect to loki datasource and logs can be viewed.
  4. Same steps (1-3) for mimir as well

I have few questions around it…

  1. Is this the right approach for the multi region / cluster architecture ?
  2. How does each cluster gets authenticated to the central loki … ? they all share same user/pass ?
  3. Need some inputs on setting the tenant value via (X-Scope-OrgID) … can we set that to the k8s cluster name ? … I need to know how it will be used for ?
  4. Do i need separate loki datasource configured for each of the X-scope-OrgID ?

Please shed some lights on it…thanks