We are operating a service where we use Grafana, Loki, and OpenTelemetry to collect and visualize logs. In our current production environment, we are using single-tenant mode with auth_enabled: false.
Now, we are planning to switch to multi-tenant mode by enabling auth_enabled: true. However, we are concerned about potential issues that may arise from this change.
For example, when switching to multi-tenancy, will existing data still be accessible in Grafana, or will we need to update all data sources with the appropriate header values (like X-Scope-OrgID)?
What are the best practices to safely transition from single-tenant to multi-tenant mode in Loki, ensuring that our monitoring services remain unaffected during the deployment process?
We would greatly appreciate any advice or recommendations based on similar experiences.
We started with auth_enabled turned on, so I’ve never attempted to migrate like this. I think your safest and easiest solution is to simply create another cluster with auth_enabled = true, then gradually migrate your log agents to the new cluster. If you are using Grafana for visualization you can have both clusters added as data sources and be able to query both, which should help with the migration.
We encountered the same issue, but starting a new cluster was not an option.
With auth_enabled: false, Grafana uses a default tenant ID called fake. As a first step, we configured promtail, loki, and Grafana to always use this tenant explicitly. We tested this setup locally with a docker-compose deployment, which you can see here
We also verified that it is possible to revert to auth_enabled: false, and it worked as well.
Then we applied these settings to our k8s dev cluster, and everything seems to be working as expected, except for some broken recording rules.
The next steps are to use at least BasicAuth for the loki endpoint and then add more tenants.
