Hi, previously we enabled multi tenancy in Loki along with conf multi_tenant_queries_enabled: true. And we designed a promtail stage in such a way that each namespace is considered as a tenant and pushes to Loki.
However, we are observing (post switching to multi tenancy), queries are 10x slower than it usually takes.
Is this expected? or anything that I was missing?
FYI, we are running ~120 namespaces (i.e. 120 tenants) and using them in X-Scope-OrgID header separated by |.
Also, it was mentioned that API call GET /loki/api/v1/tail won’t be working for multi tenant queries. This is essential for live logs in Grafana. May I know if this support is in the roadmap of future releases?
TIA.
I’ve not tried to run 120 tenants, so I can’t really comment on the performance impact, perhaps someone else with more experiences can chime in.
However, I’d say you are using tenants incorrectly. It should be used as a way to separate logs for different organization, business units, or departments. You can use label to separate logs from different namespaces, there is no need to involve tenants here. But if you have, say, 2 K8S clusters and they belong to different department that warrants different access pattern then that’s when you should use tenants.
As for Grafana integration, you can use integrate multi-tenant with Grafana, but because X-Scope-OrgID is an HTTP header you have to supply with data source, you cannot do that dynamically when querying, meaning in this case you’d need 120 data sources all with different org ID. Again, I don’t think this is what tenant is meant to be used for.
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.