Mimir remote write based on namespace

Grafana Alloy
, , ,
1

Hello .*,

I could not find an answer if the following is currently possible with alloy. Therefore I’d appreciate any help.

We have a setup, where we send metrics from different namespaces to a corresponding tenant in mimir (1:1 or n:1). E.g.

k8s namespace -> mimir tenant
tenant1              -> tenant1
tenant2              -> tenant2
...

Currently I am solving this by a prometheus.relabel PER namespace/tenant forwarding_to a prometheus.remote_write PER namespace/tenant.

I’m a wondering if a more (memory efficient) conditional routing is possible or would be a usefull feature request:
Having ONE prometheus.*selective_forward* (or similar) which routes to a specific remote_write base on a label. E.g.

prometheus.selective_forward {
  // forward all metrics for a namespace to the remote_write 
  // named like the value of the namespace label
  selector {
     label =  "namespace"
     regex = "(.*)"
     forward_to = "prometheus.remote_write.$1.receiver"
  }
}

This would save a lot of resources in alloy and I would guess it is much more memory efficient.

Or is this already possible now and I don’t see how?

Thanks a lot
Frederik

1 Like
2

First of all, I generally consider tenants as an organizational separator, and unless you actually need to enforce visibility to each namespace based on your organization unit using namespace for tenant is not ideal.

How is your Mimir configured? Are you using a frontend with authentication, or are you sending metrics directly to Mimir with an org ID header?

If you are using header, you can supply the header in prometheus.remote_write, but if you are using authentication I am not sure if that’s possible or not.

3

Hi Toni, thanks for your input.
Just wondering: what would you use to separate “tenants/org units” inside a cluster other than namespaces? We exactly do need enforce visibility per namespace.

We are sending metrics via a proxy-frontend with basic authentication to mimir.

I don’t understand how specifying headers would help (which is per remote_write identical to basic auth - how we do it now). If I could use a label and rewrite it into a header THAT would something which could be working. But I don’t see this is possible.

Thanks!

4

I typically use tenants for organization separation, and if you do need to enforce visibility per namespace then it probably fits your use case. That said, I usually consider an EKS cluster a logical entity, and having to separate metrics based on namespace is probably not a very standard use case.

The way tenant works is that when you send a request to Mimir you would have header X-Scope-OrgID in the request. The authentication reverse proxy in the front adds a real authentication layer and maps user to orgid. If you are already using the authentication frontend then header won’t work for you.

I think you probably have two realistic solution:

  1. Hard code one metrics pipeline per workspace. I am not an expert in prometheus, and I don’t think this is actually possible. But worth some exploration.
  2. Modify the authentication proxy to NOT map users, and instead supply the org headers yourself (and the authentication layer would just be a dummy auth layer that does no user/org mapping).

Hopefully someone else can think of a better solution.

5

Thanks for your thoughts. We generally have a working setup but it does not scale well with namespaces/tenants and therefore “logically unnecessary” relabelings. This is my actual concern/question.

For different scraped metrics (directly or via servicemonitor - or similar - definitions) we need to “sort by namespace (or similar labels) and forward to different tenants”. Currently we do this by sending a scraped metric to one relabel per tenant. There we keep only the tenants metrics and forward it to the corresponding remote_write.

So in pictures, e.g. ingress metrics for n tenants:

scrape: ingress ---1:n---> relabel: keep where "tenant==exported_namespace" ---1:1---> remote_write for selected tenant

With my idea from the initial post - a selective forwarded - this could be:

scrape: ingress ---1:1---> "selective_forward": forward to remote_write named like label "exported_namespace" ---1:1---> remote_write for named tenant

This would reduce the number of relabelings (and alloy RAM usage) enormously.
To me it feels that this is “codewise” not too difficult to implement on the alloy side.

Is this something which is already logically possible and I overlooked it or is this a generally useful idea? Thent I would open a feature request.

I hope this clarifies my thoughts a bit.

Thanks!

1 Like