Limit non-admin to access part of the metrics from single Prometheus data source

[robbywang]

What Grafana version and what operating system am I using?
9.5.3 docker image, running on Ubuntu 22.04 for Arm64

What am I trying to achieve?

In short, this is like a follow up question on this previous non-solved issue:

Grafana to access limited subset of metrics from Prometheus data source

Hello, We would like to expose Grafana dashboards & metrics to our end user from a well-known limited subset of our Prometheus datasource metrics. At the moment we imagine deploying a federated prometheus instance that would import a subset of our core Prometheus instance metrics and adding that customer-oriented Grafana to that federated instance. But I was wondering if there is any possibility exists in Grafana configuration to restrict the access to a limited subset of Prometheus metrics. …

So I have prometheus running on a master server, it has 5 jobs, each job gets certain metrics from different machines in real time.
I want to split this single prometheus datasource, so that if a non-admin user logged in to grafana, he can only see some parts of the data, for example only able to see data with job=1, but still be able to edit the dashboards.

How did I try to achieve it?

I know that if I run 5 prometheus instances(each has one job from the previous 5 jobs), I can accomplish this goal by creating different organizations: adding “prometheus datasource 1” to dashboards in organization 1, and “prometheus datasource 2” to dashboards in organization2, and so on. I can just give “edit” permission to the organization, and a non-admin user can edit the dashboard. Since “edit” permission does not grant user the right to add/delete datasource in the organization, this will make sure that he can only see data from that specific prometheus/job instance.

However, I find this solution not optimal, as I will have to run 5 instances of prometheus on the master server. (In real case it could even scale up to 100 or more) Is there any way to solve this data permission problem just running one prometheus instance?
(Or if anyone can reassure me that running 100 instances/containers of Prometheus is totally fine, then I’ll just do it that way.)

Possible solutions that I can think of:

1. If there’s any way to make “Every Prometheus job as a data source in Grafana”, as this question on StackOverflow asks.
   Every Prometheus job as a data source in Grafana - Stack Overflow
   But up to now I don’t think there’s a solution. I even searched the features of the Enterprise version too, but still found no relevant solution.

2. Should I use alert alternatives to prometheus? If so, what are some good recommendations?

3. Save Prometheus data to database first, then query in real time from Grafana.

Thank you very much for reading my long question and looking forward to any suggestions!