Grafana only detects the namespace/node where the prometheus server is located

I have installed a Grafana-loki stack on my Kubernetes cluster in namespace “grafana-loki”. The Kubernetes cluster has several other namespaces which are used for applications. I have exposed kube-state-metrics pod and I viewed the metrics and here is an example of what it looks like.

kube_deployment_status_replicas_updated{namespace="grafana-loki",deployment="loki-stack-prometheus-pushgateway"} 1
kube_deployment_status_replicas_updated{namespace="ns-app-dev",deployment="app-configuration-api"} 1
kube_deployment_status_replicas_updated{namespace="ns-app-dev",deployment="app-monitoring-api"} 1
kube_deployment_status_replicas_updated{namespace="grafana-loki",deployment="loki-stack-grafana"} 1
kube_deployment_status_replicas_updated{namespace="default",deployment="aws-load-balancer-controller"} 2
kube_deployment_status_replicas_updated{namespace="kube-system",deployment="ebs-csi-controller"} 2

Then, I created a stat panel in Grafana with below query:


The stat panel shows 8, which is the correct number. However, when I change the namespace to “grafana-loki”, the stat panel still shows 8, and when I change the namespace to “ns-app-dev”, “default”, or “kube-system”, the stat panel shows “no data”.

The problem is the same for every metrics on kube-state-metrics, which Grafana cannot view the correct number for other namespaces/nodes.