I have installed a Grafana-loki stack on my Kubernetes cluster in namespace “grafana-loki”. The Kubernetes cluster has several other namespaces which are used for applications. I have exposed kube-state-metrics pod and I viewed the metrics and here is an example of what it looks like.
kube_deployment_status_replicas_updated{namespace="grafana-loki",deployment="loki-stack-prometheus-pushgateway"} 1
kube_deployment_status_replicas_updated{namespace="ns-app-dev",deployment="app-configuration-api"} 1
kube_deployment_status_replicas_updated{namespace="ns-app-dev",deployment="app-monitoring-api"} 1
kube_deployment_status_replicas_updated{namespace="grafana-loki",deployment="loki-stack-grafana"} 1
kube_deployment_status_replicas_updated{namespace="default",deployment="aws-load-balancer-controller"} 2
kube_deployment_status_replicas_updated{namespace="kube-system",deployment="ebs-csi-controller"} 2
Then, I created a stat panel in Grafana with below query:
sum(kube_deployment_status_replicas{namespace=~".*"})
The stat panel shows 8, which is the correct number. However, when I change the namespace to “grafana-loki”, the stat panel still shows 8, and when I change the namespace to “ns-app-dev”, “default”, or “kube-system”, the stat panel shows “no data”.
The problem is the same for every metrics on kube-state-metrics, which Grafana cannot view the correct number for other namespaces/nodes.