Why can't connect to Prometheus in K8s created by Rancher Desktop?

Grafana Dashboards
1

I installed Prometheus and Grafana via Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus
helm repo add grafana https://grafana.github.io/helm-charts 
helm repo update
helm install grafana grafana/grafana

Forwarding

export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 9090

export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 3000

Expose services

kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext
kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext

When I login to Grafana(http://localhost:3000) and add data source with Prometheus

1
1685×535 22.9 KB

I got this error:

2
21176×232 10.5 KB

Post “http://localhost:9090/api/v1/query”: dial tcp 127.0.0.1:9090: connect: connection refused - There was an error returned querying the Prometheus API.

But I can access localhost:9090 from browser.

The kubernetes services are:

kubectl get services
NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes                            ClusterIP   10.43.0.1       <none>        443/TCP        1d
prometheus-alertmanager-headless      ClusterIP   None            <none>        9093/TCP       1h
prometheus-server                     ClusterIP   10.43.39.133    <none>        80/TCP         1h
prometheus-prometheus-pushgateway     ClusterIP   10.43.199.209   <none>        9091/TCP       1h
prometheus-prometheus-node-exporter   ClusterIP   10.43.246.176   <none>        9100/TCP       1h
prometheus-alertmanager               ClusterIP   10.43.28.143    <none>        9093/TCP       1h
prometheus-kube-state-metrics         ClusterIP   10.43.228.89    <none>        8080/TCP       1h
prometheus-server-ext                 NodePort    10.43.104.42    <none>        80:30186/TCP   1h
grafana                               ClusterIP   10.43.208.205   <none>        80/TCP         1h
grafana-ext                           NodePort    10.43.55.78     <none>        80:32765/TCP   1h

How to set Prometheus server URL correctly?

2

Don’t use localhost - each container has own localhost. Connect to ClusterIP, e.g 10.43.39.133