# Kubernetes App for Grafana :: Not able to connect to DS **URL:** \ **Category:** Grafana\ **Created:** [August 30, 2017, 7:18pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515 "2017-08-30T19:18:51Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jeunii](https://avatars.discourse-cdn.com/v4/letter/j/53a042/32.png) [@jeunii](https://community.grafana.com/u/jeunii)\ **Post date:** [August 30, 2017, 7:18pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/1 "2017-08-30T19:18:51Z")
I am trying to use the Kubernetes App in Grafana ([kubernetes plugin for Grafana | Grafana Labs](https://grafana.com/plugins/raintank-kubernetes-app)) I am entering the information for my K8S cluster. The API / K8S master node is the one that I am trying to use. ![29850839-666f54a2-8cfe-11e7-907d-0cdb8d584a7f](https://us1.discourse-cdn.com/grafana/original/2X/d/d8245f10ea0541b20f2973ff771daaa10642dd0c.JPG) But when I click on Deploy , I get “Could not connect to cluster” When I try to manually add K8S as my DataSource, I get “Unknown” Error. ![29850889-bcc39ca0-8cfe-11e7-8dfd-863861858604](https://us1.discourse-cdn.com/grafana/original/2X/0/09eba7b99fae44dfe1162775cbbeb5829dba4d2d.JPG) Does anyone know what this “Unknown Error” might be ? ---
**Author:** ![daniellee](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/daniellee/32/41_2.png) [@daniellee](https://community.grafana.com/u/daniellee)\ **Post date:** [August 31, 2017, 7:56am UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/2 "2017-08-31T07:56:08Z")
Try changing your access mode to “proxy” unless you have really good reasons for using “direct”. To see the actual error, check the Chrome Dev Tools (it is a bug in the K8s App that it does not give a better error). ---
**Author:** ![jeunii](https://avatars.discourse-cdn.com/v4/letter/j/53a042/32.png) [@jeunii](https://community.grafana.com/u/jeunii)\ **Post date:** [August 31, 2017, 1:01pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/3 "2017-08-31T13:01:49Z")
Thanks daniellee. The reason I selected “Direct Access” was that its the URL that I can access directly via browser. So I opened Chrome Dev tools and checked the Response I get. See screenshot below 🙂 ![Capture](https://us1.discourse-cdn.com/grafana/original/2X/c/cd2f58e7276a661a3d68ba886abf0b6a2014d070.JPG) Although I URL ([http://172.29.219.65:8080](http://172.29.219.65:8080)) is my K8S Matser and it is returning all the API results but I am still getting the “Unknown error”. The Header values are : ``` Request URL:http://172.29.219.65:8080/ Request Method:GET Status Code:200 OK Remote Address:172.29.219.65:8080 Referrer Policy:no-referrer-when-downgrade ``` Looking into the Console output, I do see the Error : `XMLHttpRequest cannot load http://172.29.219.65:8080/. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://172.29.219.97:30998' is therefore not allowed access.` ---
**Author:** ![daniellee](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/daniellee/32/41_2.png) [@daniellee](https://community.grafana.com/u/daniellee)\ **Post date:** [August 31, 2017, 1:07pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/4 "2017-08-31T13:07:37Z")
Proxy mode exists to prevent CORS (Cross Origin) errors which you can get if you access directly from your browser. That’s why it is the default. Direct access is only used in special circumstances (local authentication through Active Directory/LDAP for example). ---
**Author:** ![jeunii](https://avatars.discourse-cdn.com/v4/letter/j/53a042/32.png) [@jeunii](https://community.grafana.com/u/jeunii)\ **Post date:** [August 31, 2017, 1:31pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/5 "2017-08-31T13:31:17Z")
Got it. Thanks daniellee. I switched to Proxy mode and now get : `GET http://172.29.219.97:30998/api/datasources/proxy/5/ 403 (Forbidden)` 172.29.219.97:30998 is the IP:Port of my grafana server. More details say : ``` The following error was encountered while trying to retrieve the URL: http://172.29.219.65:8080/ Access Denied. Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect. ``` 172.29.219.65:8080 is the IP:Port of my K8S Master. I believe there might be some limiting factor on my corporate network. I’ll contact the network admin and question this. ---
**Author:** ![daniellee](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/daniellee/32/41_2.png) [@daniellee](https://community.grafana.com/u/daniellee)\ **Post date:** [August 31, 2017, 1:33pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/6 "2017-08-31T13:33:34Z")
Is it a problem with authentication? Usually you would authenticate with TLS certs: ![image](https://us1.discourse-cdn.com/grafana/original/2X/8/8385b6d29ca6853e00af09ea2ab31d5e7218f0b3.png) ---
**Author:** ![jeunii](https://avatars.discourse-cdn.com/v4/letter/j/53a042/32.png) [@jeunii](https://community.grafana.com/u/jeunii)\ **Post date:** [August 31, 2017, 2:24pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/7 "2017-08-31T14:24:57Z")
I setup Client Certs & Client Key using `openssl req -nodes -new -x509 -keyout clientkey.pem -out clientreq.pem -days 365` Even tried to generate CA cert and use it but did not work. Whats weird is that if I try to access the URL “[http://172.29.219.65:8080/](http://172.29.219.65:8080/)” directly, I can do it : ![Capture](https://us1.discourse-cdn.com/grafana/original/2X/7/70c924d0bf89c1b78b96cabff9fce1b27c41f12a.JPG) ![Capture1](https://us1.discourse-cdn.com/grafana/original/2X/4/4938ea35d6841380b6bb3a92b4846f03a3f0fe19.JPG) No idea why I cant upload any pictures anymore 😕 Here is the curl output. ``` # curl http://172.29.219.65:8080/ { "paths": [ "/api", "/api/v1", "/apis", "/apis/apps", "/apis/apps/v1beta1", "/apis/authentication.k8s.io", "/apis/authentication.k8s.io/v1beta1", "/apis/authorization.k8s.io", "/apis/authorization.k8s.io/v1beta1", "/apis/autoscaling", "/apis/autoscaling/v1", "/apis/batch", "/apis/batch/v1", "/apis/batch/v2alpha1", "/apis/certificates.k8s.io", "/apis/certificates.k8s.io/v1alpha1", "/apis/extensions", "/apis/extensions/v1beta1", "/apis/policy", "/apis/policy/v1beta1", "/apis/rbac.authorization.k8s.io", "/apis/rbac.authorization.k8s.io/v1alpha1", "/apis/storage.k8s.io", "/apis/storage.k8s.io/v1beta1", "/healthz", "/healthz/ping", "/healthz/poststarthook/bootstrap-controller", "/healthz/poststarthook/extensions/third-party-resources", "/healthz/poststarthook/rbac/bootstrap-roles", "/logs", "/metrics", "/swaggerapi/", "/ui/", "/version" ] } ``` But the URL is not accessible from Grafana using the Proxy method 🤔🤔🤔 ---
**Author:** ![daniellee](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/daniellee/32/41_2.png) [@daniellee](https://community.grafana.com/u/daniellee)\ **Post date:** [August 31, 2017, 4:00pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/8 "2017-08-31T16:00:22Z")
Can you ping/curl the url from the machine where Grafana is installed? ---
**Author:** ![jeunii](https://avatars.discourse-cdn.com/v4/letter/j/53a042/32.png) [@jeunii](https://community.grafana.com/u/jeunii)\ **Post date:** [August 31, 2017, 5:35pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/9 "2017-08-31T17:35:30Z")
So my grafana instance is containerized as a docker image and is deployed using Kubernetes. Below are the results from within the container/pod. ``` # kubectl get pod | grep grafana grafana-deployment-2968758414-ffvd2 1/1 Running 0 1d # kubectl exec -it grafana-deployment-2968758414-ffvd2 -- /bin/bash [root@grafana-deployment-2968758414-ffvd2 /]# [root@grafana-deployment-2968758414-ffvd2 /]# ping -c 1 172.29.219.65 PING 172.29.219.65 (172.29.219.65) 56(84) bytes of data. 64 bytes from 172.29.219.65: icmp_seq=1 ttl=63 time=0.176 ms --- 172.29.219.65 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.176/0.176/0.176/0.000 ms [root@grafana-deployment-2968758414-ffvd2 /]# [root@grafana-deployment-2968758414-ffvd2 /]# curl http://172.29.219.65:8080/ { "paths": [ "/api", "/api/v1", "/apis", "/apis/apps", "/apis/apps/v1beta1", "/apis/authentication.k8s.io", "/apis/authentication.k8s.io/v1beta1", "/apis/authorization.k8s.io", "/apis/authorization.k8s.io/v1beta1", "/apis/autoscaling", "/apis/autoscaling/v1", "/apis/batch", "/apis/batch/v1", "/apis/batch/v2alpha1", "/apis/certificates.k8s.io", "/apis/certificates.k8s.io/v1alpha1", "/apis/extensions", "/apis/extensions/v1beta1", "/apis/policy", "/apis/policy/v1beta1", "/apis/rbac.authorization.k8s.io", "/apis/rbac.authorization.k8s.io/v1alpha1", "/apis/storage.k8s.io", "/apis/storage.k8s.io/v1beta1", "/healthz", "/healthz/ping", "/healthz/poststarthook/bootstrap-controller", "/healthz/poststarthook/extensions/third-party-resources", "/healthz/poststarthook/rbac/bootstrap-roles", "/logs", "/metrics", "/swaggerapi/", "/ui/", "/version" ] } ``` ---
**Author:** ![jasl](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/jasl/32/3227_2.png) [@jasl](https://community.grafana.com/u/jasl)\ **Post date:** [March 8, 2018, 3:44pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/10 "2018-03-08T15:44:24Z")
@daniellee I have similar problem with laates version. Doesn’t matter which combination of TLS or CA Auth I use, I always get same error on save: ```auto deployments.apps is forbidden: User "system:anonymous" cannot list deployments.apps in the namespace "kube-system" ``` I tried CA certs in: - local kube config - kubernetesdash board - tiller seems like doesn’t matter which cert is used, grafa always try to use anonymous user. In my case grafana is deployed using grafana helm, so it has access to local token for auth, will be great an option to use it ---
**Author:** ![daniellee](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/daniellee/32/41_2.png) [@daniellee](https://community.grafana.com/u/daniellee)\ **Post date:** [March 9, 2018, 8:02am UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/11 "2018-03-09T08:02:56Z")
@jasl there are now two apps for k8s - one for Graphite (which is deprecated) and one for Prometheus. Which one are you using? ---
**Author:** ![adrianosantos](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@adrianosantos](https://community.grafana.com/u/adrianosantos)\ **Post date:** [March 20, 2018, 10:01pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/12 "2018-03-20T22:01:48Z")
I’m having the same problem. I’m using the new grafana kubernetes app for prometheus. ---
**Author:** ![adrianosantos](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@adrianosantos](https://community.grafana.com/u/adrianosantos)\ **Post date:** [March 21, 2018, 5:31am UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/13 "2018-03-21T05:31:11Z")
@daniellee I’m using the one for Prometheus and I’m getting the same problem: deployments.apps is forbidden: User “system:anonymous” cannot list deployments.apps in the namespace “kube-system” ---
**Author:** ![daniellee](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/daniellee/32/41_2.png) [@daniellee](https://community.grafana.com/u/daniellee)\ **Post date:** [March 22, 2018, 7:17am UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/14 "2018-03-22T07:17:11Z")
@jasl @adrianosantos which version of Kubernetes are you using? Looks like a k8s permissions problem: - [https://github.com/kubernetes/kubernetes/issues/39722](https://github.com/kubernetes/kubernetes/issues/39722) - [https://github.com/kubernetes/helm/issues/2657](https://github.com/kubernetes/helm/issues/2657) ---
**Author:** ![jasl](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/jasl/32/3227_2.png) [@jasl](https://community.grafana.com/u/jasl)\ **Post date:** [March 22, 2018, 9:18am UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/15 "2018-03-22T09:18:25Z")
@daniellee I am on 1.8.4 on azure ---
**Author:** ![daniellee](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/daniellee/32/41_2.png) [@daniellee](https://community.grafana.com/u/daniellee)\ **Post date:** [March 22, 2018, 6:32pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/16 "2018-03-22T18:32:58Z")
I don’t know the answer to this problem - I have never used RBAC with Kubernetes. But it looks like there is already an issue for adding a RBAC role to the app: > > > Hi. I tried to run this plugin on an existing kube+prometheus+grafana setup and …noticed grafana needs additional permissions to complete the setup (specifically: list deployments.apps in the namespace "kube-system"). > Would you guys be able to add an RBAC config to this repo? I can't really figure out what permissions this app needs beyond adding them one at a time, every time I get an error. ---
**Author:** ![adrianosantos](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@adrianosantos](https://community.grafana.com/u/adrianosantos)\ **Post date:** [March 22, 2018, 8:08pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/17 "2018-03-22T20:08:21Z")
@jasl @danielle I notice the permission problem and I fixed it with kubectl create clusterrolebinding grafana-admin-binding –clusterrole=cluster-admin –serviceaccount=:\ Connecting to the container with kubectl exec -it I’m able to execute and get the expected result for example to get the namespaces with the following curl: curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H “Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)” https//kubernetes.default/api/v1/namespaces But there is no place in the config page to enter the certificate and/or token when you choose direct connect. My kubernetes version is 1.9+ $ kubectl version --short Client Version: v1.9.2 Server Version: v1.9.1+2.0.2.el7 I’m able to get all deployments: curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H “Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)” https//kubernetes.default/apis/extensions/v1beta1/deployments ---
**Author:** ![adrianosantos](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@adrianosantos](https://community.grafana.com/u/adrianosantos)\ **Post date:** [March 22, 2018, 9:16pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/18 "2018-03-22T21:16:06Z")
@daniellee This is exactly the problem we are having. We are not able to pass from config page due to permissions. Our setup uses RBAC without Basic Auth and Grafana is installed with a service account by tiller. The dashboard is not honoring the service account when it tries to connect to the kubernetes. ---
**Author:** ![daniellee](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/daniellee/32/41_2.png) [@daniellee](https://community.grafana.com/u/daniellee)\ **Post date:** [March 23, 2018, 7:37am UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/19 "2018-03-23T07:37:34Z")
> [@adrianosantos](#): > > when you choose direct connect. Are you using Direct mode on the cluster config page in the app? Why? You should almost definitely be using proxy mode. (Direct means directly from your browser and is usually used with single sign on or because you have your own proxy between Grafana and k8s that handles authentication/CORS headers) ---
**Author:** ![adrianosantos](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@adrianosantos](https://community.grafana.com/u/adrianosantos)\ **Post date:** [March 23, 2018, 2:32pm UTC](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515/20 "2018-03-23T14:32:05Z")
@danillee Hi Daniel, the reason I was using direct connect is because when I try with proxy it was was not working. I reinstalled everything and I’m able to pass the config page using proxy. Thank you very much for your promptly response. Now when I go to the Dashboard it gives an error that “Datasource not found”. I checked the datasource in the list of datasources and it is there. Any clues? [Next page](https://community.grafana.com/t/kubernetes-app-for-grafana-not-able-to-connect-to-ds/2515.md?page=2)