Error E0328 with Promtail

[ftriboix]

Hi all,

I am deploying Promtail to a Kubernetes cluster (minikube for the time being), along with Loki. I can see that Promtail is showing a couple of error messages:

E0328 14:41:21.022682       1 reflector.go:127] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:451: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
E0328 14:41:22.543821       1 reflector.go:127] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:451: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused

What do those error mean? I can see a number of “Adding target” messages after that, so does it mean some retries have occurred and I can safely ignore the error messages?

Thanks a lot for any help!

[ftriboix]

Hi all,

I double-checked and I can confirm that I can curl https://kubernetes:443 and https://10.96.0.1:443 from within the Promtail container.

I do get an HTTP 403 with the following message: "forbidden: User \"system:anonymous\" cannot get path \"/\"". Is that the reason why Promtail is showing those error messages?

Thanks for your help

[ftriboix]

Hi all,

I double-checked the RBAC as well, and I can curl the “https://10.96.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&limit=500&resourceVersion=0” URL using the bearer token for the service account the Promtail pod is using.

So if there is no problem with the network aspect and the RBAC is setup correctly, what could be the reason for these error messages?

Thanks for any help