We deployed official grafana promtail docker image v2.7.7 into Azure K8s cluster.
We configured a service account with following permissions in its associated cluster role:
apiGroups: [“”]
resources: [“pods”, “nodes”, “namespaces”]
verbs: [“get”, “list”, “watch”]
We also configured it to scrape logs from pods in some namespaces.
But when promtail pod starts, we noticed following errors in its logs:
reflector.go:324] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:572: failed to list *v1.Pod: Get “https://10.0.0.1:443/api/v1/namespaces/argocd/pods?fieldSelector=spec.nodeName%3Dpromtail-5775f9bc95-8hfgt&limit=500&resourceVersion=0”: dial tcp 10.0.0.1:443: connect: connection refused.
My understanding is that promtail is querying K8s api to get list of pods in the namespace.
I did verify that 10.0.0.8 is the correct address of K8s internal api endpoint injected into the pod as env variable.
I also validated the automaically generated bearer token (from the associated service account) is injected correctly at /var/run/secrets/kubernetes.io/serviceaccount and I can use it to access the internal K8s API successfully.
So I have no clue why promtail’s attempt to access K8s api got connection refused.
Anyone has any thought on the cause that promtail get connection refused?
Thanks a lot!