I have the loki-stack helm chart installed on my k8s cluster and both prometheus and grafana seem to be running fine. Loki seems alright as well except there are a number of namespaces missing from my logs. Seeing no issues in the logs for the loki pod I checked the two promtail pods and one of them is having an issue with “creating targets”. Example log below:
level=info ts=2022-03-27T05:16:40.733709032Z caller=filetargetmanager.go:261 msg=“Adding target” key="{app_kubernetes_io_instance=“portainer”, app_kubernetes_io_name=“portainer”, container=“portainer”, job=“portainer/portainer”, namespace=“portainer”, pod=“portainer-597585d76d-n875p”, pod_template_hash=“597585d76d”}"
level=error ts=2022-03-27T05:16:40.733733039Z caller=filetargetmanager.go:265 msg=“Failed to create target” key="{app_kubernetes_io_instance=“portainer”, app_kubernetes_io_name=“portainer”, container=“portainer”, job=“portainer/portainer”, namespace=“portainer”, pod=“portainer-597585d76d-n875p”, pod_template_hash=“597585d76d”}" error=“filetarget.fsnotify.NewWatcher: too many open files”
The other promtail pod is also giving some errors:
level=error ts=2022-03-27T05:31:01.58481613Z caller=filetarget.go:295 msg=“failed to tail file, stat failed” error=“stat /var/log/pods/kube-system_etcd-server_08e28ca60fbe23d654b96ac08dc74c30/etcd/2.log: no such file or directory” filename=/var/log/pods/kube-system_etcd-server_08e28ca60fbe23d654b96ac08dc74c30/etcd/2.log
I’m not sure which is the core of the issue but I’d like to resolve both if possible.