I installed loki on to my cluster using this command:
helm upgrade --install loki grafana/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false
When I go to the explore tab to get logs for my pod, the namespace of my pod is not detected at all only these namespaces are shown:
cert-manager, default, ingress-nginx, kube-system, monitoring where as I have several more namespaces with multiple pods yet they are not detected by loki. I even tried it using grafana cloud yet I got the same result. Am I missing something? Is there any good resource that I can refer so I that I can configure loki to detect all pods
The component responsible for reading the logs and forwarding them to your local Loki (or to Grafana Cloud) is promtail.
You should be able to see a promtail pod per Kubernetes-node:
$ kubectl get pods -o wide -l app=promtail
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
loki-promtail-pzcnk 1/1 Running 0 18m 10.42.0.10 k3d-k3s-default-server-0 <none> <none>
The configuration of those promtail pods can be found in a config map called
$ kubectl edit configmaps loki-promtail
I would check the following:
I have 2 pods running and both of them are running on separate nodes
kubectl get pods -o wide -l app=promtail --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
monitoring loki-promtail-mxf26 1/1 Running 0 4h16m 10.244.0.24 pool-sxgza28n2-udrxd <none> <none>
monitoring loki-promtail-vshdt 1/1 Running 0 4h16m 10.244.0.251 pool-sxgza28n2-udrx0 <none> <none>
how do I check for pod logs in this?
The best next step would be to check the “show more” for a particular pod in one of those namespaces that you are missing. (Ideally post the output of
kubectl get pods -o wide -n my-namespace --show-labels my-pod).
I think I got it, so in the targets I found the labels for pod, so I went to the explore tab again in grafana and there I found the logs for my deployment. Thank you so much for your help!