Namespace labels not showing all namespaces

I installed loki on to my cluster using this command:

helm upgrade --install loki grafana/loki-stack  --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false

When I go to the explore tab to get logs for my pod, the namespace of my pod is not detected at all only these namespaces are shown:
cert-manager, default, ingress-nginx, kube-system, monitoring where as I have several more namespaces with multiple pods yet they are not detected by loki. I even tried it using grafana cloud yet I got the same result. Am I missing something? Is there any good resource that I can refer so I that I can configure loki to detect all pods

The component responsible for reading the logs and forwarding them to your local Loki (or to Grafana Cloud) is promtail.

You should be able to see a promtail pod per Kubernetes-node:

$ kubectl get pods -o wide -l app=promtail
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE                       NOMINATED NODE   READINESS GATES
loki-promtail-pzcnk   1/1     Running   0          18m   10.42.0.10   k3d-k3s-default-server-0   <none>           <none>

The configuration of those promtail pods can be found in a config map called loki-promtail:

$ kubectl edit configmaps loki-promtail

I would check the following:

  • Is every Kubernetes node running a promtail pod. Especially nodes that are home to the pod with the missing logs.
  • If there is a promtail on the node missing the pod logs, I would connect to the promtail web interface and doublecheck it is showing up under the scrape targets:
    # Forward port of promtail
    $ kubectl port-forward loki-promtail-pzcnk 3101:3101
    # Now you can open your browser and go to http://localhost:3101/targets

I have 2 pods running and both of them are running on separate nodes

kubectl get pods -o wide -l app=promtail --all-namespaces
NAMESPACE    NAME                  READY   STATUS    RESTARTS   AGE     IP             NODE                   NOMINATED NODE   READINESS GATES
monitoring   loki-promtail-mxf26   1/1     Running   0          4h16m   10.244.0.24    pool-sxgza28n2-udrxd   <none>           <none>
monitoring   loki-promtail-vshdt   1/1     Running   0          4h16m   10.244.0.251   pool-sxgza28n2-udrx0   <none>           <none>

image

how do I check for pod logs in this?

The best next step would be to check the “show more” for a particular pod in one of those namespaces that you are missing. (Ideally post the output of kubectl get pods -o wide -n my-namespace --show-labels my-pod).

I think I got it, so in the targets I found the labels for pod, so I went to the explore tab again in grafana and there I found the logs for my deployment. Thank you so much for your help!