Namespace labels not showing all namespaces

Grafana Loki
#1

I installed loki on to my cluster using this command:

helm upgrade --install loki grafana/loki-stack  --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false

When I go to the explore tab to get logs for my pod, the namespace of my pod is not detected at all only these namespaces are shown:
cert-manager, default, ingress-nginx, kube-system, monitoring where as I have several more namespaces with multiple pods yet they are not detected by loki. I even tried it using grafana cloud yet I got the same result. Am I missing something? Is there any good resource that I can refer so I that I can configure loki to detect all pods

#2

The component responsible for reading the logs and forwarding them to your local Loki (or to Grafana Cloud) is promtail.

You should be able to see a promtail pod per Kubernetes-node:

$ kubectl get pods -o wide -l app=promtail
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE                       NOMINATED NODE   READINESS GATES
loki-promtail-pzcnk   1/1     Running   0          18m   10.42.0.10   k3d-k3s-default-server-0   <none>           <none>

The configuration of those promtail pods can be found in a config map called loki-promtail:

$ kubectl edit configmaps loki-promtail

I would check the following:

  • Is every Kubernetes node running a promtail pod. Especially nodes that are home to the pod with the missing logs.
  • If there is a promtail on the node missing the pod logs, I would connect to the promtail web interface and doublecheck it is showing up under the scrape targets:
    # Forward port of promtail
$ kubectl port-forward loki-promtail-pzcnk 3101:3101
# Now you can open your browser and go to http://localhost:3101/targets
#3

I have 2 pods running and both of them are running on separate nodes

kubectl get pods -o wide -l app=promtail --all-namespaces
NAMESPACE    NAME                  READY   STATUS    RESTARTS   AGE     IP             NODE                   NOMINATED NODE   READINESS GATES
monitoring   loki-promtail-mxf26   1/1     Running   0          4h16m   10.244.0.24    pool-sxgza28n2-udrxd   <none>           <none>
monitoring   loki-promtail-vshdt   1/1     Running   0          4h16m   10.244.0.251   pool-sxgza28n2-udrx0   <none>           <none>

image
image925×538 34.3 KB

image

how do I check for pod logs in this?

#4

The best next step would be to check the “show more” for a particular pod in one of those namespaces that you are missing. (Ideally post the output of kubectl get pods -o wide -n my-namespace --show-labels my-pod).

#5

I think I got it, so in the targets I found the labels for pod, so I went to the explore tab again in grafana and there I found the logs for my deployment. Thank you so much for your help!