Collect log files from kubernets pods

Grafana Loki
,
1

Hello everyone,

I have some pods running in Kubernetes, which do not write to stdout/stderr, but write their own log files.
e.g.: /var/log/<program_name>/*.log
I cannot change the behaviour that they also write to stdout/stderr.
Promtail as well as Loki run in Kubernetes and were installed via Helm.
How can I collect these logs with Promtail?
I have not yet been able to find any instructions that could help me. I could find some posts from the last years, but none of them were answered or has a solution.
If this is not possible at all, I would also appreciate an answer.
I hope you can help me.

Thank you very much.

1 Like
2

If your container writs logs to filesystem you really only have three choices that I can think of:

  1. Use a sidecar container to collect logs. This is probably the most straight forward and least intrusive to both your app development and deployment process.

  2. You can mount some sort of volumes to your container’s /var/log/ directory. You can do this either by using persistent volumes, or by making your container DAEMON and mount the volume from host. Neither of which is ideal.

  3. Change your app and write everything to stdout. You can use structured output like JSON to include things such as log file name, so that can be parsed and grouped into separate log streams from your log pipeline. This of course requires cooperation from your app development team.

1 Like
3

Hi @bigflagburito ,

I take it this just mean that the logs are not discovered by your kubernetes_sd_configs. Even with kubernetes_sd_configs Promtail is really only tailing log files written to the local file system of the worker node.

For me that means that Promtail mounts the following paths

      - hostPath:
          path: /var/lib/docker/containers
          type: ""
        name: containers
      - hostPath:
          path: /var/log/pods
          type: ""
        name: pods

If you wanted to, you should be able to mount /var/log/ or /var/log/<program_name> and use static_configs to pick up the logs.

1 Like
4

I have implemented the variant with the sidecar container. I have another promtail running, but it does what it should.

Many thanks

1 Like
5

Thank you for the suggested answers @tonyswumac and the feedback @bigflagburito .

I could also get it to work this way, but I did not want to have a second promtail instance running on the pod only for this reason. I also needed to mount the container’s /var/log folder to the new promtail /var/log so that it could collect the logs.

Did you also have to mount the log files from the container to the new promtail instance /var/log folder?

I was hoping the log files from the neighboring container get discovered by promtail like @b0b suggested without adding a new volumeMount to promtail. They run on the same pod after all.

6

Your problem is that the containers are writing logs to storage. For containers that’s just not a good ideal in general. So your only option really is to make the container’s volume available to promtail somehow, which means either running promtail inside or along side containers, or make the container volume available on the host level by mounting a host path into containers (this limits your container to 1 per host).

#3 proposal from my original recommendation should still be your best long-term solution.

1 Like
7

Thanks for the reply. The logs actually get written to remote Azure Blob Containers (under an Azure Storage Account as json files). From there the Blob Container gets mounted to a persistent volume of a Kubernetes pod container so that they can be read by Promtail which sends them to Loki.

I wanted to avoid writing logs to storage and get them written to stdout instead, however it is not possible in this case I guess.

8

Because I was asked by PN what the config files look like, I’ll post them here too.

I have not changed the actual promtail that monitors all Kubernetes pods.
In the deployment for the specific pods, I added the sidecar with an configmap.

deployment.yaml

.....
      containers:
      - image: "test-logger"
        .......
        - name: shared-logs # shared space monitored with Promtail
          mountPath: /var/log/test-logger
      # Sidecar Container Promtail
      - name: promtail
        image: grafana/promtail:master
        args:
          - "-config.file=/etc/promtail/promtail.yaml" # Found in the ConfigMap
        volumeMounts:
          - name: promtail-config
            mountPath: /etc/promtail
          - name: shared-logs # shared space monitored with Promtail
            mountPath: /var/log/test-logger
      volumes:
      - name: promtail-config
        configMap:
          name: test-logger-promtail-sidecar-config-map
      - name: shared-logs  # shared space monitored with Promtail
        emptyDir: {}

The matching test-logger-promtail-sidecar-config-map.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-logger-promtail-sidecar-config-map
  namespace: test
data:
  promtail.yaml: |
      clients:
        - url: http://loki-gateway.loki.svc.cluster.local/loki/api/v1/push # URL from Loki
      scrape_configs:
        - job_name: "test-logger"
          static_configs:
            - targets:
                - localhost
              labels:
                app: "test-logger"
                namespace: "test"
                __path__: /var/log/test-logger/*.log
1 Like
9

Thanks a lot for sharing the config! I have meanwhile tried a couple of things, but mainly the two solutions below seemed to work:

1- with using a second promtail instance deployed to the same pod, it had some side effects as Tony explained. In addition, promtail sometimes silently skipped scraping logs after a while, perhaps because the size of logs is huge and they get sent quite often.
I used a deployment similar to this:

spec:
      containers:
      - args:
        - -config.file=/etc/promtail/promtail.yaml
        - name: promtail
          image: docker.io/grafana/promtail:2.9.2
          ...
         volumeMounts:
          - mountPath: /var/log/
          name: shared
          ...
  volumes:
      - name: shared
        persistentVolumeClaim:
          claimName: shared

and the promtail config-yaml similar to the one you posted.

2- tried deploying a sidecar container solution from the trailing sidecar which uses Fluent Bit under the hood. It seems to produce better results, but I still need to keep an eye on its performance. Here the scraping worked better and there is no need to deploy an extra promtail instance because the daemonset promtail can scrape the container normally like with other containers. The side car container gets deployed automatically after I added an annotation with the persistent volume name and the path of the logs directory to the deployment spec as below:

Annotations: tailing-sidecar: shared-pvc: /var/log/*/*.log

1 Like
10

No funcionaba si se creaba un enlace simbĂłlico desde el archivo donde la app escribĂ­a los log hacia /dev/stderr ??