Collect log files from kubernets pods

Grafana Loki
,
1

Hello everyone,

I have some pods running in Kubernetes, which do not write to stdout/stderr, but write their own log files.
e.g.: /var/log/<program_name>/*.log
I cannot change the behaviour that they also write to stdout/stderr.
Promtail as well as Loki run in Kubernetes and were installed via Helm.
How can I collect these logs with Promtail?
I have not yet been able to find any instructions that could help me. I could find some posts from the last years, but none of them were answered or has a solution.
If this is not possible at all, I would also appreciate an answer.
I hope you can help me.

Thank you very much.

1 Like
2

If your container writs logs to filesystem you really only have three choices that I can think of:

  1. Use a sidecar container to collect logs. This is probably the most straight forward and least intrusive to both your app development and deployment process.

  2. You can mount some sort of volumes to your container’s /var/log/ directory. You can do this either by using persistent volumes, or by making your container DAEMON and mount the volume from host. Neither of which is ideal.

  3. Change your app and write everything to stdout. You can use structured output like JSON to include things such as log file name, so that can be parsed and grouped into separate log streams from your log pipeline. This of course requires cooperation from your app development team.

3

Hi @bigflagburito ,

I take it this just mean that the logs are not discovered by your kubernetes_sd_configs. Even with kubernetes_sd_configs Promtail is really only tailing log files written to the local file system of the worker node.

For me that means that Promtail mounts the following paths

      - hostPath:
          path: /var/lib/docker/containers
          type: ""
        name: containers
      - hostPath:
          path: /var/log/pods
          type: ""
        name: pods

If you wanted to, you should be able to mount /var/log/ or /var/log/<program_name> and use static_configs to pick up the logs.

4

I have implemented the variant with the sidecar container. I have another promtail running, but it does what it should.

Many thanks