Error using Promtail-sidecar for nginx deployemnt

Grafana Loki
, ,
1

Hi All
I wanted to use promtail as a sidecar with nginx container.
I have created a deployment that does not seem to work.
Can anyone please help me correct the Yaml script?

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-sidecar-config-map
data:
  promtail.yaml: |
      server:
        http_listen_port: 9080
        grpc_listen_port: 0
        log_level: "debug"
      positions:
        filename: /tmp/positions.yaml
      clients: # Specify target
        - url: http://loki:3100/loki/api/v1/push
      scrape_configs:
        - job_name:  "nginx" 
          static_configs: 
            - targets: 
                - localhost 
              labels:
                app: "storage-service" 
                __path__: var/log/nginx/*.log # Any file .log in the EmptyDir Volume.
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-promtail
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-promtail
  template:
    metadata:
      labels:
        app: nginx-promtail
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
      - name: promtail
        image: grafana/promtail:latest
        args:
        - "--config.file=/etc/promtail/promtail.yaml" 
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: config
          mountPath: /etc/promtail
        - name: shared-logs # shared space
          mountPath: /app/logs
      volumes:
      - name: config
        configMap:
          name: promtail-sidecar-config-map
      - name: shared-logs # shared space monitored with Promtail
        emptyDir:
          sizeLimit: 500Mi

The Promtail-sidecar container starts but it is not able to push the logs to loki

PS C:\Windows\system32> kubectl logs pod/nginx-promtail-566f697b85-mrdt6 -c promtail
level=debug ts=2023-01-27T09:51:13.881537424Z caller=promtail.go:115 msg="Reloading configuration file"
level=info ts=2023-01-27T09:51:13.882148715Z caller=promtail.go:123 msg="Reloading configuration file" md5sum=963a279a498a62c329093fd2a20e1e1a
level=debug ts=2023-01-27T09:51:13.882388112Z caller=manager.go:289 component=discovery msg="Starting provider" provider=static/0 subs=map[nginx:{}]
level=info ts=2023-01-27T09:51:13.882876204Z caller=server.go:323 http=[::]:9080 grpc=[::]:35413 msg="server listening on addresses"
level=info ts=2023-01-27T09:51:13.883006603Z caller=main.go:171 msg="Starting Promtail" version="(version=2.7.2, branch=HEAD, revision=c35554d09)"
level=debug ts=2023-01-27T09:51:13.883082901Z caller=manager.go:323 component=discovery msg="Discoverer channel closed" provider=static/0
level=warn ts=2023-01-27T09:51:13.883090901Z caller=promtail.go:220 msg="enable watchConfig"
level=debug ts=2023-01-27T09:51:18.883511587Z caller=filetargetmanager.go:292 msg="new target" labels="{__address__=\"localhost\"}"
level=info ts=2023-01-27T09:51:18.883616885Z caller=filetargetmanager.go:352 msg="Adding target" key="var/log/nginx/*.log:{app=\"storage-service\"}"
level=debug ts=2023-01-27T09:51:18.883738283Z caller=filetarget.go:229 msg="no files matched requested path, nothing will be tailed" path=var/log/nginx/*.log pathExclude=
PS C:\Windows\system32> notepad .\config.yaml

Loki and Grafana is running and In grafana dashboard i can see that there are no logs as promtail is not able to push any logs to loki.
Please help me in configuring the manifest to resolve the issue

2

I Found out that were few mistakes in the config and the deployment yaml files that i made which i am listing below:

  1. changing the path to an correct absolute path
    __path__: var/log/nginx/*.log

correct : __path__: /var/log/nginx/*.log
2) Adding the missing Volume block of code to the nginx container
Below is the block of code for Nginx contianer

      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: shared-logs # shared space
          mountPath: /var/log/nginx

This fixed the issue and i saw that the logs started showing up correctly.

3

I would recommend a different pattern. There is no reason to couple promtail container with your nginx container. You can have nginx output to json file (stored on each node locally), then mount the /var/lib/containers directory into a DAEMON set of promtail containers (so it runs one count on each host). The benefit is you can now use promtail potentially for other containers as well.

4

Thanks, @tonyswumac for looking into my query.
Yes, you are correct, the best way to use promtail is to use it as a demonset if the application sends the output to stdout and stderr.

5

This is a good idea, but in our case we do not have enough permissions to set the RBAC for the daemon set, can the same tactics be applied with lower permissions? We only want the stdout and stderr to be logged, but it appears that the daemonset wants to be able to access k8s cluster logs too. Or am I missing something? Thanks!

6

The daemon needs the log files from host mounted so it can read, if you don’t have permission to do that then you probably don’t have a lot of options other than either embedding the promtail agent inside your application container, or run it as a sidecar, neither is great in my opinion.

Logging is a function that should be provided by the platform (whoever manages the Kubernetes cluster). Just my opinion :slight_smile: