Error using Promtail-sidecar for nginx deployemnt

Grafana Loki
, ,
1

Hi All
I wanted to use promtail as a sidecar with nginx container.
I have created a deployment that does not seem to work.
Can anyone please help me correct the Yaml script?

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-sidecar-config-map
data:
  promtail.yaml: |
      server:
        http_listen_port: 9080
        grpc_listen_port: 0
        log_level: "debug"
      positions:
        filename: /tmp/positions.yaml
      clients: # Specify target
        - url: http://loki:3100/loki/api/v1/push
      scrape_configs:
        - job_name:  "nginx" 
          static_configs: 
            - targets: 
                - localhost 
              labels:
                app: "storage-service" 
                __path__: var/log/nginx/*.log # Any file .log in the EmptyDir Volume.
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-promtail
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-promtail
  template:
    metadata:
      labels:
        app: nginx-promtail
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
      - name: promtail
        image: grafana/promtail:latest
        args:
        - "--config.file=/etc/promtail/promtail.yaml" 
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: config
          mountPath: /etc/promtail
        - name: shared-logs # shared space
          mountPath: /app/logs
      volumes:
      - name: config
        configMap:
          name: promtail-sidecar-config-map
      - name: shared-logs # shared space monitored with Promtail
        emptyDir:
          sizeLimit: 500Mi

The Promtail-sidecar container starts but it is not able to push the logs to loki

PS C:\Windows\system32> kubectl logs pod/nginx-promtail-566f697b85-mrdt6 -c promtail
level=debug ts=2023-01-27T09:51:13.881537424Z caller=promtail.go:115 msg="Reloading configuration file"
level=info ts=2023-01-27T09:51:13.882148715Z caller=promtail.go:123 msg="Reloading configuration file" md5sum=963a279a498a62c329093fd2a20e1e1a
level=debug ts=2023-01-27T09:51:13.882388112Z caller=manager.go:289 component=discovery msg="Starting provider" provider=static/0 subs=map[nginx:{}]
level=info ts=2023-01-27T09:51:13.882876204Z caller=server.go:323 http=[::]:9080 grpc=[::]:35413 msg="server listening on addresses"
level=info ts=2023-01-27T09:51:13.883006603Z caller=main.go:171 msg="Starting Promtail" version="(version=2.7.2, branch=HEAD, revision=c35554d09)"
level=debug ts=2023-01-27T09:51:13.883082901Z caller=manager.go:323 component=discovery msg="Discoverer channel closed" provider=static/0
level=warn ts=2023-01-27T09:51:13.883090901Z caller=promtail.go:220 msg="enable watchConfig"
level=debug ts=2023-01-27T09:51:18.883511587Z caller=filetargetmanager.go:292 msg="new target" labels="{__address__=\"localhost\"}"
level=info ts=2023-01-27T09:51:18.883616885Z caller=filetargetmanager.go:352 msg="Adding target" key="var/log/nginx/*.log:{app=\"storage-service\"}"
level=debug ts=2023-01-27T09:51:18.883738283Z caller=filetarget.go:229 msg="no files matched requested path, nothing will be tailed" path=var/log/nginx/*.log pathExclude=
PS C:\Windows\system32> notepad .\config.yaml

Loki and Grafana is running and In grafana dashboard i can see that there are no logs as promtail is not able to push any logs to loki.
Please help me in configuring the manifest to resolve the issue

2

I Found out that were few mistakes in the config and the deployment yaml files that i made which i am listing below:

  1. changing the path to an correct absolute path
    __path__: var/log/nginx/*.log

correct : __path__: /var/log/nginx/*.log
2) Adding the missing Volume block of code to the nginx container
Below is the block of code for Nginx contianer

      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: shared-logs # shared space
          mountPath: /var/log/nginx

This fixed the issue and i saw that the logs started showing up correctly.

3

I would recommend a different pattern. There is no reason to couple promtail container with your nginx container. You can have nginx output to json file (stored on each node locally), then mount the /var/lib/containers directory into a DAEMON set of promtail containers (so it runs one count on each host). The benefit is you can now use promtail potentially for other containers as well.

4

Thanks, @tonyswumac for looking into my query.
Yes, you are correct, the best way to use promtail is to use it as a demonset if the application sends the output to stdout and stderr.