Grafana via Promtail is showing intitial log content but then is not updating when new recs are added

Grafana Loki
1

Hi. I am brand new to Grafana and Promtail / Loki. I am testing with a Quarkus Getting Started app with logging enabled. I have Promtail:latest running in a container.

My log file is at ./var/log/app.log and whatever is in it when the Promtail container starts shows in the Grafana Loki explorer. All good.

However, if I do something in my Quarkus app that adds records to the myapp.log file, those new records do not appear when I click the Run Query button in the Loki explorer. If I restart the Promtail container, and click the Run Query button, those new records appear.

My config.yaml is in ./promtail/config.yaml

My docker compose file in my Quarkus project:

version: '3'
services:
  promtail:
    image: grafana/promtail:main
    volumes:
      - ./promtail:/etc/promtail
      - ./var/log:/var/log
      - ./promtail/config.yaml:/etc/promtail/config.yaml
    command: -config.file=/etc/promtail/config.yaml

The .promtail/config.yaml (API key truncated here)

server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /etc/promtail/positions.yaml

client:
  url: https://765067:glc_eyJvIjoiMTAxM...c3QtMSJ9fQ==@logs-prod-026.grafana.net/api/prom/push
  basic_auth:
      username: 'any-string'
      password: 'glc_eyJvIjoiMTAxM...c3QtMSJ9fQ=='
scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/myapp.log

The Promtail container log has this is the final few lines, and no errors:

getting-started-promtail-1  | level=info ts=2023-12-14T05:39:51.444173471Z caller=filetargetmanager.go:361 msg="Adding target" key="/var/log/myapp.log:{job=\"varlogs\"}"
getting-started-promtail-1  | level=info ts=2023-12-14T05:39:51.453864013Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log
getting-started-promtail-1  | level=info ts=2023-12-14T05:39:51.463944805Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/myapp.log
getting-started-promtail-1  | ts=2023-12-14T05:39:51.464024721Z caller=log.go:168 level=info msg="Seeked /var/log/myapp.log - &{Offset:5764 Whence:0}"

I notice the Promtail container logs are in UTC and my Quarkus app and Grafana are in UTC+11 but that shouldn’t affect anything as far as I can see. After all, the initial data appears with a query time span of Last 5 minutes. Just nothing after that.

What have I done wrong in my setup, do you think?
Thanks,
Murray

2

First, you might want to remove the credentials from your post, and cycle the keys.

Second, sounds like you are mounting /var/log to your promtail container. When the log file changes, do you see the change reflected in the promtail container as well? Also, it might be worth a try setting your promtail container to the same time zone, just in case.

3

Thanks!

  1. Re credentials. Yes, indeed. I intend to throw all this away once I have my “hello world” test working with Grafana.
  2. Thanks for the tip re checking the docker bind mount. This is indeed the issue. If I manually save a log entry in the host log file, it immediately shows in Grafana. But if I use Quarkus logging to write to the host log file, it is in the file but doesn’t trigger Docker to notice! If I manually save the log file via text editor, all the new log entries turn up in Grafana.

So, I am experimenting with using a volume instead of a bind mount and/or other Quarkus options. I will report back once I have solved it.

Thanks for your help,
Much appreciated,
Murray

4

Hello, I am running exactly in the same issue.

when I start the promtail image with:

docker-compose up promtail

I see the logs are being detected:

promtail_1  | level=info ts=2024-05-14T19:39:32.980862331Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/rails/development.log

And they are digested by Loki and presented in Grafana.

But new logs are not being detected.

If I bash into the promtail image I see the /var/log/rails/*.log files are frozen and they are not being updated with my locals /railsapp/logs/*.log.

This looks to be a limitation with Docker containers: Docker: Files from volume not updated in target - Stack Overflow

Did you manage to make this work?

This is the promtail section of my docker-compose.yml:

  promtail:
    image: grafana/promtail:3.0.0
    volumes:
      - ./docker/promtail/config.yml:/etc/promtail/config.yml
      - ./log:/var/log/rails
    command: -config.file=/etc/promtail/config.yml