Grafana via Promtail is showing intitial log content but then is not updating when new recs are added

Hi. I am brand new to Grafana and Promtail / Loki. I am testing with a Quarkus Getting Started app with logging enabled. I have Promtail:latest running in a container.

My log file is at ./var/log/app.log and whatever is in it when the Promtail container starts shows in the Grafana Loki explorer. All good.

However, if I do something in my Quarkus app that adds records to the myapp.log file, those new records do not appear when I click the Run Query button in the Loki explorer. If I restart the Promtail container, and click the Run Query button, those new records appear.

My config.yaml is in ./promtail/config.yaml

My docker compose file in my Quarkus project:

version: '3'
services:
  promtail:
    image: grafana/promtail:main
    volumes:
      - ./promtail:/etc/promtail
      - ./var/log:/var/log
      - ./promtail/config.yaml:/etc/promtail/config.yaml
    command: -config.file=/etc/promtail/config.yaml

The .promtail/config.yaml (API key truncated here)

server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /etc/promtail/positions.yaml

client:
  url: https://765067:glc_eyJvIjoiMTAxM...c3QtMSJ9fQ==@logs-prod-026.grafana.net/api/prom/push
  basic_auth:
      username: 'any-string'
      password: 'glc_eyJvIjoiMTAxM...c3QtMSJ9fQ=='
scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/myapp.log

The Promtail container log has this is the final few lines, and no errors:

getting-started-promtail-1  | level=info ts=2023-12-14T05:39:51.444173471Z caller=filetargetmanager.go:361 msg="Adding target" key="/var/log/myapp.log:{job=\"varlogs\"}"
getting-started-promtail-1  | level=info ts=2023-12-14T05:39:51.453864013Z caller=filetarget.go:313 msg="watching new directory" directory=/var/log
getting-started-promtail-1  | level=info ts=2023-12-14T05:39:51.463944805Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/var/log/myapp.log
getting-started-promtail-1  | ts=2023-12-14T05:39:51.464024721Z caller=log.go:168 level=info msg="Seeked /var/log/myapp.log - &{Offset:5764 Whence:0}"

I notice the Promtail container logs are in UTC and my Quarkus app and Grafana are in UTC+11 but that shouldn’t affect anything as far as I can see. After all, the initial data appears with a query time span of Last 5 minutes. Just nothing after that.

What have I done wrong in my setup, do you think?
Thanks,
Murray

First, you might want to remove the credentials from your post, and cycle the keys.

Second, sounds like you are mounting /var/log to your promtail container. When the log file changes, do you see the change reflected in the promtail container as well? Also, it might be worth a try setting your promtail container to the same time zone, just in case.

Thanks!

  1. Re credentials. Yes, indeed. I intend to throw all this away once I have my “hello world” test working with Grafana.
  2. Thanks for the tip re checking the docker bind mount. This is indeed the issue. If I manually save a log entry in the host log file, it immediately shows in Grafana. But if I use Quarkus logging to write to the host log file, it is in the file but doesn’t trigger Docker to notice! If I manually save the log file via text editor, all the new log entries turn up in Grafana.

So, I am experimenting with using a volume instead of a bind mount and/or other Quarkus options. I will report back once I have solved it.

Thanks for your help,
Much appreciated,
Murray