How can I scrape content from dmesg

Grafana Loki
1

I wanted to send the kernel logs to Loki with Promtail, but haven’t figured out how to do it. Here is my attempt:

scrape_configs:
- job_name: kernel
  static_configs:
    - labels:
        job: kernel
        __path__: /dev/kmsg

But Promtail is giving out an error:

level=info ts=2024-02-25T18:11:00.07763986Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=a809dbf068b9203bbe47c13d4223722a
level=info ts=2024-02-25T18:11:00.078824545Z caller=main.go:174 msg="Starting Promtail" version="(version=2.9.4, branch=HEAD, revision=f599ebc535)"
level=warn ts=2024-02-25T18:11:00.078929408Z caller=promtail.go:260 msg="disable watchConfig" reason="promtailServer cast fail"
level=info ts=2024-02-25T18:11:05.079104171Z caller=filetargetmanager.go:361 msg="Adding target" key="/dev/kmsg:{job=\"kernel\"}"
level=info ts=2024-02-25T18:11:05.079211544Z caller=filetarget.go:313 msg="watching new directory" directory=/dev
level=info ts=2024-02-25T18:11:05.079359504Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/dev/kmsg
ts=2024-02-25T18:11:05.079389133Z caller=log.go:168 level=info msg="Seeked /dev/kmsg - &{Offset:0 Whence:0}"
level=info ts=2024-02-25T18:11:05.079431812Z caller=tailer.go:163 component=tailer msg="tail routine: tail channel closed, stopping tailer" path=/dev/kmsg reason="seek /dev/kmsg: invalid argument"
level=info ts=2024-02-25T18:11:05.079476836Z caller=tailer.go:154 component=tailer msg="tail routine: exited" path=/dev/kmsg
level=info ts=2024-02-25T18:11:05.079489187Z caller=tailer.go:118 component=tailer msg="position timer: exited" path=/dev/kmsg
level=info ts=2024-02-25T18:11:15.080093129Z caller=tailer.go:206 component=tailer msg="skipping update of position for a file which does not currently exist" path=/dev/kmsg
level=info ts=2024-02-25T18:11:15.080286179Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/dev/kmsg
ts=2024-02-25T18:11:15.080342544Z caller=log.go:168 level=info msg="Seeked /dev/kmsg - &{Offset:0 Whence:0}"
level=info ts=2024-02-25T18:11:15.080399575Z caller=tailer.go:163 component=tailer msg="tail routine: tail channel closed, stopping tailer" path=/dev/kmsg reason="seek /dev/kmsg: invalid argument"
level=info ts=2024-02-25T18:11:15.080418681Z caller=tailer.go:154 component=tailer msg="tail routine: exited" path=/dev/kmsg
level=info ts=2024-02-25T18:11:15.080430076Z caller=tailer.go:118 component=tailer msg="position timer: exited" path=/dev/kmsg

Seems like the device file cannot be accessed by simply passing it as a static file. Is there any workaround for this? Thanks

2

I’ve never tried this before, so I could be wrong, but couple of things to try:

First make sure promtail is running as root. If that still doesn’t work, then it’s probably because /dev/kmsg being a device rather than a normail file that can be tailed. If that’s the case you can try one of two things:

  1. Kernel logs go through syslog (if i remember correctly). So you could change your syslog configuration and send kernel logs to a file too. I forgot the flag, but you should be able to find it in documentation.

  2. Rung dmesg periodically and output to a file then pick it up from promtail. This is probably less ideal.

3

Hi, sorry for the late reply. Debian 12 no longer have syslog and all the logs are going to the journald. I experienced a lot of other problems after this one so I decided to ditch Loki and installed GrayLog, which comes with FileBeat that can easily collect all logs from journald.