Use loki docker plugin to label python logger

My goal: Label the python logger from docker containers, and also keep the docker container label.

What I did and what works for now:
I have a working Loki instance and installed Loki as a docker plugin on the local environment.
Here is what I’ve added to my daemon.json:

  "debug" : true,
  "log-driver": "loki",
  "log-opts": {
     "loki-url": "http://loki.local/loki/api/v1/push",
     "loki-batch-size": "400"

I can see logs in Loki :grinning:, and they look like that:

[INFO][2023-08-15T16:23:22.856312+03:00][MainThread]: RB operation has been completed. The next operation is schedule to start in 3600 seconds. (
[INFO][2023-08-15T16:23:22.856201+03:00][MainThread]: No jobs were found in db (
[INFO][2023-08-15T16:23:22.854520+03:00][MainThread]: Wake-up and run RB operation... (

each log have container_name, container_id, stack_name (those are enough for me).

What missing: I want to parse the severity of the log level ( INFO,DEBUG,WARNING,ERROR,FATAL ), Thread (we can see MainThread on this log, but can be something else), and the python file (in this example it’s

What I’ve tried: Found this article: Configuration | Grafana Loki documentation, but not sure how to work with loki-pipeline-stages on JSON, and what regex is good for me.
What is the best practice here? what is the regex that can fit here?

The regexp is probably your best bet. Something like:

| regexp `^\[(?P<log_level>\S[^\[]+)\]\[(?:\S[^\[]+)\]\[(?P<thread>\S[^\[]+)\].+\((?P<file_name>\S+\.\S+):`

You can also try it here: LogQL Analyzer | Grafana Loki documentation

This answer is accepted even if the regex won’t work. because I wasn’t familiar with the simulator page and it’s very helpful.
Hope someone can help in the future to know to use daemon.json properly.

The regex is working well :nerd_face:
For anyone who’s going to face that, I’ve added config.yaml inside /var/lib/docker/plugins/0dc182b0a98f3689ea8d5c74d9ce568e698a342d3ca6aecf6411b18c7d6a8e14/rootfs/home/config.yaml (you might need replace 0dc182b0a98f3689ea8d5c74d9ce568e698a342d3ca6aecf6411b18c7d6a8e14 with your plugin ID)
and it’s looking like that now:

- regex:
     expression: '^\[(?P<log_level>\S[^\[]+)\]\[(?:\S[^\[]+)\]\[(?P<thread>\S[^\[]+)\].+\((?P<file_name>\S+\.\S+):'
- labels:

and added loki-pipeline-stage-file section to my docker daemon (/etc/docker/daemon.json):

"debug" : true,
  "log-driver": "loki",
  "log-opts": {
     "loki-url": "http://my-loki.local/loki/api/v1/push",
     "loki-batch-size": "400",

Disable & Enable your plugin to make sure it’s working.