Extract Metrics using Loki Docker Driver and Container

[fmoledina]

Loki Docker Driver plug-in: grafana/loki-docker-driver:2.1.0
Loki Docker image: grafana/loki:2.1.0

I’m using Loki to parse JSON formatted log entries from Docker containers with the Loki Docker Driver plugin. My goal is to create labels that I can use in Loki, as well as produce metrics that I can export to Prometheus.

A sample log entry is as follows:

{"L":"INFO","T":"2021-01-30T06:17:18.876Z","N":"piecestore","M":"download started","Piece ID":"6CYNNMUS6DVVCXOE4N7JNFKXETIZVYBX4PSAVKPY3WPN7VKLVUPQ","Satellite ID":"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs","Action":"GET_REPAIR"}

The docker-compose.yml file includes the following pipeline stages for the logging: section:

x-logger: &logger
  logging:
    driver: loki
    options:
      loki-url: "http://172.18.0.250:3100/loki/api/v1/push"  # Static IP for Loki container on user-defined bridge
      loki-external-labels: job=storj
      loki-pipeline-stages: |
        - json:
            expressions:
              loglevel: L
              timestamp: T
              message: M
              satellite: '"Satellite ID"'
              action: Action
              error: error
              errorVerbose: errorVerbose
              available_space: '"Available Space"'
              size: Size
        - labels:
            loglevel:
            satellite:
            action:
            error:
        - metrics:
            download_started:
              type: counter
              description: "Total number of downloads started"
              prefix: storj_pieces_
              max_idle_duration: 24h
              source: message
              config:
                value: "download started"
                action: inc

Since there isn’t a Promtail container in this setup, I would expect that the metrics would populate at the Loki container which is accessible on http://localhost:3100/metrics. However, I see no metrics beginning with the storj_pieces_ prefix as specified above. I can confirm that the JSON parsing and label assignment is successful, having queried the logs using LogQL in Grafana.

Is there a step I have missed in order to get metrics to be available at the Loki container target?

[fmoledina]

I’m learning now that Loki cannot expose metrics directly via the Docker driver and so it first needs to be pushed to Promtail.

Allow docker driver to send logs to Promtail

Configuration

[fmoledina]

I have solved this by using the loki_push_api in Promtail to first send the logs from the Docker driver. I then run all my pipeline stages in Promtail, including exposing the metrics from Prometheus. Then the logs are pushed from Promtail to Loki.

[ewelch]

Really appreciate you posting your follow-up and solution @fmoledina this is what I would have suggested for you as well! Thanks!