Prometheus.exporter.cadvisor for all container

Grafana Alloy
1

Hi all,

I’m trying to set up prometheus.exporter.cadvisor to monitor all containers. However, I only get metrics for the alloy container itself.

I’m running alloy inside docker container, mounted /var/run/docker.sock inside container

config file:

prometheus.exporter.cadvisor "example" {
  docker_host = "unix:///var/run/docker.sock"

  storage_duration = "5m"
}

// Configure a prometheus.scrape component to collect cadvisor metrics.
prometheus.scrape "scraper" {
  targets    = prometheus.exporter.cadvisor.example.targets
  forward_to = [ prometheus.remote_write.demo.receiver ]
}

prometheus.remote_write "demo" {
  endpoint {
    url = PROMETHEUS_REMOTE_WRITE_URL

    basic_auth {
      username = USERNAME
      password = PASSWORD
    }
  }
}

image
image1874×860 62.2 KB

image
image1123×232 6.13 KB

2

I’ve not tried this myself, but couple of things you could try:

  1. You might need to mount /var/lib/docker as RO as well.
  2. Make sure your alloy container is privileged.
3

Hi, I’ve encountered the same problem (I am running Alloy in compose with ‘pid’ and ‘network_mode’ set to ‘host’, without ‘privileged: true’). To get metrics from all containers (via cadvisor component) on the host, I had to add the following volumes:

  • /:/rootfs
  • /var/run:/var/run
  • /sys:/sys
  • /var/lib/docker/:/var/lib/docker
4

Bonjour, :wave:

I have the same problem, but, I not sure of your advice @jakubzukal, or I simply don’t understand, sorry… :worried:

This is a part of my docker compose:

alloy:
    image: grafana/alloy:latest
    container_name: alloy
    restart: "always"
    command:
      - run
      - /etc/alloy/
      - --storage.path=/var/lib/alloy/data
      - --server.http.listen-addr=0.0.0.0:12345
      - --stability.level=experimental
    ports:
      - 12345:12345
      - 9001:9001
      - 9002:9002
      - 4318:4318
      - 4317:4317
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/var/log:/host-var-log"
      - "/proc:/host-proc"
      - "/sys:/host-sys"
      - "/:/host-root"
      - "/run/udev/data:/host-udev"
      - "/data/etc/alloy:/etc/alloy"
      - "/data/var/lib/alloy/data:/var/lib/alloy/data"
    labels:
      - "traefik.enable=false"
    cap_add:
      - SYS_PTRACE
      - SYS_TIME
    privileged: true

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

If I have already mount in my alloy configuration /, sys to be used with prometheus.exporter.unix . How I can use it for cAdvisor?

Thanks!

5

Hi,
you dont have to run cadvisor sideways. Use alloy cadvisor component, if possible.

I am using the following compose (template) with other components (exporters: cadvidor, kafka, unix, loki: remote syslog, journald)

version: '3'

services:
  grafana-alloy:
    container_name: grafana-alloy
    hostname: {{ alloy_hostname }}
    image: grafana/alloy:{{ alloy_docker_tag }}
    ports:
      - "{{ alloy_http_listen_port }}:{{ alloy_http_listen_port }}/tcp"
    volumes:
      - /var/lib/alloy/data
      - type: bind
        source: ./confs
        target: /etc/alloy/confs
        read_only: true
      - type: bind
        source: /
        target: /rootfs
        read_only: true
      - type: bind
        source: /var/run
        target: /var/run
        read_only: true
      - type: bind
        source: /var/log
        target: /var/log
        read_only: true
      - type: bind
        source: /sys
        target: /sys
        read_only: true
      - type: bind
        source: /var/lib/docker/
        target: /var/lib/docker/
        read_only: true
      - type: bind
        source: /run/udev/data
        target: /run/udev/data
        read_only: true
    environment:
      - ...
    pid: host
    network_mode: host
    cap_add:
      - SYS_TIME
      - SYS_PTRACE
    security_opt:
      - apparmor=unconfined
    command: run --server.http.listen-addr=0.0.0.0:{{ alloy_http_listen_port }} --storage.path=/var/lib/alloy/data /etc/alloy/confs
    restart: unless-stopped

and Alloy’s cAdvisor config

prometheus.exporter.cadvisor "docker" {
  docker_host       = "unix:///var/run/docker.sock"
  docker_only       = true
}

prometheus.scrape "cadvisor" {
  targets    = prometheus.exporter.cadvisor.docker.targets
  forward_to = [prometheus.relabel.cadvisor.receiver]
}

prometheus.relabel "cadvisor" {
  forward_to = [prometheus.remote_write.grafana_cloud.receiver]

  rule {
    regex   = "^(container_label_com_docker_compose_.+|id|container_label_maintainer|container_label_org_opencontainers_.+)"
    action  = "labeldrop"
  }

  rule {
    source_labels = ["__name__"]
    regex         = "^(container_cpu_usage_seconds_total|container_fs_inodes_free|container_fs_inodes_total|container_fs_limit_bytes|container_fs_usage_bytes|container_last_seen|container_memory_usage_bytes|container_network_receive_bytes_total|container_network_tcp_usage_total|container_network_transmit_bytes_total|container_spec_memory_reservation_limit_bytes|machine_memory_bytes|machine_scrape_error)$"
    action        = "keep"
  }
}