Labels seem to get lost with cadvisor export and prometheus scraping

Grafana Alloy
, , ,
1

Dear community,

I currently have the problem that labels for e.g. per pod CPU usage metrics in Prometheus seem to get lost. The metric itsself is there, but additonal labels seem to be lost.

My river.config for the components looks as follows. I am only allowing one metric to pass through at the moment, so that I can gradually add additional metrics

prometheus.exporter.cadvisor "default" {
        store_container_labels = true
        enabled_metrics = ["cpu", "cpuLoad", "percpu", "memory", "memory_numa", "referenced_memory", "cpu_topology", "cpuset"]
        storage_duration = "3m"
      }

      prometheus.scrape "cadvisor" {
        targets    = prometheus.exporter.cadvisor.default.targets
        forward_to = [prometheus.relabel.add_cluster_label.receiver]
      }

   prometheus.relabel "add_cluster_label" {
        forward_to = [prometheus.remote_write.grafana_cloud.receiver]
        
         rule {
          action        = "keep"
          source_labels = ["__name__"]
          regex         = "container_cpu_usage_seconds_total"
        }
        
        rule {
          replacement     = "test"
          target_label    = "cluster"
        }
      }
      
      prometheus.remote_write "grafana_cloud" {
        endpoint {
          url = "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"
      
          basic_auth {
            username = secret
            password = secret
          }
        }
      }

However, when checking the UI for grafana agent, the arguments look a little different:

image
image1646Ă—813 66.9 KB

And the result in grafana looks like this:

image
image1596Ă—725 64.5 KB

From other resources I saw that the cadvisor metric container_cpu_usage_seconds_total can have additonal labels for containers or pods. But those seem to be missing and I am only getting information per worker node of the cluster. For grafana agent I thought that the config store_container_labels would add those labels.

Am I missing something, or is my interpretation of the metric wrong? Is there an alternative metric I could use?

1 Like
2

Hello,

I have exactly the same behavior and I was wondering if you found any solution. I find it quite weird that my only series are from the worker nodes and nothing from what I usually get with the kube-prom-stack on pods with several labels (images, namespace, etc)

3

We are having the same issue - any solutions?

4

I haven’t found one yet, sorry.

5

Ouff :confused: thanks for the answer though :slight_smile:

6

Hello,
I also spend quite a time with this issue. In my case, I missed to add some folder mounts to my Alloy Docker container, that were needed by cAdvisor in order to get information about container metadata and labels.

These mounts are documented in the cAdvisor documentation.

In the end, the following compose configuration for my Alloy container worked and labels were shown:

services:
  alloy:
    image: grafana/alloy:v1.5.1
    hostname: monitor-alloy
    restart: unless-stopped
    command: run /etc/alloy/config
    volumes:
      - "/:/rootfs:ro"
      - "/var/run:/var/run:ro"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
      - /var/run/docker.sock:/var/run/docker.sock
      - ./files/config.alloy:/etc/alloy/config/config.alloy

Alloy config snippet:

prometheus.exporter.cadvisor "local_system" {
    store_container_labels = true
    docker_only = true
    enabled_metrics = ["cpu", "memory", "network"]
}
1 Like
7

I am still experiencing this and I already tried all these volume mounts and flags.

This is my config in config.alloy:

prometheus.exporter.cadvisor "docker" {
  docker_host = "unix:///var/run/docker.sock"
  storage_duration = "5m"
}

// Configure a prometheus.scrape component to collect cadvisor metrics.
prometheus.scrape "scraper" {
  targets    = prometheus.exporter.cadvisor.docker.targets
  forward_to = [ prometheus.remote_write.metrics_service.receiver ]
  scrape_interval = "10s"
}

prometheus.remote_write "metrics_service" {
  endpoint {
      url = sys.env("PROMETHEUS_URL")+ "/api/v1/write"
  }
}

and my docker compose:

  alloy:
    privileged: true

    image: grafana/alloy:${ALLOY_IMAGE_TAG:-v1.11.0}

    ports:
      - "${ALLOY_EXPOSED_PORT:-12345}:${ALLOY_INTERNAL_PORT:-12345}"

    networks:
      - observability_network

    hostname: ${HOSTNAME} # required to show correct host instead of the one generated by docker

    pid: host 

    cap_add:
      - SYS_TIME
      - SYS_PTRACE

    command:
      - run
      - --server.http.listen-addr=0.0.0.0:12345
      - --storage.path=/var/lib/alloy/data
      - --server.http.ui-path-prefix=/alloy
      - --stability.level=experimental
      - /etc/alloy/config.alloy

    volumes:
      - alloy_data:/var/lib/alloy/data
      - ${OBSERVABILITY_CONFIG_DIR}/alloy:/etc/alloy # config.alloy is stored here
      - /var/run/docker.sock:/var/run/docker.sock # allow access to docker socket
      - /var/run:/var/run:ro
      - /:/rootfs:ro
      - /sys:/sys:ro
      - /dev/disk/:/dev/disk:ro

    environment:
      PROMETHEUS_URL: ${PROMETHEUS_URL}

In Grafana explore Prometheus I only see these labels being available:

image
image772Ă—431 31.6 KB

What I notice quite a lot in the logs of alloy is this:

ts=2025-10-02T21:01:07.137285312Z level=error msg=“Failed to create existing container: /docker/a65bd67e236649d7a3d3e0fba3453184790a00a6b361d1f9f6edb8c1945162e2: failed to identify the read-write layer ID for container “a65bd67e236649d7a3d3e0fba3453184790a00a6b361d1f9f6edb8c1945162e2”. - open /rootfs/var/lib/docker/image/overlayfs/layerdb/mounts/a65bd67e236649d7a3d3e0fba3453184790a00a6b361d1f9f6edb8c1945162e2/mount-id: no such file or directory” component_path=/ component_id=prometheus.exporter.cadvisor.docker

8

This is probably the reason. Try adding a couple of more mounts from host. I don’t remember exactly what those are, but I’d start with these:

/:/rootfs:ro
/var/run:/var/run:ro
/sys:/sys:ro
/var/lib/docker/:/var/lib/docker:ro
/dev/disk/:/dev/disk:ro

I am sure the cAdvisor documentation would have a more precise list.