Grafana agent cgroup problem on CentOS7 (Docker integration)

Hello!

I am following this new Grafana blog article to monitor my docker instance on a Debian and a CentOS7. On Debian I don’t have any issue when I’m trying to launch Grafana Agent in a container. On the contrary, I have a cgroup problem when I try on CentOS7. I get this :

ts=2022-08-24T18:32:26.528195902Z caller=klog.go:108 level=warn integration=cadvisor func=Warningf msg="Error getting perf_event cgroup path: \"could not find path for resource \\\"perf_event\\\" for container \\\"/\\\"\""

I have tried a lot of configuration and the latest one look like this:

docker run \
  -v /tmp/agent:/etc/agent/data \
  -v /home/grafanaagent/agent/agent.yaml:/etc/agent/agent.yaml \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --rm \
  --expose 12345 \
  --privileged=true \
  --volume=/cgroup:/cgroup:ro \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  grafana/agent:v0.26.1

Does anyone have any idea how I can solve this problem?

Thank you in advance,
Cybo

Hi @cybo12!

Would it be possible for you to share your agent.yaml config file?

Also the log you are sharing is just a warning log.

Does the agent fail to run in the container? Or you do not see the required metrics?

Hello,

Here is my agent.yaml (it’s the one generated by the instalation). The container is still running (ingesting logs and revoking old logs) but I don’t see my metrics.

integrations:
  agent:
    enabled: true
    relabel_configs:
    - action: replace
      source_labels:
      - agent_hostname
      target_label: instance
  cadvisor:
    docker_only: true
    enabled: true
    instance: localhost:9090
    relabel_configs:
    - action: replace
      replacement: integrations/docker
      target_label: job
  prometheus_remote_write:
  - basic_auth:
      password: 
      username: 
    url: https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push
logs:
  configs:
  - clients:
    - basic_auth:
        password: 
        username: 
      url: https://logs-prod3.grafana.net/loki/api/v1/push
    name: integrations
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
    - docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
      job_name i:ntegrations/docker
      relabel_configs:
      - action: replace
        replacement: integrations/docker
        source_labels:
        - __meta_docker_container_id
        target_label: job
      - action: replace
        replacement: localhost:9090
        source_labels:
        - __meta_docker_container_id
        target_label: instance
      - regex: /(.*)
        source_labels:
        - __meta_docker_container_name
        target_label: container
      - source_labels:
        - __meta_docker_container_log_stream
        target_label: stream
    - job_name: integrations/agent
      journal:
        labels:
          instance: <hostname>
          job: integrations/agent
        max_age: 24h
      pipeline_stages:
      - match:
          action: drop
          drop_counter_reason: only_keep_grafana_agent_logs
          selector: '{unit!="grafana-agent.service"}'
      - regex:
          expression: (level=(?P<log_level>[\s]*debug|warn|info|error))
      - labels:
          level: log_level
      relabel_configs:
      - source_labels:
        - __journal__systemd_unit
        target_label: unit
    - job_name: integrations/agent
      pipeline_stages:
      - json:
          expressions:
            message: message
      - regex:
          expression: (level=(?P<log_level>[\s]*debug|warn|info|error))
      - labels:
          level: log_level
      - output:
          source: message
      relabel_configs:
      - source_labels:
        - computer
        target_label: instance
      - replacement: grafana-agent.service
        target_label: unit
      windows_events:
        bookmark_path: ./bookmark-application.xml
        eventlog_name: Application
        labels:
          job: integrations/agent
        use_incoming_timestamp: true
        xpath_query: '*[System[Provider[@Name=''Grafana Agent'']]]'
    target_config:
      sync_period: 10s
metrics:
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: 
        username: 
      url: https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push
  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal

Unrelated to metrics but in the logs config - line 36 - I do see a small typo:
job_name i:ntegrations/docker (colon not placed correctly)

Have you configured the agent to run as root user?
As per documentation here:

Privileged access is required by the agent to access docker daemon on the machine.