Network metrics not available with kubernetes node exporter

Grafana Alloy
1

I’ve been running the Grafana Alloy agent on my Ubuntu machine that runs Kubernetes and receive all system metrics including node_network_receive_bytes_total. When I install the helm chart for the grafana agent with this script and stop alloy as a system service, (note the username and password fields are removed), and I use this dashboard (Node Exporter Full | Grafana Labs) certain panels such as the one that uses node_network_receive_bytes_total no longer have data.

helm repo add grafana https://grafana.github.io/helm-charts &&
  helm repo update &&
  helm upgrade --install --atomic --timeout 300s grafana-k8s-monitoring-2 grafana/k8s-monitoring \
    --namespace "grafana" --create-namespace --values - <<EOF
cluster:
  name: k8s-ga-c01
externalServices:
  prometheus:
    host: https://prometheus-prod-13-prod-us-east-0.grafana.net
    basicAuth:
  loki:
    host: https://logs-prod-006.grafana.net
    basicAuth:
  tempo:
    host: https://tempo-prod-04-prod-us-east-0.grafana.net:443
    basicAuth:
metrics:
  enabled: true
  cost:
    enabled: true
  node-exporter:
    enabled: true
logs:
  enabled: true
  pod_logs:
    enabled: true
  cluster_events:
    enabled: true
traces:
  enabled: true
receivers:
  grpc:
    enabled: true
  http:
    enabled: true
  zipkin:
    enabled: true
opencost:
  enabled: true
  opencost:
    exporter:
      defaultClusterId: k8s-ga-c01
    prometheus:
      external:
        url: https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom
kube-state-metrics:
  enabled: true
  service:
    isTLS: false
prometheus-node-exporter:
  enabled: true
prometheus-operator-crds:
  enabled: true
alloy-events: {}
alloy:
  configMap:
    create: false
    name: alloy-config
    key: config.alloy
alloy-logs: {}
EOF
2

Hi, @mwimpelberg ! The Kubernetes Monitoring Helm chart does deploy Node Exporter and scrapes some of its metrics, but not as many as the node exporter integration. Thats to keep the number of active series low, but still usable for Kubernetes monitoring purposes.

You can easily enable the full set of metrics for the dashboards by setting:

metrics:
  node-exporter:
    metricsTuning:
      useIntegrationAllowList: true

For more information, see this example: k8s-monitoring-helm/examples/custom-metrics-tuning at main · grafana/k8s-monitoring-helm · GitHub

Or this page about the allow lists: k8s-monitoring-helm/charts/k8s-monitoring/default_allow_lists/README.md at main · grafana/k8s-monitoring-helm · GitHub