A lot of node exporter metrics suddenly missing after upgrade to Grafana Alloy

Grafana Prometheus
,
1

Hello everybody,
I just migrated my monitoring setup from Grafana Agent to Grafana Alloy.
I used the converter built into Alloy to convert the configuration file. That worked so far, although I made some changes to consolidate some of the blocks.

While logs properly end up in Loki, there are suddenly a lot of metrics missing which should be sent by the node exporter. node_cpu_seconds_total is quite critical and used in some of my alerting rules.

Bildschirmfoto 2025-06-21 um 14.28.20
Bildschirmfoto 2025-06-21 um 14.28.201702×530 65.4 KB

It seems that with Alloy, Grafana now uses a fork of the node_exporter. However, there is zero information if that changed anything. Also, there is zero mention in the Alloy upgrade guide if anything related to the node exporter changed.
Why is this happening? And how do I get the metrics back?

Grafana Agent: agent, version v0.44.3 (branch: HEAD, revision: 3657b1d0a)
Grafana Alloy: alloy, version v1.9.1 (branch: HEAD, revision: d3d7931)

Configuration is straight forward with no spcific changes to the exporter itself:

prometheus.exporter.unix "agent" { }

prometheus.scrape "agent" {
  targets    = prometheus.exporter.unix.agent.targets
  forward_to = [prometheus.relabel.mimir.receiver]
}

prometheus.remote_write "mimir" {
	endpoint {
		name = "mimir"
		url  = "http://mimir:9009/api/v1/push"
	}
}

prometheus.relabel "mimir" {
    forward_to = [prometheus.remote_write.mimir.receiver]
    rule {
        action = "labeldrop"
        regex  = "instance"
    }
}
2

So, it seems that the metrics are there. I noticed that the integration label was changed. A bit unfortunate, that such a change wasn’t cought by the Alloy converter and not made public anywhere else.
Also, the Drilldown feature of Grafana seems to be either broken or if defaults to wrong settings. Because when searching for node_cpu_seconds_total in Explore, I indeed find the metrics for my hosts. But in Drilldown, all charts er empty, no matter what settings and filters I select.

TLDR: Use Explore mode in Grafana and double-check the labels used in dashboards.

1 Like