Different data from http_sd vs file_sd when using rate

Hello everyone.

I’m having a bad time trying to figure what’s going on.

I have two instances of Grafana/Prometheus (one in prod, one in dev).

The prod instance is using file_sd_configs to get the targets and data and the dev instance uses http_sd_configs. Same targets, same returned metrics.

When I try to query using “rate” I seem to have different results; if I simply use the counter metric, both instances return the same values.

Query: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=“idle”}[2m])) * 100) > 70

Prod:

Dev:

The only real difference between instances is the way of gathering the targets, with other queries there are no issues, only when using rate.

Thanks in advance.