Group_left() generates incorrect values?

Grafana Prometheus
1

I have two timeline panels with the same source data but on one I have joined two metrics and that results in the values graphed being 10000 time higher? heres the two graphs:

image
image2831×546 93.9 KB

Top one is just a single metric query : NutanixVms_stats_hypervisor_cpu_usage_ppm{hostUuid=“0166200e-9b72-49a2-96d8-0f45ad5ed5c2”}/10000

this panel is displaying the correct CPU values for the three vms

the lower panel i have tried to pull the hostname in from another set of metrics using this query:

(NutanixVms_stats_hypervisor_cpu_usage_ppm{hostUuid=“0166200e-9b72-49a2-96d8-0f45ad5ed5c2”} * on(hostUuid) group_left(host) (label_replace(NutanixHosts_stats_hypervisor_cpu_usage_ppm, “hostUuid”, “$1”, “uuid”, “(.*)”)))/10000

but this messes up all the values for some reason ? is there another way to pull the hostname into these metrics without messing up the values ?

Thanks

2

Hi, by using * on()... you’re actually executing multiplication - that’s why you have the values messed up. You could do

(NutanixVms_stats_hypervisor_cpu_usage_ppm{hostUuid=“0166200e-9b72-49a2-96d8-0f45ad5ed5c2”} + (on(hostUuid) group_left(host) (label_replace(NutanixHosts_stats_hypervisor_cpu_usage_ppm, “hostUuid”, “$1”, “uuid”, “(.*)”)) * 0))/10000

(or so with parentheses) so that the right-side value does not give any values (addition with 0 will be the same).

1 Like
3

Thanks, works a treat.

4

Just a follow on, when a VM moves between Hosts the graph still displays the last value making it look like the CPU for example is running constantly at 25% :

image
image1552×696 62.3 KB

these two graphs show one VM moving between two hosts

I am guessing its related to the group_left function somehow?

Thanks

5

I’m not quite sure - have you tried to remove the whole +... expression? I would guess that it’s more of how the metrics are reported? Maybe even after the move, the VM is still reported as constant in the element of the infrastructure that reports metrics? It would be best to see at the reporter (source) if the series for the VM are still there.

6

Bingo, the problem is with the collector as it only builds the indexes on startup, if i restart the collector its all fixed. So all good with the querys.

Thanks

1 Like