How Can I match PromQL Query Result with no label in Grafana

Grafana Dashboards
,
1

Hi, I’m new in PromQL & Grafana.

I’m using Grafana v10.3.3 (252761264e) in docker container(base OS is UB20.04)

I want to make visualize dashboard, which contains “average CPU Usage by team in last month”

  • I have lots of VMs, almost 300. each VM has prometheus label named “team”
  • (1) I add each team’s VMs cpu usage(team A, team B, team C, …)
  • (2) I add all VM’s cpu usage(team A+team B+team C+team D+…)
  • (3) so I want (1) / (2) * 100 and show it one visualize dashboard with 1d/M Time Shift Option

PromQL Query that i tested below :
(1) each team’s VMs avg
avg by(team) ((100 - sort(avg by (instance,team (irate(node_cpu_seconds_total{team=“A”,mode=“idle”}[3m])) * 100)))

(2) all VMs avg
sum ((100 - sort(avg by (instance,team) (irate(node_cpu_seconds_total{team!=“”,mode=“idle”}[3m])) * 100)))

(3)
avg((100 - sort(avg by (instance,team) (irate(node_cpu_seconds_total{team=“A”,mode=“idle”}[3m])) * 100))) / (sum ((100 - sort(avg by (instance,team) (irate(node_cpu_seconds_total{team!=“”,mode=“idle”}[3m])) * 100)))) * 100

but, as you know,
(2) has just value, no label… so i can’t calculate (1) / (2) because of label match problem

if i use “avg” instance of “avg by (team)” in front of (1), it’ll calculated (3),
but i can’t match that data in Grafana side.

So… The Question is:

  1. How can i Add label when just use “sum”, not “sum by” in PromQL Query Result.
  2. Is another option in Grafana match with no label?
2

Hi,

Update: I think that creating a scalar from the query without labels would also work but I’m not sure about that (see this link for how to turn instant vector to a scalar).

As far as I understand, you have a problem that the left side of the / operator has label team and the right side doesn’t have a single label? As you’ve correctly pointed out, you have a problem with vector matching - Prometheus does not match any of the series on right and left side, so it drops the vectors. What you can do is to use ignoring operator with group_left operator, making it so Prometheus will ignore team label for calculation time and then will group the result by labels on the left (maybe that’s not exactly what happens but I do understand it in such a way :sweat_smile:).

Sample query (please ignore random metrics and labels, I don’t use prometheus that often :smile:):

image
image1884×798 66.4 KB

It should be somewhat the same problem as you have, just insert the metrics and labels you need.
Here’s the query for a quick copy-paste shenanigans, if you need so:

avg(go_gc_duration_seconds{}) by (quantile) / ignoring(quantile) group_left sum(go_gc_duration_seconds{})

Notice that PromQL docs say that group_.* is a costly operation. I’d also like to point out that VictoriaMetrics (alternative to Prometheus) dropped the distinction of Instant Vector and a Scalar number, so a query like avg(go_gc_duration_seconds{}) by (quantile) / sum(go_gc_duration_seconds{}) would work out of the box, since the right side of the / operator is a number (a little bit of trivia :smile:).

Hope that helps!