How to filter query results based on labels in another query?

Grafana Dashboards
, ,
1

Hi All!

Please help me with following problem regarding Grafana dashboard development.

  • What Grafana version and what operating system are you using?
    Grafana v9.2.4 on Linux/k8s from Helm repo prometheus-community.github.io.

  • What are you trying to achieve?
    I’m trying to create a panel that displays a stacked time-series graph of CPU usage of multiple pods in Kubernetes, but the pod list should be filtered based on labels of another metric.

  • How are you trying to achieve it?
    I have created a time series panel with two queries:

  • max by (pod) (kube_pod_labels{my_label_a=“$a”, my_label_b=“$b”})
  • sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate) by (pod)

Then I tried to use various transformations to filter out CPU data that don’t have a matching kube_pod_labels metric.

I was actually able to get it working for instant tables but not for time-series.

Also directly in Prometheus following query produces the graph I want:

max by (pod) (kube_pod_labels{my_label_a=“$a”, my_label_b=“$b”}) * sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate) by (pod)

Unfortunatelly, I cannot use the multiplicator in Grafana and I cannot find a transformation that would do the trick.

  • What happened?
    I couldn’t get the data displayed correctly.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No errors.

  • Did you follow any online instructions? If so, what is the URL?
    No.