Calculation on Multidimensional Queries for an Alert do not work in Math Expression

[dhillonhs06]

I am having a similar issue with Prometheus Data Source

I am trying to set an alert on the memory usage of the Kubernetes pod if it breaches 80 percent.

I have 2 queries, one returning the data of the current usage and the other query returning the resource limit, see below
Query A is getting data from Prometheus cAdvisor:

Query B is getting data from Prometheus kube-state-metrics

I have reduced the query results using reduce functions to get the last values, see below

My desired result is to be able to calculate the usage percentage from Expressions C and D, and then use a threshold Expression to check if it breached 80 percent, but that doesn’t work in the case of multidimensional Queries, it results in no data

Options Explored

Option 1:
Calculating 80 percent value from Expression D using math Expression in Panel E and then using Threshold Expression on Expression C to check if it breached the value attained from Expression E.

Drawback/Blocker:
Unable to dynamically use the result from Expression E in Threshold Expression, as I want this calculation to be on the fly, instead of manually entering the value.
The reason I want this to be done dynamically is that I am using Terraform provider to provision my alerts as per my team’s requirement and use-case.

Option 2:
Using a Math Expression to check if values from Expression C are Greater than values from Expression E, See below:

Drawback/Blocker:
Again this results in No data as the math expression is not working with multidimensional queries

Environment:

Grafana Version: 9.3.6
Data source type & version: Prometheus
User OS & Browser: MacOS/Chrome

[dhillonhs06]

Image for Option 2:

[jangaraj]

I guess those “last” timeseries results have different timestamps, maybe also a labels. That can be a problem for math expression.

[dhillonhs06]

@jangaraj
I thought so too but when I change the container in the queries which has only one pod running in Kubernetes CLuster, then the math expression works like a charm. See below:

Also any idea, as to how I restraint the labels to be populated from Query A and Query B into Expression C and D, maybe then the math expression will work, but need to check it out.

[dhillonhs06]

@jangaraj,
Thanks for your suggestion, it worked.
I added sum by(pod,cluster_name,namespace,cluster_id) in the queries, to get the common labels propagated to the dependent expressions and then it worked like a charm.

This solution is working for containers having multiple pods and containers having single pods and in the alert where the container is having multiple pods, the math calculation is done by the pods respectively.

[jangaraj]

I would say you can do more calculations directly in the Prometheus. Example:

100 
* 
max(   
    max( 
        container_memory_working_set_bytes{
            image!="",
            ...
        }
    ) by (pod) 
    /
    max(
        kube_pod_container_resource_limits{
            ...
            resource="memory"
        }
    ) by (pod)
) by (pod)

For the inspiration - I used table with thresholds for percentage values for my pod mem utilizations: