Combine multiple metrics in a single status panel

Grafana Prometheus
#1

Hi,
I would like to combine multiple queries into one single status panel that should color green or red.
My application exists out of a couple micro services and a frontend that goes along. I would like to combine the readiness of all these containers with http request increase of error statuses.
I can for instance do a query in a dashboard variable that returns me the number of unique containers that are in ready state, in my case 12. If that sum goes to 11 or lower, my panel should color red indicating that my app needs attention. Also I want my tile to color red if there is an increase of http errors, but I do not immediately see how I can combine and evaluate 2 constant numbers.
If for instance http errors increase I would like to evaluate something as 12 - 1 (-1 indicating the increase of http status error)

Anyone an idea?
Thanks in advance,
Koen

#2

PromQL supports a bunch of arithmetic and boolean operations which can be combined to create a single query which works sort of like what you’re describing. Since boolean operators in Prometheus returns 1 on true and 0 when false, you can simply subtract the boolean check for increase of errors from the number of containers in ready state.

Your query could end up looking something like this:

sum(containers{status="ready"}) - (sum(increase(http_errors_total[5m])) > 10)

In this example you start by summing up the gauge of the number of containers currently reporting as ready to create your first scalar value/number and then subtracting 0 if your error counter hasn’t increased by at least 10 during the last five minutes and 1 otherwise.