Prometheus - How to “join” two metrics and calculate the difference?

Grafana Prometheus
1

How would I join metrics by their similar labels and calculate the difference?

Example I have metric for tracking the inventory from two systems. I have the numbers push to the same metric but use labels to identify where the metric came from.

inventory_quantities{sku="ABC", system="warehouse1"} value = 5
inventory_quantities{sku="ABC", system="warehouse2"} value = 15

I want to join on the SKU and calculate the difference between the two. From there I can set alerts that if the difference is ever greater than a threshold.

I have control how the metrics are published so if my data model is bad, I can change it. I would want it be extensible if I have many warehouses and independently time scraped.

2

Hello @michaelblack! Check out this thread where was similar question discussed. I hope it is going to help!

3

I think I got it working, returning the SKUs where there’s a difference between

((inventory_quantities{system="warehouse2"}) - on(sku) (inventory_quantities{system="warehouse1"})) != 0

What I would like to show on my graph is value that each system has? Is there where I need to use this first query as a subquery?? I’ve very new to PromQL and I haven’t found any good tutorials to practice the syntax. If this was SQL I would write a query like this

SELECT time, 
		sku,
		
		system2.name,
		system2.quantity,
		
		system1.name
		system1.quantity
	FROM system1
		JOIN system2 ON system1.sku = system2.sku
	WHERE (system2.quantity - system1.quantity) != 0
	ORDER BY time

My goal if I were to look at a single SKU, the graph would show 2 lines, one for each system’s value. Then I could observe over time their relative difference.

4

What I would like to show on my graph is value that each system has?

Try the following PromQL query:

inventory_quantities{system=~"warehouse1|warehouse2"}
and on(sku) (
  (inventory_quantities{system="warehouse2"} - on(sku) inventory_quantities{system="warehouse1"}) != 0
)

See more details about vector matching in PromQL and about and operator.