I am using Grafana to display performance data from a few IBM i systems (formerly known as AS/400).
The data itself comes from Prometheus.
One of the things I display is the current load of active jobs in the system in %.
You can only have so much processes in the system before it shuts down.
I calculate this by using the following formula:
(Current Active Jobs * 100) / Maximum Possible Jobs
The actuall Math expression I use is this:
($CURJOBS * 100) / $MAXJOBS
I had Grafana v10.1.5 and updated to version Grafana v10.2.3.
After updating, I suddenly get the following error message for this calculation:
The thing is that I can’t seem to turn the warning off, nor do I have any idea how to resolve it.
The exclamation-mark-sign gets plastered all over my dashboard, which looks bad.
The calculation works and displays correct data - I have no use for this warning, it’s not even clear what it’s supposed to mean.
Googling the error message yields very few results from topics I made and one other guy on stackoverflow, nothing besides that.
I’m baffled almost noone else has this problem. I’ve rolled back to v10.1.5 and the problem is gone, aside some more super weird side effects like drop-down-menus not loading correctly.
I guess 10.2.3 is just a bad release not much people have yet.
Unions in SSE operations currently drop items when doing a binary operation (e.g. $A + $B) from either A or B when there isn’t a label “match” between them.
This is confusing, so to aid in this, we warn and display the count of items that where dropped, and which items they were.
IMHO you can have that math on the PromQL level, so you don’t need need to use expression.
so in your case, you have completely different labelsets. Why does it matter? When prometheus (and Grafana follows this rule too) tries to do math operations between two vectors, it needs to decide between which vectors it should operate. For example consider your queries return two series per each query:
Now, how Grafana will know which serie should be added to which serie? It will try to perform label matching (meaning it will try to check which series have the same labels) and do operations on those. The series that do not match are dropped, hence the warning (at least I think so) in your case.
Unfortunately, I don’t know if it’s possible to overcome in Grafana editors, but you can certainly overcome it in promql in couple of ways:
Ensure the labelsets are the same - using e.g. sum(<metric>) by (interesting labels) (if you do it right, i.e. the resulting labels will be the same, Grafana will do the addition).
Do the addition in promql with on() operator, e.g. site_P_PV{job=...} + on(label1, label2) current_power_currentpower(job=...}
The on() operator allows you to tell prometheus to do its label matching but to limit the matching to only labels you explicite set in the parentheses. Since your job labels are different, those are what you don’t want in on().
Hope that helps. If you need more assisatance, don’t hesitate to comment, but I think we’ll need more information, e.g. how those series look (labels etc.) and on what labels you’d like to to the addition.
there will be no site_P_PV{job="json_solaredge"}, that metric (its name/label? correct me) is only available for the other job “json_fronius1”.
And vice versa: current_power_currentpower (ugly name, other topic) exists only for the job “json_solaredge”.
Basically I read in two time series containing power values from 2 appliances. And I need to calculate/visualize the added power values. To me that sounds simple, but I see it somehow isn’t
Let me read and understand the rest of your posting first, especially your points 1. and 2. … I will try to apply these tips. Thanks so far!
pt 2. sounds promising
EDIT: As far as I see the value is calculated already, despite the warning … hmmm
and you do site_P_PV + current_power_currentpower you’ll get one resulting series - the one with job="json_solaredge" with value 12. The other ones will be dropped from the result.
After some tests on my end, I see that the warning might not specifically refer to the series themselves, look at the example below:
Here, two of my labels were different (name and type) and two items were dropped. Probably the warning tells that there are missing fields in result series (notice that I also got Series 1 name). That’s when you have only one series in each side of the equation. But when you have more series:
You’ll get no data as the result (which is actually reasonable, because Grafana doesn’t know how to connect the dots).
So, to sum up:
in your case, since you have only one resulting series in both queries, Grafana can omit the vector matching and just adds values. It will still warn you that it’s something you should look into, since labels have been dropped from the result (if I were to guess - your job label was dropped).
if you would add another series, it might stop working, so I would still try to fix the queries. If sum is what you need, all it takes would be to wrap the series in the sum, sum(site_P_PV) + sum(current_power_currentpower) since then you have equal (empty) labelsets.
at first: thank you
second: I understand the direction of all this, but not yet the details. I will read this a few times and try to learn from it. The current value I get for the addition seems to be “correct” even with this warning … So I could leave it like that. Or try my best to do it “the correct way”.
thanks. That sounds even more complicated to me. I will try that, but it sounds somehow redundant or even wrong for my task.
Let me describe it in non-grafana words.
I collect metrics (ok, prometheus words ) from two inverters delivering power from two photovoltaic installations. They both pipe their output into some accumulators. For an overview the customer wants to see a sum of the overall power coming from PV right now.
The first metric is site_P_PV{job="json_fronius1"}, the second is current_power_currentpower{job="json_solaredge"}. The unit of both is kW. So far that sounds simple to me: take value1 and value2 at the same point in time (in this case “now”), add them up, display the result.
You write sum(current_power) + sum(site_P_PV). Isn’t that somehow redundant? sum plus sum, why?
Basically I would expect this to be a simple and plain task in grafana. I see that I might have to learn some more basics, but this can’t be that complicated, right?
And what about subtracting then? For another value I have to subtract one metric from the other …
Understand the Warning: Recognize that the warning about “items dropped from union(s)” is due to mismatched label sets in time series data when performing mathematical operations.
Align Timestamps: Ensure that the timestamps of the series you are trying to operate on are aligned to avoid unnecessary warnings.
Use PromQL for Calculations: Consider performing calculations at the PromQL level to avoid issues with Grafana’s expression handling.
Resample Data: If necessary, use resampling techniques to align data points before performing operations.
Label Matching: Use PromQL’s on() operator to specify which labels should be matched during operations, or use sum() to ensure label sets are consistent.
Summary
The thread discusses a warning encountered after upgrading to Grafana v10.2.3, where users see “2 items dropped from union(s)” when performing mathematical operations on time series data. This warning arises due to mismatched label sets between the series involved in the operation. Users shared experiences and solutions, including using PromQL to handle calculations, aligning timestamps, and employing the on() operator for label matching. Some users found that resampling data helped resolve the issue. The discussion highlights the importance of understanding how Grafana handles time series data and the need for consistent label sets when performing operations. Users are encouraged to perform calculations at the PromQL level to avoid such warnings and ensure accurate results.
So, if you had series like this:
