How do you sum all like-named rows?

  • What Grafana version and what operating system are you using?

Grafana open source v9.4.7, OS some form of Linux

  • What are you trying to achieve?

I have data that has a label ‘version’ with a string of the version of the reporting asset. The metric to query us as simple boolean (basically, it is reporting data). This version string is major.minor.point (e.g. 9.9.2, 8.5.1-custom, etc.).

I want to create a pie chart (or any visualization, really) of the percentage of the major.minor versions currently reporting data, e.g. ignoring the ‘point’ part. So if the results of the query had 5 rows of version 9.1.0, 5 rows of version 9.1.2, 10 rows of version 9.8.0, and 5 rows of version 9.8.1, the end result would be

9.1 40% (10 of 25)
9.8 60% (15 of 25)

  • How are you trying to achieve it?

Getting the base PromQL query is deceptively easy:

count by (version)(metric query with other filters)[24h]

of type instant (don’t need to graph this over time, just need the last report; systems report data every 24 hours which is why the [24h] is used from VictoriaMetrics)

The Rename by Regex makes it easy to change the row labels (count by version) to be major.minor version. But I haven’t found how to take all of the unique-before-rename-but-not-unique-now rows and sum them up. I thought Group By would do what I want, but it doesn’t.

(as in, what used to be result rows of 9.1.0, 9.1.0, 9.1.0, etc., are just multiple rows of 9.1, 9.1, 9.1, etc.)

The output of multiple ‘add field from calculation’ reduce row for each major.minor version, creating a new field called “major.minor Total” (which is not ideal as the rows will be different depending on the filters in the query) and then ending with a filter by name transform to only include the new .* Total ones gives me a pie chart that looks reasonable with percentages, but the underlying numbers are way off (32% of a given major.minor is close to a manual calculation by exporting CSV into Excel, but the underlying number of 7 should be more like 3400).

I’ve gone through all the transforms that looked like they might get me somewhere but no combination has given me what I want yet.

  • What happened?

Unpredictable results which won’t work for different filters. Depending on filters, the version numbers will be different and they will change over time.

The reported numbers in the pie chart labels change unexpectedly too; I know that changing query time will do that, but I would not expect that much of a change when doing that (also assuming a refresh implicitly changes query time as it’s “last N hours”).

  • What did you expect to happen?

All duplicate major.minor rows get summed. Unicorns and sunshine too.

Below is the query JSON. Assuming what I’m trying to do is possible, I would welcome any suggestions on how to accomplish it. Thank you.

{
“datasource”: {
“uid”: “redacted”,
“type”: “prometheus”
},
“fieldConfig”: {
“defaults”: {
“custom”: {
“hideFrom”: {
“tooltip”: false,
“viz”: false,
“legend”: false
}
},
“color”: {
“mode”: “palette-classic”
},
“mappings”:
},
“overrides”:
},
“gridPos”: {
“h”: 9,
“w”: 12,
“x”: 0,
“y”: 0
},
“id”: 2,
“options”: {
“reduceOptions”: {
“values”: false,
“calcs”: [
“lastNotNull”
],
“fields”: “”
},
“pieType”: “pie”,
“tooltip”: {
“mode”: “single”,
“sort”: “none”
},
“legend”: {
“showLegend”: true,
“displayMode”: “list”,
“placement”: “bottom”
},
“displayLabels”: [
“percent”
]
},
“title”: “Reporting Data”,
“type”: “piechart”,
“targets”: [
{
“datasource”: {
“type”: “prometheus”,
“uid”: “yKjGXOB4z”
},
“refId”: “A”,
“editorMode”: “code”,
“expr”: “count by (cpr_version)({name="is_online",product="$productfamily",model=~".$productmodel.",version=~".$productversion."})[24h]”,
“legendFormat”: “__auto”,
“range”: false,
“instant”: true,
“exemplar”: false,
“format”: “time_series”
}
],
“transparent”: true,
“transformations”: [
{
“id”: “renameByRegex”,
“options”: {
“regex”: “([0-9]+)\.([0-9]+)\.(.)",
“renamePattern”: “$1.$2”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “sum”,
“include”: [
“9.1”
]
},
“alias”: “9.1 Total”,
“replaceFields”: false
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “sum”,
“include”: [
“9.2”
]
},
“alias”: “9.2 Total”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “sum”,
“include”: [
“9.5”
]
},
“alias”: “9.5 Total”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “sum”,
“include”: [
“9.6”
]
},
“alias”: “9.6 Total”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “sum”,
“include”: [
“9.7”
]
},
“alias”: “9.7 Total”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “sum”,
“include”: [
“9.8”
]
},
“alias”: “9.8 Total”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “sum”,
“include”: [
“9.9”
]
},
“alias”: “9.9 Total”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “sum”,
“include”: [
“9.10”
]
},
“alias”: “9.10 Total”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “sum”,
“include”: [
“9.12”
]
},
“alias”: “9.12 Total”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “count”,
“include”: [
“9.14”
]
},
“alias”: “9.14 Total”
}
},
{
“id”: “calculateField”,
“options”: {
“mode”: “reduceRow”,
“reduce”: {
“reducer”: “count”,
“include”: [
“9.15”
]
},
“alias”: “9.15 Total”
}
},
{
“id”: “filterFieldsByName”,
“options”: {
“include”: {
“names”: [
“9.1 Total”,
“9.2 Total”,
“9.5 Total”,
“9.6 Total”,
“9.7 Total”,
“9.8 Total”,
“9.9 Total”,
“9.10 Total”,
“9.12 Total”,
“9.14 Total”,
“9.15 Total”
],
“pattern”: ".
Total”
}
}
}
],
“pluginVersion”: “9.4.7”
}

Update: this does in fact work, at least for known-in-advance version numbers. The counting issue was just using Last instead of Total as the number to display. But I’m still looking for a way to do this without having to know what the version numbers are in advance / require adding more transforms to handle each major.minor version number.

Have you looked at using apache echart grafana plugin where you have a ton more control on manipulating data than grafana transformation?
Can you please post sample data coming from your promql so we can emulate it on our grafana?

Is it json format?
regex it, group it, count it in js

Thanks for the response. Haven’t heard of that plugin, is that what your screen shot came from? Looks pretty much exactly like what I’m looking for.

The output of the PromQL is just a label “version” of A.B.C* where A and B are integers and C is an integer followed by other integers or strings, with a timestamp (hourly over the last 24 hours in default use). While I don’t see anything dangerous about that output as it’s not anything near being sensitive, we do have a privacy policy and this is production data so I am not comfortable with posting it.

Only way I could export the data is CSV series joined by time.

then obfuscate the data and post the structure at least. why would anyone ask you to post sensitive data?

Given that the data isn’t really all that complicated and posting any data would simply be a time series of A.B.stuff, I didn’t think it was all that worthwhile an exercise as anything I’d make up would be pretty much the same as anything you make up. This isn’t about sensitive data, it’s about being cautious about our privacy policy.

Turns out my colleague found a way to do this using built-in transforms.
Series to rows
Group by (time ignore, metric group by value calculate total)
Rows to fields (auto for both fields)
Rename by regex to turn the version into just major.minor
Reduce, series to rows, calculate total
Group by (field group by, total calculate total)
sort by total (sum)
Rows to fields again (auto for both fields)

1 Like