Apologies for the wall of text, but I’m trying to be as thorough as I can…
I’m on Grafana 11.2 running on Windows. I’m successfully collecting Windows Performance Counters and sending them to Prometheus and seeing the data in Grafana.
I’ve successfully defined a $CounterNames variable and used a repeating panel to generate a Time Series panel and repeat it for every uniquely named counter.
I have 48 such counters, so I get 48 panels, which is fine. As the counter names are sorted alphabetically (as per my $CounterNames definition), that’s the order in which the names are iterated, and the panels appear.
The problem is, because they’re simply sorted by name, the panels don’t have any sort of logical grouping.
Alongside its unique CounterName, each perf counter has an ObjectName, which defines such groups (a group has many perf counters). For example, I have 6 perf counters that have “Memory” for their ObjectName, 5 perf counters for the object named “Process”, 5 perf counters for “LogicalDisk”, etc. It’s not the same counters for each object. I have 17 unique object names, each with a varying number of counters.
Starting over, I defined an $ObjectNames variable holding all the unique object names. I successfully created a repeated row - 17 of them, each showing its own unique object name for its title.
Now I’d like to have a repeated panel for each row. I thought I could somehow have row 1 (showing “LogicalDisk”) repeat a time series panel 5 times: one for every unique CounterName that has its ObjectName set to LogicalDisk.
The second row should show “Memory”. It should contain 6 time series panels, since I have 6 unique CounterName values that have their ObjectName value set to Memory. So on and so forth. This is dynamic, so I can’t assume ahead of time any sort of fixed number for either object names or counter names.
My ObjectNames variable is defined as:
label_values(MyPerfMetrics,ObjectName)
I tried to set a filter on a new counter names variable, CounterNamesFilteredByObjectName, as:
label_values(MyPerfMetrics{ObjectName=“$ObjectNames”},CounterName)
That is:
Query = Label values
Label = CounterName
Metric = MyPerfMetrics
Label filters = ObjectName = $ObjectNames
In both cases, I have both “Multi-value” and “Include All option” checked.
As for the time series panel definition:
On the right side of the screen, “Repeat by Variable” is set to CounterNamesByObjectName.
The query portion on the left side of the panel definition screen is set to:
MyPerfMetrics{ObjectName=“$ObjectNames”,CounterName=~“$CounterNamesByObjectName”}
The problem is, even though I have my properly repeated rows, each row contains a single panel and not showing any data.
I’m not necessarily trying to allow the user to select objects or counters himself through the dropdowns; ideally they should both be set to “All”, and preferably hidden, so the rows are always iterating over all objects, and the panels are always iterating over all the counters (for that row’s object).
I just don’t think Grafana is intended to do what I’m trying to accomplish; I’m thinking, rather, that what I should expect is to have the same panels repeated for every row. But that’s not what I’m getting. The feeling I have is that each panel is not getting the one ObjectName value that is “assigned” to its containing row and thus not showing the subset of items with the matching ObjectName value.
I’ve been fiddling with this for a while, using references to things such as “$CounterNames” (the variable) instead of “CounterName” (the label), as well as “=” vs “=~”…and the best I got at one point was 17 rows rendered, each repeating all 48 perf counters - in other words, the repeating panel in each row ignored the ObjectName “assigned” to the row and just iterated over all CounterNames without any filtering.
Is what I’m trying to do feasible? Am I on the right track?
If it’s not feasible, and there are a fixed number of ObjectName values, I wouldn’t mind all that much hardcoding my rows and assigning a fixed filter for the repeating panel in each row. However, both ObjectName and CounterName are entirely dynamic…