Graphing usage percentage for multiple cgroups with grafana and elasticsearch

davidspencerward · March 21, 2021, 1:59am

I’m on a linux system and I have three cgroups - /Detection, /System, /Database. I have a metric in elasticsearch for the usage percentage for each cgroup.

system.cgroup.memory.path:"/Detection"
system.cgroup.memory.usagePercent:21.2
system.cgroup.memory.path:"/System"
system.cgroup.memory.usagePercent:31.2
system.cgroup.memory.path:"/Database"
system.cgroup.memory.usagePercent:11.2

I have a line graph with a query:

system.cgroup.memory.path.keyword:("/Detection" OR “/System” OR “/Database”)

and is graphing the system.cgroup.memory.usagePercent metric as an Average using a Date Histogram.

Obviously, this will only plot a single line with the average of the three cgroups.

But that’s not what I want - I want it to graph three lines - one for each cgroup. The only way I have been successful is to create three separate queries - an explicit one for each cgroup. But, that’s not very maintainable because if I add a fourth cgroup, I would have to come back to grafana and add another query. So, I want the query to return three data sets - one for each cgroup. This works with a table because you can use the “Group By” feature. But, that doesn’t work for graphs because it doesn’t generate a time sequence.

I’ve looked at a whole bunch of threads online but there doesn’t seem to be an answer to this question.

Is it possible to use a single query to graph multiple lines?

Merci.

davidspencerward · March 21, 2021, 3:14am

I think I just answered my own question. You use “Group By” and specify Terms and the field name. The key was then to click on the “+” on the right hand side of “Group By” and then it gives you a line called “Then By”. On the “Then By” line, you select Date Histogram. And it creates the graph. OMG - I have spent like 4 or 5 nights trying to figure it out and it is all in a hidden field. Uggghhhh.