How can I get CPU usage for multiple servers within a single panel?

Hi,

I’m relatively new to Grafana\Prometheus & I’m trying to build a server monitoring dashboard using the Boom Table Panel with the goal at this very early stage to have a row for each server selected, showing memory & CPU usage. When multiple servers are selected, memory usage works as expected & we get a row for each server, however CPU usage appears to be aggregated & this is also the case if we take the Boom Table out of the equation. I suspect that it’s something to do with the metric query I’m using, which works absolutely fine for a single server, but as soon as I add others, it provides a single aggregated number rather than the usage for each server.

The query I’m using is :

100-(avg(irate(windows_cpu_time_total{instance=~“$instance”,mode=“idle”}[2m])))*100

I’d really appreciate it if someone could point the error of my ways :slight_smile:

Thanks!

Hi,

If you want to show the all instance, please use this:


Then select the instance, so it’ll show the repeat of all instances…

Regards,
Fadjar

Hi Fadjar,

Thanks for replying!

That approach works fine for multiple panels, but I’m looking to have a single panel with a table showing all servers & their stats, so another column here showing CPU usage…

Like I say, I’m pretty sure that the query for CPU is taking an average across all servers & I’m trying to achieve something like this…

Thanks,

Neil

Hi…

I build this panel that suitable with my requirement

This panel just using Table with 3 transform, Merged, Group By and Organized Fields.
The Datasource using Prometheus

Regards,
Fadjar

Hello Fadjar,

This panel you made is great! How can I make/download it? Don’t leave us hanging! :slight_smile:

Regards,
Seth

Below the panel json if you want to use:

{
  "datasource": {
    "type": "prometheus",
    "uid": "djoI8bg4k"
  },
  "fieldConfig": {
    "defaults": {
      "custom": {
        "align": "center",
        "cellOptions": {
          "type": "auto"
        },
        "inspect": false
      },
      "mappings": [],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "semi-dark-green",
            "value": null
          },
          {
            "color": "semi-dark-yellow",
            "value": 75
          },
          {
            "color": "semi-dark-red",
            "value": 80
          }
        ]
      },
      "color": {
        "mode": "thresholds"
      },
      "max": 100
    },
    "overrides": [
      {
        "matcher": {
          "id": "byName",
          "options": "CPU Load"
        },
        "properties": [
          {
            "id": "custom.cellOptions",
            "value": {
              "mode": "lcd",
              "type": "gauge"
            }
          },
          {
            "id": "thresholds",
            "value": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "#EAB839",
                  "value": 75
                },
                {
                  "color": "red",
                  "value": 95
                }
              ]
            }
          },
          {
            "id": "custom.width",
            "value": 231
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "RAM"
        },
        "properties": [
          {
            "id": "custom.cellOptions",
            "value": {
              "mode": "basic",
              "type": "gauge"
            }
          },
          {
            "id": "thresholds",
            "value": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "#EAB839",
                  "value": 75
                },
                {
                  "color": "red",
                  "value": 95
                }
              ]
            }
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "Disk Usage"
        },
        "properties": [
          {
            "id": "custom.cellOptions",
            "value": {
              "mode": "lcd",
              "type": "gauge"
            }
          },
          {
            "id": "thresholds",
            "value": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "#EAB839",
                  "value": 80
                },
                {
                  "color": "red",
                  "value": 95
                }
              ]
            }
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "Node"
        },
        "properties": [
          {
            "id": "mappings",
            "value": [
              {
                "options": {
                  "pattern": "(.*):9796",
                  "result": {
                    "index": 0,
                    "text": "$1"
                  }
                },
                "type": "regex"
              }
            ]
          },
          {
            "id": "custom.width",
            "value": 140
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "Cores"
        },
        "properties": [
          {
            "id": "custom.cellOptions",
            "value": {
              "mode": "basic",
              "type": "color-background"
            }
          },
          {
            "id": "color",
            "value": {
              "fixedColor": "semi-dark-blue",
              "mode": "fixed"
            }
          },
          {
            "id": "custom.width",
            "value": 86
          }
        ]
      }
    ]
  },
  "gridPos": {
    "h": 15,
    "w": 19,
    "x": 5,
    "y": 9
  },
  "id": 12,
  "options": {
    "showHeader": true,
    "cellHeight": "sm",
    "footer": {
      "show": false,
      "reducer": [
        "sum"
      ],
      "countRows": false,
      "fields": ""
    },
    "frameIndex": 0,
    "sortBy": [
      {
        "desc": true,
        "displayName": "RAM"
      }
    ]
  },
  "pluginVersion": "9.5.2",
  "targets": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "djoI8bg4k"
      },
      "editorMode": "code",
      "expr": "(sum by (instance) (node_cpu_seconds_total{job=\"linux_vm\",mode!=\"idle\"})) / (sum by (instance) (node_cpu_seconds_total{job=\"linux_vm\"}) ) * 100 ",
      "format": "table",
      "hide": false,
      "legendFormat": "CPU",
      "range": true,
      "refId": "B"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "djoI8bg4k"
      },
      "editorMode": "code",
      "expr": "((((sum by (instance) (node_memory_MemTotal_bytes{job=\"linux_vm\"})) - (sum by (instance)(node_memory_MemFree_bytes{job=\"linux_vm\"}))) / (sum by (instance) (node_memory_MemTotal_bytes{job=\"linux_vm\"}))) * 100)",
      "format": "table",
      "hide": false,
      "legendFormat": "Memory",
      "range": true,
      "refId": "A"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "djoI8bg4k"
      },
      "editorMode": "code",
      "expr": "((((sum by (instance) (node_filesystem_size_bytes{job=\"linux_vm\"})) - (sum by (instance) (node_filesystem_free_bytes{job=\"linux_vm\"}))) / (sum by (instance) (node_filesystem_size_bytes{job=\"linux_vm\"}))) * 100)",
      "format": "table",
      "hide": false,
      "legendFormat": "Disk",
      "range": true,
      "refId": "C"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "djoI8bg4k"
      },
      "editorMode": "code",
      "expr": " sum by (instance)(count without (job,mode) (node_cpu_seconds_total{job=\"linux_vm\",mode=\"idle\"}))",
      "format": "table",
      "hide": false,
      "range": true,
      "refId": "E"
    }
  ],
  "title": "Server Capacity",
  "transformations": [
    {
      "id": "merge",
      "options": {}
    },
    {
      "id": "groupBy",
      "options": {
        "fields": {
          "Value #A": {
            "aggregations": [
              "lastNotNull"
            ],
            "operation": "aggregate"
          },
          "Value #B": {
            "aggregations": [
              "lastNotNull"
            ],
            "operation": "aggregate"
          },
          "Value #C": {
            "aggregations": [
              "lastNotNull"
            ],
            "operation": "aggregate"
          },
          "Value #E": {
            "aggregations": [
              "lastNotNull"
            ],
            "operation": "aggregate"
          },
          "instance": {
            "aggregations": [],
            "operation": "groupby"
          }
        }
      }
    },
    {
      "id": "organize",
      "options": {
        "excludeByName": {},
        "indexByName": {
          "Value #A (lastNotNull)": 3,
          "Value #B (lastNotNull)": 2,
          "Value #C (lastNotNull)": 4,
          "Value #E (lastNotNull)": 1,
          "instance": 0
        },
        "renameByName": {
          "Value #A (lastNotNull)": "RAM",
          "Value #B (lastNotNull)": "CPU Load",
          "Value #C (lastNotNull)": "Disk Usage",
          "Value #E (lastNotNull)": "Cores",
          "instance": "Node"
        }
      }
    }
  ],
  "type": "table"
}

Just tune the query for your prometheus data.

Regards,
Fadjar

2 Likes

Hey Fadjar, I finally got a chance to try this out. It worked well once I got the data source right.
Big thanks!

Hi There,

I am looking for something similar as well. I tried to copy this Json for 1 of my panel (after changing the datasource uid) but there is no data in the table (the table shows “no data”). I am also not getting any errors and it seems that grafana finds the parameters and the methods (group by, merge) properly. Do you have some idea what could go wrong? I am very new to grafana and prometheus, just trying to figure out basic stuff that I need.

Thank you for the help!

First of all, try to remove all the transformation, then check the data in the table view.
Check the query and make sure the data already show up.
Then step by step, put the transformation as per your requirement…
Perhaps the transformation is not suitable for your query…
Please pay attention in the job value… It;s depends on your data value in the Prometheus

Regards,
Fadjar

I am afraid that I am a bit of a newbie to properly understand everything that you just typed. I removed the “transformations” part from the json code, but it is still not working and the table view says “no data”. I am using prometheus 2.45.0 and grafana 10.0.1.

What can I do? By the way are you available for hire?

Please check the expression of the query:

“((((sum by (instance) (node_memory_MemTotal_bytes{job=“linux_vm”})) - (sum by (instance)(node_memory_MemFree_bytes{job=“linux_vm”}))) / (sum by (instance) (node_memory_MemTotal_bytes{job=“linux_vm”}))) * 100)”,`

At this query, look at the job.
That says: “linux_vm”.
Is your data in the Prometheus have job like that?
If no, change that suitable with your data.

Regards,
Fadjar

PS: message me for the job… :slight_smile:

Thank you!

Indeed after I changed the job name it started to work properly. Thank you! I am going to send a PM in the meantime.

Hey there! It sounds like you’re encountering aggregation issues with your Grafana\Prometheus setup. Make sure your metric query is properly scoped to individual servers to avoid aggregation child recovery. Double-check the query syntax and ensure it’s configured to fetch metrics for each selected server separately. You might need to adjust your query to fetch data per server explicitly.

This json import not working, error Dashborad not found when import

I was able to import the JSON panel but I couldn’t find any job. I am getting ‘No options’ for job. Most of our queries are filtered using - {project_id=~“$project_id”, namespace_name=~“$tenant_name”,pod_name=~“$deployment.*”}

You need to create the Variables accordingly…
Just change the variables that suitable with your variables such as project_id, tenant_name and deployment. Find the right query to get the proper result in the variables.

Regards,
Fadjar