Multiple Targets on a Single Dashboard

[ionicowl]

Hi, new to Grafana so I’m still getting to grips with setting up dashboards. I’ve made some good progress so far, managing to get Prometheus to scrape various resources and graph them in simple dashboards in Grafana. I’m hitting a problem that I’m sure has a more elegant solution than whatever horrible things I’m going to try in the meantime.

Basic overview of my setup is as follows:

• I have a server running NGINX. Let’s say its IP address is 100.100.100.100.
• That server is also running node_exporter, listening on port 9100, as well as nginx_prometheus_exporter, listening on port 9113.
• There is a single Prometheus instance that successfully scrapes both system metrics from the node_exporter and NGINX metrics from the nginx_prometheus_exporter with no issues. I have individual dashboards set up to monitor both aspects of the server independently.

What I want to do is create a single dashboard that allows me to display both system info and NGINX metrics simultaneously. The dashboard I have has a variable set up which allows me to select a target from the dropdown list at the top (there is only a single target at the moment but more will be added later). This is the query for the variable:

label_values(node_uname_info{job="$job"}, instance)

I can select 100.100.100.100:9100 to display metrics scraped by node_exporter, but I can’t then display anything reported by the nginx exporter, which talks over port 9113.

An example query for displaying load average:

node_load1{instance="$node",job="$job"}
(where $node is the target, in this case 100.100.100.100:9100)

This works fine, as expected. The problem arises when I want to graph something from the nginx_exporter. Since this communicates over another port, using $node won’t work, since that will always mean 9100. To get it to work, I have to remove the variable from the query and define it absolutely, like this example:

irate(nginx_http_requests_total{instance=~"100.100.100.100:9113"}[5m])

This works, but now I no longer have the ability to change the target on the dashboard and it update both graphs for node_exporter and nginx_exporter. It’ll update the system info but not the NGINX info because that’s hard-coded.

I guess the short version of what I’m asking is if there is a way that I can make the IP address of the target a variable in my query, whilst making the port number absolute, without breaking everything?

[ionicowl]

Ok, I guess writing that post allowed me to figure out what I was actually asking for, so after some better Googling I think I’ve found the answer.

What I needed to do is configure the variable to only return the IP/hostname of the target, leaving the port out. This means I can keep the target IP the same across all panels, but define the port number absolutely, for example:

For something from node_exporter: node_load1{instance="$node:9100",job="$job"}

For something from nginx_exporter: irate(nginx_http_requests_total{instance=~"$node:9113"}[5m])

This seems to work fine. I guess I’ll know for certain when I add more servers.

For posterity, I found the solution at this forum thread:

Using regex in Grafana templating - how?

I think this should work: /([a-zA-Z0-9\.]+)/ [a-zA-Z0-9\.]+ matches lower and uppercase letters, numbers and dots. The brackets around them create a capture group so that it only returns the part that matches.

I added the following regex to the variable in Dashboard Settings.
/(.*[^:])[:]/