Extract labels values from prometheus metrics

[slynickel]

;tldr You can make columns a type of “hidden” on the column style page, you just regex away everything you don’t want or hide one by one. Or you just use max(metric) by (LabelICareAbout,LableICareAbout)

Using Grafana Version: v5.2.1

I created an account just to respond to this because I too spent a bunch of time on this and the answer ended up being pretty easy.

I think this is what you are trying to do

Okay you go into Prometheus and you query a single metric in my case kube_node_info. If that is all you query you get a bunch of columns back. I added line breaks below, in reality this is just a single long string. Now in my case (and I think your case) you just want a few of these, one, maybe two in a table. I want to have a table that is just Kernel version, node, name, and OS image.

kube_node_info{
container_runtime_version="docker://1.13.1",
instance="10.0.0.0:8080",
job="k8s-pods",
k8s_node_name="aks-agent-123456789-1",
k8s_pod_name="kube-state-metrics",
kernel_version="4.15.0-1032-azure",
kubelet_version="v1.11.4",
kubeproxy_version="v1.11.4",
node="aks-agent-123456789-0",
os_image="Ubuntu 16.04.5 LTS",
provider_id="<Redacted>"
}

If you put just the metric in it looks gross, I don’t want all those columns.

I would put an image here of how that looks but but I'm a *new user* and *new users* only get 2 photos per post.

Solutions

There’s two ways I found to get what you want.

1. Use aggregation: I only care about the most recent reading so first thing is to check the “Instant”. Then you do max(metric) by (LabelICareAbout,LableICareAbout)

   

2. The other way is to use column style page and regex out the columns you don’t want. In this one my query is just kube_node_info no max or group, with instant checked (or without up to you). On the Column styles tab I enter \b(?!node|kernel_version|os_image)\b\S+¹ for the "apply to columns named field. And specify the type to “Hidden”. Picture is probably easier to see what is going on. I also added a few other column styles to make the names pretty.

   

¹ I don’t know regex and never will. If you don’t just google “regex everything that isn’t” or something similar to get the regex to regex everything else away.