Extract labels values from prometheus metrics

[codecap]

Hi guys,

I’ve already spent a lot of time , but still have no solution.

Our Prometheus instance stores some interesting metrics, where the relevant information saved as labels and not as metric value:

cadvisor_version_info{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",dockerVersion="17.06.0-ce",instance="minion-005056b311e7",job="kubernetes-nodes",kernelVersion="3.10.0-514.26.2.el7.x86_64",kubernetes_io_hostname="minion-005056b311e7",osVersion="Debian GNU/Linux 8 (jessie)",vendor="vmware",virtualization_role="guest"}

kubernetes_build_info{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",buildDate="2017-01-12T04:52:34Z",compiler="gc",gitCommit="08e099554f3c31f6e6f07b448ab3ed78d0520507",gitTreeState="clean",gitVersion="v1.5.2",goVersion="go1.7.4",instance="minion-005056b369fc",job="kubernetes-nodes",kubernetes_io_hostname="minion-005056b369fc",major="1",minor="5",platform="linux/amd64",vendor="vmware",virtualization_role="guest"}

In both cases the metric value is 1, so it looks to me like nobody is interested in values but labels.

Is there any possibility to extract values from the metrics?

We would like to have a dashboard, where we could see per node kubernetes and docker versions. What would be the best solution for that?

Thanx a lot in advance

[torkel]

did you checkout the docs?

Prometheus data source | Grafana documentation

There is a templating query called label_values that allows you to query label values for a template variable.

@codecap

Is there any possibility to extract values from the metrics?

@codecap

We would like to have a dashboard, where we could see per node kubernetes and docker versions. What would be the best solution for that?

Write a prometheus query and visualize data in table or graph panel? group by kubernetes node & docker version

[codecap]

Hi @torkel,

thanx for your reply . Yes, I could use label_values(metric, label) for templating. But I still couldn’t visualize metric labels …

I tried tables, it looks pretty good, but I wasn’t able to reduce number of columns …

also no success with single stat … only the metric value was shown

[dimitardanev]

Hi @torkel,
any update on display label values inside metrics?

Maybe I am doing something wrong but I am not able to display the value of label coming from Prometheus.

For example Grafana/Singlestat panel which will display the version as text “v.9.4.0” .

nodejs_version_info{group=“axsportal”,instance=“localhost:6600”,job=“prometheus”,major=“9”,minor=“4”,patch=“0”,version=“v9.4.0”} 1

Similar labels are coming from this popular package -> prom-client

---

Update
After struggling for a couple of hours reading how to do it , it finally worked.
It was already explained at least a couple of times.
In my case to get it display v9.4.0 into Singlestat Panel from
nodejs_version_info{group=“axsportal”,instance=“localhost:6600”,job=“prometheus”,major=“9”,minor=“4”,patch=“0”,version=“v9.4.0”} 1

I had to go to

1. Data sources make sure I have Scrape interval set to : 60s
2. In my dashboard Add row -> SingleStat -> Metrics tab -> Query is : nodejs_version_info , now value displayed is 1
3. SingleStat -> Options -> Value Stat (name)
4. Go back to SingleStat Metrics -> Legend format {{version}} You can pick any label of the nodejs_version_info query.

Hopefully this will help someone else.

[mikkop71]

Thanks for the tip using Value Stat and name option. It worked almost like a charm!

There is one problem it doesn’t work with if I repeat that Singlestat panel.
Should it work or is it impossible?

[dimitardanev]

Do you mean duplicate the panel when you say repeat ?
On my end duplicate or creating new Panel from scratch with same settings works.

That is how it looks on my end:

[benkeil]

Its important that only one series is returned from the query. My query:

count(some_stat{app=~"$Application", environment=~"$Environment", namespace=~"$Namespace"}) by (appVersion)

[simontheleg]

has anyone found a solution to displaying @codecap’s original query in a Grafana table? I’ve spend a couple hours on this one as well, but to no avail

[slynickel]

;tldr You can make columns a type of “hidden” on the column style page, you just regex away everything you don’t want or hide one by one. Or you just use max(metric) by (LabelICareAbout,LableICareAbout)

Using Grafana Version: v5.2.1

I created an account just to respond to this because I too spent a bunch of time on this and the answer ended up being pretty easy.

I think this is what you are trying to do

Okay you go into Prometheus and you query a single metric in my case kube_node_info. If that is all you query you get a bunch of columns back. I added line breaks below, in reality this is just a single long string. Now in my case (and I think your case) you just want a few of these, one, maybe two in a table. I want to have a table that is just Kernel version, node, name, and OS image.

kube_node_info{
container_runtime_version="docker://1.13.1",
instance="10.0.0.0:8080",
job="k8s-pods",
k8s_node_name="aks-agent-123456789-1",
k8s_pod_name="kube-state-metrics",
kernel_version="4.15.0-1032-azure",
kubelet_version="v1.11.4",
kubeproxy_version="v1.11.4",
node="aks-agent-123456789-0",
os_image="Ubuntu 16.04.5 LTS",
provider_id="<Redacted>"
}

If you put just the metric in it looks gross, I don’t want all those columns.

I would put an image here of how that looks but but I'm a *new user* and *new users* only get 2 photos per post.

Solutions

There’s two ways I found to get what you want.

1. Use aggregation: I only care about the most recent reading so first thing is to check the “Instant”. Then you do max(metric) by (LabelICareAbout,LableICareAbout)

   

2. The other way is to use column style page and regex out the columns you don’t want. In this one my query is just kube_node_info no max or group, with instant checked (or without up to you). On the Column styles tab I enter \b(?!node|kernel_version|os_image)\b\S+¹ for the "apply to columns named field. And specify the type to “Hidden”. Picture is probably easier to see what is going on. I also added a few other column styles to make the names pretty.

   

¹ I don’t know regex and never will. If you don’t just google “regex everything that isn’t” or something similar to get the regex to regex everything else away.

[amjad489]

@dimitardanev dimitardanev solution works perfect thank you

[tenriola]

Hi, as we know singlestats is deprecated and will no longer available in Grafana 7.0, instead we have to use Stats panel. Do we have any clue how to do similar in Stats panel? Thx. @dimitardanev

[xqos]

My label value no longer works with 7 Stats panel, how do I get them back? I used to be able to choose the column I wanted.

[yejk2k]

Grafana 7.0, You should use Legend

==============

Repeat Options: use Repeat by variable targets, If repeat is used, the Legend configuration will not take effect

Legend is not work

[mattabrams]

Dear all, we now have an official blog post that details how to extract label values from Prometheus metrics:

How to extract label values from Prometheus metrics in Grafana | Grafana Labs