I’m not sure if this is a bug or intended, and I’ve done a bit of searching and can’t find anyone else asking this question.
When querying for variables, I believe it’s common to use something like this to extract the actual values for use later on in the dashboard:
label_values(container_last_seen{pod_name=~".*something.*"}, env)
For our purposes, I’m trying to make a “container / pod” variable that only shows Kubernetes containers that are actually alive. For some reason, cAdvisor gives metrics for containers that are no longer up, however the container_last_seen metric will not be updated. For this reason, I want to use the following query in Grafana to properly limit the containers found in the query to ones that have been recently seen:
label_values((container_last_seen{env="k8s-uat"} >= (time() -30)), pod_name)
The query contained in the brackets as the first argument of the label_values function is a perfectly valid query to run in Prometheus, and produces identically formatted results as if I were to just type “container_last_seen”, however when attempting to use the above in Grafana I get the following error:
Templating
Template variables could not be initialized: parse error at char 1: vector selector must contain label matchers or metric name
Any ideas as to why I can’t use a valid - albeit more complex - query as the first argument to the label_values function?