We are heavily using the function of "label_values"in our dashboards.
As we have millions of metric and values are in every metrics, the load of variables and also dashboards are too heavy. We wait for the dashboards to load approximately 30 secs or more. Even after the dashboards are loaded, when we change a variable, we again wait for seconds. Here is the example of our usage :
variable : pod
query : label_values({squad=“XXX”, name=~".YYY"}, label_env)
refresh : on dashboard load
variable : hostname
query : label_values({label_env=~"$pod",name=~".YYY"}, sysname)
refresh : on dashboard load
variable : interface
query : query_result(interface_ifAdminStatus{sysname="$hostname"})
refresh : on dashboard load
How can I speed up these queries?
I tried to transfer the queries into Prometheus. But as the “label_values” function is not defined in Prometheus, the label related queries (label_values(), label_names(), label_values(metric, label)) are not working.
Is there a way to transfer label related queries to Prometheus?
Thank you in advance!
Cagdas