Metric labels for returned data not in the original order

Grafana Dashboards
1

We send the prometheus metrics from our application to Grafana SaaS Cloud instance -

On the Grafana Cloud dashboard we setup the retrieved data line is too long (please see below) and its not possible to see the most important metrics which are the “qmgr” and “queue”. We had to painfully copy the entire retrieved string below which you can see when you hover mouse over the graph.

The display ordering we think Grafana decides to format/sort a map: it appears to be alphabetic. But it’s certainly not the same order in which the labels were added

Please suggest a way to fix it.

How it is today:

ibmmq_queue_mqput_mqput1_count{description=“-”, endpoint=“metrics”, instance=“10.131.0.244:9157”, job=“mq-metric-service-qmgr04”, namespace=“mq”, platform=“UNIX”, pod=“mq-metric-qmgr04-f549b4684-wghww”, prometheus=“openshift-user-workload-monitoring/user-workload”, prometheus_replica=“prometheus-user-workload-1”, qmgr=“QMGR04”, queue=“TIP230.HEALTHCHECK”, service=“mq-metric-service-qmgr04”, usage=“NORMAL”}

How it should be:
ibmmq_queue_mqput_mqput1_count{qmgr=“QMGR04”, queue=“TIP230.HEALTHCHECK”, description=“-”, endpoint=“metrics”, instance=“10.131.0.244:9157”, job=“mq-metric-service-qmgr04”, namespace=“mq”, platform=“UNIX”, pod=“mq-metric-qmgr04-f549b4684-wghww”, prometheus=“openshift-user-workload-monitoring/user-workload”, prometheus_replica=“prometheus-user-workload-1”, service=“mq-metric-service-qmgr04”, usage=“NORMAL”}

Environment (with versions):

  • Grafana version: Whatever is your latest on grafana cloud
  • Operating system: RHEL
  • Datasource version(s) (prometheus, graphite, etc.): Built-in Prometheus instance on Openshift cluster
2

Is there any Prometheus doc, specification, which promise to keep label order?
IMHO not and that shouldn’t be causing a problem, because labels are used for filtering and aggregation

3

I am not sure about that. But the people who created the metrics argue that it was not in that order. Is there any way to change /fix this behavior?

4

i would submit a feature request if nothing specific in doc states it should be ordered or not

maybe a config setting that says honor_source_sort

there are other such config settings

honor_☆₩●

5

Absolutely, they have to write labels in alphabetical order :joy:

Why that label order matters anyway?

6

It matters as I mentioned before that I have to hover the mouse over the graph, scroll all the way to the right to see the most important metric

7

OK, so then IMHO Prometheus (and also InfluxDB which has the same feature) is not right choice for your case.