Alert not placing quotes around label values

We’re using blackbox exporter for some endpoint monitoring and when creating an alert the json that is created doesn’t place quotes around the label values. This results in message templates that don’t show the values for the instance or other labels.

e.g. {{ .labels.instance }} site is DOWN

I’ve tried with both .Labels and .labels, doesn’t matter.

The data as output in the query and expressions is how I expect it to be.

The previewed alert values seem fine, until I take a look at Info

What Info looks like:

[ var='B' labels={__name__=probe_success, container=blackbox-exporter, endpoint=http, instance=https://removed.com/health, job=prometheus-blackbox-exporter, namespace=blackbox, pod=prometheus-blackbox-exporter-5cdf9d69df-bxktw, service=prometheus-blackbox-exporter, target=removed-api-internal} value=1 ], [ var='C' labels={__name__=probe_success, container=blackbox-exporter, endpoint=http, instance=https://removed.com/health, job=prometheus-blackbox-exporter, namespace=blackbox, pod=prometheus-blackbox-exporter-5cdf9d69df-bxktw, service=prometheus-blackbox-exporter, target=removed-api-internal} value=0 ]

What I believe should be happening:

[ var='B' labels={__name__="probe_success", container="blackbox-exporter", endpoint="http", instance="https://removed.com/health", job="prometheus-blackbox-exporter", namespace="blackbox", pod="prometheus-blackbox-exporter-5cdf9d69df-bxktw", service="prometheus-blackbox-exporter", target="removed-api-internal"
    } value=1
],
[ var='C' labels={__name__="probe_success", container="blackbox-exporter", endpoint="http", instance="https://removed.com/health", job="prometheus-blackbox-exporter", namespace="blackbox", pod="prometheus-blackbox-exporter-5cdf9d69df-bxktw", service="prometheus-blackbox-exporter", target="removed-api-internal"
    } value=0
]

I’m not sure whether quotes around the instance are really the issue, but unless I am mistaken it appears to be breaking the JSON data due to the colon.

This was not an issue with Alertmanager/Prometheus rules, but we are moving away from Alertmanager to get native support for MS teams alerts and we’d like to centralize our rules.

Currently on Grafana 9.0.5.

Hey @kbick675 ,

Can you try with {{ $labels.instance }}.

Here’s a doc on how to use labels and its values : Labels in Grafana Alerting | Grafana documentation

Thanks @tanmaybhat24, that was the correct answer for the up/down site check alert! Can’t believe I missed something that obvious.

However, I have some certificate expiration alerts where I reference the value of the alert and that is still broken.

Same apparent issue with the JSON:

[ var='B' labels={container=blackbox-exporter, endpoint=http, instance=https://removed.com/up.txt, job=prometheus-blackbox-exporter, namespace=blackbox, pod=prometheus-blackbox-exporter-5cdf9d69df-bxktw, service=prometheus-blackbox-exporter, target=prod-admin-internal} value=1.5405972e+07 ], [ var='C' labels={container=blackbox-exporter, endpoint=http, instance=https://removed.com/up.txt, job=prometheus-blackbox-exporter, namespace=blackbox, pod=prometheus-blackbox-exporter-5cdf9d69df-bxktw, service=prometheus-blackbox-exporter, target=prod-admin-internal} value=0 ]

Template text looks like this when rendered:

{{ $labels.instance }} cert is expiring in {{ humanizeDuration $values.B }}.

Any ideas?

Perhaps try this?

{{ $values.B.Labels.instance}} cert is expiring in {{ $values.B }}. 

and if that works, then I think the humanizeDuration part is something like

{{ humanizeDuration .Value}}
1 Like

@grant2 That works! Unfortunately I’ve been unable to get humanizeDuration to work despite trying a number of permutations.

1 Like