Grafana Alert Manager: I would like to use a threshold which is stored as label for the Alert, how is this possible?

Grafana Alerting
, ,
1

I have results of time measuring (using blackbox exporter) in prometheus for different services. Now I want to implement an Alert if the service is slower than <limit_warn> seconds.

The limit has to differ for different services so I added the concrete limit for each service as an prometheus label called “limit_warn=1” to make an alert if this service talkes more than 1 second.

Now to define the rule in Grafana Alertmanager I have:

  • query SECONDS (reading the metrics value from prometheus)
  • reduce expression LAST_SECONDS
  • threshold expression THRESHOLD = LAST_SECONDS is above $limit_warn

This won’t work, so how to refer to the label value in this threshold expression?
I also tried a math expression with no more luck :frowning:

Has anyone of you done a similiar approach which works?