Hi Folks,
I’m being driven mad with this one.
I’ve set up my Grafana instance to get data from the Blackbox Exporter to probe some external domains.
Using this, I’ve set up a basic Slack notification for downtime alerts. Blackbox handily also includes the HTTP status code.
I’ve got the alert triggering on probe_success - Which returns a 0 or 1 depending. This is fine, but what I’m trying to do is include a second query in my alert which pulls the actual http response.
However, when I try to get the value for the second query in my alert it simply isn’t there.
My notification message. as added via the Alert Rule config section looks like the below:
But the lack notification looks like this:
Looking at the map[ ] values, it looks like B isn’t anywhere to be found.
So, the question is this, what on earth have I done wrong here?!
Unfortunately, this is not possible at this moment. When rule is evaluated, Grafana expression engine builds dependency graph of queries and expressions, and executes it. All orphaned nodes that are not used in the condition are ignored.
@yuriy.tseretyan - Thanks for the reply! It’s definitely a shame, because this would certainly give us some feature rich alerts!
Are there any plans to add a feature like this, and if so, a roadmap? Alternatively, is there any other alerting I could use to trigger this?
We have some plans to support alert enrichment in the future. It’s in a design phase right now but we will come up with something soon-ish. No specific dates, sorry.
You can achieve similar if you implement your own http server and point a webhook integration to it. It will receive a notification, take labels and enrich alerts with some data and send to the end user.
@yuriy.tseretyan - That’s very handy to know, thank you!
Don’t suppose there is any documentation that would say how to handle your webhook suggestion?
There is a format of the webhook payload Configure the webhook notifier for Alerting | Grafana documentation. But other than that it’s up to you how to handle it.
