No data alert rule for multiple tag sets in same measurement

Grafana Alerting
1

I get data from many sensors in a single measurement. Sensors are distinguished by “name” label so that a single query with group by name returns one timeseries per sensor.

What I would like to do is create an alert rule that would fire an alert for each sensor with no data, to identified lost sensors.

A naive approach is to query the measurement, group by name, count and threshold on each timeseries. Except when there is no value, the query doesn’t even return the timeseries, so the alert doesn’t fire.

I can manually create an alert rule per sensor name, that would fire in case of “no data”. I could even do it programmatically using the API. But I was hoping the web UI offers a nicer way to do that.

Am I missing something?

I understand how this can be difficult to achieve. When evaluating the rule, Grafana doesn’t know about the labels that don’t appear in the time range of the query.

We could use another query on a wider time range to get all values for a label, kinda like what is done to fill the dropdowns in the InfluxQL query editor. But this might necromantiate long forgotten values remaining in the base we don’t want to monitor anymore.

It sounds saner to use a user-defined list of label values. Is there a way to define the alert as a template and instantiate it for each sensor name (i.e. for a given set of values for a given label)?

2

You could achieve what you’re doing by programmatically creating alerts, as you’ve mentioned before. What is your datasource?

If it’s prometheus (group by name suggests it is), you can do something like or on() vector(-1) which would return the data from the sensors or, if they are not there, will return -1. Then you could alert on -1 value. I’m not sure how many series like that would it catch though (meaning how long ago your sensor died), so that’s something you’d have to experiment with.

3

Datasource is InfluxDB.

Sure, I could do anything programmatically. Since this might be a common need, I figured there could be a GUI way.

I’m afraid what I’d like doesn’t exist (yet). It would be variables in rules and the possibility to create many rule instances from a list of values for rule variables. In my case, the variable would be the name tag values.

Multi-dimensional alerts work great but not so much for the No Data case.