I get data from many sensors in a single measurement. Sensors are distinguished by “name” label so that a single query with group by name returns one timeseries per sensor.
What I would like to do is create an alert rule that would fire an alert for each sensor with no data, to identified lost sensors.
A naive approach is to query the measurement, group by name, count and threshold on each timeseries. Except when there is no value, the query doesn’t even return the timeseries, so the alert doesn’t fire.
I can manually create an alert rule per sensor name, that would fire in case of “no data”. I could even do it programmatically using the API. But I was hoping the web UI offers a nicer way to do that.
Am I missing something?
I understand how this can be difficult to achieve. When evaluating the rule, Grafana doesn’t know about the labels that don’t appear in the time range of the query.
We could use another query on a wider time range to get all values for a label, kinda like what is done to fill the dropdowns in the InfluxQL query editor. But this might necromantiate long forgotten values remaining in the base we don’t want to monitor anymore.
It sounds saner to use a user-defined list of label values. Is there a way to define the alert as a template and instantiate it for each sensor name (i.e. for a given set of values for a given label)?