My Setup
I am using Grafana v9.3.2 with data stored in InfluxDB.
One of my panels is a grouping of readings from multiple different temperature sensors. I have used thresholds to colour the gauges when the temperature goes outside the acceptable range, but because each sensor is measuring something different, the acceptable range is different for each one and I have used the field override to set individual thresholds.
My Issue
I would also like to set up alerts for when any of the temperatures cross this threshold. However, I quickly ran into two problems:
There does not seem to be any way to reference these overrides from the alert creation and editing section of Grafana.
If I set up a query-based alert with a threshold, it seems I can only choose a single threshold for the entire query, meaning there is no way to group every sensor into a single temperature alerts query.
A solution?
There’s a clear workaround here - 5 different alerts, each querying a single sensor, with the temperature thresholds set individually in the alert rules page. But this is a lot of manual work, which introduces more room for error. (The temperature panel given as example here is not the only instance of this problem for me). It also duplicates the threshold information across two very different places, making it difficult to change if we decide the thresholds need to be updated or if another sensor is added.
Ideally, I would like some sort of Alert when a specified panel passes a threshold option, but I could settle for anything that can reduce either the duplication of threshold information or the number of individual alert rules created. I’d prefer a solution that works entirely within Grafana, if possible.
Does anyone know if this can be done? Or do I have to add additional fields/data to the database itself?
Thanks for the link. This, especially example 5, looks like a great way to do it if I were to add additional information to the database containing my thresholds. Ideally, I was looking for some sort of… global threshold variables I could set in Grafana? So that I did not have to modify the database structure in any way. But if the suggested way to solve this problem is through changes to the database instead I will take that route.
Remember that Grafana is not a database, so other than being able to select a threshold in certain panels and define a threshold in an alert formula, one would really need to store the setpoint in a database (and ideally, with the same timestamp as the actual value).
I came to this conclusion a while back when just getting started with Grafana & InfluxDB and storing temperature data. So I set up Node-RED to transmit our temperature readings to InfluxDB whereby the actual and setpoint were recorded at the exact same time.
The temperature readings are collected every few seconds with a simple script that reads sensors and pushes the data to InfluxDB with timestamp, sensor name, temperature reading, and so on. It should not be too difficult to include thresholds in this information as well, which would ensure the timestamps match as well.
I had been thinking about InfluxDB as a data readings storage and Grafana as a tool to display data in various ways based on conditions. But you’re correct that when the conditions become complicated enough, they are essentially “data” of their own and should be tracked in the database rather than in Grafana.
This is not the sort of answer I was expecting, but it’s a perfect way to address the core issue at hand rather than the question I originally asked. I’ll have to pay attention in the future to what information belongs in InfluxDB vs Grafana.
