I’m not sure about the preferred way in which I should set queries and alerts conditions. Let’s imagine the following scenario:
I wan to to get a mail alert, when e.g. memory usage of a VM is 80%+ for an hour.
I set condition in query pattern and use expression in next step to reduce input of A
It looks nice, the data displayed is transparent, if I expand the alert I get only entries were the problem exist, but I think I get a lot of false alerts, because, even if we have a VM that consumes more resources for an hour with a certain frequency, and it happens that in a 15-minute window, there will be this increase even for 5 minutes, it is treated as an alert.
First, I set query, then I reduce the input of A and finally I use expression to set the math condition of B.
In this way, there is a lot of data to display when I expand the alert name, but I think, in this option I get less false-positive alerts.
I’m using latest grafana (v9) and unified alerting.
Which option is better?