we are trying to achieve the below with same dashboard panel using different ALERT conditions WITH AND/OR operator in automated way in stead of manually pause/resume alert. can this possible by using grafana dashboard or grafana API?
Requirement:
"Given the current levels of noise from alerts generated by Grafana due to tests failing to complete within the defined timescales we need to have some more flexible/staggered alerting within Grafana in order to be able to reduce the noise to the support team while the underlying issues are worked on.
What we need to achieve is some form of staggered alerting process where there are different trigger points for the same failure.
For example, if we took the cluster creation process then for AWS for example the SLA value is 15 minutes. Currently if creation time takes more than this then we have alerts that are continually generated. What we need to be able to do is to have the ability for the alert to have different trigger points, such as 15 minutes, 20 minutes and 50 minutes for example. The support team need the ability to be able to silence alerts at the various levels for a fixed period of time - say 7 days. After this period of time then the alert should become active again.
This will give the engineering team time to investigate underlying issues without the support team being continually bombarded by alerts and they have the ability to silence the lower thresholds if required.
This should initially focus on cluster creation tests as these seem the most problematic…
Grafana alerts to have staggered alerting thresholds for cluster creation tests.
Support team have the ability to silence the lower threshold alerts for a fixed period of time so they don’t get bombarded with noise.
This should be initially applied to cluster creation tests.
After the fixed period of time the lower thresholds are automatically re-activated (the support team can silence again if necessary).
"
forum,