I am trying to create my first alert in Grafana with AWS CloudWatch data source. There are two settings related to time that needs to be configured.
The query section has the “Time Range”
Also, the “Set evaluation behaviour” section has the evaluation interval to be configured.
Do we need to consider any rules while setting these values, to ensure that there will be no duplicate alerts. For instance, should “Time range” be equal or less than the evaluation interval. Please advise.
That’s a good question!
To ensure there are no duplicate alerts, Grafana uses a process called de-duplication based on a unique fingerprint. This fingerprint is essentially a hash of all the alert’s labels. When an alert notification is sent, it gets logged in the notification log.
During the next evaluation, Alertmanager checks this log. If it finds an entry for the same alert, it skips sending a new notification unless the alert’s labels have changed. This means the query itself doesn’t influence the de-duplication process—alerts are only sent again if their labels change or after the specified repeat_interval has passed.
In theory yes, but it depends.
I would keep your config for CloudWatch case, because CloudWatch can be slow - it may need a few minutes to publish metric. If that few minutes is 5+ minutes, then you will have nodata error if you keep both settings 5min. So overlap make sense in this particular case.
You may have also alert logic which may need longer query period. So it depends on the use case.