I have an alert defiined below that send me a 3 notifications (1 for every query a,b,c) . My question is 1) - why and 2) how do i stop grafana doing that. I only want the one notification.
1 Like
It is not visible, but I guess your metric has query label and alerting consider each timeserie as different instance - just make sure that no additional labels are returned from your alert query.
sorry as with alot of the question on here im not the best with grafana/observability tyring to discover more as i go along…
the raw metric that i’m basing this all of is :
track_vulnerabilities{cluster_arn=“arn:aws:ecs:eu-west-2:456:cluster/scan-tools-cluster”, container_arn=“arn:aws:ecs:eu-west-2:456:container/scan-tools-cluster/abc/cba”, container_name=“exporter”, docker_image=“456.dkr.ecr.eu-west-2.amazonaws.com/track-exporter:latest”, instance=“10.0.0.5:9916”, job=“ecs”, name=“aet”, severity=“CRITICAL”, task_arn=“arn:aws:ecs:eu-west-2:456:task/scan-tools-cluster/6yujhgfrtyhgr6t44f”, task_group=“service:Track”, task_name=“track”, task_revision=“68”, uuid=“123”, version=“1.0.0”} 1
Update:
When i look at the history of the state of the alert i can see that its firing 3 times… I would have thought that should just be once… ?
This is old, but I am pretty sure we are facing the same issue. Looks like you might be running several containers, e.g. in ECS, without any specific HA config (Configure high availability | Grafana documentation) We also think this is annoying and would have hoped that an internal “at-most-once” mechanism could have been part of Grafana itself without the need for external caches. But it appears we will need such a cache.
Anyone following this link to find the solution.
In my situation I’m running the Amazon managed Grafana and this is a known issue with its implementation.
To resolve this I implemented simple a mechanism using aws Lambda,DynamoDB to deduplicate the alerts before processing them.
Hey @simonhall I have the exact same issue with AMG, and AWS has not solved the issue as of yet, can you please share more details or steps on how to implement this strategy in order to get only ONE alert at a time. Thanks!