Grafana Alerts getting sent multiple times

We use Grafana to manage our logs, we use alerts to be notified for every ERROR log.

We use this expression:
sum by (line) (count_over_time({swarm_service="api"} |= "ERROR" | pattern "<line>" [2m])) > 0

with “Alert evaluation behavior” of 1min.
The “Rule group evaluation interval” is 1 min.
This leads to three alerts being sent for a single error line. We tried tweaking the alert duration or count_over_time duration, but this didn’t lead to changes.

We are also not able to assess the impact of changing duration values, it seems to have no measurable impact, e.g. changing “Alert evaluation behavior” to a shorter or longer value. We also couldn’t find any documentation that is helping here.

We use Grafana Cloud and this used to work, but updates seem to have broken it.

We believe this is a bug in Grafana Cloud, but greatly appreciate help to try to debug this.

Thanks.