Grafana Alerting from CloudWatch failed to fetching of query results exceeded max number of attempts

Grafana Alerting
, , , ,
1

Hello,

Could any help me with the following problem? We have Grafana alert based on AWS Cloudwatch Logs query, but sometimes the alert fails with the errors.

The error messages are below:
logger=ngalert.scheduler rule_uid=oEDtfR2Vk org_id=1 version=13 attempt=0 now=2023-02-13T02:09:00Z t=2023-02-13T02:09:12.596886776Z level=error msg="Failed to evaluate rule" error="[plugin.downstreamError] failed to query data: fetching of query results exceeded the max number of attempts" duration=0s

Sometimes all are good, data is returned and the alert works fine.
Does anybody have an idea?

Other useful information:
Grafana version: 9.3.6

2

As per error messages, I doubt sometimes connection timeout happens here but cannot get which one.

3

The answer can be here:

https://github.com/grafana/grafana/issues/63305