Problem generating alerts "msg=Alert Rule returned no data"

Hi,

I am having troubles in creating Grafana alerts. Raising this topic here to get some help in identifying the issue.

Below are the details.

Grafana version: 7.5.9
Data source: AWS Elasticsearch(Open Distro for Elasticsearch).
Query:

Alert:

Test rule:
{
“firing”: false,
“state”: “no_data”,
“conditionEvals”: “false = false”,
“timeMs”: “46.340ms”,
“logs”: [
{
“message”: “Condition[0]: Query”,
“data”: {
“from”: 1631105903722,
“queries”: [
{
“refId”: “A”,
“model”: {
“alias”: “”,
“bucketAggs”: [
{
“field”: “@timestamp”,
“id”: “1”,
“settings”: {
“interval”: “1m”,
“min_doc_count”: “1”
},
“type”: “date_histogram”
}
],
“format”: “table”,
“metrics”: [
{
“field”: “response_time_ms”,
“id”: “1”,
“meta”: {
“max”: true,
“std_deviation_bounds_lower”: false,
“std_deviation_bounds_upper”: false
},
“type”: “extended_stats”
}
],
“query”: “message.raw:"Request finished" AND app_name.keyword:pharmacy-api”,
“queryType”: “lucene”,
“refId”: “A”,
“timeField”: “@timestamp
},
“datasource”: {
“id”: 11,
“name”: “Elasticsearch - health-prod”
},
“maxDataPoints”: 0,
“intervalMs”: 0
}
],
“to”: 1631106203722
}
},
{
“message”: “Condition[0]: Query Result”,
“data”: {
“fromDataframe”: true,
“series”:
}
},
{
“message”: “Condition: Eval: false, Query Returned No Series (reduced to null/no value)”,
“data”: null
}
]
}

Debug log related to Alert:
t=2021-09-08T12:55:40+0000 lvl=dbug msg=“Scheduler: Putting job on to exec queue” logger=alerting.scheduler name=“TestAlert alert” id=128
t=2021-09-08T12:55:40+0000 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=126 name=“Pharmacy API response times alert” changing state to=no_data
t=2021-09-08T12:55:40+0000 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=128 name=“TestAlert alert” changing state to=no_data
t=2021-09-08T12:55:40+0000 lvl=info msg=“New state change” logger=alerting.resultHandler ruleId=128 newState=no_data prev state=unknown
t=2021-09-08T12:55:40+0000 lvl=info msg=“Database locked, sleeping then retrying” logger=sqlstore error=“database is locked” retry=0
t=2021-09-08T12:55:40+0000 lvl=info msg=“Alert already updated” logger=alerting.resultHandler
t=2021-09-08T12:55:40+0000 lvl=dbug msg=“Job Execution completed” logger=alerting.engine timeMs=56.802 alertId=128 name=“TestAlert alert” firing=false attemptID=1

I have tried multiple conditions as well like,
query(A,1m,now-1m),
query(A,1m,now-5m)
I have upgraded to latest version as well but same issue then i reverted the version.

This same alert works perfect when i change the data source to dev index and fails when i change it back to point to prod index.

Please let me know if any further details are needed to provide support on this.

Reconfigured everything from scratch and it has worked.