Observed in Grafana v10.1.1 but also in v8/9. Alerts created that reference OpenSearch data source and have queries missing surround double quotes, as well as ‘Group by’ as ‘Filters’ cause error below:
[ss.dataQueryError] failed to execute query [A]:
[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = invalid character ‘U’ looking for beginning of value
New alerts created that are known to work, and have correct query format and use ‘Group by’ as ‘Data histogram’, produce same error.
By removing all data sources and alerts in org, then re-adding, and creating alerts known to work that query OpenSearch data source now work. Seems to indicate that specific query configuration break the data sources or alert system.