"Elasticsearch error: 404 page not found" when save/test new datasource

Grafana Elasticsearch
1
  • What Grafana version and what operating system are you using?

Grafana version - v9.3.8
OpenSearch version - 2.3.0

  • What are you trying to achieve?
    Adding new datasource

  • How are you trying to achieve it?
    From UI

  • What happened?

“Elasticsearch error: 404 page not found” when save/test new datasource

  • What did you expect to happen?

No errors

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

Error in UI:
“Elasticsearch error: 404 page not found”

In Grafana pod logs:
t=2024-02-16T16:43:17+0000 lvl=info msg=“Request Completed” logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/xy-xyz-abc-abcdef-*/_mapping status=404 remote_addr=10.244.1.0 time_ms=106 size=19 referer=https://abc.xyz.com/datasources/edit/1/

  • Did you follow any online instructions? If so, what is the URL?
2

This works on one Grafana instance and fails on another Grafana instance (two separate regions).

The configurations are the same on both instances.

Would like to know what may be causing the error, we have turned on debug logs but there is not sufficient information as to why it’s failing in one region.

We have checked connectivity from the pod (nc -zv) to the opensearch endpoints and they are open from both.