I recently upgraded Grafana from 10.0.9 to 10.2.3 and I’m observing the following errors for each of my elasticsearch datasources:
message: “An error occurred within the plugin”
messageId: “plugin.downstreamError”
GET https://grafana.dev.leoncorp.net/api/datasources/uid/PE4E1D10B840E7EAC/resources/ 500 (Internal Server Error)
My gitlab datasource config:
- name: “Stasaoo_Test”
type: “elasticsearch”
access: “proxy”
url: “My URL”
basicAuth: true
basicAuthUser: “elast”
jsonData:
includeFrozen: false
index: “[myindex-*]YYYY.MM.DD”
interval: “Daily”
timeField: “@timestamp”
maxConcurrentShardRequests: “256”
oauthPassThru: false
sigV4Auth: false
secureJsonData:
basicAuthPassword: “{{ grafana_elasticsearch_password }}”
For all my panels that use elk datasource work, alerting is ok but I’m curious what is the reason for such an error?
might want to post in grafana github
Can you try 10.2.2?
How do you know that’s a bug? It can be just configuration issue. Pls show server logs?
Below you can find logs, which appeared by testing datasource ( 1. Connections - Data Sources - Your datasource - Test)
Jan 19 11:27:30 grafana[27894]: logger=tsdb.elasticsearch endpoint=callResource pluginId=elasticsearch dsName=Stasaoo_Test dsUID=PE4E1D10B840E7EAC uname=stanislav.polynkov t=2024-01-19T11:27:30.898016996+03:00 level=error msg="Error received from Elasticsearch" error="Get \"https://os.dev\": 403 Forbidden" status=error duration=12.077755ms stage=databaseRequest resourcePath=
Jan 19 11:27:30 grafana[27894]: logger=context userId=34 orgId=1 uname=stanislav.polynkov t=2024-01-19T11:27:30.898580935+03:00 level=error msg="Internal server error" error="[plugin.downstreamError] client: failed to call resources: Get \"https://os.dev\": 403 Forbidden" remote_addr=x.x.x.x traceID=
Jan 19 11:27:30 grafana[27894]: logger=context userId=34 orgId=1 uname=stanislav.polynkov t=2024-01-19T11:27:30.898643961+03:00 level=error msg="Request Completed" method=GET path=/api/datasources/uid/PE4E1D10B840E7EAC/resources/ status=500 remote_addr=x.x.x.x time_ms=25 duration=25.522481ms size=116 referer=https://grafana.dev/connections/datasources/edit/PE4E1D10B840E7EAC handler=/api/datasources/uid/:uid/resources
Jan 19 11:27:30 grafana[27894]: logger=tsdb.elasticsearch endpoint=callResource pluginId=elasticsearch dsName=Stasaoo_Test dsUID=PE4E1D10B840E7EAC uname=stanislav.polynkov t=2024-01-19T11:27:30.971197525+03:00 level=info msg="Response received from Elasticsearch" statusCode=200 status=ok duration=19.150655ms stage=databaseRequest contentLength= resourcePath=ldap2pg-*2024.01.19/_mapping
Fixed by adding additional permission for my grafana user in ELK in security roles, “cluster:monitor/main”