Hi all
After upgrading from Grafana 5.0.0 to 5.2.2 I’m unable to connect to our Elasticsearch (version 6.2) datasource. When I configure the datasource and hit the save & test button I’ll get a 502 error back, if I look in the Elasticsearch logs I’ll find a message “Received fatal alert: bad_certificate”. Except from updating Grafana nothing else did change.
Is there anything changed on how Grafana will connect to Elasticsearch in the recent update? Or any suggestions how to fix this?
Thanks in advance.
Regards
Davy
What access mode are you using, server or browser? Have you changed your secret_key in configuration between updates? Are you using alerting feature?
Hi Marcus
Thanks a lot for your reply. In the main time I did some extra testing and found out that this specific issue was introduced between version 5.0.4 and 5.1.0 of Grafana. Thus in version 5.0.4 everything is working while in version 5.1.0 the problem as described above occurs.
I use the Basic Auth mechanism combined with a CA Cert (thus no TLS client Auth). The username, password and CA certificate are not changed in between updates. Further I’m using Server (Default) access and no alerting features.
KR
Davy
That sounds weird. Have you tried to create a complete new ES datasource - does that make any difference?
Looking at the code I cannot see any specific change.
Would it be possible for you to include the json response of your configured datasource http://docs.grafana.org/http_api/data_source/#get-a-single-data-sources-by-id?
Enable debug logging and check if Grafana server log include any interesting information.
I tried adding a new ES datasource but with the same result.
This is the json config of my datasource:
{ "access": "proxy", "basicAuth": true, "basicAuthPassword": "***", "basicAuthUser": "grafana", "database": "[metricbeat-]YYYY.MM.DD", "id": 1, "isDefault": true, "jsonData": { "esVersion": 56, "interval": "Daily", "keepCookies": [], "maxConcurrentShardRequests": 256, "timeField": "@timestamp", "tlsAuth": false, "tlsAuthWithCACert": true }, "name": "Elasticsearch", "orgId": 1, "password": "", "readOnly": true, "secureJsonFields": { "tlsCACert": true }, "type": "elasticsearch", "typeLogoUrl": "", "url": "https://fqdn:9200", "user": "", "version": 17, "withCredentials": false }
And this is what gets logged by Grafana when I hit the save & test button:
t=2018-08-28T13:31:35+0200 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/metricbeat-2018.08.28/_mapping status=502 remote_addr=10.0.2.2 time_ms=38 size=0 referer=http://grafana.staged-by-discourse.com/datasources/edit/1
To be clear this is with Grafana version 5.1.0
How have you setup SSL in ES - using X-Pack? You’re sure that the certificate haven’t expired or you’ve done changes to ES (upgraded for example)?
Quite hard for me to test/verify this. Do you have any good advice/pointers that could help me setup a reproducible test environment?
Think I never seen anyone using only the CA cert without TLS client auth, but according to our code it should work.
I’m 100% sure that there are no certificate nor ES config changes between the Grafana updates. Further we have indeed setup SSL in ES using X-Pack, so I guess if you want a reproducible environment you’ll need a trial license.
I understand it’s quite hard to test/verify for you guys, but it’s also quite hard for me to explain our whole ES setup here online.
For the moment I suggest I will try to connect to our ES using the TLS client auth, if it solves the problem it’s OK for us. I will post my experiences here.
1 Like
Hi @mefraimsson
I did a lot of testing in the mean time, unfortunately without solving the issue. I’ll try to summarise the problem below as detailed as possible and hopefully we could start from there to fix the issue.
Thus, I have a working setup with a Elasticsearch 6.2 (also used ES 5.6 with exact the same results) datasource in combination with Grafana 5.0.4. When I update to Grafana 5.1.0 (aptitude install grafana=5.1.0) and use exact the same values for configuring the Elasticsearch datasource from scratch I’ll get a 502 error when I hit the Save&test button in Grafana and I’ll see a Bad certificate error in the log on the Elasticsearch side. Downgrading Grafana to version 5.0.4 (aptitude install grafana=5.0.4) solves the issue again. The certificate we are talking about is the CA Cert certificate which will look like -----BEGIN CERTIFICATE----- … -----END CERTIFICATE------.
I my opinion there must be changed something in the way Grafana presents the CA Cert to the Elasticsearch node between Grafana version 5.0.4 and 5.1.0. Thus Grafana presents something to Elasticsearch but Elasticsearch does not see that the presented content is a certificate. This has nothing to do with the real content of the certificate if this is the right one or not nor with the fact which authentication mechanism is used. It’s the presented format that isn’t recognized.
If I toggle the xpack setting xpack.security.http.ssl.enabled from true to false on the Elasticsearch side the whole secure connection setup will be bypassed and Grafana 5.1.+ works like a charm.
Can you please let me know with which info or actions I can help you guys so this issue can be solved?
KR
Davy
@mefraimsson I just tested the 5.3.0-beta1 version of Grafana and I can confirm the issue is resolved here.
1 Like
What? That’s really good news but at the same time very weird 
The only possible explanation I can think of is different go versions. Grafana v5.3.0-beta1 was built using go v1.11, Grafana 5.0.4 was built using go v1.9.x and Grafana v5.1.0 was build using go v1.10. Must have been some bug in go v1.10 that was resolved in go v1.11. However the only somewhat related release note I’ve found is this: https://golang.org/doc/go1.11#crypto/x509
