Unable to connect to OpenLDAP 2.5 via SSL/TLS - Connection error EOF

Grafana Authentication
,
1
  • What Grafana version and what operating system are you using?

Official Grafana docker image (grafana:11.2.0) on Host Ubuntu 22.04

  • What are you trying to achieve?

Connecting Grafana to LDAPS on port 636, instead of 389 with StartTLS

  • How are you trying to achieve it?

Reconfiguration of ldap.toml to use our new LDAP host and SSL relevant options.

LDAP means openLDAP (slapd with GnuTLS) 2.5.18 on Ubuntu 22.04.

Every other tool (like Jenkins, Artifactory etc.) can connect to LDAPS without issues, Grafana is the only tool which is unable.

  • What happened?

When trying to login (UI), this error appears:

Login failed

Invalid username or password

In the logfile you’ll find this:


Unauthorized error="[password-auth.failed] failed to authenticate identity: unable to create LDAP client

LDAP with start_tls=true on port 389 is working. Only LDAPS on 636 doesn’t.

  • What did you expect to happen?

Connection to LDAPS on port 636 should work, Login should be possible.

  • Can you copy/paste the configuration(s) that you are having problems with?

grafana.ini


[auth.ldap]

enabled = true

config_file = /etc/grafana/ldap.toml

allow_sign_up = true

ldap.toml


host = "<hostname>"

port = 636

use_ssl = true

start_tls = false

ssl_skip_verify = false

root_ca_cert = "/etc/grafana/ssl/internalCA.pem"

Also tried with additional options after trying to connect with gnutls-cli, also no success.


tls_ciphers = "TLS_AES_256_GCM_SHA384"

min_tls_version = "TLS1.2"

OpenLDAP relevant configuration:


olcTLSCACertificateFile: /etc/ssl/certs/internalCA.pem

olcTLSCertificateFile: /etc/ssl/openldap.crt

olcTLSCertificateKeyFile: /etc/ssl/openldap.key

olcTLSDHParamFile: /etc/ssl/dhparams

olcTLSCipherSuite: NORMAL

olcTLSProtocolMin: 3.3

Also tried olcTLSCipherSuite: LEGACY which had no effect.

2

Enable debug logging and then check debug logs.

To enable debug logging edit the configuration file grafana.ini:

[log]
# Either "console", "file", "syslog". Default is console and file
# Use space to separate multiple modes, e.g. "console file"
mode = console file

# Either "debug", "info", "warn", "error", "critical", default is "info"
level = debug

Then restart grafana for the setting to go into effect.

3

Screenshot 2024-10-15 100014
Screenshot 2024-10-15 1000141426Ă—303 73.1 KB

4 
logger=authn.password t=2024-10-15T08:04:22.210755896Z level=debug msg="Failed to authenticate password identity" client=ldap error=EOF
logger=authn.password t=2024-10-15T08:04:22.223910195Z level=debug msg="Failed to authenticate password identity" client=grafana error="[password-auth.invalid] invalid password"
logger=authn.service t=2024-10-15T08:04:22.227934089Z level=info msg="Failed to authenticate request" client=auth.client.form error="[password-auth.failed] failed to authenticate identity: EOF\n[password-auth.invalid] invalid password"
logger=context userId=0 orgId=0 uname= t=2024-10-15T08:04:22.228139647Z level=info msg=Unauthorized error="[password-auth.failed] failed to authenticate identity: EOF\n[password-auth.invalid] invalid password" remote_addr=xxx.xxx.xxx.xxx traceID=
5

Don’t check only logs, which contain “ldap” string. There can be meaningful error without ldap string, so check all logs, not just some.

6

Unfortunately there is not much more to find in the log

logger=ngalert.scheduler rule_uid=ae851867-8ac5-49f2-9af9-1d96abf1b039 org_id=1 version=1 fingerprint=1083d905f7e5681e now=2024-10-15T07:59:40Z t=2024-10-15T07:59:42.015330673Z level=debug msg="Tick processed" attempt=1 duration=10.455701ms
logger=authn.password t=2024-10-15T07:59:42.337236035Z level=debug msg="Failed to authenticate password identity" client=ldap error=EOF
logger=authn.password t=2024-10-15T07:59:42.350435051Z level=debug msg="Failed to authenticate password identity" client=grafana error="[password-auth.invalid] invalid password"
logger=authn.service t=2024-10-15T07:59:42.354751116Z level=info msg="Failed to authenticate request" client=auth.client.form error="[password-auth.failed] failed to authenticate identity: EOF\n[password-auth.invalid] invalid password"
logger=context userId=0 orgId=0 uname= t=2024-10-15T07:59:42.354875608Z level=info msg=Unauthorized error="[password-auth.failed] failed to authenticate identity: EOF\n[password-auth.invalid] invalid password" remote_addr=xxx.xxx.xxx.xxx traceID=logger=context userId=0 orgId=0 uname= t=2024-10-15T07:59:42.354948997Z level=info msg="Request Completed" method=POST path=/login status=401 remote_addr=104.155.23.55 time_ms=26 duration=26.128656ms size=107 referer=https://grafana.domain.io/login handler=/login status_source=server
logger=ngalert.scheduler rule_uid=af3b7999-a3d4-46cd-b6e3-758f83334179 org_id=1 version=1 fingerprint=4aa2397b3faa60f2 now=2024-10-15T07:59:40Z t=2024-10-15T07:59:42.504568639Z level=debug msg="Processing tick"
logger=datasources rule_uid=af3b7999-a3d4-46cd-b6e3-758f83334179 org_id=1 t=2024-10-15T07:59:42.50479957Z level=debug msg="Querying for data source via SQL store" uid=000000004 orgId=1
logger=ngalert.eval rule_uid=af3b7999-a3d4-46cd-b6e3-758f83334179 org_id=1 t=2024-10-15T07:59:42.505882326Z level=debug msg="Executing pipeline" commands=classic_condition datasources=prometheus
logger=secrets.kvstore t=2024-10-15T07:59:42.506026789Z level=debug msg="got secret value from cache" orgId=1 type=datasource namespace=prod
7

meanwhile updated to Grafana 11.2.2, issue still occurs

8

bump
someone got an idea?

9

I did setup a similar sandbox environment as we have in production
1 VM Ubuntu 22.04 with OpenLDAP 2.5.18 (slapd with GnuTLS)
1 VM Ubuntu 22.04 with Grafana 11.2.2 as Docker container

Connection via LDAP / 389 works perfectly. As soon as I switch to LDAPS / 636 its broken.
Debug log is not very helpful. Thats everything from the moment, I press on login.

logger=ngalert.scheduler t=2024-10-23T10:06:30.001023705Z level=debug msg="Alert rules fetched" rulesCount=0 foldersCount=0 updatedRules=0
logger=authn.password t=2024-10-23T10:06:34.16168956Z level=debug msg="Failed to authenticate password identity" client=ldap error=EOF
logger=authn.password t=2024-10-23T10:06:34.16861171Z level=debug msg="Failed to authenticate password identity" client=grafana error="[password-auth.invalid] invalid password"
logger=authn.service t=2024-10-23T10:06:34.173517425Z level=info msg="Failed to authenticate request" client=auth.client.form error="[password-auth.failed] failed to authenticate identity: EOF\n[password-auth.invalid] invalid password"
logger=context userId=0 orgId=0 uname= t=2024-10-23T10:06:34.173683023Z level=info msg=Unauthorized error="[password-auth.failed] failed to authenticate identity: EOF\n[password-auth.invalid] invalid password" remote_addr=104.155.23.55 traceID=
logger=context userId=0 orgId=0 uname= t=2024-10-23T10:06:34.173745675Z level=info msg="Request Completed" method=POST path=/login status=401 remote_addr=xxx.xxx.xxx.xxx time_ms=19 duration=19.89487ms size=107 referer=https://xxx.xxx.xxx.xxx/login handler=/login status_source=server

ldap.toml:

[[servers]]

# Ldap server host (specify multiple hosts space separated)
host = "openldap-test.internal"

# Default port is 389 or 636 if use_ssl = true
port = 636

# Set to true if ldap server supports TLS
use_ssl = true

# Set to true if connect ldap server with STARTTLS pattern (create connection in insecure, then upgrade to secure connection with TLS)
start_tls = false

# set to true if you want to skip ssl cert validation
ssl_skip_verify = false

# set to the path to your root CA certificate or leave unset to use system defaults
root_ca_cert = "/etc/grafana/ssl/internalCA.pem"

Also the UI is showing a connection error on the LDAP view (/admin/authentication/ldap):

Connection error
openldap-test.internal:636
EOF

10

Test TCP connectivity, e.g. telnet openldap-test.internal 636
Test TLS connectivity, e.g. openssl s_client -connect openldap-test.internal:636.
Test TLS on that ldap server, e.g. testssl.sh ... - prove/test that configured cipher is supported, that configured CA cert is correct, that certs are valid, …

Of course test it from the Grafana container, not from the host. Of course if test utilities are not there, then install them, …

You have many moving gears in the infra, but no proves that they are moving correctly - verify them first.

11

telnet is working.

openssl “censored” output:

CONNECTED(00000003)
depth=1 C = DE, ST = Berlin, L = Berlin, O = <Company>, CN = <Company> RSA Certification Authority
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 C = DE, ST = Berlin, L = Berlin, O = <Company>, CN = <Company> RSA Certification Authority
verify return:1
depth=0 C = DE, ST = Berlin, L = Berlin, O = <Company>, OU = IT, CN = openldap-test.internal
verify return:1
---
Certificate chain
 0 s:C = DE, ST = Berlin, L = Berlin, O = <Company>, OU = IT, CN = openldap-test.internal
   i:C = DE, ST = Berlin, L = Berlin, O = <Company>, CN = <Company> RSA Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct 22 13:10:07 2024 GMT; NotAfter: Nov 21 13:10:07 2024 GMT
 1 s:C = DE, ST = Berlin, L = Berlin, O = <Company>, CN = <Company> RSA Certification Authority
   i:C = DE, ST = Berlin, L = Berlin, O = <Company>, CN = <Company> RSA Certification Authority
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA384
   v:NotBefore: Oct 20 13:23:52 2017 GMT; NotAfter: Oct 18 13:23:52 2027 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
<...>
-----END CERTIFICATE-----
subject=C = DE, ST = Berlin, L = Berlin, O = <Company>, OU = IT, CN = openldap-test.internal
issuer=C = DE, ST = Berlin, L = Berlin, O = <Company>, CN = <Company> RSA Certification Authority
---
No client certificate CA names sent
---
SSL handshake has read 2934 bytes and written 651 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.2, Cipher is AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-GCM-SHA384
    Session-ID: E5F743EF73ACB1F620C5C4E4943BC9ADC0C2092C0CA5FC6A52293A3ED8753E0A
    Session-ID-ctx:
    Master-Key: 967142E1CAC1665174A35D6FC4F14A886232F3720B778F54B88B141EC671780D6F582601BC6BBDCFF7CD1C96363CA5AD
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1729686237
    Timeout   : 7200 (sec)
    Verify return code: 19 (self-signed certificate in certificate chain)
    Extended master secret: yes
---

and testssl:

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      offered (deprecated)
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    not offered and downgraded to a weaker protocol
 NPN/SPDY   not offered
 ALPN/HTTP2 not offered

 Testing robust (perfect) forward secrecy, (P)FS -- omitting Null Authentication/Encryption, 3DES, RC4

 No ciphers supporting Forward Secrecy offered

 Testing server preferences

 Has server cipher order?     no (NOT ok)
 Negotiated protocol          TLSv1.2
 Negotiated cipher            AES128-GCM-SHA256 -- inconclusive test, matching cipher in list missing, better see below
 Negotiated cipher per proto  (matching cipher in list missing)
     AES256-SHA:                    TLSv1, TLSv1.1
     AES256-GCM-SHA384:             TLSv1.2
 No further cipher order check has been done as order is determined by the client


 Testing server defaults (Server Hello)

 TLS extensions (standard)    "renegotiation info/#65281" "encrypt-then-mac/#22" "extended master secret/#23" "max fragment length/#1"
 Session Ticket RFC 5077 hint no -- no lifetime advertised
 SSL Session ID support       yes
 Session Resumption           Tickets no, ID: no
 TLS clock skew               Random values, no fingerprinting possible
 Signature Algorithm          SHA256 with RSA
 Server key size              RSA 2048 bits
 Server key usage             Key Encipherment, Data Encipherment
 Server extended key usage    TLS Web Server Authentication
 Serial                       xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (OK: length 20)
 Fingerprints                 SHA1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                              SHA256 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 Common Name (CN)             openldap-test.internal
 subjectAltName (SAN)         missing -- no SAN is deprecated
 Issuer                       Company RSA Certification Authority
 Trust (hostname)             certificate does not match supplied URI (same w/o SNI)
 Chain of trust               NOT ok (self signed CA in chain)
 EV cert (experimental)       no
 ETS/"eTLS", visibility info  not present
 Certificate Validity (UTC)   expires < 30 days (29) (2024-10-22 13:10 --> 2024-11-21 13:10)
 # of certificates provided   2
 Certificate Revocation List  --
 OCSP URI                     --
                              NOT ok -- neither CRL nor OCSP URI provided
 OCSP stapling                not offered
 OCSP must staple extension   --
 DNS CAA RR (experimental)    not offered
 Certificate Transparency     N/A

 Testing 370 ciphers via OpenSSL plus sockets against the server, ordered by encryption strength

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256      TLS_RSA_WITH_AES_256_GCM_SHA384
 xc09d   AES256-CCM                        RSA        AESCCM      256      TLS_RSA_WITH_AES_256_CCM
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 xc09c   AES128-CCM                        RSA        AESCCM      128      TLS_RSA_WITH_AES_128_CCM
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128      TLS_RSA_WITH_AES_128_GCM_SHA256
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
12

Yeah, but you must to create valid TLS connection with your /etc/grafana/ssl/internalCA.pem because you configured that. Of course test if from the Grafana container under Grafana user to simulate Grafana’s case (e.g. problem with cert permissions, …)

13

looks also good to me:

grafana@493ac6dd0edf:/usr/share/grafana$ openssl s_client -CAfile /etc/ssl/certs/internalCA.pem openldap-test.internal:636
CONNECTED(00000003)
depth=1 C = DE, ST = Berlin, L = Berlin, O = <company>, CN = internal RSA Certification Authority
verify return:1
depth=0 C = DE, ST = Berlin, L = Berlin, O = <company>, OU = IT, CN = openldap-test.internal
verify return:1
---
Certificate chain
 0 s:C = DE, ST = Berlin, L = Berlin, O = <company>, OU = IT, CN = openldap-test.internal
   i:C = DE, ST = Berlin, L = Berlin, O = <company>, CN = internal RSA Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct 22 13:10:07 2024 GMT; NotAfter: Nov 21 13:10:07 2024 GMT
 1 s:C = DE, ST = Berlin, L = Berlin, O = <company>, CN = internal RSA Certification Authority
   i:C = DE, ST = Berlin, L = Berlin, O = <company>, CN = internal RSA Certification Authority
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA384
   v:NotBefore: Oct 20 13:23:52 2017 GMT; NotAfter: Oct 18 13:23:52 2027 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
#
#
#
#
#
-----END CERTIFICATE-----
subject=C = DE, ST = Berlin, L = Berlin, O = <company>, OU = IT, CN = openldap-test.internal
issuer=C = DE, ST = Berlin, L = Berlin, O = <company>, CN = internal RSA Certification Authority
---
No client certificate CA names sent
---
SSL handshake has read 2934 bytes and written 651 bytes
Verification: OK
---
New, TLSv1.2, Cipher is AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-GCM-SHA384
    Session-ID: CBEA95F6195732AF8AC58E0B6B26A0DE517FF7775AF5E599870D5B7CA6C87D94
    Session-ID-ctx:
    Master-Key: 69FEF041E144BA3B82A182F84A95666FC80A84B173FA47E4D5EF8ECB5DA5E3E287D4ABB2357B5D775E6DF454E9F93287
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1729748842
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
1 Like
14

1.) don’t configure any specific cipher, tls version on the Grafana side, try ssl_skip_verify=true
2.) enable otel tracing in the Grafana and inspect traces - there can be more details about issue
3.) dig into source code and find where it’s failing and why

15

I just found this comment on Configure LDAP authentication | Grafana documentation

# Starting with Grafana v11.0 only ciphers with ECDHE support are accepted for TLS 1.2 connections.

I assume thats the issue, as the LDAPS server doesn’t provide any ECDHE ciphers.

Already tried to use GODEBUG="tlsrsakex=1" in Grafana container as mentioned here: Auth: LDAP failing TLS handshake · Issue #89828 · grafana/grafana · GitHub - but this also didn’t had any effect.

16

Make sure you have EC ciphers on the ldap side available. EC is better than RSA from the cryptography point of view, so it looks like Golang is pushing users to be more secure. Or enable tls 1.3, where you don’t have cipher configuration. Generaly, make your tls better.

17

Hello again,

after a couple of days and further testing, we’re able to find a solution.

We tried to play around with OpenLDAPs default implementation of GnuTLS, but we could not change the offered ciphers in a way, that EC ciphers will be available.

I then started to compile OpenLDAP myself, but using OpenSSL implementation instead of GnuTLS - and look there … we got a useful error message:

client=auth.client.form error="[password-auth.failed] failed to authenticate identity: tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead\n[password-auth.invalid] invalid password"

After adding the missing SANs to the certificate, the connection between Grafana and OpenLDAP works like a charm. TLS 1.2 uses ECDHE-RSA-AES256-GCM-SHA384 and even TLS 1.3 is working with TLS_AES_256_GCM_SHA384.

We then switched back to OpenLDAP with GnuTLS and added the updated certificate with SANs to OpenLDAP and see: It’s still not working - I guess because of missing EC cipher support.

BUT I remembered the environment variable GODEBUG: "tlsrsakex=1" and tried it once again. And then the connection is working between Grafana and OpenLDAP (with GnuTLS).

So final result:

Grafana 11.3 → OpenLDAP 2.6 (GnuTLS) only works with GODEBUG: "tlsrsakex=1"
Grafana 11.3 → OpenLDAP 2.6 (OpenSSL) works like a charm

So basically the “root cause” of this issue were the missing SANs in the certificate
and maybe even the implementation of GnuTLS a bit.

May this will help others.
Thanks for your support.