X509: certificate signed by unknown authority with autentication by keycloak

Grafana Authentication
,
1

Hello
I have :

  • a Ubuntu VM with Grafana v8.3.3
  • a Ubuntu VM with keycloak 17.0.0

Everything works (autentication + grafana use) when the parameter
tls_skip_verify_insecure is set to true.
(but of course security is not nice…)
When I set it to false I get an error message (and can’t connect to grafana…) in the log :

lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=0 orgId=0 uname= error=“Post "https://auth.myDomain.net/realms/myRealm/protocol/openid-connect/token\”: x509: certificate signed by unknown authority"

in the grafana.ini I have the following conf :

[auth.generic_oauth]
enabled = true
tls_skip_verify_insecure = true
tls_client_cert = /etc/ssl/certs/grafana/Grafana-SSL.crt
tls_client_key = /etc/ssl/certs/grafana/Grafana-SSL.key
tls_client_ca = /etc/ssl/certs/grafana/CA-myDomain.crt

the two .crt and the .key where localy produced and “Grafana-SSL.crt” is successfuly verify against “CA-myDomain.crt” with openssl

I have try like describe in the forum to add the “CA-myDomain.crt” content in the grafana “/etc/ssl/certs/ca-certificates.crt”

any idea to know wath is the check perform by grafana ? with witch certificate?

Thanks for help

2

1.) Make sure you have right CA cert(s) = you must be able to verify issuer of “HTTPS” certificate used on https://auth.myDomain.net (Keycloak domain). It may be one CA cert, but it can be more - google Chain of Trust - for example Let’s Encrypt uses also intermediate certificate, so additonal CA cert(s) are required to verify also them.

2.) Check Grafana logs for TLS errors - CA cert format may not be in the right format or CA certs may not be in the right order there

3.) Unset tls_client_cert/tls_client_key- I would say they are usefull only for mutual TLS

3

Thanks for the Help
Indeed I did not fully understand the check of the TLS certificate…

Some more information on my configuration :
I have a pfsense that host the letsencrypt certificate for “auth.myDomain.net”
In this pfsense I have a HAproxy that listen in https and forward to the keycloak VM (with SSL).

So which cert will check the option tls_skip_verify_insecure = false ?
=> The cert “grafana” agains the CA myDomain?
=> The letsencrypt certificate send from the Pfsense against the CA local to Grafana?
=> The cert from keycloak against the CA local to Grafana

PS : with your 3rd response it seem’s to mean that I have to comment

tls_client_cert = /etc/ssl/certs/grafana/Grafana-SSL.crt
tls_client_key = /etc/ssl/certs/grafana/Grafana-SSL.key

what about the “tls_client_ca = /etc/ssl/certs/grafana/CA-myDomain.crt” ?
=> and you sugest to comment them because not used in my case ?

4

Are you sure that you installed LE certificate correctly? Make sure that TLS testers, e.g. SSL Server Test (Powered by Qualys SSL Labs) doesn’t detect “certificate chain is incomplete”.

If yes, then you installed LE cert incorrectly = it is not a Keycloak/Grafana config issue, but your pfsence/HAproxy config problem. Their config is out of scope for this forum, but your will find many results letsencrypt chain issues incomplete on the Google, which may help you.

5

Thanks for help,
I have check with SSL server test and it is 100% on certificate notation.
I am confident on the ouside configuration (LE certificate).
My doubt are more on the inside configuration (Auto sign certificate)

And I don’t know exactly how it works between Keycloak and grafana but I think it stays in the local network.

Have you an idea to see which certificate get the error “x509: certificate signed by unknown authority”
a local to grafana VM ?
a one send by keycloak?
Or maybe a one from HA proxy?

6

Sounds similar to what I was getting with JSON datasource except ignore TLS didn’t work. My issue seem to have been Grafana (public server) was using an wildcard public certifcate but the backend (internal server) was using a self-signed cert. When I changed the backend to use the official wildcard cert all was good.

Not sure if that is your isssue or not but I got zero useful help via the community :slight_smile:

7

Thanks for helping
But not sure what you mean by the Grafana (public server) and backend (internal server) ?
I have a grafana server running on a vm in my private network.
The grafana is in my local networks.
I put the CA crt and key on the server with the path fill in grafana.ini

8

OK in my world, the grafana server is publicly available and uses a wildcard domain cert from Digicert. The database, accessed via JSON, is on an internal server that can not seen from the web. When the DB server used a self-signed cert then the error was “Bad Gateway http: proxy error: x509: certificate is valid for Unknown, not FQDN”, I think the Unknown was the self-sign and not a valid FQDN, so I used the DigiCert across both servers. Cos it was JSON I could use curl for the debug.

Unlike your case setting tls_skip_verify_insecure made no difference.

9

Ok in my case I have a pfsense in front of the grafana server that hold the letsencrypt certificate.
And in the local networks I use some self sign certificate…

10

Sounds very similar though, like I said my certs are all for the same domains, no self-signed

11

Hi

i have same issue with cert…

i have public IP within ingress grafana ,when tls check is turned off, login auth grafana against keycloak works,

but how to add my wildcard certificate for backend ?

thanks