Fluentd to Loki mTLS - unable to verify the first certificate

Grafana Loki
,
1

I’m configuring Fluentd to forward logs to Loki using the fluent-plugin-grafana-loki plugin. Loki is exposed via a gateway that is secured with mutual TLS (mTLS). Both the Fluentd client certificate and the Loki server certificate are signed by the same intermediate ClusterIssuer (cert-manager).

However, when Fluentd tries to send logs to Loki, the connection fails with the following error:

client SSL certificate verify error: (21:unable to verify the first certificate) while reading client request headers

Validations I’ve done:

  • I used the same Fluentd client certificate to connect from Grafana to Loki, and it works without issues.
  • The Fluentd and Loki certificates are signed by the same trusted internal CA.
  • The Loki gateway is reachable and responds over HTTPS.

Fluentd Config Snippet:

<match py_app>
      <match py_app>
        @type loki
        url "https://loki-gateway"
        extra_labels {"logger_name": "myservice"}
        cert /opt/bitnami/fluentd/certs/client/tls.crt
        key /opt/bitnami/fluentd/certs/client/tls.key
        ca_cert /opt/bitnami/fluentd/certs/client/ca.crt
</match>

Any guidance, examples, or suggestions from someone who’s configured Fluentd with mTLS to Loki would be super helpful!

2

How response from the command openssl s_client -showcerts -connect loki-gateway:443 looks like and which CAs are in the /opt/bitnami/fluentd/certs/client/ca.crt?

3

Output of the command. (I have removed sensitive info and replaced with <>)

#openssl s_client -showcerts -connect loki-gateway:443
CONNECTED(00000003)
Can't use SSL_get_servername
verify error:num=20:unable to get local issuer certificate
verify return:1
verify return:1
---
Certificate chain
 0 s:<>
   i:<> CN = intermediate-ca
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 23 17:58:34 2025 GMT; NotAfter: Apr 23 17:58:34 2027 GMT
-----BEGIN CERTIFICATE-----
<>
-----END CERTIFICATE-----
 1 s:<> CN = intermediate-ca
   i:<> CN = top-level-ca
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 23 14:59:37 2025 GMT; NotAfter: Apr 21 14:59:37 2035 GMT
-----BEGIN CERTIFICATE-----
<>
-----END CERTIFICATE-----
---
Server certificate
<>
<> CN = intermediate-ca
---
Acceptable client certificate CA names
<> CN = top-level-ca
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3012 bytes and written 402 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---
4007193D017F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:322

CA in /opt/bitnami/fluentd/certs/client/ca.crt is top-level-ca

1 Like
4

ok, so is

openssl s_client -showcerts -connect loki-gateway:443 -CAfile /opt/bitnami/fluentd/certs/client/ca.crt

successful?

5

Yes, the output suggests its successful:

    Start Time: 1745520923
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes

I can see an error message before command exits though
40C7C4B5197F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:322:

6

OK, so next level is to create mTLS, not just TLS - -cert -key parameters

7

The mTLS command also successful

openssl s_client -showcerts -connect loki-gateway:443 -CAfile /opt/bitnami/fluentd/certs/client/ca.crt -cert /opt/bitnami/fluentd/certs/client/tls.crt -key /opt/bitnami/fluentd/certs/client/tls.key

    Start Time: 1745522357
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
8

Dunno. Permissions (e.g. fluentd user can’t read those cert files, …), old fluentd version, cert restriction (e. g. keyCertSign usage is missing), …
You declared your certs as sensitive, so only you know.