Client certificate authentication between promtail and loki

[hollomyfoolish]

I want to setup an Loki system (single Loki and single promtail run on the same machine) with enabling client certificate authentication, here is my steps:
Step 1: follow this Securing Grafana Mimir communications with TLS | Grafana Mimir documentation to generate server certificate and client certificate , the server certificate is for loki, and the client certificate is for promtail
Step 2: config loki as below
auth_enabled: false

server:
##http_listen_address: 127.0.0.1
  http_listen_port: 3100
  grpc_listen_port: 9096
  http_tls_config:
    cert_file: /usr/allen/loki/cert/server.crt
    key_file: /usr/allen/loki/cert/server.key
    client_auth_type: RequireAndVerifyClientCert
    client_ca_file: /usr/allen/loki/cert/root.crt

Step 3 start loki
Step 4: config promtail as below:

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https ://localhost:3100/loki/api/v1/push
    tls_config:
      ca_file: /usr/allen/loki/cert/root.crt
      cert_file: /usr/allen/loki/cert/client.crt
      key_file: /usr/allen/loki/cert/client.key
      server_name: localhost

Step 5: after I start promtail, I got below error

level=warn ts=2022-12-21T12:47:42.902570068Z caller=client.go:379 component=client host=localhost:3100 msg=“error sending batch, will retry” status=-1 error=“Post "https ://localhost:3100/loki/api/v1/push": x509: certificate relies on legacy Common Name field, use SANs instead”

Step 6: then I set the insecure_skip_verify to be true in promtail config file:

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https ://localhost:3100/loki/api/v1/push
    tls_config:
      ca_file: /usr/allen/loki/cert/root.crt
      cert_file: /usr/allen/loki/cert/client.crt
      key_file: /usr/allen/loki/cert/client.key
      server_name: localhost
      insecure_skip_verify: true

Step 7: After restart promtail, I got below error

level=warn ts=2022-12-21T12:50:34.649898921Z caller=client.go:379 component=client host=localhost:3100 msg=“error sending batch, will retry” status=-1 error=“Post "https ://localhost:3100/loki/api/v1/push": remote error: tls: bad certificate”

Can someone tell me is there anything wrong with my configuration? How can I make it work? Thanks a lot.

[hollomyfoolish]

Finally, I find the root cause about this issue is that my certificate missed the attribute of subjectAltName, after I add this attribute I can enable mutual authentication between loki and promtail successfully.
Here is the detail certificate generation steps:

• ca

openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 365 -key ca.key -subj "/C=CN/ST=GD/L=SZ/O=Acme, Inc./CN=Acme Root CA" -out ca.crt

• loki

openssl req -newkey rsa:2048 -nodes -keyout server.key -subj "/C=CN/ST=GD/L=SZ/O=Acme, Inc./CN=*.lokiserver.com" -out loki.server.csr
openssl x509 -req -extfile <(printf "subjectAltName=DNS:lokiserver.com,DNS:www.lokiserver.com") -days 1365 -in loki.server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out loki.server.crt

• promtail

openssl req -newkey rsa:2048 -nodes -keyout client.key -subj "/C=CN/ST=GD/L=SZ/O=Acme, Inc./CN=*.promtailclient.com" -out promtail.client.csr
openssl x509 -req -extfile <(printf "subjectAltName=DNS:promtailclient.com,DNS:www.promtailclient.com") -days 1365 -in promtail.client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out promtail.client.crt

And the config

• loki

auth_enabled: false

server:
##http_listen_address: 127.0.0.1
  http_listen_port: 3100
  grpc_listen_port: 9096
  http_tls_config:
    cert_file: /usr/allen/loki/cert/loki.server.crt
    key_file: /usr/allen/loki/cert/server.key
    client_auth_type: RequireAndVerifyClientCert
    client_ca_file: /usr/allen/loki/cert/ca.crt

• promtail

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https://lokiserver.com:3100/loki/api/v1/push
    tls_config:
      ca_file: /usr/allen/loki/cert/ca.crt
      cert_file: /usr/allen/loki/cert/promtail.client.crt
      key_file: /usr/allen/loki/cert/client.key
      server_name: lokiserver.com
      insecure_skip_verify: false