Secure loki and promtail via TLS

Grafana Loki
1

I’m trying to establish a secure connection via TLS between my promtail client and loki server. I am unable to figure out how to make this happen. It appears I’m able to get promtail configure to send content via TLS with the below block within the config file. However when I try to configure loki for TLS I’m hitting a road block, and I’m unable to find the documentation stating how to.
Promtail snip it that seems to be working:
clients:

  • url: http://10.0.0.111:3100/loki/api/v1/push
    batchwait: 1s
    batchsize: 102400 # ~100KB
    tls_config:
    ca_file: /etc/loki/certs/myCA.pem
    cert_file: /etc/loki/certs/loki-local.crt
    key_file: /etc/loki/certs/loki-local.key

Loki attempts.
ExecStart=/usr/local/bin/loki -config.file /etc/loki/loki-config.yaml -server.http-tls-ca-path /etc/loki/certs/myCA.pem -server.http-tls-cert-path /etc/loki/certs/loki-local.crt -server.http-tls-key-path /etc/loki/certs/loki-local.key
other various configuration changes within the yaml config, but non have worked. From what I’ve read setting this in systemd is proper, but the error I get is below.
caller=log.go:106 msg=“error running loki” err="error generating http tls config: Client CA’s have been configured without a Client Auth Policy\nerror initialising module: server\ngith>

Any help is appreciated.
Thanks
Zack

1 Like
2

how to run loki over https (TLS) via the configuration settings that go in local-config.yaml …in this example, the endpoint is using 8443:

server:
  http_listen_port: 8443
  http_tls_config: &tls_server_config
    cert_file: /etc/loki/myserver.cert.pem
    key_file: /etc/loki/myserver.key.pem

this is an alternative to using a reverse proxy.
The above is based on the fact that loki uses github.com/weaveworks/common/server/server.go which in turn uses github.com/prometheus/node_exporter/https/tls_config.go

3

Hi all,

I could configure TLS in Promtail and Loki, but now the communication between them does not work.

Configurations for Loki:

loki-distributed:
  loki:
    config: |
      auth_enabled: false
      ...
      server:
        http_listen_port: 3100
        http_tls_config: &tls_server_config
          cert_file: /etc/loki/cert.pem
          key_file: /etc/loki/key.pem

Configurations for Promtail:

clients:
  uri: http://<Redacted>/loki/api/v1/push
  tls_config:
    ca_file: /etc/loki/certs/ca.crt
    cert_file: /etc/loki/certs/cert.pem
    key_file: /etc/loki/certs/key.pem

extraVolumes:
  - name: inf-secret
    secret:
      secretName: obs-secret
      optional: false

extraVolumeMounts:
  - name: inf-secret
    mountPath: "/etc/loki/certs"

I am using secrets to store the certificates and the services can find them. But I am getting the following errors:

Loki Distributed:

ts=2022-08-12T13:52:30.424816874Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve obs-loki-distributed-memberlist: lookup obs-loki-distributed-memberlist on 10.152.183.10:53: no such host"

Promtail:

level=warn ts=2022-08-12T14:00:25.150418634Z caller=client.go:369 component=client host=pt-rdl2617-vm.infinera.com:3101 msg="error sending batch, will retry" status=-1 error="Post \"http://pt-rdl2617-vm.infinera.com:3101/loki/api/v1/push\": dial tcp 10.46.100.17:3101: connect: connection refused"
level=warn ts=2022-08-12T14:01:21.774977915Z caller=client.go:369 component=client host=pt-rdl2617-vm.infinera.com:3101 msg="error sending batch, will retry" status=-1 error="Post \"http://pt-rdl2617-vm.infinera.com:3101/loki/api/v1/push\": dial tcp 10.46.100.17:3101: connect: connection refused"

PS: I am configuring through Helm charts.

4

Hi Zach,

Have you tried the ExecStart parameter -server.http-tls-client-auth=RequestClientCert? according to the weaveworks/weave code in GitHub, these are the possible values:

  • “RequestClientCert”
  • “RequireAnyClientCert”
  • “VerifyClientCertIfGiven”
  • “RequireAndVerifyClientCert”
  • “” or “NoClientCert”

I checked the code here: tls_config.go.
PS: Another useful link https://grafana.com/docs/mimir/latest/operators-guide/securing/securing-communications-with-tls/

5

@ancarvalho it looks like your configuration for promtail specifies that the client uri should use http in lieu of https which is probably the cause of your connection refused in the promtail logs.

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.