Promtail 401 errors when Loki ingress is configured for basic auth

Grafana Loki
1

I am trying to configure basic authentication for Loki, and am running into an issue with the Promtail side.
I am using the Helm charts to deploy Loki on a K8s cluster.
Authentication is configured on the Loki ingress like this:

loki:
  auth_enabled: false
  serviceAccountName: loki 
.
.
.
ingress:
  enabled: true
  ingressClassName: "nginx-default"
  hosts:
    - loki.hostname.com
  annotations: {
    nginx.ingress.kubernetes.io/auth-type: basic,
    nginx.ingress.kubernetes.io/auth-secret: loki-basic-auth,
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
  }

Promtail is configured with the username and password in plain text (ideally this should be pulled from a K8s secret):

config:
  enabled: true
  logLevel: info
  serverPort: 3100
  clients:
    - url: http://loki.hostname.com/loki/api/v1/push
      basic_auth:
        username: loki-user
        password: loki-password

Promtail fails to send logs with the following error:

level=error ts=2023-11-20T13:09:02.857907174Z caller=client.go:430 component=client host=loki.hostname.com msg="final error sending batch" status=401 tenant= error="server returned HTTP status 401 Unauthorized (401): <html>"

Having the same username and password configured for the logcli user works just fine, so I have excluded possible password mismatch issues as the cause.

Any suggestions on how to fix this would be appreciated.

2

I’d double check your endpoint. Looks like your auth is happening on an Nginx proxy, but your promtail is still configured to use port 3100, which most likely is incorrect (unless you purposely configured Nginx to take 3100 as well).

3

Thank you for your reply. As far as I understand the serverPort config in the Promtail values file is there to specify on which port Promtail would listen to receive logs as a server.
The same configuration works just fine when authentication is not enabled on the ingress.

4

I see. In that case I’d test the authentication with a curl command using Loki API. That should tell you whether it’s a problem with promtail or with how authentication is configured on nginx.