Authorization Required 401 when send logs from Fluentbit to Loki gateway with ingress and basic-auth

Grafana Loki
1

Hello,

I am using the grafana/loki Helm Chart. Since it is going to be in production I am trying to use Loki gateway ingress with and basic-auth. When I enable ingress and basic auth I get the following error:

[2024/04/16 11:40:03] [error] [output:loki:loki.0] loki-gateway.com:443, HTTP status=401
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.24.0</center>
</body>
</html>

My configuration looks like below:
Loki:

    gateway:
      enabled: true
      replicas: 1
      verboseLogging: true
      ingress:
        enabled: true
        ingressClassName: "nginx"
        annotations: 
          nginx.ingress.kubernetes.io/auth-secret: loki-gateway-tls
          nginx.ingress.kubernetes.io/auth-type: basic
        labels: {}
        hosts:
          - host: loki-gateway.com
            paths:
              - path: /
                pathType: Prefix
        tls:
          - secretName: loki-gateway-tls
            hosts:
              - loki-gateway.com

Secret(loki-gateway-tls):

apiVersion: v1
data:
  auth: dXNlcm5hbWU6JDJ5JDA1JGdvbWR2cUtkQjd4U0UxNFp5ZllhYWVjOE1tWk5WSGFBblRkcnRpcVp4MkpFVG9pY1RlSmlX
kind: Secret
metadata:
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"
  creationTimestamp: "2024-04-15T07:04:25Z"
  name: loki-gateway-tls
  namespace: logs-and-monitoring
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: loki-gateway-tls
type: Opaque

Fluentbit:

        [OUTPUT]
            Name                   loki
            Match                  *
            Host                   loki-gateway.com
            Port                   443
            Http_User              username
            Http_Passwd            password
            TLS                    On
            tls.verify             On
            Labels                 job=fluentbit
            Auto_Kubernetes_Labels on
            Compress               gzip

I tested the ingress with username and password and it works fine:

curl -v -u username:password https://loki-gateway.com:443

What am I missing?

/Angelos

2

I would try to remove special characters from the password first. It is a question if each component handle (de/en-code) them properly. Maybe not and that can be a root cause why correct password is not accepted.

3

Hey @jangaraj,

Thanks for your fast response!
Hmm but I am using only numbers and letters(uppercase and lowercase) when I run the htpasswd.

/Angelos

4

But your secret:

dXNlcm5hbWU6JDJ5JDA1JGdvbWR2cUtkQjd4U0UxNFp5ZllhYWVjOE1tWk5WSGFBblRkcnRpcVp4MkpFVG9pY1RlSmlX

after base64 decode:

username:$2y$05$gomdvqKdB7xSE14ZyfYaaec8MmZNVHaAnTdrtiqZx2JEToicTeJiW
5

Yes the password is “hashed” when I used htpasswd:

htpasswd -B -c ./auth username

Should I not use htpasswd?

1 Like
6

nope, you are right. I don’t know.

7

It seems like in Loki values, if I disable gateway and enable ingress and then use tls/basicauth there it works fine.

However, I don’t know the reason. Is there anyone that knows?