[promtail] basic_auth problem using Kubernetes secret in the Helm chart

[zoltan6]

I’m pulling my hair so some support would be greatly appreciated…

Promtail Helm chart version: 6.8.1.

I define the env vars:

    extraEnv:
      - name: LOKI_USERNAME
        valueFrom:
          secretKeyRef:
            name: loki-secrets
            key: USERNAME
      - name: LOKI_PASSWORD
        valueFrom:
          secretKeyRef:
            name: loki-secrets
            key: PASSWORD

And then this setup doesn’t work (401 Unauthorized):

    config:
      clients:
        - url: https://loki-url/loki/api/v1/push
          basic_auth:
            username: ${LOKI_USERNAME}
            password: ${LOKI_PASSWORD}

But this one does:

    config:
      clients:
        - url: https://loki-url/loki/api/v1/push
          basic_auth:
            username: user
            password: pass@

Yes, the password has special characters in it but I generated the secret in a way that my shell didn’t mess with them. And also, decoding the secret gives the correct values.
The env vars are present in the pods and the generated promtail.yaml looks ok too:

root@promtail-c2rkw:/# env | grep LOKI                   
LOKI_PASSWORD=user
LOKI_USERNAME=pass@

clients:
  - basic_auth:
      password: ${LOKI_PASSWORD}
      username: ${LOKI_USERNAME}
    url: https://loki-url/loki/api/v1/push