Promtail could not successfully authenticate to Loki

Grafana Loki
,
1

Prerequisites: My test data source at grafan is fine
I deployed the simple scalable model (no tenants configured) where the VALUES about gateway are as follows.

...
gateway:
  enabled: true
  ingress:
    enabled: true
    ingressClassName: "monitoring-nginx"
    annotations: {}
    hosts:
      - host:xxxxx.com
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: xxx-secret
        hosts:
          -xxxxx.com
  basicAuth:
    enabled: true
    # -- The basic auth username for the gateway
    username: xloki
    # -- The basic auth password for the gateway
    password: tW0eFx7uPD
...

The specific service is as follows

root@qcloud-singapore-3-dhysrhw-web-1:~/v1.30/addons/promtail# kubectl  -n  monitoring get svc
NAME                                            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
grafana                                         ClusterIP      192.168.0.73    <none>          80/TCP                       5d2h
loki-backend                                    ClusterIP      192.168.2.152   <none>          3100/TCP,9095/TCP            7m20s
loki-backend-headless                           ClusterIP      None            <none>          3100/TCP,9095/TCP            7m20s
loki-canary                                     ClusterIP      192.168.2.227   <none>          3500/TCP                     7m20s
loki-chunks-cache                               ClusterIP      None            <none>          11211/TCP,9150/TCP           7m20s
loki-gateway                                    ClusterIP      192.168.2.251   <none>          80/TCP                       7m20s
loki-memberlist                                 ClusterIP      None            <none>          7946/TCP                     7m20s
loki-query-scheduler-discovery                  ClusterIP      None            <none>          3100/TCP,9095/TCP            7m20s
loki-read                                       ClusterIP      192.168.1.137   <none>          3100/TCP,9095/TCP            7m20s
loki-read-headless                              ClusterIP      None            <none>          3100/TCP,9095/TCP            7m20s
loki-results-cache                              ClusterIP      None            <none>          11211/TCP,9150/TCP           7m20s
loki-write                                      ClusterIP      192.168.3.94    <none>          3100/TCP,9095/TCP            7m20s
loki-write-headless                             ClusterIP      None            <none>          3100/TCP,9095/TCP            7m20s
monitoring-ingress-nginx-controller             LoadBalancer   192.168.3.59    xxxx            80:30372/TCP,443:31335/TCP   4d22h
monitoring-ingress-nginx-controller-admission   ClusterIP      192.168.3.161   <none>          443/TCP                      4d22h

The values of my promtail are as follows

config:
  clients:
    - url: http://loki-gateway/loki/api/v1/push
      basic_auth:
        username: "xloki"
        password: "tW0eFx7uPD"

But I can check the logs of ds promtail and there is the following error reported. I don’t know what the problem is.

level=error ts=2025-02-11T08:01:36.555565365Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=401 tenant= error="server returned HTTP status 401 Unauthorized (401): <html>"
2

I see you have cert in ingress, shouldn’t your URL be https instead then? You might be able to find better information by looking at logs from ingress controller or gateway container.

Also I hope that is not your real password :wink:

3

My promtail is deployed in the same namespace as loki, so I don’t need to use ingress, the ingress you see is for grafana!

4

I tried the multi-tenant setup again and here is my VALUES file

root@qcloud-singapore-3-dhysrhw-web-1:~/v1.30/addons/loki# cat values.yaml
loki:
  auth_enabled: true
  schemaConfig:
    configs:
      - from: "2024-10-30"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  tenants:
    - name: proj-hd
      password: testpassword
  ingester:
    chunk_encoding: snappy
  tracing:
    enabled: true
  querier:
    max_concurrent: 2
  query_range: 
    align_queries_with_step: true
  storage:
    bucketNames:
      #主存储桶
      chunks: xxxxxxxxxxx
      .....
deploymentMode: SimpleScalable
backend:
  replicas: 2
read:
  replicas: 2
  resources:
    limits:
      cpu: 1
      memory: 1Gi
    requests:
      cpu: 50m
      memory: 100Mi
write:
  replicas: 2
gateway:
  enabled: true
  ingress:
    enabled: true
    ingressClassName: "monitoring-nginx"
    annotations: {}
    hosts:
      - host: xxxxx.com
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: monitoring-ingress-secret
        hosts:
          - xxxxx.com
  basicAuth:
    enabled: true
    htpasswd: >-
      {{ if .Values.loki.tenants }}
        {{- range $t := .Values.loki.tenants }}
      {{ htpasswd (required "All tenants must have a 'name' set" $t.name) (required "All tenants must have a 'password' set" $t.password) }}
        {{- end }}
      {{ else }} {{ htpasswd (required "'gateway.basicAuth.username' is required" .Values.gateway.basicAuth.username) (required "'gateway.basicAuth.password' is required" .Values.gateway.basicAuth.password) }} {{ end }}
....

Above is the yaml file for my helm deployment of grafana loki. Below is the yaml file for my promtail.

root@qcloud-singapore-3-dhysrhw-web-1:~/v1.30/addons/promtail# cat values.yaml 
extraVolumes:
  - name: node-logs
    hostPath:
      path: /var/local
extraVolumeMounts:
  - name: node-logs
    mountPath: /var/local 
config:
  clients:
    # My promtail is deployed in the same namespace as loki, so I don’t need to use ingress
    - url: http://loki-gateway/loki/api/v1/push
      headers:
        X-Scope-OrgID: proj-hd
      basic_auth:
        username: "proj-hd"
        password: "testpassword"
  snippets:
    pipelineStages:
      - cri: {}
    extraScrapeConfigs: |
    ............

But I’m looking at the logs and I’m seeing. Did I write something wrong?

kubectl  -n monitoring  logs daemonsets/promtail  -f  
level=error ts=2025-02-12T06:22:16.306182317Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=401 tenant= error="server returned HTTP status 401 Unauthorized (401): <html>"
5

I found the problem and it was with my promtail chart. It has been solved so far