Can't connect to Loki datasource

Grafana
, ,
1

I have installed loki and grafana in the monitoring namespace using helm charts. I’ve included the values.yaml I used for them.
I am trying to add the Loki datasource using the Grafana UI, however I keep getting an error.

Loki values.yaml

deploymentMode: SingleBinary
loki:
  commonConfig:
    replication_factor: 2
  schemaConfig:
    configs:
    - from: "2024-01-01"
      store: tsdb
      index:
        prefix: loki_index_
        period: 24h
      object_store: s3
      schema: v13
  storage:
    type: 's3'
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin
    s3:
      endpoint: <endpoint>
      region: <region>
      accessKeyId: <access-key>
      secretAccessKey: <secret>
      s3ForcePathStyle: false
      insecure: false
test:
  enabled: false
singleBinary:
  replicas: 2
  persistence:
    size: 20Gi
    storageClass: alicloud-disk-essd
lokiCanary:
  enabled: false
gateway:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    labels: {}
    hosts:
      - host: loki.example.com
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: loki-gateway-tls
        hosts:
          - loki.example.com
  basicAuth:
    enabled: true
    existingSecret: loki-gateway-ingress-basic-auth
read:
  replicas: 0
backend:
  replicas: 0
write:
  replicas: 0

Grafana values.yaml

replicas: 2

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  path: /
  pathType: Prefix

  hosts:
    - grafana.example.com
  tls:
   - secretName: grafana-tls
     hosts:
       - grafana.example.com

persistence:
  type: pvc
  enabled: true
  storageClassName: alicloud-disk-essd
  accessModes:
    - ReadWriteOnce
  size: 20Gi

admin:
  existingSecret: grafana-admin-creds
  userKey: admin-user
  passwordKey: admin-password

Grafan UI

Screenshot from 2024-10-07 14-55-35-redacted
Screenshot from 2024-10-07 14-55-35-redacted1936×1906 222 KB

Loki Gateway logs

192.168.0.62 - lokiuser [07/Oct/2024:07:43:57 +0000]  401 "GET /loki/api/v1/query?direction=backward&query=vector%281%29%2Bvector%281%29&time=4000000000 HTTP/1.1" 10 "-" "Grafana/11.2.1" "-"

If I try to fetch /loki/api/v1/status/buildinfo from my laptop or the grafana Pod I get a 200 response but 401 response if I try to add the loki datasource from the UI

Grafana Pod shell

grafana-7cd7f5c57d-fvwgr:/usr/share/grafana$ curl -u lokiuser:lokipasswd http://loki-gateway/loki/api/v1/status/buildinfo
{"version":"release-3.1.x-89fe788","revision":"89fe788d","branch":"release-3.1.x","buildUser":"root@buildkitsandbox","buildDate":"2024-08-08T14:25:46Z","goVersion":""}

Port Forwarding Loki gateway to localhost 3100

curl -u lokiuser:lokipasswd http://localhost:3100/loki/api/v1/status/buildinfo
{"version":"release-3.1.x-89fe788","revision":"89fe788d","branch":"release-3.1.x","buildUser":"root@buildkitsandbox","buildDate":"2024-08-08T14:25:46Z","goVersion":""}

Did I miss some configuration?

2

Hi @delwinj :slight_smile:
I am using different deployment modes other than SingleBinary, however I think the default listen port should still be 3100:
image

In Grafana:

image

Have you considered adding the port number to the URL?

3

I tried that too, but the loki-gateway service uses port 80.
I have also tried using http://loki-gateway.monitoring.svc.cluster.local/ and https://loki.example.com/ both of which use port 80
Also, when I use curl to fetch the status, port 80 works fine

4

While trying to post this question on stackoverflow I found a duplicate post that solved my issue (Unable to add Grafana Loki datasource in Kubernetes - Stack Overflow). The Helm chart deployed Loki by default in multi-tenant mode since I’m using basic auth. I needed to add a custom HTTP header X-Scope-OrgID, didn’t need to change anything else.

1 Like
5

Found the documentation that it defaults to multi-tenancy :+1:

I was using “auth_enabled: false”, therefore the X-Scope-OrgID header was not required.

1 Like