Grafana is unable to connect with Loki

Grafana Loki
1

I’m running kube-prometheus-stack, loki, and promtail and I’m having trouble connecting Loki as a Grafana datasource. I get the error message “Unable to connect with Loki. Please check the server logs for more details.”

I’ve shelled into the grafana pod and I can curl http://loki-gateway.monitoring.svc.cluster.local/ no problem. Funny thing is, I get 404s when I try to hit some endpoints like /ready, but others work totally fine like /loki/api/v1/status/buildinfo. This issue seems similar to Can't connect Loki to Grafana in k8s · Issue #13179 · grafana/loki · GitHub.

Any idea what could be going on? Have I misconfigured something? Any help you can provide is much appreciated.

I’m running the latest Loki chart (6.x), using the single binary deployment mode, and using the loki gateway. Here are my values.

deploymentMode: SingleBinary
loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: "filesystem"
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        index:
          prefix: loki_index_
          period: 24h
        object_store: filesystem
        schema: v13
singleBinary:
  replicas: 1
read:
  replicas: 0
backend:
  replicas: 0
write:
  replicas: 0

Here’s my Grafana datasource:

grafana:
  additionalDataSources:
    - name: Loki
      access: proxy
      isDefault: false
      orgId: 1
      type: loki
      url: http://loki-gateway.monitoring.svc.cluster.local/
2

Looking through the Loki auth docs, I came across the requirement to pass the org ID in X-Scope-OrgID header, Multi-tenancy | Grafana Loki documentation. This is required if multi-tenancy is enabled, which it is by default in the helm chart. I wonder if this is causing the issue. I’ll give it a try.

3

Yep, the missing X-Scope-OrgID header was the issue. Here’s the fix. I added the header in jsonData and the value in secureJsonData.

  grafana:
    additionalDataSources:
      - name: Loki
        access: proxy
        isDefault: false
        jsonData:
          httpHeaderName1: X-Scope-OrgID
        secureJsonData:
          httpHeaderValue1: 1
        orgId: 1
        type: loki
        url: http://loki-gateway.monitoring.svc.cluster.local/

You can also just disable multi-tenancy as well with auth_enabled: false.

deploymentMode: SingleBinary
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: "filesystem"
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        index:
          prefix: loki_index_
          period: 24h
        object_store: filesystem
        schema: v13
singleBinary:
  replicas: 1
read:
  replicas: 0
backend:
  replicas: 0
write:
  replicas: 0