Loki: read-only account

Grafana Loki
1

Our use case: One account with admin access and second account with read-only access to query logs.

When loki.auth_enabled: true and basicAuth.enabled are set, with a user (e.g., admin) and password configured, logs can be separated using the X-Scope-OrgID header. With the admin account, you can push logs using different X-Scope-OrgID values, and those logs will be isolated accordingly.

However, once we added loki.tenants.{list-of-users} and user/password removed from basicAuth.enabled, the X-Scope-OrgID header is ignored. As a result, regardless of which tenant pushed the logs, they become visible to the user who pushed them, and tenant separation no longer applies.

We pushed logs to two different X-Scope-OrgID values: dc1 and dc2. However, when querying logs from dc1, it shows logs from both dc1 and dc2.

curl -X POST \
  -u "${ADMIN_USER}:${ADMIN_PASS}" \
  -H "Content-Type: application/json" \
  -H "X-Scope-OrgID: dc1" \
  "${LOKI_URL}/loki/api/v1/push" \
  -d '{
    "streams": [
      {
        "stream": {
          "job": "curl-test",
          "instance": "localhost",
          "level": "info",
          "service": "web-app"
        },
        "values": [
          ["'$(date +%s%N)'", "DC1: User login successful - user_id=12345, ip=192.168.1.100 - DC1"],
          ["'$(($(date +%s%N) + 1000000))'", "DC1: Database query executed - query_time=45ms, table=users"]
        ]
      },
      {
        "stream": {
          "job": "curl-test",
          "instance": "localhost",
          "level": "error",
          "service": "web-app"
        },
        "values": [
          ["'$(($(date +%s%N) + 2000000))'", "ERROR: Failed to connect to payment gateway - timeout after 30s"]
        ]
      }
    ]
  }'
---
curl -X POST \
  -u "${ADMIN_USER}:${ADMIN_PASS}" \
  -H "Content-Type: application/json" \
  -H "X-Scope-OrgID: dc2" \
  "${LOKI_URL}/loki/api/v1/push" \
  -d '{
    "streams": [
      {
        "stream": {
          "job": "curl-test",
          "instance": "localhost",
          "level": "info",
          "service": "web-app"
        },
        "values": [
          ["'$(date +%s%N)'", "DC2: User login successful - user_id=12345, ip=192.168.1.100 - DC2"],
          ["'$(($(date +%s%N) + 1000000))'", "DC2: Database query executed - query_time=45ms, table=users"]
        ]
      },
      {
        "stream": {
          "job": "curl-test",
          "instance": "localhost",
          "level": "error",
          "service": "web-app"
        },
        "values": [
          ["'$(($(date +%s%N) + 2000000))'", "ERROR: Failed to connect to payment gateway - timeout after 30s"]
        ]
      }
    ]
  }'
---
curl -G \
  -u "${ADMIN_USER}:${ADMIN_PASS}" \
  -H "X-Scope-OrgID: dc1" \
  "${LOKI_URL}/loki/api/v1/query_range" \
  --data-urlencode 'query={job="curl-test"}' \
  --data-urlencode 'start='$(($(date +%s) - 3600)) \
  --data-urlencode 'end='$(date +%s) \
  --data-urlencode 'limit=100'
{"status":"success","data":{"resultType":"streams","result":[{"stream":{"detected_level":"info","instance":"localhost","job":"curl-test","level":"info","service":"web-app","service_name":"web-app"},"values":[["1755860467526188249","DC2: Database query executed - query_time=45ms, table=users"],["1755860467524190100","DC2: User login successful - user_id=12345, ip=192.168.1.100 - DC2"],["1755860449487778409","DC1: Database query executed - query_time=45ms, table=users"],["1755860449485766180","DC1: User login successful - user_id=12345, ip=192.168.1.100 - DC1"],["1755860026592361995","INFO: Database query executed - query_time=45ms, table=users"],["1755860026590401377","INFO: User login successful - user_id=12345, ip=192.168.1.100-readonly"],["1755860006162838316","INFO: Database query executed - query_time=45ms, table=users"],["1755860006159350985","INFO: User login successful - user_id=12345, ip=192.168.1.100"],["1755858282089758518","INFO: Database query executed - query_time=45ms, table=users"],["1755858282086312820","INFO: User login successful - user_id=12345, ip=192.168.1.100"]]},{"stream":{"detected_level":"error","instance":"localhost","job":"curl-test","level":"error","service":"web-app","service_name":"web-app"},"values":[["1755860467528158039","ERROR: Failed to connect to payment gateway - timeout after 30s"],["1755860449489722964","ERROR: Failed to connect to payment gateway - timeout after 30s"],["1755860026594362093","ERROR: Failed to connect to payment gateway - timeout after 30s"],["1755860006166398737","ERROR: Failed to connect to payment gateway - timeout after 30s"],["1755858282093245841","ERROR: Failed to connect to payment gateway - timeout after 30s"]]}],"stats":{"summary":{"bytesProcessedPerSecond":25445,"linesProcessedPerSecond":361,"totalBytesProcessed":1057,"totalLinesProcessed":15,"execTime":0.04154,"queueTime":0.001987,"subqueries":0,"totalEntriesReturned":15,"splits":5,"shards":5,"totalPostFilterLines":15,"totalStructuredMetadataBytesProcessed":120},"querier":{"store":{"totalChunksRef":0,"totalChunksDownloaded":0,"chunksDownloadTime":0,"queryReferencedStructuredMetadata":false,"chunk":{"headChunkBytes":0,"headChunkLines":0,"decompressedBytes":0,"decompressedLines":0,"compressedBytes":0,"totalDuplicates":0,"postFilterLines":0,"headChunkStructuredMetadataBytes":0,"decompressedStructuredMetadataBytes":0},"chunkRefsFetchTime":7842832,"congestionControlLatency":0,"pipelineWrapperFilteredLines":0}},"ingester":{"totalReached":10,"totalChunksMatched":8,"totalBatches":13,"totalLinesSent":15,"store":{"totalChunksRef":0,"totalChunksDownloaded":0,"chunksDownloadTime":0,"queryReferencedStructuredMetadata":false,"chunk":{"headChunkBytes":1057,"headChunkLines":15,"decompressedBytes":0,"decompressedLines":0,"compressedBytes":0,"totalDuplicates":0,"postFilterLines":15,"headChunkStructuredMetadataBytes":120,"decompressedStructuredMetadataBytes":0},"chunkRefsFetchTime":0,"congestionControlLatency":0,"pipelineWrapperFilteredLines":0}},"cache":{"chunk":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0,"queryLengthServed":0},"index":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0,"queryLengthServed":0},"result":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0,"queryLengthServed":0},"statsResult":{"entriesFound":1,"entriesRequested":1,"entriesStored":1,"bytesReceived":137,"bytesSent":0,"requests":2,"downloadTime":2256913,"queryLengthServed":0},"volumeResult":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0,"queryLengthServed":0},"seriesResult":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0,"queryLengthServed":0},"labelResult":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0,"queryLengthServed":0},"instantMetricResult":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0,"queryLengthServed":0}},"index":{"totalChunks":0,"postFilterChunks":0,"shardsDuration":0,"usedBloomFilters":false}}}}

You expect to see logs based on the X-Scope-OrgID value, not based on which user pushed them.

Additionally, even though we currently have multiple users with the same permissions to push and read logs, users cannot see logs they did not push themselves. We expect to be able to read logs according to the X-Scope-OrgID value, regardless of which user pushed them.

Environment:

  • Infrastructure: Kubernetes

  • Deployment tool: helm [loki-6.37.0]

  • Helm values:

    loki:
      auth_enabled: true
      # Configure multiple tenants/users
      tenants:
        - name: "loki-admin"
          password: "${loki_admin_password}"
        - name: "loki-readonly"
          password: "${loki_readonly_password}"
    gateway:
      basicAuth:
        enabled: true
        htpasswd: |
          {{- with $tenants := .Values.loki.tenants }}
            {{- range $t := $tenants }}
              {{- $username := required "All tenants must have a 'name' set" $t.name }}
              {{- if $passwordHash := $t.passwordHash }}
                {{- printf "%s:%s\n" $username $passwordHash }}
              {{- else if $password := $t.password }}
                {{- printf "%s\n" (htpasswd $username $password) }}
              {{- else }}
                {{- fail "All tenants must have a 'password' or 'passwordHash' set" }}
              {{- end }}
            {{- end }}
          {{- end }}
2

As far as I know, Loki doesn’t have the concept of read write or readonly users. In fact, Loki itself doesn’t do authentication, and the part that does authentication is the gateway, which is just nginx.

You should look at the helm chart and see if your use case is supported, if not, you might consider baking your own nginx gateway and not use the one included by the helm chart, then you can do pretty much whatever you want. For example:

  1. Configure list of users with htpasswd
  2. Create a map of read write users and readonly users
  3. Allow readonly users to hit the read path only
  4. Allow readwrite users to hit both the read and write paths

You can see a list of Loki APIs here and which ones are read or read write: Loki HTTP API | Grafana Loki documentation

Personally, I don’t think there is much point to readonly user for Loki. As long as you don’t expose the delete endpoint, users can’t overwrite existing logs anyway. Sure, they could intentionally send logs with the same label and try to mess things up by adding junk logs, but that would have to be very intentional and not something normal users would do.