Multi-tenant queries execution failure with multiple org IDs present

Grafana Loki
1

Hi team

I’m following this blog post, Grafana Loki 2.6 release: multi-tenant queries and targeted log line deletion to perform multi-tenant queries or even have a multi-tenant Datasource in Grafana.

When I query logs for my tenant my_org I have a successful request

curl -H 'X-Scope-OrgID:my_org' --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' http://10.10.100.189:30813/loki/api/v1/query
{"status":"success","data":{"resultType":"vector","result":[],"stats":{"summary"...

When I do the same for my_org2 I again have a successful request

curl -H 'X-Scope-OrgID:my_org2' --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' http://10.10.100.189:30813/loki/api/v1/query
{"status":"success","data":{"resultType":"vector","result":[],"stats"......

But when I tried to use the multi-tenant query following the blog post, I get this error:

curl -H 'X-Scope-OrgID:my_org|my_org2' --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' http://10.10.100.189:30813/loki/api/v1/query
multiple org IDs present

logs:

ts=2024-10-17T17:12:32.960843268Z caller=spanlogger.go:109 middleware=QueryShard.astMapperware level=warn msg="failed mapping AST" err="rpc error: code = Code(500) desc = multiple org IDs present" query="sum(rate({job=\"varlogs\"}[10m])) by (level)"
level=warn ts=2024-10-17T17:12:32.961056115Z caller=logging.go:126 orgID=my_org2|myorg msg="POST /loki/api/v1/query (500) 18.368797037s Response: \"multiple org IDs present\" ws: false; Accept: */*; Content-Length: 74; Content-Type: application/x-www-form-urlencoded; User-Agent: curl/7.81.0; X-Scope-Orgid: my_org2|myorg; "

Is there anything I can do to solve the issue?

More context from the environment

I have Loki running on release 3.1

curl http://10.10.100.189:30813/loki/api/v1/status/buildinfo
{"version":"release-3.1.x-89fe788","revision":"89fe788d","branch":"release-3.1.x","buildUser":"root@buildkitsandbox","buildDate":"2024-08-08T14:25:46Z","goVersion":""}

Setup, Pods:

loki-backend-0                                                  2/2     Running   0              10m
loki-canary-frkqn                                               1/1     Running   0              10m
loki-chunks-cache-0                                             2/2     Running   0              10m
loki-gateway-56954586c5-2fpf9                                   0/1     Pending   0              10m
loki-gateway-6b6c5667db-vx4mg                                   1/1     Running   0              104m
loki-minio-76b595b85c-m6f4h                                     1/1     Running   0              104m
loki-read-6c67fd8b7-b65md                                       1/1     Running   0              10m
loki-results-cache-0                                            2/2     Running   0              10m
loki-write-0                                                    1/1     Running   0              10m

Config settings:

loki:
  deploymentMode: SimpleScalable
  loki:
    commonConfig:
      replication_factor: 1
    storage:
      type: s3
      s3:
        region: eu-central-1
      bucketNames:
        chunks: loki-traces/chunks
        ruler: ruler
        admin: admin
    schemaConfig:
      configs:
      - from: "2024-01-01"
        store: tsdb
        index:
          prefix: loki_index_
          period: 24h
        object_store: s3
        schema: v13
  auth_enabled: true
  read:
    replicas: 1
  write:
    replicas: 1
  backend:
    replicas: 1

  minio:
    enabled: true
2

Did you enable multi_tenant_queries_enabled? See Multi-tenancy | Grafana Loki documentation

1 Like