Promtail tenant stage

[dmitryrint]

I’m trying to setup a multi-tenant log collection in AWS EKS.
Promtail match stage works well, JSON and labels, but the tenant doesn’t apply.
There’s a single fake directory in the S3 bucket and it seems it sends an empty tenant value, I can receive logs using this query:
{namespace="podinfo", __tenant_id__=""}
This query returns no logs:
{namespace="podinfo", __tenant_id__="admin"}
Also, I can query all logs even with X-Scope-OrgID header set in Grafana and address Loki directly, or send header with reverse proxy.

I’m using this chart

grafana/helm-charts/blob/main/charts/loki-stack/Chart.yaml

version 2.8.7
Here are my Helm chart values:

loki:
  auth_enabled: true
  extraVolumes:
    - name: reverse-proxy-auth-config
      secret:
        secretName: reverse-proxy-auth-config
  extraContainers:
    - name: reverse-proxy
      image: k8spin/loki-multi-tenant-proxy:v1.0.0
      args:
        - "run"
        - "--port=3101"
        - "--loki-server=http://localhost:3100"
        - "--auth-config=/etc/reverse-proxy-conf/authn.yaml"
      ports:
        - name: http
          containerPort: 3101
          protocol: TCP
      resources:
        limits:
          cpu: 250m
          memory: 200Mi
        requests:
          cpu: 50m
          memory: 40Mi
      volumeMounts:
        - name: reverse-proxy-auth-config
          mountPath: /etc/reverse-proxy-conf  
  extraPorts:
    ## Additional ports to the loki services. Useful to expose extra container ports.
    - port: 3101
      protocol: TCP
      name: http
      targetPort: http
  config:
    schema_config:
      configs:
        - from: 2022-12-07
          store: boltdb-shipper
          object_store: s3
          schema: v11
          index:
            prefix: loki_index_
            period: 24h
    compactor:
      working_directory: /tmp/loki/compactor
      shared_store: s3
    common:
      ring:
        kvstore:
          store: inmemory
      path_prefix: /tmp/loki
      storage:
        s3:
          bucketnames: logging-storage-loki
          region: ap-northeast-1
          access_key_id: null
          secret_access_key: null
    storage_config:
      aws:
        s3: s3://ap-northeast-1/logging-storage-loki
        s3forcepathstyle: true
        bucketnames: logging-storage-loki
        region: ap-northeast-1
        insecure: false
        sse_encryption: false
      boltdb_shipper:
        shared_store: s3
        cache_ttl: 24h

prometheus:
  enabled: false

test:
  enabled: false

grafana:
  enabled: true
  image:
    repository: grafana/grafana
    tag: "9.3.1"
    pullPolicy: IfNotPresent

promtail:
  enabled: true
  config:
    snippets:
      relabel_configs:
          - action: replace
            target_label: namespace
            source_labels:
                - __meta_kubernetes_namespace
      pipelineStages:
        - cri: {}
        - match:
            selector: '{namespace=~"podinfo|test"}'
            stages:
                - json:
                    expressions:
                      caller: caller
                - labels:
                    caller:
                - tenant:
                    value: admin

[dmitryrint]

I’ve tried to push logs by myself:

curl -X POST "http://localhost:5000/loki/api/v1/push" \
-H "Content-Type: application/json" \
-H "X-Scope-OrgID: admin" \
--data-raw '{"streams": [{ "stream": { "foo": "bar" }, "values": [ [ "1670948226475000000", "example4" ] ] }]}'

And I’m still not able to restrict access to these logs.
The tenant query still returns no logs.
{foo="bar", __tenant_id__="admin"}

[tonyswumac]

Check your reverse proxy and make sure it’s either forming it’s own X-Scope-OrgID, or it’s forwarding the value somehow.

You can also do the API call directly to your loki container instead of the nginx endpoint to ensure the auth_enabled option is actually turned on.