Trying to configure Loki/Promtail in multiple K8s clusters aggregating to a single Grafana instance

Grafana Loki
1

I’m running Loki/Promtail on multiple K8s clusters, but want to only run Grafana on one and aggregate logs there. I’ve tried a number of different config options, but I can’t get logs to the Grafana cluster.

I’m deploying using ArgoCD to run Helm charts into my clusters and the Loki instances are deploying just fine, but not using the ingress and not available from outside the cluster.

#values.yaml:
loki:
  schema_config:
    configs:
    - from: "2022-01-11"
      index:
        period: 24h
        prefix: index_
      object_store: azure
      schema: v11
      store: boltdb-shipper
  storage_config:
    azure:
      account_name: STORAGE_NAME_VALUE
      account_key: STORAGE_ACCOUNT_KEY_VALUE
      container_name: loki-data
      use_managed_identity: false
      request_timeout: 0
      endpoint_suffix: blob.core.windows.net
    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-shipper-active
      cache_location: /data/loki/boltdb-shipper-cache
      cache_ttl: 24h
      shared_store: azure
    filesystem:
      directory: /data/loki/chunks
  write: 
    autoscaling:
      enabled: true
      minReplicas: 3
      behavior:
        scaleUp:
          policies:
            - type: Pods
              value: 1
              periodSeconds: 900
        scaleDown:
          policies:
            - type: Pods
              value: 1
              periodSeconds: 1800
          stabilizationWindowSeconds: 3600
    extraEnvFrom:
      - secretRef:
          name: azurestorage-secrets

#dev.yaml
loki:
  ingress:
    hosts: ['loki-dev.internal.org.com']
    path: "/"
    tls:
      - secretName: wildcard-internal-org-com
        hosts:
        - loki-dev.internal.org.com
2

I was able to make this work by getting rid of the loki on the remote instance and just running promtail sending logs to a loki gateway ingress. This is not covered in documentation anywhere, but seems like a reasonable use case that really should be covered.