Loki rulerConfig block definition

Grafana Loki
1

Hi all. I have a configuration question about Loki rulerConfig block.

I am using grafana/loki helm chart . On our test cluster, I am running Loki with single tenant (multi tenancy enabled). rulerConfig block is as follows for this test cluster:

rulerConfig:
  external_url: https://external-grafana-url.example.com
  evaluation_interval: 1m
  poll_interval: 1m
  rule_path: "/var/loki/rules"
  alertmanager_url: "http://mimir-gateway.namespace.svc.cluster.local/alertmanager"
  enable_alertmanager_discovery: false
  enable_alertmanager_v2: false
  notification_queue_capacity: 10000
  notification_timeout: 10s
  alertmanager_client:
    type: Basic
    credentials_file: /etc/loki/secrets/alertmanager.credentials
  flush_period: 1m
  enable_api: true
  enabled_tenants: ""
  query_stats_enabled: true
  disable_rule_group_label: false
  wal:
    dir: "/var/loki/ruler-wal"
  remote_write:
    enabled: true
    clients:
      tenant2:
        url:  "http://mimir-gateway.namespace.svc.cluster.local/api/v1/push"
        headers:
          X-Scope-OrgId: tenant1
        authorization:
          type: basic-auth
          credentials_file: /etc/loki/secrets/prometheus_remotewrite.credentials

This works with 1 tenant, as expected.

Now the problem: In our production environment we have two tenants for Mimir, Loki and Alert Manager and two different Grafana instances for these tenants.

In the above rulerConfig block:

  • How should I define Grafana External URL? There is only one in the documentation
  • How should I define Alert Manager tenant ID? There is no header option
  • How should I define second tenant’s Alert Manager configuration?
  • I believe remote_write.clients will be OK with another block, isn’t it?

I need this rulerConfig block to route alerts to Grafana OnCall and also to have recording rules on Loki queries…

Any help is appreciated.

2

We run Grafana and Alertmanager separately from our Loki and Mimir stacks, so I am not very familiar with running those within the Mimir / Loki stacks. I will offer some recommends, but I could be wrong.

Grafana can have multiple org, and you can add Loki / Mimir data sources to each org within Grafana according to tenant ID. Is there a good reason to want to run one Grafana per tenant? I don’t know how to go about doing this either way from helm chart. We run our Grafana instance completely separately.

I believe Loki automatically add the tenant ID header if tenant mode is enabled. Have you tested and see how it behaves?

I think you can override this with limits_config.ruler_remote_write_config. Try adding an override config per tenant and define the alertmanager’s client info there.