Loki Ruler API not discovered by Grafana

Grafana Loki
1

Hello all,

I’m trying to get Loki Ruler to work with Grafana.

Everything running in Kubernetes
Loki version 2.8.2 in namespace loki
Grafana version 9.5.5 in namespace grafana

Loki Ruler config

ruler:
  alertmanager_url: http://alertmanager-main.monitoring:9093
  ring:
    kvstore:
      store: memberlist
  rule_path: /tmp/loki/scratch
  storage:
    s3:
      bucketnames: my-bucket
      insecure: false
      region: my-region
      s3: s3://my-region
      s3forcepathstyle: false
  enable_api: true

I can curl the Loki Ruler API from the grafana namespace

# curl http://loki-distributed-ruler.loki:3100/loki/api/v1/rules
no rule groups found

The Loki datasource in Grafana is using http://loki-distributed-query-frontend.loki:3100.
I have enabled Manage alerts via Alerting UI

How is Grafana supposed to know the URL of Loki Ruler?

2

Figured it out… This could be better documented.

What tipped me off was the following paragraph, found here (for Grafana v10) and here (for Grafana v9.5) for Mimir alerts

… Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.

Going back to the loki-distributed Helm chart, I found that there is a shared gateway that you can deploy for this.

You deploy the gateway, fix the Nginx config as needed and then use the URL to the gateway service (in my case http://loki-distributed-gateway.loki:80) for the Loki datasource.