Loki is not executing recording rules or sending them to Prometheus

Grafana Loki
1

I’m trying to figure out why my Loki setup is not running recording rules and not sending the resulting metrics to the Prometheus remote write endpoint. The rules do get added to the /rules directory by the sidecar container, but I don’t see anything related in the logs of the loki container in the loki-backend pod or the loki-sc-rules container, even after enabling debug logging for both. Of course, there are no new metrics either. I’m starting to think that the recording rules might not actually be running on the Loki backend ruler component at all.

I’m using the Loki Helm chart, version 6.25.0 with Flux (S3 bucket and region values redacted).

Any insights would be greatly appreciated; I’ve tried everything in the Grafana forums or Github issues but nothing seems to work.

Loki config:

    global:
      dnsService: coredns
    chunksCache:
      enabled: false
    resultsCache:
      enabled: false
    gateway:
      enabled: false
    test:
      enabled: false
    lokiCanary:
      enabled: false
    backend:
      extraArgs:
        - "-log.level=debug"
    sidecar:
      rules:
        logLevel: DEBUG
    loki:
      auth_enabled: false
      ingester:
        chunk_encoding: snappy
      storage:
        type: s3
      limits_config:
        volume_enabled: true
        query_timeout: 10m
      schemaConfig:
        configs:
          - from: "2024-01-01"
            index:
              period: 24h
              prefix: loki_index_
            object_store: s3
            schema: v13
            store: tsdb
      rulerConfig:
        remote_write:
          enabled: true
          clients:
            main:
              url: http://prometheus.monitoring:9090/api/v1/write
    ruler:
      enabled: true
      persistence:
        enabled: true

Prometheus config:

prometheus:
  prometheusSpec:
    enableRemoteWriteReceiver: true

Some of the dummy rules I tried:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aaa
  namespace: monitoring
  labels:
    loki_rule: ""
data:
  aaa.yaml: |
    groups:
      - name: aaa
        limit: 10
        interval: 1m
        rules:
          - record: aaa:aaa:rate1m
            expr: |
              sum(
                rate({container="aaa"}[1m])
              )

or

kind: ConfigMap
metadata:
  name: aaab
  namespace: monitoring
  labels:
    loki_rule: ""
data:
  aaab.yaml: |
    namespace: rules
    groups:
      - name: aaab
        interval: 1m
        rules:
          - record: aaab:aaab:rate1m
            expr: |-
              sum(rate({service="aaab"}[1m]))