Loki - Dynamic Alerting Rules generation

Grafana Loki
1

Hello everyone,
I’m trying to achieve the same kind of behaviour that we have with PrometheusRules in the sense that we are able to attach new PrometheusRules to an already existing Prometheus cluster, but for Loki Alerting Rules. The idea is to enable applications generate a new Alerting Rule on installation (or any other lifecycle step). We’ve tried to explore the existing OperatorHub.io | The registry for Kubernetes Operators to make use of their AlertingRule object, but following this comment Loki rule operator · Issue #3456 · grafana/loki · GitHub it would seem that is not possible to connect the k8 resources generated by the operator with a previously existing Loki solution.

My question is, if this is not possible, what is the best way to dynamically generate new Alerting Rules based on Loki log content? Through Grafana Alerting? Directly injecting the alerts to Loki via curl ?

Thanks in advance for your time. Appreciate it

2

I use PrometheusRule custom resources that are read by Alloy and provisioned to Loki ruler.

This is my Alloy config

loki.rules.kubernetes "local" {
    address = "http://loki-distributed-ruler.loki.svc.cluster.local:3100"
    tenant_id = "fake"

    rule_selector {
        match_labels = {
            role = "loki-rulefiles",
        }
    }
}

More info on Alloy loki.rules.kubernetes

I have configured Loki with shared storage.

My ruler config looks like this (part of the full Loki config)

ruler:
  alertmanager_url: http://alertmanager-main.system-monitoring:9093
  remote_write:
    enabled: true
    clients:
      thanos-receive:
        url: http://thanos-receive-router.thanos:19291/api/v1/receive
  ring:
    kvstore:
      store: memberlist
  rule_path: /tmp/loki/scratch
  storage:
    s3:
      bucketnames: my-loki-ruler-bucket
      insecure: false
      region: eu-west-1
      s3: s3://eu-west-1
      s3forcepathstyle: false
  enable_api: true
  wal:
    dir: "/var/loki/ruler-wal"
3

Thanks for the quick reply @b0b.

Interesting, I see that the loki.rules.kubernetes it’s also compatible with grafana-agent, which would fit our needs. Although I still don’t see how can I generate alerts based on log content since PrometheusRules would generate alerts based on the Prometheus datasource , not the Loki one, is that correct?

4

You can use PrometheusRule resources for Loki rules as well.

With the rule_selector you choose which PrometheusRule resources should go to Loki ruler. I use the label role: loki-rulefile

Here is an example PrometheusRule for Loki

---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  annotations:
    description: Provisioning Loki rule with Alloy test
  labels:
    role: loki-rulefiles
  name: alloy-test
  namespace: grafana
spec:
  groups:
  - name: AlloyPrometheusRuleTest
    rules:
    - alert: AlloyLogLineCount
      annotations:
        description: Grafana Alloy is logging more than usual
        summary: Grafana Alloy is logging more than usual
      expr: count_over_time({namespace="grafana", container="alloy"}[1m]) > 100
      for: 10m
      labels:
          severity: warn

Because the rule will be provisioned to Loki ruler, the rule expression will be queried against Loki.

1 Like
5

oh wow, thanks for the indications @b0b , this is exactly what we wanted to achieve.

Thanks!