Hi all,
I am trying to add sending alerts to AlertManager from Loki.
I have loki deployed via helm chart loki-stack and documentation is unclear whether to add “ruler” or “rulerConfig” into the helm values. Also, there is no good example of how to create rules or how to create a ConfigMap such that rules will be discovered by Loki.
Can anyone help me out here?
Does anyone maybe have a sample repo where a helm chart is configured with the ruler, either with loki-stack or loki as the base chart?
Thanks!!
Thomas
There are two parts when it comes to configuring ruler in Loki. First is the configuration of the ruler component itself, second is the actual rules.
To configure the ruler itself, you’ll want to use the ruler block in Loki configuration (see Grafana Loki configuration parameters | Grafana Loki documentation). This includes settings such as alertmanager_url, evaluation_interval, and storage.
Once you have that working, you need to create the actual rules. First I’d recommend reading through this: Alerting and recording rules | Grafana Loki documentation. There are two ways to create rules, one is local storage, one is object storage such as S3. The local storage one is straight forward, you mount the rules into the ruler or backend container at the path specified in rule_path, and all is well. Caveat is the ruler API would be readonly, meaning you can’t change the rules through API call. If you want to use object storage for rules, then you’d need to use cortextool to modify the rules via Ruler API, which may require external pipelines depending on how you do your deployment.
In terms of the rule itself, the structure of Loki alerting rules are identical to Prometheus alerting rules (except the query part of course). You should be able to find plenty of examples online.
@tonyswumac Hi Tony, thank you, I appreciate your reply. However, I understand all that… I know that the ruler and rules have to be configured, yet the documentation and internet posts are confusing with regards to helm configuration… some say that the loki-stack chart does not have a way of configuring the ruler, others says you have to set the “rulerConfig” block in the values for the chart, yet others say you have to configure the “ruler” block, and even others say you need to configure for the “read” pods only. Do you know of any example config e.g. a helm values configuration configuring both ruler and rules?
Unfortunately I don’t have an example for you, hopefully someone else might. And since I am not actually running Loki on Kubernetes, I could be wrong on some of my comments below.
There are several Loki helm charts. I believe the one inside the Loki repo is the one generally recommended. If you are using that helm chart, then you’ll want to use rulerConfig, since that’s the configuration object used by the helm chart.
If you are writing your own helm chart, then you can simply add a ruler block in your existing loki configuration that you are already using. Generally speaking you can keep configuration from all components in one file, and distribute that to all your components. Meaning that the configuration file mounted into your read, write, and backend components can be identical.
Ruler used to be part of the reader component in simple scalable mode, but this is no longer the case (see loki/pkg/loki/loki.go at 24c760a75e600d07548ac5aadb56712f9865af38 · grafana/loki · GitHub). Now ruler is part of backend component if you are using simple scalable mode.
