Grafana Alerting Issue with Loki

Grafana Loki
,
1

How is Grafana alerting supposed to work with Loki when they are deployed via two helm charts (one kube-prometheus-stack and one loki with storage in s3)? Depending on what we try we can get different errors, but we can’t get it to work:

  • If we deploy the charts, set loki as a datasource in grafana, and try and create an alert it says: Failed to load the data source configuration for loki: Unable to fetch alert rules. Is the Loki data source properly configured . The ruler bucket in s3 is empty, and the logs for loki-backend say: ruler up and running
  • If we change the labels of the PrometheusRules that Loki creates to match the labels for the kube-prometheus-stack deployment, the rules themselves show in grafana, but there is no data for them

How is this supposed to work? As far as I can tell, Loki can’t see the PrometheusRule that it creates (the ClusterRole it creates doesn’t have rights to see them), but “something” has to run the rules and store that data into prometheus.

2

I don’t deploy Loki on Kubernetes, so I can’t tell you exactly how to fix it. But I can tell you how it’s supposed to work.

There are two ways you can run Loki ruler. First way is to use a local filesystem to store the rules (on kubernetes this most likely means configmap). With this method Loki ruler will NOT accept API call, can you cannot modify the rules. Which means this is not ideal if you intent to see Loki ruler rules from Grafana.

The second way to operate Ruler is to use an S3 storage. With this method Loki ruler will accept API call, hence you can enable alerting view for Loki in Grafana, but you cannot manually update / modify the rules in S3 bucket (unless you know the format, which is compressed protobuf if I remember correctly). With this method you need some sort of CI/CD to change the rules via Ruler API.

3

We are using S3 Storage. Is the real way to do this to use cortex-tool and upload the resulting files to the bucket. Separately, why is Grafana showing us an error? Does Grafana automatically show an error if there are no rules configured?

4

  1. If you can show the error I might be able to tell you why.

  2. You are not uploading files to the S3 bucket, you are using cortextool to perform API call against the ruler, and ruler will then upload to S3 bucket.

Personally I prefer the API route with S3 storage. It allows you to see the rules in Grafana, and it’s very useful in my opinion, and worth the additional setup (CI/CD or however you want to interact with the ruler API).