Grafana Alerting for Loki Datasource: Failed to load the data source configuration for Loki: Unable to fetch alert rules. Is the Loki data source properly configured?

[alxsuvorov]

We have installed Loki Stack in Kubernetes cluster (without Helm). In the Alerting section of Grafana we see error:

Errors loading rules
Failed to load the data source configuration for Loki: Unable to fetch alert rules. Is the Loki data source properly configured?

This is not really a very descriptive error message. What do we have in our configuration set improperly? I Googled this error and there are instructions to update Loki config file, but these instructions are still very vague and after putting those parameters in our Loki we still see this error message.

Can anyone please point us in direction with some guidance how this needs to be set up properly? (Alerting in Grafana for Loki Datasource)

[alxsuvorov]

So, we had a look at our Loki Gateway log and found error that Nginx doesn’t know how to process route “/prometheus/api/v1/rules”.

In the Nginx config file I added the routing for this path as following and the error message in Grafana is gone:

        location ~ /prometheus/api/v1/rules.* {
            proxy_pass       http://loki-stack-read$cluster_domain:3100$request_uri;
        }