Setup Loki receiver with Thanos

Grafana Loki
1

Hello,

I try to setup rules in Loki. I’ve followed the documentation and setup my ruler and configured thanos as receiver.

I’ don’t have errors on startup and I can see that my rules are charged correctly. But I don’t see the end result in Thanos.

I’ve enabled debug log in thanos and loki but didn’t get more informations.

Is it possible to test my setup, sending directly a request to thanos receiver ?

1 Like
2

Hey @thomasboutelier

Is it possible to test my setup, sending directly a request to thanos receiver ?

I think you’d have to consult the document the Thanos docs for that.

But let’s first determine whether the rules are being evaluated and sent to remote storage at all, to see if the issue is on Loki’s side or on Thanos’.

Which version of Loki are you running?

If you’re running >2.4.0? If so, you can consult this page which details some metrics about the remote-write feature.

You could also change the remote-write URL to a request bin (https://requestbin.net/) and see what requests, if any, get sent there.

3

Hey @dannykopping, thank’s for the reply.

I didn’t knew requestbin, great tool.

So I’ve configured the remote url to request bin and didn’t have any request coming in, even after ten+ minutes. I tested my rule in grafana and I have metrics.

My loki version is 2.3.0

4

OK, so that indicates that either you have no samples produced by your recording rules or your rules are not being evaluated correctly.

Please enable debug logging in Loki, grep for your query and send through the relevant logs - as well as your config please.

5

I don’t see the rule in the logs with debug mode enabled.

This ruler part configuration of my loki.yaml:

ruler:
  storage:
    type: local
    local:
      directory: /mnt/loki/
  rule_path: /data/loki/rules/
  ring:
    kvstore:
      store: inmemory
  remote_write:
    enabled: true
    client:
      url: ...

My rules are in /mnt/loki/ directory, is that the right one ?

6

Looks fine to me @thomasboutelier

Do you see any values for these two metrics?

cortex_prometheus_rule_evaluations_total
cortex_prometheus_rule_evaluation_failures_total
7

(even though it says cortex, these are metrics from Loki since we reuse a lot of cortex)

8

I have setuped an exporter for loki, I can now see some metrics in prometheus but not

cortex_prometheus_rule_evaluations_total
cortex_prometheus_rule_evaluation_failures_total

Did I have to enable something else ?

9

Ok nevermind. Those metrics are now available but there is no data…

10

I’ve worked on my configuration and I’m now able to see the metrics:

cortex_prometheus_rule_evaluations_total{container="loki",endpoint="http-metrics",instance="10.0.2.251:3100",job="loki",namespace="logging",pod="loki-0",rule_group="/tmp/scratch/test/rules.yaml;LogstashRules",service="loki",user="test"}	5
cortex_prometheus_rule_evaluation_failures_total{container="loki",endpoint="http-metrics",instance="10.0.2.251:3100",job="loki",namespace="logging",pod="loki-0",rule_group="/tmp/scratch/test/rules.yaml;LogstashRules",service="loki",user="test"}	0

So I assume that my present configuration work, but I still see nothing in the request bin.

11

it’s working now !!

I had an error on startup, loki was not able to create the /ruler-wal directory on root.

Once this directory was created, I was able to see my metrics in Thanos.

12

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.