About ruler remote query ingester or store

Does the current Loki 2.7.4 version support running a separate instance of ruler(-target=ruler) to remotely query an ingester or store?

We have an all(-target=all) instance and want to run a separate instance of ruler(-target=ruler) to perform log queries and monitor alarms, but it seems that this cannot be achieved in the test (did we run it incorrectly?). Can we run the query-frontend/querier component at the same time on the ruler instance to implement this? (such as -target=ruler,query-frontend,querier)

Secondly, we saw on github that the upcoming Loki 2.8.x ruler configuration seems to support this (ruler.evaluation.mode: remote). Should we wait for its release?

Looking forward to your reply, thank you.

The following is the configuration we run:

loki-ruler.yaml
server:
  http_listen_port: 13100
  grpc_listen_port: 19095
  log_level: debug
common:
  path_prefix: /data
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
    heartbeat_period: 15s
    heartbeat_timeout: 1m
  instance_addr: 127.0.0.1
limits_config:
  ruler_remote_write_disabled: false
  max_query_lookback: 168h
  split_queries_by_interval: 5m
  query_timeout: 10s
#frontend:
#  downstream_url: http://loki.domain.com:3100
#querier:
#  query_ingesters_within: 168h
#  max_concurrent: 10
#  query_ingester_only: true
#  multi_tenant_queries_enabled: true
ruler:
  #external_url:
  evaluation_interval: 30s
  enable_sharding: false
  enable_api: true
  storage:
    type: local
    local:
      directory: /data/rules
  remote_write:
    enabled: false
    clients:
      local:
        url: http://prom_host:prom_port/api/v1/write
        remote_timeout: 30s
        follow_redirects: true
        queue_config:
          capacity: 1
          batch_send_deadline: 0s
  wal:
    dir: /data/ruler-wal
    min_age: 5m
    max_age: 2h
  wal_cleaner:
    min_age: 6h
  enable_alertmanager_discovery: true
  alertmanager_refresh_interval: 1m
  enable_alertmanager_v2: true
chunk_store_config:
  chunk_cache_config:
    enable_fifocache: false
  write_dedupe_cache_config:
    enable_fifocache: false
schema_config:
  configs:
  - from: 2020-01-01
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h
    chunks:
      period: 168h
runtime_config:
  file: "/data/overrides.yaml"
  period: 10s
tracing:
  enabled: false```

Secondly, we saw on github that the upcoming Loki 2.8.x ruler configuration seems to support this (ruler.evaluation.mode: remote ). Should we wait for its release?

The new remote rule evaluation feature will probably only be released with 2.9 in a couple month’s time. The ruler currently embeds a querier (this is the default local evaluation mode) which can query the ingesters or the store.

What part did not work? Please elaborate on the observed behaviour.
You can provide the same config file to the -target=all and -target=ruler. This way you can have a single config which defines all the components’ config, and if querying works on the -target=all instance then it will work on the -target=ruler.

I must however caution against running one all and one ruler. all already contains the ruler, so it’s probably unnecessary to do so unless you’re finding that your rule evaluation is affecting the performance of the overall Loki instance.

1 Like

Thank you for your reply. I think I understand. Running a single instance of ruler(-target=ruler) to perform query operations on all(-target=all) instances requires shared storage (because the default evaluation mode for ruler is local), but we use the local filesystem. Therefore, running the ruler component as a single instance is not feasible, is it?

At first, we learned about Loki microservices mode, so we wanted to separate the ruler component and run it like promtail, the ruler component only performs log metrics and writes them to Victoria Metrics, alarms are implemented by vmalert, so we separated the ruler component. From your reply now, it seems unnecessary, and we will make modifications.

Thank you again for your reply!
We are also looking forward to the release of a new version of Loki.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.