Memberlist join_members configuration for read-write deployments

Grafana Loki
1

I see many examples of Grafana Loki memberlist.join_members configuration is set with all writes and reads deployments, including here https://github.com/grafana/loki/blob/v2.8.3/production/docker/config/loki.yaml#L24

My question:

  1. Is it possible if I want to make sure that all Loki write components is separated with read components? I mean, why should I define the write component address into read component’s memberlist.join_members and vice versa? Could it be separated, so when I want drop all my read component (or if outage occurs in the read component), my write component don’t have to rebalance the memberlist group.

  2. I want to scale my read and write component independently, how can it be achieved if in the memberlist.join_members it defined all of the Loki component address?

I have tried to remove write address from read component configuration, but it seems that read component can’t process the query, said “empty ring”. But the write components works normally, it can persist data into Minio. I just want the read and write components works independently, and any query should go to the object storage (index and chunks).

Thanks!

2

Let me answer question #2 first. You can scale writers and readers separately by using different service discovery for them. For example:

memberlist:
  <OTHER_PROPERTIES>
  join_members:
    - writers.loki.local
    - readers.loki.local

In terms of why you should define both, I don’t actually know if it’s necessary to have the reader members, but you have to have writer members in reader configuration because readers query writers for new logs that haven’t been written into chunk storage (see max_chunk_age and query_ingesters_within). And there doesn’t seem to be any adverse effect having reader membership in the midst anyway (for example if you configure both and query /ring endpoint, you’ll only ever see writers in there, which would imply that readers simply doesn’t register to be a member). To keep thing simple we simply configure a global memberlist with both readers and writers and use that for all components.

1 Like