Loki Read/Write Split Configuration Issue - Frontend Service Connection Error

Grafana Loki
1

I’m trying to set up Loki with read/write separation, where one node (target: all ) can handle both read and write operations, and another node (target: read ) is read-only. However, I’m encountering an error when launching the read-only node.
I host on AWS EC2 and run in docker compsoe.

loki_rw_mode
loki_rw_mode471×321 5.75 KB

Current Setup

I have two nodes:

  • Node 1 (172.31.87.229): target: all - handles both read and write operations
  • Node 2 (172.31.87.205): target: read - read-only node

Configuration Files

Node 1 (target: all) Configuration:

yaml

target: all
server:
  http_listen_address: 0.0.0.0
  http_listen_port: 3100

frontend:
  compress_responses: true
  log_queries_longer_than: 10s

memberlist:
  join_members: ["172.31.87.229","172.31.87.205"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946
  gossip_interval: 2s
  advertise_addr: "172.31.87.229"

schema_config:
  configs:
    - from: 2023-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h
common:
  path_prefix: /loki
  replication_factor: 1
  compactor_address: http://172.31.87.229:3100
  storage:
    s3:
      endpoint: minio:9000
      insecure: true
      bucketnames: loki-data
      access_key_id: loki
      secret_access_key: supersecret
      s3forcepathstyle: true
  ring:
    kvstore:
      store: memberlist
ruler:
  storage:
    s3:
      bucketnames: loki-ruler

compactor:
  working_directory: /tmp/compactor

Node 2 (target: read) Configuration:

target: read
server:
  http_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_port: 9096

memberlist:
  join_members: ["172.31.87.229"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946
  gossip_interval: 2s
  advertise_addr: "172.31.87.205"

schema_config:
  configs:
    - from: 2023-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h
common:
  path_prefix: /loki
  replication_factor: 1
  compactor_address: http://172.31.87.229:3100
  storage:
    s3:
      endpoint: 172.31.87.229:9000
      insecure: true
      bucketnames: loki-data
      access_key_id: loki
      secret_access_key: supersecret
      s3forcepathstyle: true
ruler:
  storage:
    s3:
      bucketnames: loki-ruler

frontend_worker:
  frontend_address: "172.31.87.229:9095"
  grpc_client_config:
    max_send_msg_size: 104857600
    max_recv_msg_size: 104857600

frontend:
  downstream_url: "http://172.31.87.229:3100"

compactor:
    working_directory: /tmp/compactor

When launching the read-only node, I get this error:

read-1  | level=info ts=2025-04-01T15:33:07.831606109Z caller=worker.go:231 component=querier msg="adding connection" addr=172.31.87.229:9095
read-1  | level=info ts=2025-04-01T15:33:07.831938189Z caller=loki.go:545 msg="Loki started" startup_time=31.11932ms
read-1  | level=info ts=2025-04-01T15:33:07.838484306Z caller=memberlist_client.go:588 phase=startup msg="joining memberlist cluster succeeded" reached_nodes=1 elapsed_time=7.131725ms
read-1  | level=error ts=2025-04-01T15:33:07.839350039Z caller=frontend_processor.go:75 component=querier msg="error processing requests" address=172.31.87.229:9095 err="rpc error: code = Unimplemented desc = unknown service frontend.Frontend"

The error suggests that the frontend service is not available at the specified address. I suspect there might be an issue with my frontend configuration or the GRPC port settings.

Has anyone successfully set up Loki with read/write separation? What am I missing in my configuration? Any guidance would be greatly appreciated.

2

You want to make sure the two nodes have identical configurations. Loki read also needs to be part of the cluster.

Unless cost is a concern, I would recommend you to use S3 (if you are already in AWS), it’ll likely be cheaper and less headache for you. Also, perhaps moving into ECS with simple scalable deployment and take scaling into considerations (if you don’t need scaling then by all means stay on EC2).

1 Like
3

Hi everyone,

I wanted to provide an update regarding the issue I faced with setting up read/write separation in Loki. I have successfully resolved the problem, and I’d like to share my current configuration along with some explanations.

Current Configuration

Here’s my updated configuration for both nodes:

memberlist:
  join_members: ["172.31.87.229:7946","172.31.87.205:7946"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946
  gossip_interval: 2s
  advertise_addr: "172.31.87.205"
  advertise_port: 7946

common:
  path_prefix: /loki
  replication_factor: 1
  compactor_address: http://172.31.87.229:3100
  ring:
    kvstore:
      store: memberlist
  instance_addr: "172.31.87.205"

Key Changes and Their Purpose

  1. instance_addr:
  • This setting specifies the IP address that the node will advertise in the ring. It is crucial for the memberlist to identify this instance in the cluster. By setting it correctly, I ensured that the other nodes can communicate with this node without issues.
  1. advertise_addr:
  • This is the address that will be advertised to other members in the cluster, particularly useful for NAT traversal. By configuring this, I allowed the node to be reachable by other nodes even if it is behind a NAT.

Conclusion

By addressing these two settings, I was able to resolve the frontend service connection error that was previously causing issues. If anyone else is experiencing similar problems, I highly recommend checking these configuration options.

Thank you for the community’s support!

1 Like