Dynamically pass memberlist details such as ip address in a scalable loki cluster

Hi Everyone,
I would like to setup scalable and highly available loki cluster with multiple nodes establishing a ring and adding a memberlist config but only problem i face here is i had to add the node ip in all the loki config for a new loki instance,
Is there any easy way to dynamically pass or any other way we can implement instead of passing ip address of the loki instance?

I’m thinking to add a autoscaling group with target groups and scale-in and scale-out based on demand of resources.

Appreciate your help

Are you using ECS? If so the discussions here might be helpful to you: Loki 2.4.1 empty ring Code(500) error for "GET /loki/api/v1/labels" API on AWS ECS - #10 by tonyswumac

If not, please specify what deployment method you are using.

i’m referring standalone ec2 deployment and already setup monolothic loki deployment mode with 2 nodes forming a cluster but want to add a autoscaling feature for this deployment since scaling is dynamic how can we add a new ip address each time instance is scaled-out and removing ip address when scaled in in asg

If you are doing monolithic deployment on EC2, you can create a DNS record with multiple values pointing to both of your nodes, and use that for your ring membership.

Hi @tonyswumac
I have setup the similar way along with ALB pointing to 2 nodes as a part of target group.
I feel DNS with multiple values are always cached so enhanced the setup with ALB with disabling sticky sessions

Was thinking to enhance it more with auto scaling setup if there is no easy way I’m thinking to write a script to get the ip and update it in each loki instance when there is a scaling threshold matches

You don’t want to use ALB for ring membership. Load balancer is an endpoint itself. You hit the load balancer, and the load balancer decides which backend to forward your request to, that’s not what you want.

For a ring membership, every member has to know every other member. In Docker environment this would be done via service discovery. In your case DNS record with multiple values. To be clear. ring membership isn’t doing DNS round robin, it’s parsing the values returned from the DNS record so it knows what members are within the ring.

Edit: If you think about it, it’s actually the same thing as hard coding your two instance IPs in the configuration, except in this case you are using a DNS record so it’s dynamic and you can change your instance without worrying about IP changing (as long as you update the DNS record).

1 Like

Setting up a scalable and highly available Loki cluster with dynamic IP addressing can be achieved using service discovery mechanisms like DNS or Kubernetes services. Instead of hardcoding IP addresses in the Loki configuration, you can use DNS names or Kubernetes service names that dynamically resolve to the IP addresses of the Loki instances.
If you are running Loki on Kubernetes ciclo di sustanon 350, you can leverage Kubernetes services, which provide stable DNS names that automatically resolve to the IP addresses of the Loki pods.

1 Like