Loki microservice with memberlist ring store

Has anyone got an example configuration for running an ingester, distributor, and querier as microservices on the same server using memberlist as the ring store?

Using consul I have this working - well, logs get to S3, and I can query them. However, it would be good to have a zero-dependencies setup, so we’d like to use memberlist for ring storage. With consul, I can browse to /ring, and see the cortex ring status, with 3 members (I’m running the a set of ingester/distributor/querier on each of 3 VMs).

If I try to use memberlist, I constantly get errors about the ring being empty, and the /ring url shows no members.

In the ingester.yaml, I have this:
store: memberlist
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
max_transfer_retries: 0

node_name: ${HOSTNAME}
bind_port: 7946
- loki1:7946
- loki2:7946
- loki3:7946
abort_if_cluster_join_fails: false

which is the same on all 3 servers, and for the distributor.yaml, I have the same stanza. All 3 components run on each of 3 servers.

Only the ingester process shows port 7946 - the (bind port for the ring) in the output of ss -ntlp, irrespective of changing the bind address in the other component config files, which is actually what I expected, but I also expected that the other components would join the ring by talking to that port. I’m very confused, and suspect that a working example using memnerlist - even for a single component to VM, but with more than one vm.

By running 2 ingesters on one host (with different ports), I have come to the conclusion that the memberlist ring does not update the /ring URL - would that be correct.


To summarise: should I have a memberlist: block in every component, or just the ingester component?
Either way, I see this error:
evel=error ts=2020-09-11T15:56:32.447558222Z caller=pool.go:161 msg=“error removing stale clients” err=“empty ring”
in the distributor log.

Whereas the ingesters seem to talk to each other - this is from an ingester log:

ts=2020-09-11T16:00:31.134947018Z caller=memberlist_logger.go:74 level=debug msg=“Initiating push/pull sync with:”
ts=2020-09-11T16:00:31.135230033Z caller=memberlist_logger.go:74 level=debug msg=“Stream connection from=”
ts=2020-09-11T16:00:31.139790956Z caller=memberlist_logger.go:74 level=debug msg=“Initiating push/pull sync with:”
ts=2020-09-11T16:00:31.145082204Z caller=memberlist_logger.go:74 level=debug msg=“Initiating push/pull sync with:”
level=info ts=2020-09-11T16:00:31.147945381Z caller=memberlist_client.go:460 msg=“joined memberlist cluster” reached_nodes=3


A long shot but… have you figured whats up?

I have ingester and distributor in a GKE cluster but the ring is empty even though both advertise ports - both on different nodes.