Scalable design - can’t query ingesters

Does anyone have example configs to have a remote querier to be able to query ingesters for current logs. I’m able to query logs after they have been written to minio, but I need current logs. Logs within 2 hours, live log streaming, etc. I can’t figure out how to get that working. I have 2 target=write nodes and 1 target=all node configured in memberlist. For some reason I can’t set the all node to read. When I do, the querier fails to execute any queries with ‘rpc error: code = Unimplemented desc = unknown service logproto.Querier’.

@tonyswumac I was able to get this working by switching network mode to host. I was previously using slirp4netns with podman and had the cluster communicating using advertise_addr and advertise_port. gossip was working fine. But I could not get querier to query ingesters. I believe the issue to be that you can’t configure grpc_advertise_addr. With slirp4netns addresses and ports are translated. I think its working now though with host mode. Initial tests are looking good. I will reply back if I run into any issues. Thanks for the help.

1 Like