i am running grafna loki in a microservice model however the deployment is not containarized … instead i am using indivisual VMs and F5 load balancer
currently i am running
4x ingester
2x qurier
1x query-frontend
3x distributor
we did all sorts of load test everything is working as expected
now i want to add 1 more query frontend
problem is unlike schedular, frontend does not have ring based discovery
so i have to depend on load balancer
i configured
frontend_worker.frontend_address=<load_balancer_domain>:<grpc_service_port>
this load balancer resolves to
frontend1: <grpc_service_port>
frontend2: <grpc_service_port>
now problem is i think this f5 balancer (tcp protocol) works fine for http but when it comes to grpc … since its terminating connection at balancer level and resolving to one of the host from backend pool … all queriers always connect to frontend1 it shows status ready … other one shows no queriers connected
how to get around this problem ? do i need DNS that returns multiple A records ? i also checked push model of frontend instead pull (as it uses http) i.e. using
frontend.downstream_url
however it seems like with this approach frontned doesnt do queues , sharding nothing … it just proxies request to querier nodes … ? thats what doc says Query frontend example | Grafana Loki documentation so its like disabling frontend all together - i do not want that