The plan is to build the cluster of 3 nodes, one for running queries and two for ingesting data to object storage. All 3 nodes are members of memberlist
All works perfect except whenever I try to run a query on querier loki1.env.infra.domain, the logs come with random delay 2-7 min. Is it the way it just it is? Can I get logs instantly from what was shipped by fluentbit to logproxy1,2.env.infra.domain or is it impossible? I also observing
Unfortunately, I don’t know the answer to your question, but I am intrigued that it seems you are running Loki cluster outside of Kubernetes? Is that true?
If yes, I need some guidance on how to do that as I am unable to find any docs for such.
rizwankh, what exactly you are willing to know? We are not running any production Loki yet, just a test mode. And yes, we use no k8s for it. We use single binary from loki release, logs are sent to loki with fluent bit and then stored to s3. We use grafana to see the logs.
Hello. I have the same problem. I use Loki in HA mode with 2 replicas and s3 as storage. Loki is installed into the k8s cluster. When I use Explore in Grafana to see logs, I get this error in Loki logs:
Was there ever a resolution to this? I am experiencing a similar issue, but it doesn’t seem to impact the data that is returned. Seems when the query is issued in grafana, data returns, then a querier logs the errors you referenced.