It might be a simple question. I am trying to create a monolithic cluster with two nodes running all components. I have a replication_factor of 2, and store my logs in s3(minio), however i have an issue. When ingesters flush logs to the object storage i can query all data from any of the node, but when querying data not yet flushed i will only get half of the data. All writes and read go through an nginx in front of the nodes, and requests are split in round robin. What am i missing here? Isn’t the distributor supposed to send writes to both ingesters since i have an replication_factor of 2?
Please share your Loki configuration and a screenshot of the /ring API output.
Usually when you see split data like that it means your ring membership isn’t configured properly. But then again you have replication_factor or 2, and if the quorum is less than 2 in this case it would simply not work.
Your problem is that your ring members have the identical IP address. The ring membership is supposed to be a way for each ring member to discover and connect to each other, if the IPs are the same then they are always connecting to itself, that’s probably why you only ever see half of the logs unless they are committed to the storage.
In order to resolve this you should ideally have some sort of service discovery. But if you are running two bare metal nodes with static IP then you can configure the static IP instead. Make sure the two nodes are able to communicate with each other as well, in terms of network connectivity.
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.