Loki querier queries the writer instances within configured query_ingesters_within. If your querier can’t reach the remote writers then that would explain it.
You could try to lower it, but a better option would be to re-design your architecture. You should keep Loki cluster container within one location, then from your remote accounts setup promtail or fluend forwarders to forward logs to your Loki cluster. The connectivity can be facilitated with VPC endpoint or transit gateway.
There is a limit on how often writers can write to chunk storage, it’s controlled either by size, idle period, or max wait period. Typically you’d configure query_ingesters_within within the max wait period.
I don’t know why the cost would be a lot more, you’d be running the same number of hosts (maybe even less), and I am not sure why it would be a nightmare to manage.
VPC endpoint service might be an alternative then.
It’s a trade-off, if you think about it. You either accept that there will always be some logs living on the ingester for a period of time before it’s sent to chunk storage, or you try to justify the cost.
@tonyswumac how do I configure the querier to query the ingesters? How does the querier learn about the ingesters? Over cluster gossip? I can’t figure this part out. I’m able to query logs from minio but not from the ingesters. I’ve tried everything I can think of and nothing has worked for me. My query_ingesters_within is default of 3h.
When you say “it doesn’t propagate the port which means you have to use native port (meaning no bridge) for your writers.”, what writer port are you referring to? gRPC port? or gossip port? I’m assumnig gRPC. I have all memberlist members communicating successfully over gossip. We are using podman with slirp4netns network mode and port translation. For gossip to work, we had to configure advertise_addr and advertise_port. I just can’t seem to figure out how to get the querier to query the ingesters.
Just in general, you’d want all open ports from writers to be directly accessible within the cluster. I see you created another topic, perhaps we should move this conversation there (just tag me) instead of hijacking someone else’s topic.
Also please include your loki configuration (particularly the ring memberlist part), and test telnet from your querier to writer and see if they work or not.