Query ingesters in different clusters for in-memory/WAL logs without forcing membership

Grafana Loki
,
1

I’ve made a configuration for Loki that consists of 2 Loki clusters that are separated from eachother in each network zone and they use the same storage platform.
This architecture somewhat works… But there’s a problem when using a single querier and trying to retrieving in-memory/WAL logs from seperate Loki clusters.
Will refer to a picture below…
Protected Network-Zone are allowed to send egress traffic to Exposed Network-Zone.
Exposed Network-Zone are not allowed to send ingress traffic to Protected Network-Zone.
My goal is I want to be able to read live-logs from either ingester in each network-zone from the querier.
Problem is I can’t read in-memory/WAL logs from the Exposed Network-Zone. I have to wait until ingester-B have filled out it’s Chunks and sent it to the cloud storage.

For querier-A to query ingester-B inmemory/WAL logs I have to add ingester-B to querier-A’s memberlist, i could not figure out any other way, the querier will however unpremediated tell the other Loki components to add it to their ring, and that declares all the components in every cluster to think they’re in the same cluster
And it will error and make the cluster unhealthy since the components in Exposed Network-Zone can’t start and send new network traffic to components in the Protected Network Zone.

I’m looking for a suggestion on a alternative configuration on how to accomplish my goal with the same architecture.
Otherwise the solution i’d like would be a configuration option for querier component to read from additional ingester without forcing membership or a ring.

Screenshot 2024-06-29 000652
Screenshot 2024-06-29 0006521209×721 124 KB

2

This is purely my opinion. You already kinda answered your own question: you can’t form membership ring without the connectivity.

I am sure you are doing the network segregation for a reason, do you have a good reason to want to try and work around it?

Couple of alternatives that I think are potentially easier:

  1. Simply operate two Loki clusters, and if you wish to visualize them together put your Grafana instance in your protected zone and add both Loki clusters as data sources.
  2. Operate one Loki cluster in your exposed network zone.
  3. Add another in-between zone in your network topology where both protected and exposed zones can send traffic to it, but it cannot sent traffic to others, and operate your Loki cluster there.

Personally I’d prefer #1.

1 Like
3

Thanks alot for looking at my issue Tony.
Solution 1 will probably work for us. I’ll add the querier to the exposed zone and add to its datasource and see how stuff works.
We looked at solution 2, it will not be able to satisfy our security requirements…
Solution 3 would be my favorite but can’t be done in our niche situation due to service provider reasons…

If solution 1 becomes a hassle, i can just configure the ingester in the exposed zone to flush its logs very rapidly so we can retrieve somewhat fresh logs.
Hopefully the compactor component will then be able correct and optimize all those very small chunks.