I am designing a monitoring backend cluster that needs to get data from child kube clusters (which are already running the prom/loki/otel stack) and then store these in a local mimir/loki stack. My main concern is that this cluster is hosted in a quite high security zone and cannot have incoming connections. As far as I understand Alloy and Opentelemetry does not have a native way to be scraped by another instance of Alloy, or directly scrape Loki/Prom.
Does anyone can think of a architecture solution to handle this?
TLDR: Can Alloy exporter work in listen mode instead of push?
Which cluster is in the security zone? The child kube clusters or your local stack?
Are you looking to replicate data from loki logs to another loki (and mimir metrics to another mimir), or are you looking to just send metric and logs of a remote EKS cluster to a local Loki/Mimir stack?
The centralized backend (it is not local, but on a secured private cloud), the one running mimir. Not the child cluster.
I want all my child clusters logs/metrics to be sent to the secured backend as well. The only thing is, initiative should be from the backend (pull). I guess replication is an option. But I don’t think mimir can replicate a prom instance? (Missing X-Org-Id information?)
As far as I know neither Loki nor Mimir has functionality to replicate data on a cluster level. You’ll probably have to come up with a creative solution.
If you are just looking to have all your metrics and logs on one centralized cluster, perhaps you can consider standing up your loki and mimir clusters in a DMZ zone where both your secure and non-secure environments can connect to:
non-secure → DMZ ← secure
Then, in your Kubernetes cluster, setup some sort of forwarder (you can use Alloy for this) such as alloy-singleton that can be the forward targets of all your other agents, and then the alloy-singleton then forwards to the loki/mimir cluster in the DMZ zone.
If that doesn’t fit your use case please elaborate and we’ll see if there are other things we can try.
