How to use Loki in a high-security environment?

A high-security environment divides system based on their security level. In those environments it is often allowed to initiate a connection from a system with a higher security level to a system with a lower security level. But the opposite direction is often forbidden to prevent the possibility that a system with a lower security level can attack a system with a higher security level. Systems with lower security level are placed in a so called demilitarized zone while systems with a higher security level are placed in the internal network. This means you can initiate a connection (TCP-SYN) from the internal network to the DMZ but you can not initiate a connection from the DMZ into the internal network.

Grafana collects all monitoring information. Therefor it needs the highest possible protection, because the information is necessary for forensic investigations in case of a security incident. It is not possible to put Grafana into the DMZ, instead it has to be put into the internal network. Logging data originates from the DMZ and has to be send to Grafana. It seems to me that the current implementation means that Promtail initiates a connection from the DMZ to Grafana in the internal network. This is definitely not allowed in high-security environments.

How is it possible to run Loki in a high-security environment where it would be allowed for Loki to initiate a connection to Promtail, but where it is not allowed for Promtail to initiate a connection to Loki?

That depends how you design your solution. I would say it is possible. You just need to think outside of standard Loki collection model + you need some queueing/streaming.
Maybe:

  • low sec. env: agent/promtail → Opentelemetry Collector → AWS Kinesis → Firehose
  • high sec. env: Firehose → Opentelemetry Collector → Loki