Hi,
Due to security reason I need to delay the logs that are viewed on loki. I am using fluent-bit to push the logs.
The fluen-bit has an option of flush time. But this will push the logs only on that flush interval.
Is there a way in which logs can be streamed in loki with N minutes of delay?
Thanks in advance.
I can’t think of a neat solution for this. Hopefully others can chime in.
One thing you can do is set up a fluentd cluster, and introduce two caching layers. First layer is strictly accept and cache, second layer is to retrieve from first layer, cache, then process. By doing this you introduce some delay naturally on when logs will be pushed. Needless to say this won’t be reliable, and if you are doing this for security reason you probably want something with more control.
You can of course go with Kafka and a custom extractor, but that depends on how much resource you can invest into this.