I recently configured Loki Distributed with Minio S3 with a retention period of 30 days.
The minio cluster has a storage of 160 GB.
The log is getting filled up rapidly 15GB/day and we will probably hit the limit before reaching 30 days. What happens when loki couldn’t write to minio anymore?
Does it automatically schedule the deletion of older logs, or does it stop working?
It would stop working.
So, I have to expand minio by adding cluster pools.
According to minio expansion,
Adding a new server pool requires restarting all MinIO nodes in the deployment at around same time.
MinIO strongly recommends restarting all nodes simultaneously. MinIO operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.
While each pool is an independent set of servers with distinct erasure sets for availability, the complete loss of one pool results in MinIO stopping I/O for all pools in the deployment.
During the degradation of size, would loki cache the logs that it couldn’t set at the time of minio degradation?
How do I do this properly?
I have very little experience in Minio, so probably not the best person to answer your question.