Loki high throughput and disk size usage in less than 5 minutes

Hello

I had an incident with loki yesterday where the disk went from 2% usage to 100% usage in less than 5 minutes. This is the second time it happened the first one being approx 2 months before the second.
The disk usage is usually around 2% to 20% max and I also checked that there are not many logs ingested during that period.
What could be the source of this problem ?



Hard to say without more information.

What’s in the disk? How much log was being ingested? Any error message from any of the Loki container?