I have been experiencing a strange behaviour from loki (ingesters) which is overwriting old logs/data with new ones after the /wal storage is full so what is happening as follow:
some ingester containers are restarting showing in the begging this log “level=error ts=2025-06-04T10:30:38.652380247Z caller=instance.go:261 msg=“Error writing to WAL, disk full, no further messages will be logged for this error”” then it start ingesting data.
Q) How did I know it is overwriting old logs/data?
A) when I search grafana on different times, oldest log is always about a fixed time delay from the Now date and can’t find any old data, which indecate that old data is overwritten.
Despite this documentation statement
No space left on disk
In the event the underlying WAL disk is full, Loki will not fail incoming writes, but neither will it log them to the WAL. In this case, the persistence guarantees across process restarts will not hold.
I am not quite sure I understand what your question is, please let me know if I misunderstood.
Logs that come into loki flow like this → logs → WAL → send to backend
Once logs are sent to the backend then it persists, it cannot be overwritten. If you cannot see older logs in Grafana it is not related to WAL.
Also you should try and make sure to not let it fill up.
Thank you for the respond, I maybe didn’t mention that but I intentionally removed inesters access permissions to S3 backend so they are not flushing now, what I expected is after /wal is full ingesters refuse accepting data or fail at least but not to overwrite on old data.
I see, so you are intentionally looking to see how WAL behaves when it’s faull?
In my opinion WAL should be treated as a temporary storage, almost like a cache. It’s not an uncommon practice to cycle out temporary storage when it’s full, and honestly either approach is somewhat reasonable since if it’s full then you should already expect something to break, whether it breaks new logs or old logs.
I agree with you that something will break but as in Docs it was mentioned that it won’t log new data Write Ahead Log | Grafana Loki documentation but as I see it does and overwrite old ones