Can you help, detail logs error:
level=error ts=2023-08-29T03:42:05.034852437Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="open /data/loki/tmp/chunks/ZmFrZS81NjgxN2YyMm: no space left on device"
however disk /data space is still quite a lot
@harapan00bangsa disks have two finite resources: disk space and inodes. I suspect you’ve run out of inodes.
You can run
df -hi /data to confirm this.
Can you please send your Loki config? This should not be happening anymore since we made some changes a while ago to prevent this.
Used inode /data 43%
- from: 2020-10-24
Is Loki running in the container, cgroup?
loki & grafana running in pm2
No idea if pm2 uses container, cgroups under the hood.
Check OS logs, you may reach OS limits, e. g. number of open files
Help, how to check logs OS reach limit the number of open files how?
This is not a limit of open files error; this is a “no space left on device” error.
Are you checking the disk space and inode usage from within your container, or from the host OS?
my service grafana and loki running in the pm2.
i have checked disk and inode
Don’t follow error message literally. See for example python - multiprocessing: No space left on device - Stack Overflow - number of posix semaphores was reached
The ENOSPC (“No space left on device”) error will be triggered in any situation in which the data or the metadata associated with an I/O operation can’t be written down anywhere because of lack of space. This doesn’t always mean disk space – it could mean physical disk space, logical space (e.g. maximum file length), space in a certain data structure or address space. For example you can get it if there isn’t space in the directory table (vfat) or there aren’t any inodes left. It roughly means “I can’t find where to write this down”.
Hhmm, good to know. We’ve never encountered this scenario before, though.
@harapan00bangsa are you checking inside the container or in the host OS?
What filesystem are you using?
I would like to see outputs from:
lsof | wc -l
lsof +D /data/loki/tmp
netstat -tanp | wc -l
netstat -uanp | wc -l
Using VM (Ubuntu 20.04.5 LTS) don’t have access to host, file system:
/dev/mapper/vg_ os-data and Mount
Try put his in /etc/sysctl.conf
vm.max_map_count = 262144
fs.file-max = 10000000
fs.nr_open = 10000000