How to fix the error there is no space left on the device but there are still plenty of disks available?

Can you help, detail logs error:

level=error ts=2023-08-29T03:42:05.034852437Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="open /data/loki/tmp/chunks/ZmFrZS81NjgxN2YyMm: no space left on device"

however disk /data space is still quite a lot

@harapan00bangsa disks have two finite resources: disk space and inodes. I suspect you’ve run out of inodes.

You can run df -hi /data to confirm this.

Can you please send your Loki config? This should not be happening anymore since we made some changes a while ago to prevent this.

Used inode /data 43%

Config : /data/loki/loki-local-config1.yaml

auth_enabled: false

  http_listen_port: 3100
  grpc_listen_port: 9096
  grpc_server_max_concurrent_streams: 0

  path_prefix: /data/loki/tmp
      chunks_directory: /data/loki/tmp/chunks
      rules_directory: /data/loki/tmp/rules
  replication_factor: 1
      store: inmemory

    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
        prefix: index_
        period: 24h

  alertmanager_url: 'http://localhost:9093'

Is Loki running in the container, cgroup?

loki & grafana running in pm2

No idea if pm2 uses container, cgroups under the hood.

Check OS logs, you may reach OS limits, e. g. number of open files

Help, how to check logs OS reach limit the number of open files how?

This is not a limit of open files error; this is a “no space left on device” error.
Are you checking the disk space and inode usage from within your container, or from the host OS?

my service grafana and loki running in the pm2.
i have checked disk and inode



Don’t follow error message literally. See for example python - multiprocessing: No space left on device - Stack Overflow - number of posix semaphores was reached

The ENOSPC (“No space left on device”) error will be triggered in any situation in which the data or the metadata associated with an I/O operation can’t be written down anywhere because of lack of space. This doesn’t always mean disk space – it could mean physical disk space, logical space (e.g. maximum file length), space in a certain data structure or address space. For example you can get it if there isn’t space in the directory table (vfat) or there aren’t any inodes left. It roughly means “I can’t find where to write this down”.

Hhmm, good to know. We’ve never encountered this scenario before, though.

@harapan00bangsa are you checking inside the container or in the host OS?
What filesystem are you using?

I would like to see outputs from:

uname -a
sysctl -a
lsof | wc -l
lsof +D /data/loki/tmp
netstat -tanp | wc -l
netstat -uanp | wc -l
df -T

Using VM (Ubuntu 20.04.5 LTS) don’t have access to host, file system: /dev/mapper/vg_ os-data and Mount /data

can you help me?


Try put his in /etc/sysctl.conf

vm.max_map_count = 262144
fs.file-max = 10000000
fs.nr_open = 10000000


For anyone else who comes across this issue of the Loki logs complaining there’s no space left on device.

Please refer to this Git issue

For me, I am running Grafana/Promtail/Loki/Syslog-ng in Docker container environment on a single vm with 32Gb mem, 8 cores with a 8Tb bind mount. Ingesting upwards of 4000 network devices.

enabling large_dir and a restart of the Loki container fixed it instantly.