How to fix the error there is no space left on the device but there are still plenty of disks available?

Grafana Loki
1

Can you help, detail logs error:

level=error ts=2023-08-29T03:42:05.034852437Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="open /data/loki/tmp/chunks/ZmFrZS81NjgxN2YyMm: no space left on device"

however disk /data space is still quite a lot

image
image891×24 796 Bytes

2

@harapan00bangsa disks have two finite resources: disk space and inodes. I suspect you’ve run out of inodes.

You can run df -hi /data to confirm this.

Can you please send your Loki config? This should not be happening anymore since we made some changes a while ago to prevent this.

4

Used inode /data 43%

image
image974×44 2 KB

Config : /data/loki/loki-local-config1.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  grpc_server_max_concurrent_streams: 0

common:
  path_prefix: /data/loki/tmp
  storage:
    filesystem:
      chunks_directory: /data/loki/tmp/chunks
      rules_directory: /data/loki/tmp/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: 'http://localhost:9093'
5

Is Loki running in the container, cgroup?

6

loki & grafana running in pm2

image
image856×85 3.34 KB

7

No idea if pm2 uses container, cgroups under the hood.

Check OS logs, you may reach OS limits, e. g. number of open files

8

Help, how to check logs OS reach limit the number of open files how?

9

This is not a limit of open files error; this is a “no space left on device” error.
Are you checking the disk space and inode usage from within your container, or from the host OS?

10

my service grafana and loki running in the pm2.
i have checked disk and inode

disk

image
image913×27 1.32 KB

inodes

image
image974×44 2 KB

11

Don’t follow error message literally. See for example python - multiprocessing: No space left on device - Stack Overflow - number of posix semaphores was reached

The ENOSPC (“No space left on device”) error will be triggered in any situation in which the data or the metadata associated with an I/O operation can’t be written down anywhere because of lack of space. This doesn’t always mean disk space – it could mean physical disk space, logical space (e.g. maximum file length), space in a certain data structure or address space. For example you can get it if there isn’t space in the directory table (vfat) or there aren’t any inodes left. It roughly means “I can’t find where to write this down”.

12

Hhmm, good to know. We’ve never encountered this scenario before, though.

@harapan00bangsa are you checking inside the container or in the host OS?
What filesystem are you using?

13

I would like to see outputs from:

uname -a
sysctl -a
lsof | wc -l
lsof +D /data/loki/tmp
netstat -tanp | wc -l
netstat -uanp | wc -l
df -T
14

Using VM (Ubuntu 20.04.5 LTS) don’t have access to host, file system: /dev/mapper/vg_ os-data and Mount /data

15

can you help me?

16

Hi,

Try put his in /etc/sysctl.conf

vm.max_map_count = 262144
fs.file-max = 10000000
fs.nr_open = 10000000

Regards
Fadjar

17

For anyone else who comes across this issue of the Loki logs complaining there’s no space left on device.

Please refer to this Git issue

For me, I am running Grafana/Promtail/Loki/Syslog-ng in Docker container environment on a single vm with 32Gb mem, 8 cores with a 8Tb bind mount. Ingesting upwards of 4000 network devices.

enabling large_dir and a restart of the Loki container fixed it instantly.