Can you help, detail logs error: level=error ts=2023-08-29T03:42:05.034852437Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="open /data/loki/tmp/chunks/ZmFrZS81NjgxN2YyMm: no space left on device" however disk /data space is still quite a lot [image]

@harapan00bangsa disks have two finite resources: disk space and inodes. I suspect you’ve run out of inodes. You can run df -hi /data to confirm this. Can you please send your Loki config? This should not be happening anymore since we made some changes a while ago to prevent this.

Used inode /data 43% [image] Config : /data/loki/loki-local-config1.yaml auth_enabled: false server: http_listen_port: 3100 grpc_listen_port: 9096 grpc_server_max_concurrent_streams: 0 common: path_prefix: /data/loki/tmp storage: filesystem: chunks_directory: /data/loki/tmp…

Is Loki running in the container, cgroup?

loki & grafana running in pm2 [image]

No idea if pm2 uses container, cgroups under the hood. Check OS logs, you may reach OS limits, e. g. number of open files

Help, how to check logs OS reach limit the number of open files how?

This is not a limit of open files error; this is a “no space left on device” error. Are you checking the disk space and inode usage from within your container, or from the host OS?

my service grafana and loki running in the pm2. i have checked disk and inode disk [image] inodes [image]

[image] anon74857911: This is not a limit of open files error; this is a “no space left on device” error. Don’t follow error message literally. See for example https://stackoverflow.com/questions/70175977/multiprocessing-no-space-left-on-device - number of posix semaphores was reached The E…

Hhmm, good to know. We’ve never encountered this scenario before, though. @harapan00bangsa are you checking inside the container or in the host OS? What filesystem are you using?

How to fix the error there is no space left on the device but there are still plenty of disks available?

Grafana Loki

harapan00bangsa September 10, 2023, 12:41pm 15

can you help me?

Topic		Replies	Views
Loki space issue Grafana Loki	1	2701	March 23, 2023
LOKI Consumes Large Disk Space in /share/containers/storage Grafana Loki	4	1449	January 15, 2025
Loki scalable helm chart read pod stop working Grafana Loki	2	1664	January 22, 2022
Loki wal use big part of diskspace Grafana Loki loki	0	573	July 14, 2022
'du' and 'df' showing different usage Grafana Loki loki	10	280	February 7, 2025

How to fix the error there is no space left on the device but there are still plenty of disks available?

Related topics