Hello everyone,
Facing an issue where prometheus is up for sometime and is able to write in chunk_heads, but after some interval it throws err “Operation not permitted” even though I checked all dir/file permissions are ok, after restart its solved runs again for sometime and goes down again,
Any idea?
prometheus, version 2.37.0
level=error component=tsdb msg=“compaction failed” err="compact head: persist head block: 2 errors: populate block: write chunks: write /data/lib/prometheus/01JYKA4QJ10TR0XFKJYZBQH2BQ.tmp-for-creation/chunks/000001
Jun 25 17:00:00 botitsappv300 prometheus[16944]: panic: write /data/lib/prometheus/chunks_head/000006: operation not permitted
Jun 25 17:00:00 botitsappv300 prometheus[16944]: goroutine 869 [running]:
Jun 25 17:00:00 botitsappv300 prometheus[16944]: github.com/prometheus/prometheus/tsdb.handleChunkWriteError({0x395e8c0?, 0xc03f1c46c0?})
Jun 25 17:00:00 botitsappv300 systemd[1]: prometheus.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 25 17:00:00 botitsappv300 systemd[1]: Unit prometheus.service entered failed state.
Jun 25 17:00:00 botitsappv300 systemd[1]: prometheus.service failed
Adding more logs for reference, I have enough memory and have limited max-block-duration to 30mins
Jul 4 16:20:01 botitsappv300 kernel: Mem-Info:
Jul 4 16:20:01 botitsappv300 kernel: active_anon:1648526 inactive_anon:346964 isolated_anon:0#012 active_file:566372 inactive_file:1263242 isolated_file:0#012 unevictable:3 dirty:726 writeback:0 unstable:0#012 slab_reclaimable:74738 slab_unreclaimable:11327#012 mapped:158733 shmem:193052 pagetables:6960 bounce:0#012 free:72576 free_pcp:29 free_cma:0
Jul 4 16:20:01 botitsappv300 kernel: Node 0 DMA free:15900kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jul 4 16:20:01 botitsappv300 kernel: lowmem_reserve: 0 2830 15866 15866
Jul 4 16:20:01 botitsappv300 kernel: Node 0 DMA32 free:98684kB min:12044kB low:15052kB high:18064kB active_anon:1206024kB inactive_anon:455468kB active_file:416044kB inactive_file:633788kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2898784kB mlocked:0kB dirty:436kB writeback:0kB mapped:117480kB shmem:107488kB slab_reclaimable:52268kB slab_unreclaimable:6444kB kernel_stack:1008kB pagetables:4524kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 4 16:20:01 botitsappv300 kernel: lowmem_reserve: 0 0 13035 13035
Jul 4 16:20:01 botitsappv300 kernel: Node 0 Normal free:175720kB min:55468kB low:69332kB high:83200kB active_anon:5388080kB inactive_anon:932388kB active_file:1849444kB inactive_file:4419180kB unevictable:12kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13351608kB mlocked:12kB dirty:2468kB writeback:0kB mapped:517452kB shmem:664720kB slab_reclaimable:246684kB slab_unreclaimable:38856kB kernel_stack:4256kB pagetables:23316kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 4 16:20:01 botitsappv300 kernel: lowmem_reserve: 0 0 0 0
Jul 4 16:20:01 botitsappv300 kernel: Node 0 DMA: 14kB (U) 18kB (U) 116kB (U) 032kB 264kB (U) 1128kB (U) 1256kB (U) 0512kB 11024kB (U) 12048kB (M) 34096kB (M) = 15900kB
Jul 4 16:20:01 botitsappv300 kernel: Node 0 DMA32: 67314kB (UEM) 23888kB (UEM) 95216kB (UEM) 27332kB (UEM) 2864kB (UEM) 12128kB (UEM) 10256kB (UEM) 7512kB (UEM) 51024kB (UM) 52048kB (M) 14096kB (M) = 98924kB
Jul 4 16:20:01 botitsappv300 kernel: Node 0 Normal: 78074kB (UEM) 71548kB (UEM) 291116kB (UEM) 39632kB (UEM) 3564kB (UEM) 27128kB (UEM) 10256kB (UEM) 5512kB (M) 51024kB (UM) 42048kB (M) 1*4096kB (M) = 175932kB
Jul 4 16:20:01 botitsappv300 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jul 4 16:20:01 botitsappv300 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 4 16:20:01 botitsappv300 kernel: 2022666 total pagecache pages
Jul 4 16:20:01 botitsappv300 kernel: 0 pages in swap cache
Jul 4 16:20:01 botitsappv300 kernel: Swap cache stats: add 0, delete 0, find 0/0
Jul 4 16:20:01 botitsappv300 kernel: Free swap = 16773116kB
Jul 4 16:20:01 botitsappv300 kernel: Total swap = 16773116kB
Jul 4 16:20:01 botitsappv300 kernel: 4194174 pages RAM
Jul 4 16:20:01 botitsappv300 kernel: 0 pages HighMem/MovableOnly
Jul 4 16:20:01 botitsappv300 kernel: 127599 pages reserved
Jul 4 16:20:01 botitsappv300 prometheus: panic: write /data/lib/prometheus/chunks_head/000018: operation not permitted
