172.16.1.250: kern: info: [2024-12-28T17:56:53.513297666Z]: oom_reaper: reaped process 206218 (cdi-uploadserve), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731047666Z]: cdi-uploadserve invoked oom-killer: gfp_mask=0x101c4a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE), order=2, oom_score_adj=998 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731058666Z]: CPU: 3 UID: 107 PID: 206736 Comm: cdi-uploadserve Not tainted 6.12.5-talos #1 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731062666Z]: Hardware name: System manufacturer System Product Name/PRIME Z370-P, BIOS 3004 07/12/2021 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731065666Z]: Call Trace: 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731068666Z]: 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731072666Z]: dump_stack_lvl+0x5d/0x80 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731077666Z]: dump_header+0x44/0x181 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731081666Z]: oom_kill_process.cold+0x8/0x9d 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731084666Z]: out_of_memory+0x215/0x4f0 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731088666Z]: mem_cgroup_out_of_memory+0x12d/0x160 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731092666Z]: try_charge_memcg+0x49b/0x690 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731096666Z]: charge_memcg+0x2f/0x80 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731100666Z]: __mem_cgroup_charge+0x30/0x90 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731103666Z]: filemap_add_folio+0x46/0xe0 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731107666Z]: __filemap_get_folio+0x169/0x2c0 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731110666Z]: iomap_write_begin+0x40d/0x610 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731114666Z]: iomap_file_buffered_write+0x1ab/0x480 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731122666Z]: xfs_file_buffered_write+0x84/0x2e0 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731125666Z]: vfs_write+0x280/0x430 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731130666Z]: ksys_write+0x6d/0xf0 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731133666Z]: do_syscall_64+0x6d/0x110 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731141666Z]: entry_SYSCALL_64_after_hwframe+0x76/0x7e 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731146666Z]: RIP: 0033:0x40bcee 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731149666Z]: Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731154666Z]: RSP: 002b:000000c0005ab130 EFLAGS: 00000212 ORIG_RAX: 0000000000000001 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731158666Z]: RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 000000000040bcee 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731161666Z]: RDX: 0000000000004000 RSI: 000000c00063c000 RDI: 0000000000000009 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731164666Z]: RBP: 000000c0005ab170 R08: 0000000000000000 R09: 0000000000000000 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731167666Z]: R10: 0000000000000000 R11: 0000000000000212 R12: 000000c0005ab2a0 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731169666Z]: R13: 000000000000000a R14: 000000c0008e2380 R15: 0000000000004011 172.16.1.250: kern: warning: [2024-12-28T18:01:10.731173666Z]: 172.16.1.250: kern: info: [2024-12-28T18:01:10.731175666Z]: memory: usage 585936kB, limit 585936kB, failcnt 35435 172.16.1.250: kern: info: [2024-12-28T18:01:10.731177666Z]: swap: usage 0kB, limit 9007199254740988kB, failcnt 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731180666Z]: Memory cgroup stats for /kubepods/burstable/podf63d5ec3-8aa5-4308-8b2f-2a67b1d8a272: 172.16.1.250: kern: info: [2024-12-28T18:01:10.731216666Z]: anon 13213696 172.16.1.250: kern: info: [2024-12-28T18:01:10.731220666Z]: file 576897024 172.16.1.250: kern: info: [2024-12-28T18:01:10.731222666Z]: kernel 3784704 172.16.1.250: kern: info: [2024-12-28T18:01:10.731224666Z]: kernel_stack 180224 172.16.1.250: kern: info: [2024-12-28T18:01:10.731225666Z]: pagetables 364544 172.16.1.250: kern: info: [2024-12-28T18:01:10.731227666Z]: sec_pagetables 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731229666Z]: percpu 24256 172.16.1.250: kern: info: [2024-12-28T18:01:10.731230666Z]: sock 6103040 172.16.1.250: kern: info: [2024-12-28T18:01:10.731232666Z]: vmalloc 65536 172.16.1.250: kern: info: [2024-12-28T18:01:10.731233666Z]: shmem 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731235666Z]: file_mapped 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731237666Z]: file_dirty 576897024 172.16.1.250: kern: info: [2024-12-28T18:01:10.731239666Z]: file_writeback 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731240666Z]: swapcached 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731242666Z]: anon_thp 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731243666Z]: file_thp 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731245666Z]: shmem_thp 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731247666Z]: inactive_anon 13176832 172.16.1.250: kern: info: [2024-12-28T18:01:10.731248666Z]: active_anon 36864 172.16.1.250: kern: info: [2024-12-28T18:01:10.731250666Z]: inactive_file 576897024 172.16.1.250: kern: info: [2024-12-28T18:01:10.731252666Z]: active_file 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731253666Z]: unevictable 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731255666Z]: slab_reclaimable 2928360 172.16.1.250: kern: info: [2024-12-28T18:01:10.731257666Z]: slab_unreclaimable 186856 172.16.1.250: kern: info: [2024-12-28T18:01:10.731259666Z]: slab 3115216 172.16.1.250: kern: info: [2024-12-28T18:01:10.731260666Z]: workingset_refault_anon 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731262666Z]: workingset_refault_file 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731264666Z]: workingset_activate_anon 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731266666Z]: workingset_activate_file 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731267666Z]: workingset_restore_anon 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731269666Z]: workingset_restore_file 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731271666Z]: workingset_nodereclaim 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731273666Z]: pgdemote_kswapd 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731274666Z]: pgdemote_direct 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731276666Z]: pgdemote_khugepaged 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731278666Z]: pgscan 223220 172.16.1.250: kern: info: [2024-12-28T18:01:10.731279666Z]: pgsteal 223220 172.16.1.250: kern: info: [2024-12-28T18:01:10.731281666Z]: pgscan_kswapd 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731283666Z]: pgscan_direct 223220 172.16.1.250: kern: info: [2024-12-28T18:01:10.731284666Z]: pgscan_khugepaged 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731286666Z]: pgsteal_kswapd 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731288666Z]: pgsteal_direct 223220 172.16.1.250: kern: info: [2024-12-28T18:01:10.731289666Z]: pgsteal_khugepaged 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731291666Z]: pgfault 12269 172.16.1.250: kern: info: [2024-12-28T18:01:10.731293666Z]: pgmajfault 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731294666Z]: pgrefill 88045500 172.16.1.250: kern: info: [2024-12-28T18:01:10.731296666Z]: pgactivate 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731297666Z]: pgdeactivate 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731299666Z]: pglazyfree 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731301666Z]: pglazyfreed 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731302666Z]: swpin_zero 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731304666Z]: swpout_zero 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731306666Z]: thp_fault_alloc 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731307666Z]: thp_collapse_alloc 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731309666Z]: thp_swpout 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731310666Z]: thp_swpout_fallback 0 172.16.1.250: kern: info: [2024-12-28T18:01:10.731312666Z]: Tasks state (memory values in pages): 172.16.1.250: kern: info: [2024-12-28T18:01:10.731314666Z]: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name 172.16.1.250: kern: info: [2024-12-28T18:01:10.731318666Z]: [ 206192] 65535 206192 255 173 0 173 0 36864 0 -998 pause 172.16.1.250: kern: info: [2024-12-28T18:01:10.731323666Z]: [ 206715] 107 206715 489755 12040 3136 8904 0 327680 0 998 cdi-uploadserve 172.16.1.250: kern: info: [2024-12-28T18:01:10.731328666Z]: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=65544ffad882fc381ed6ec84e48879c2983005aa5033c1a1182d2201019071d8,mems_allowed=0,oom_memcg=/kubepods/burstable/podf63d5ec3-8aa5-4308-8b2f-2a67b1d8a272,task_memcg=/kubepods/burstable/podf63d5ec3-8aa5-4308-8b2f-2a67b1d8a272/65544ffad882fc381ed6ec84e48879c2983005aa5033c1a1182d2201019071d8,task=cdi-uploadserve,pid=206715,uid=107 172.16.1.250: kern: err: [2024-12-28T18:01:10.731351666Z]: Memory cgroup out of memory: Killed process 206715 (cdi-uploadserve) total-vm:1959020kB, anon-rss:12544kB, file-rss:35616kB, shmem-rss:0kB, UID:107 pgtables:320kB oom_score_adj:998 172.16.1.250: kern: info: [2024-12-28T18:01:10.731418666Z]: Tasks in /kubepods/burstable/podf63d5ec3-8aa5-4308-8b2f-2a67b1d8a272/65544ffad882fc381ed6ec84e48879c2983005aa5033c1a1182d2201019071d8 are going to be killed due to memory.oom.group set 172.16.1.250: kern: err: [2024-12-28T18:01:10.731424666Z]: Memory cgroup out of memory: Killed process 206733 (cdi-uploadserve) total-vm:1959020kB, anon-rss:12544kB, file-rss:35616kB, shmem-rss:0kB, UID:107 pgtables:320kB oom_score_adj:998