general:ipkm_it_post_mortem
This is an old revision of the document!
2025-02-06 mail server crash
Mail server crashed. Reason not fully clear, was working until around 15:05. Mail host was responding to pings but ssh and all mail/LDAP related stuff was not working.
Node was moved to pve-gustav (which ran a different qemu-kvm version, wee below) the evening before.
Somebody rebooted the MV around 15:08:
Feb 6 15:08:10 mail kernel: [ 0.000000] Linux version 5.10.0-33-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.226-1 (2024-10-03) <pre><span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">811753</span>] Hardware name: QEMU Standard PC (i440FX + PIIX, <span style="color:#FF00FF">1996</span>), BIOS rel-<span style="color:#FF00FF">1</span>.<span style="color:#FF00FF">16</span>.<span style="color:#FF00FF">2</span>-<span style="color:#FF00FF">0</span>-gea1b7a073390-prebuilt.qemu.org <span style="color:#FF00FF">04</span>/<span style="color:#FF00FF">01</span>/<span style="color:#FF00FF">2014</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">811783</span>] Call Trace: <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">813554</span>] dump_stack+<span style="color:#FF00FF">0x6b</span>/<span style="color:#FF00FF">0x83</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">814022</span>] dump_header+<span style="color:#FF00FF">0x4a</span>/<span style="color:#FF00FF">0x1f4</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">814448</span>] oom_kill_process.cold+<span style="color:#FF00FF">0xb</span>/<span style="color:#FF00FF">0x10</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">814827</span>] out_of_memory+<span style="color:#FF00FF">0x1bd</span>/<span style="color:#FF00FF">0x4e0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">815223</span>] __alloc_pages_slowpath.constprop.<span style="color:#FF00FF">0</span>+<span style="color:#FF00FF">0xc02</span>/<span style="color:#FF00FF">0xcc0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">815580</span>] __alloc_pages_nodemask+<span style="color:#FF00FF">0x2de</span>/<span style="color:#FF00FF">0x310</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">815942</span>] pagecache_get_page+<span style="color:#FF00FF">0x175</span>/<span style="color:#FF00FF">0x390</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">816294</span>] filemap_fault+<span style="color:#FF00FF">0x6a2</span>/<span style="color:#FF00FF">0x900</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">816655</span>] ? xas_load+<span style="color:#FF00FF">0x5</span>/<span style="color:#FF00FF">0x80</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">817069</span>] ext4_filemap_fault+<span style="color:#FF00FF">0x2d</span>/<span style="color:#FF00FF">0x50</span> [ext4] <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">817430</span>] __do_fault+<span style="color:#FF00FF">0x37</span>/<span style="color:#FF00FF">0x170</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">817754</span>] handle_mm_fault+<span style="color:#FF00FF">0x124d</span>/<span style="color:#FF00FF">0x1c00</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">818145</span>] do_user_addr_fault+<span style="color:#FF00FF">0x1b8</span>/<span style="color:#FF00FF">0x400</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">818484</span>] exc_page_fault+<span style="color:#FF00FF">0x78</span>/<span style="color:#FF00FF">0x160</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">818785</span>] ? asm_exc_page_fault+<span style="color:#FF00FF">0x8</span>/<span style="color:#FF00FF">0x30</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">819121</span>] asm_exc_page_fault+<span style="color:#FF00FF">0x1e</span>/<span style="color:#FF00FF">0x30</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">819439</span>] RIP: <span style="color:#FF00FF">0033</span>:<span style="color:#FF00FF">0x7f8e1fe9d386</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">819740</span>] Code: Unable to access opcode bytes at RIP <span style="color:#FF00FF">0x7f8e1fe9d35c</span>. <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">820066</span>] RSP: <span style="color:#FF00FF">002b</span>:<span style="color:#FF00FF">00007fff7fde5570</span> EFLAGS: <span style="color:#FF00FF">00010202</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">820387</span>] RAX: <span style="color:#FF00FF">0000000000000000</span> RBX: <span style="color:#FF00FF">0000000000000000</span> RCX: <span style="color:#FF00FF">0000000000000004</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">820707</span>] RDX: <span style="color:#FF00FF">00007f8e1feb09a0</span> RSI: <span style="color:#FF00FF">0000000000000000</span> RDI: <span style="color:#FF00FF">00005576b32a0270</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">821115</span>] RBP: <span style="color:#FF00FF">00005576b32a0270</span> R08: <span style="color:#FF00FF">00005576b32a0270</span> R09: <span style="color:#FF00FF">00007f8e1fe73be0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">821465</span>] R10: <span style="color:#FF00FF">00005576b32a0170</span> R11: <span style="color:#FF00FF">0000000000000070</span> R12: <span style="color:#FF00FF">00007fff7fde55bc</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">821764</span>] R13: <span style="color:#FF00FF">0000000000000004</span> R14: <span style="color:#FF00FF">0000000000000000</span> R15: <span style="color:#FF00FF">00007fff7fde5890</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822111</span>] Mem-Info: <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>] active_anon:<span style="color:#FF00FF">62</span> inactive_anon:<span style="color:#FF00FF">4025</span> isolated_anon:<span style="color:#FF00FF">0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>] active_file:<span style="color:#FF00FF">132</span> inactive_file:<span style="color:#FF00FF">35</span> isolated_file:<span style="color:#FF00FF">0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>] unevictable:<span style="color:#FF00FF">0</span> dirty:<span style="color:#FF00FF">0</span> writeback:<span style="color:#FF00FF">0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>] slab_reclaimable:<span style="color:#FF00FF">3337</span> slab_unreclaimable:<span style="color:#FF00FF">7300</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>] mapped:<span style="color:#FF00FF">138</span> shmem:<span style="color:#FF00FF">739</span> pagetables:<span style="color:#FF00FF">315</span> bounce:<span style="color:#FF00FF">0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>] free:<span style="color:#FF00FF">11707</span> free_pcp:<span style="color:#FF00FF">0</span> free_cma:<span style="color:#FF00FF">0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">824222</span>] Node <span style="color:#FF00FF">0</span> active_anon:<span style="color:#FF00FF">248</span>kB inactive_anon:<span style="color:#FF00FF">16100</span>kB active_file:<span style="color:#FF00FF">284</span>kB inactive_file:<span style="color:#FF00FF">88</span>kB unevictable:<span style="color:#FF00FF">0</span>kB isolated(anon):<span style="color:#FF00FF">0</span>kB isolated(file):<span style="color:#FF00FF">0</span>kB mapped:<span style="color:#FF00FF">552</span>kB dirty:<span style="color:#FF00FF">0</span>kB writeback:<span style="color:#FF00FF">0</span>kB shmem:<span style="color:#FF00FF">2956</span>kB shmem_thp: <span style="color:#FF00FF">0</span>kB shmem_pmdmapped: <span style="color:#FF00FF">0</span>kB anon_thp: <span style="color:#FF00FF">0</span>kB writeback_tmp:<span style="color:#FF00FF">0</span>kB kernel_stack:<span style="color:#FF00FF">2832</span>kB all_unreclaimable? no <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">824849</span>] Node <span style="color:#FF00FF">0</span> DMA free:<span style="color:#FF00FF">4128</span>kB min:<span style="color:#FF00FF">788</span>kB low:<span style="color:#FF00FF">984</span>kB high:<span style="color:#FF00FF">1180</span>kB reserved_highatomic:<span style="color:#FF00FF">0</span>KB active_anon:<span style="color:#FF00FF">0</span>kB inactive_anon:<span style="color:#FF00FF">4</span>kB active_file:<span style="color:#FF00FF">0</span>kB inactive_file:<span style="color:#FF00FF">0</span>kB unevictable:<span style="color:#FF00FF">0</span>kB writepending:<span style="color:#FF00FF">0</span>kB present:<span style="color:#FF00FF">15992</span>kB managed:<span style="color:#FF00FF">5076</span>kB mlocked:<span style="color:#FF00FF">0</span>kB pagetables:<span style="color:#FF00FF">0</span>kB bounce:<span style="color:#FF00FF">0</span>kB free_pcp:<span style="color:#FF00FF">0</span>kB local_pcp:<span style="color:#FF00FF">0</span>kB free_cma:<span style="color:#FF00FF">0</span>kB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">825570</span>] lowmem_reserve[]: <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">844</span> <span style="color:#FF00FF">844</span> <span style="color:#FF00FF">844</span> <span style="color:#FF00FF">844</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">825933</span>] Node <span style="color:#FF00FF">0</span> DMA32 free:<span style="color:#FF00FF">42896</span>kB min:<span style="color:#FF00FF">42940</span>kB low:<span style="color:#FF00FF">53672</span>kB high:<span style="color:#FF00FF">64404</span>kB reserved_highatomic:<span style="color:#FF00FF">0</span>KB active_anon:<span style="color:#FF00FF">248</span>kB inactive_anon:<span style="color:#FF00FF">16096</span>kB active_file:<span style="color:#FF00FF">48</span>kB inactive_file:<span style="color:#FF00FF">292</span>kB unevictable:<span style="color:#FF00FF">0</span>kB writepending:<span style="color:#FF00FF">0</span>kB present:<span style="color:#FF00FF">1032040</span>kB managed:<span style="color:#FF00FF">652036</span>kB mlocked:<span style="color:#FF00FF">0</span>kB pagetables:<span style="color:#FF00FF">1260</span>kB bounce:<span style="color:#FF00FF">0</span>kB free_pcp:<span style="color:#FF00FF">0</span>kB local_pcp:<span style="color:#FF00FF">0</span>kB free_cma:<span style="color:#FF00FF">0</span>kB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">827042</span>] lowmem_reserve[]: <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">827439</span>] Node <span style="color:#FF00FF">0</span> DMA: <span style="color:#FF00FF">26</span>*<span style="color:#FF00FF">4</span>kB (M) <span style="color:#FF00FF">13</span>*<span style="color:#FF00FF">8</span>kB (UM) <span style="color:#FF00FF">13</span>*<span style="color:#FF00FF">16</span>kB (M) <span style="color:#FF00FF">10</span>*<span style="color:#FF00FF">32</span>kB (UM) <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">64</span>kB (M) <span style="color:#FF00FF">4</span>*<span style="color:#FF00FF">128</span>kB (UM) <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">256</span>kB (U) <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">512</span>kB (M) <span style="color:#FF00FF">0</span>*<span style="color:#FF00FF">1024</span>kB <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">2048</span>kB (M) <span style="color:#FF00FF">0</span>*<span style="color:#FF00FF">4096</span>kB = <span style="color:#FF00FF">4128</span>kB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">828236</span>] Node <span style="color:#FF00FF">0</span> DMA32: <span style="color:#FF00FF">476</span>*<span style="color:#FF00FF">4</span>kB (UME) <span style="color:#FF00FF">201</span>*<span style="color:#FF00FF">8</span>kB (ME) <span style="color:#FF00FF">136</span>*<span style="color:#FF00FF">16</span>kB (ME) <span style="color:#FF00FF">87</span>*<span style="color:#FF00FF">32</span>kB (UME) <span style="color:#FF00FF">86</span>*<span style="color:#FF00FF">64</span>kB (UME) <span style="color:#FF00FF">48</span>*<span style="color:#FF00FF">128</span>kB (UME) <span style="color:#FF00FF">19</span>*<span style="color:#FF00FF">256</span>kB (UME) <span style="color:#FF00FF">3</span>*<span style="color:#FF00FF">512</span>kB (UM) <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">1024</span>kB (U) <span style="color:#FF00FF">8</span>*<span style="color:#FF00FF">2048</span>kB (UME) <span style="color:#FF00FF">0</span>*<span style="color:#FF00FF">4096</span>kB = <span style="color:#FF00FF">43928</span>kB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">829155</span>] Node <span style="color:#FF00FF">0</span> hugepages_total=<span style="color:#FF00FF">0</span> hugepages_free=<span style="color:#FF00FF">0</span> hugepages_surp=<span style="color:#FF00FF">0</span> hugepages_size=<span style="color:#FF00FF">1048576</span>kB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">829594</span>] Node <span style="color:#FF00FF">0</span> hugepages_total=<span style="color:#FF00FF">0</span> hugepages_free=<span style="color:#FF00FF">0</span> hugepages_surp=<span style="color:#FF00FF">0</span> hugepages_size=<span style="color:#FF00FF">2048</span>kB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">830021</span>] <span style="color:#FF00FF">913</span> total pagecache pages <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">830427</span>] <span style="color:#FF00FF">0</span> pages in swap cache <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">830779</span>] Swap cache stats: add <span style="color:#FF00FF">0</span>, delete <span style="color:#FF00FF">0</span>, find <span style="color:#FF00FF">0</span>/<span style="color:#FF00FF">0</span> <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">831302</span>] Free swap = <span style="color:#FF00FF">0</span>kB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">831726</span>] Total swap = <span style="color:#FF00FF">0</span>kB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">832224</span>] <span style="color:#FF00FF">262008</span> pages RAM <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">832793</span>] <span style="color:#FF00FF">0</span> pages HighMem/MovableOnly <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">833272</span>] <span style="color:#FF00FF">97730</span> pages reserved <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">833657</span>] <span style="color:#FF00FF">0</span> pages hwpoisoned <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">834067</span>] Unreclaimable slab info: <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">834458</span>] Name Used Total <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">834929</span>] ext4_system_zone <span style="color:#FF00FF">3</span>KB <span style="color:#FF00FF">3</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">835368</span>] scsi_sense_cache <span style="color:#FF00FF">400</span>KB <span style="color:#FF00FF">400</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">835764</span>] RAWv6 <span style="color:#FF00FF">30</span>KB <span style="color:#FF00FF">30</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">836221</span>] UDPv6 <span style="color:#FF00FF">94</span>KB <span style="color:#FF00FF">94</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">836631</span>] mqueue_inode_cache <span style="color:#FF00FF">31</span>KB <span style="color:#FF00FF">31</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">837034</span>] UNIX <span style="color:#FF00FF">382</span>KB <span style="color:#FF00FF">382</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">837435</span>] RAW <span style="color:#FF00FF">32</span>KB <span style="color:#FF00FF">32</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">837808</span>] hugetlbfs_inode_cache <span style="color:#FF00FF">30</span>KB <span style="color:#FF00FF">30</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">838234</span>] eventpoll_pwq <span style="color:#FF00FF">47</span>KB <span style="color:#FF00FF">47</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">838639</span>] request_queue <span style="color:#FF00FF">411</span>KB <span style="color:#FF00FF">506</span>KB <span style="color:#FF00FF">Feb 6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[ <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">839075</span>] biovec-max <span style="color:#FF00FF">480</span>KB <span style="color:#FF00FF">480</span>KB </pre>
Going in to the Proxmox web interface and attach the console to see if there is any output on the terminal revealed some kind of memory issues:
Fix: Disable the option for memory hotplug!
hotplug: disk,network,usb,cpu # was additionally with memory
pve-donna.cluster: ii pve-qemu-kvm 8.1.5-5 amd64 Full virtualization on x86 hardware pve-emil.cluster: ii pve-qemu-kvm 8.1.5-5 amd64 Full virtualization on x86 hardware pve-franz.cluster: ii pve-qemu-kvm 9.0.2-4 amd64 Full virtualization on x86 hardware pve-gustav.cluster: ii pve-qemu-kvm 8.1.5-5 amd64 Full virtualization on x86 hardware pve-hans.cluster: ii pve-qemu-kvm 8.1.5-5 amd64 Full virtualization on x86 hardware
general/ipkm_it_post_mortem.1738918907.txt.gz · Last modified: 2025-02-07 10:01 by Markus Rosenstihl