IPKM DokuWiki

User Tools

Site Tools

Trace:
general:ipkm_it_post_mortem

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
general:ipkm_it_post_mortem [2025-02-07 10:01] – created Markus Rosenstihlgeneral:ipkm_it_post_mortem [2025-02-07 11:09] (current) – Markus Rosenstihl
Line 3: Line 3:
 Mail host was responding to pings but ssh and all mail/LDAP related stuff was not working. Mail host was responding to pings but ssh and all mail/LDAP related stuff was not working.
  
-Node was moved to pve-gustav (which ran a different qemu-kvm version, wee below) the evening before.+<del>Node was moved to pve-gustav (which ran a different qemu-kvm version, wee below) the evening before.</del> Not according to the logfiles. 
 + 
 +Graylog search: "update VM 105"  
 +The hotplug setting has not been changed either.
  
 Somebody rebooted the MV around 15:08: Somebody rebooted the MV around 15:08:
  
-<code>+<code syslog>
 Feb  6 15:08:10 mail kernel: [    0.000000] Linux version 5.10.0-33-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.226-1 (2024-10-03) Feb  6 15:08:10 mail kernel: [    0.000000] Linux version 5.10.0-33-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.226-1 (2024-10-03)
- +... 
-<pre><span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">811753</span>] Hardware name: QEMU Standard PC (i440FX + PIIX, <span style="color:#FF00FF">1996</span>), BIOS rel-<span style="color:#FF00FF">1</span>.<span style="color:#FF00FF">16</span>.<span style="color:#FF00FF">2</span>-<span style="color:#FF00FF">0</span>-gea1b7a073390-prebuilt.qemu.org <span style="color:#FF00FF">04</span>/<span style="color:#FF00FF">01</span>/<span style="color:#FF00FF">2014</span> +Feb  6 15:08:10 mail kernel: [   26.811753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">811783</span>] Call Trace: +Feb  6 15:08:10 mail kernel: [   26.811783] Call Trace: 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">813554</span>]  dump_stack+<span style="color:#FF00FF">0x6b</span>/<span style="color:#FF00FF">0x83</span> +Feb  6 15:08:10 mail kernel: [   26.813554]  dump_stack+0x6b/0x83 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">814022</span>]  dump_header+<span style="color:#FF00FF">0x4a</span>/<span style="color:#FF00FF">0x1f4</span> +Feb  6 15:08:10 mail kernel: [   26.814022]  dump_header+0x4a/0x1f4 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">814448</span>]  oom_kill_process.cold+<span style="color:#FF00FF">0xb</span>/<span style="color:#FF00FF">0x10</span> +Feb  6 15:08:10 mail kernel: [   26.814448]  oom_kill_process.cold+0xb/0x10 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">814827</span>]  out_of_memory+<span style="color:#FF00FF">0x1bd</span>/<span style="color:#FF00FF">0x4e0</span> +Feb  6 15:08:10 mail kernel: [   26.814827]  out_of_memory+0x1bd/0x4e0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">815223</span>]  __alloc_pages_slowpath.constprop.<span style="color:#FF00FF">0</span>+<span style="color:#FF00FF">0xc02</span>/<span style="color:#FF00FF">0xcc0</span> +Feb  6 15:08:10 mail kernel: [   26.815223]  __alloc_pages_slowpath.constprop.0+0xc02/0xcc0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">815580</span>]  __alloc_pages_nodemask+<span style="color:#FF00FF">0x2de</span>/<span style="color:#FF00FF">0x310</span> +Feb  6 15:08:10 mail kernel: [   26.815580]  __alloc_pages_nodemask+0x2de/0x310 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">815942</span>]  pagecache_get_page+<span style="color:#FF00FF">0x175</span>/<span style="color:#FF00FF">0x390</span> +Feb  6 15:08:10 mail kernel: [   26.815942]  pagecache_get_page+0x175/0x390 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">816294</span>]  filemap_fault+<span style="color:#FF00FF">0x6a2</span>/<span style="color:#FF00FF">0x900</span> +Feb  6 15:08:10 mail kernel: [   26.816294]  filemap_fault+0x6a2/0x900 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">816655</span>]  ? xas_load+<span style="color:#FF00FF">0x5</span>/<span style="color:#FF00FF">0x80</span> +Feb  6 15:08:10 mail kernel: [   26.816655]  ? xas_load+0x5/0x80 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">817069</span>]  ext4_filemap_fault+<span style="color:#FF00FF">0x2d</span>/<span style="color:#FF00FF">0x50</span> [ext4] +Feb  6 15:08:10 mail kernel: [   26.817069]  ext4_filemap_fault+0x2d/0x50 [ext4] 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">817430</span>]  __do_fault+<span style="color:#FF00FF">0x37</span>/<span style="color:#FF00FF">0x170</span> +Feb  6 15:08:10 mail kernel: [   26.817430]  __do_fault+0x37/0x170 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">817754</span>]  handle_mm_fault+<span style="color:#FF00FF">0x124d</span>/<span style="color:#FF00FF">0x1c00</span> +Feb  6 15:08:10 mail kernel: [   26.817754]  handle_mm_fault+0x124d/0x1c00 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">818145</span>]  do_user_addr_fault+<span style="color:#FF00FF">0x1b8</span>/<span style="color:#FF00FF">0x400</span> +Feb  6 15:08:10 mail kernel: [   26.818145]  do_user_addr_fault+0x1b8/0x400 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">818484</span>]  exc_page_fault+<span style="color:#FF00FF">0x78</span>/<span style="color:#FF00FF">0x160</span> +Feb  6 15:08:10 mail kernel: [   26.818484]  exc_page_fault+0x78/0x160 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">818785</span>]  ? asm_exc_page_fault+<span style="color:#FF00FF">0x8</span>/<span style="color:#FF00FF">0x30</span> +Feb  6 15:08:10 mail kernel: [   26.818785]  ? asm_exc_page_fault+0x8/0x30 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">819121</span>]  asm_exc_page_fault+<span style="color:#FF00FF">0x1e</span>/<span style="color:#FF00FF">0x30</span> +Feb  6 15:08:10 mail kernel: [   26.819121]  asm_exc_page_fault+0x1e/0x30 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">819439</span>] RIP: <span style="color:#FF00FF">0033</span>:<span style="color:#FF00FF">0x7f8e1fe9d386</span> +Feb  6 15:08:10 mail kernel: [   26.819439] RIP: 0033:0x7f8e1fe9d386 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">819740</span>] Code: Unable to access opcode bytes at RIP <span style="color:#FF00FF">0x7f8e1fe9d35c</span>. +Feb  6 15:08:10 mail kernel: [   26.819740] Code: Unable to access opcode bytes at RIP 0x7f8e1fe9d35c. 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">820066</span>] RSP: <span style="color:#FF00FF">002b</span>:<span style="color:#FF00FF">00007fff7fde5570</span> EFLAGS: <span style="color:#FF00FF">00010202</span> +Feb  6 15:08:10 mail kernel: [   26.820066] RSP: 002b:00007fff7fde5570 EFLAGS: 00010202 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">820387</span>] RAX: <span style="color:#FF00FF">0000000000000000</span> RBX: <span style="color:#FF00FF">0000000000000000</span> RCX: <span style="color:#FF00FF">0000000000000004</span> +Feb  6 15:08:10 mail kernel: [   26.820387] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000004 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">820707</span>] RDX: <span style="color:#FF00FF">00007f8e1feb09a0</span> RSI: <span style="color:#FF00FF">0000000000000000</span> RDI: <span style="color:#FF00FF">00005576b32a0270</span> +Feb  6 15:08:10 mail kernel: [   26.820707] RDX: 00007f8e1feb09a0 RSI: 0000000000000000 RDI: 00005576b32a0270 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">821115</span>] RBP: <span style="color:#FF00FF">00005576b32a0270</span> R08: <span style="color:#FF00FF">00005576b32a0270</span> R09: <span style="color:#FF00FF">00007f8e1fe73be0</span> +Feb  6 15:08:10 mail kernel: [   26.821115] RBP: 00005576b32a0270 R08: 00005576b32a0270 R09: 00007f8e1fe73be0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">821465</span>] R10: <span style="color:#FF00FF">00005576b32a0170</span> R11: <span style="color:#FF00FF">0000000000000070</span> R12: <span style="color:#FF00FF">00007fff7fde55bc</span> +Feb  6 15:08:10 mail kernel: [   26.821465] R10: 00005576b32a0170 R11: 0000000000000070 R12: 00007fff7fde55bc 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">821764</span>] R13: <span style="color:#FF00FF">0000000000000004</span> R14: <span style="color:#FF00FF">0000000000000000</span> R15: <span style="color:#FF00FF">00007fff7fde5890</span> +Feb  6 15:08:10 mail kernel: [   26.821764] R13: 0000000000000004 R14: 0000000000000000 R15: 00007fff7fde5890 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822111</span>] Mem-Info: +Feb  6 15:08:10 mail kernel: [   26.822111] Mem-Info: 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>] active_anon:<span style="color:#FF00FF">62</span> inactive_anon:<span style="color:#FF00FF">4025</span> isolated_anon:<span style="color:#FF00FF">0</span> +Feb  6 15:08:10 mail kernel: [   26.822431] active_anon:62 inactive_anon:4025 isolated_anon:0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>]  active_file:<span style="color:#FF00FF">132</span> inactive_file:<span style="color:#FF00FF">35</span> isolated_file:<span style="color:#FF00FF">0</span> +Feb  6 15:08:10 mail kernel: [   26.822431]  active_file:132 inactive_file:35 isolated_file:0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>]  unevictable:<span style="color:#FF00FF">0</span> dirty:<span style="color:#FF00FF">0</span> writeback:<span style="color:#FF00FF">0</span> +Feb  6 15:08:10 mail kernel: [   26.822431]  unevictable:0 dirty:0 writeback:0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>]  slab_reclaimable:<span style="color:#FF00FF">3337</span> slab_unreclaimable:<span style="color:#FF00FF">7300</span> +Feb  6 15:08:10 mail kernel: [   26.822431]  slab_reclaimable:3337 slab_unreclaimable:7300 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>]  mapped:<span style="color:#FF00FF">138</span> shmem:<span style="color:#FF00FF">739</span> pagetables:<span style="color:#FF00FF">315</span> bounce:<span style="color:#FF00FF">0</span> +Feb  6 15:08:10 mail kernel: [   26.822431]  mapped:138 shmem:739 pagetables:315 bounce:0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">822431</span>]  free:<span style="color:#FF00FF">11707</span> free_pcp:<span style="color:#FF00FF">0</span> free_cma:<span style="color:#FF00FF">0</span> +Feb  6 15:08:10 mail kernel: [   26.822431]  free:11707 free_pcp:0 free_cma:0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">824222</span>] Node <span style="color:#FF00FF">0</span> active_anon:<span style="color:#FF00FF">248</span>kB inactive_anon:<span style="color:#FF00FF">16100</span>kB active_file:<span style="color:#FF00FF">284</span>kB inactive_file:<span style="color:#FF00FF">88</span>kB unevictable:<span style="color:#FF00FF">0</span>kB isolated(anon):<span style="color:#FF00FF">0</span>kB isolated(file):<span style="color:#FF00FF">0</span>kB mapped:<span style="color:#FF00FF">552</span>kB dirty:<span style="color:#FF00FF">0</span>kB writeback:<span style="color:#FF00FF">0</span>kB shmem:<span style="color:#FF00FF">2956</span>kB shmem_thp: <span style="color:#FF00FF">0</span>kB shmem_pmdmapped: <span style="color:#FF00FF">0</span>kB anon_thp: <span style="color:#FF00FF">0</span>kB writeback_tmp:<span style="color:#FF00FF">0</span>kB kernel_stack:<span style="color:#FF00FF">2832</span>kB all_unreclaimable? no +Feb  6 15:08:10 mail kernel: [   26.824222] Node 0 active_anon:248kB inactive_anon:16100kB active_file:284kB inactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:552kB dirty:0kB writeback:0kB shmem:2956kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:2832kB all_unreclaimable? no 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">824849</span>] Node <span style="color:#FF00FF">0</span> DMA free:<span style="color:#FF00FF">4128</span>kB min:<span style="color:#FF00FF">788</span>kB low:<span style="color:#FF00FF">984</span>kB high:<span style="color:#FF00FF">1180</span>kB reserved_highatomic:<span style="color:#FF00FF">0</span>KB active_anon:<span style="color:#FF00FF">0</span>kB inactive_anon:<span style="color:#FF00FF">4</span>kB active_file:<span style="color:#FF00FF">0</span>kB inactive_file:<span style="color:#FF00FF">0</span>kB unevictable:<span style="color:#FF00FF">0</span>kB writepending:<span style="color:#FF00FF">0</span>kB present:<span style="color:#FF00FF">15992</span>kB managed:<span style="color:#FF00FF">5076</span>kB mlocked:<span style="color:#FF00FF">0</span>kB pagetables:<span style="color:#FF00FF">0</span>kB bounce:<span style="color:#FF00FF">0</span>kB free_pcp:<span style="color:#FF00FF">0</span>kB local_pcp:<span style="color:#FF00FF">0</span>kB free_cma:<span style="color:#FF00FF">0</span>kB +Feb  6 15:08:10 mail kernel: [   26.824849] Node 0 DMA free:4128kB min:788kB low:984kB high:1180kB reserved_highatomic:0KB active_anon:0kB inactive_anon:4kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:5076kB mlocked:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">825570</span>] lowmem_reserve[]: <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">844</span> <span style="color:#FF00FF">844</span> <span style="color:#FF00FF">844</span> <span style="color:#FF00FF">844</span> +Feb  6 15:08:10 mail kernel: [   26.825570] lowmem_reserve[]: 0 844 844 844 844 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">825933</span>] Node <span style="color:#FF00FF">0</span> DMA32 free:<span style="color:#FF00FF">42896</span>kB min:<span style="color:#FF00FF">42940</span>kB low:<span style="color:#FF00FF">53672</span>kB high:<span style="color:#FF00FF">64404</span>kB reserved_highatomic:<span style="color:#FF00FF">0</span>KB active_anon:<span style="color:#FF00FF">248</span>kB inactive_anon:<span style="color:#FF00FF">16096</span>kB active_file:<span style="color:#FF00FF">48</span>kB inactive_file:<span style="color:#FF00FF">292</span>kB unevictable:<span style="color:#FF00FF">0</span>kB writepending:<span style="color:#FF00FF">0</span>kB present:<span style="color:#FF00FF">1032040</span>kB managed:<span style="color:#FF00FF">652036</span>kB mlocked:<span style="color:#FF00FF">0</span>kB pagetables:<span style="color:#FF00FF">1260</span>kB bounce:<span style="color:#FF00FF">0</span>kB free_pcp:<span style="color:#FF00FF">0</span>kB local_pcp:<span style="color:#FF00FF">0</span>kB free_cma:<span style="color:#FF00FF">0</span>kB +Feb  6 15:08:10 mail kernel: [   26.825933] Node 0 DMA32 free:42896kB min:42940kB low:53672kB high:64404kB reserved_highatomic:0KB active_anon:248kB inactive_anon:16096kB active_file:48kB inactive_file:292kB unevictable:0kB writepending:0kB present:1032040kB managed:652036kB mlocked:0kB pagetables:1260kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">827042</span>] lowmem_reserve[]: <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">0</span> <span style="color:#FF00FF">0</span> +Feb  6 15:08:10 mail kernel: [   26.827042] lowmem_reserve[]: 0 0 0 0 0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">827439</span>] Node <span style="color:#FF00FF">0</span> DMA: <span style="color:#FF00FF">26</span>*<span style="color:#FF00FF">4</span>kB (M) <span style="color:#FF00FF">13</span>*<span style="color:#FF00FF">8</span>kB (UM) <span style="color:#FF00FF">13</span>*<span style="color:#FF00FF">16</span>kB (M) <span style="color:#FF00FF">10</span>*<span style="color:#FF00FF">32</span>kB (UM) <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">64</span>kB (M) <span style="color:#FF00FF">4</span>*<span style="color:#FF00FF">128</span>kB (UM) <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">256</span>kB (U) <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">512</span>kB (M) <span style="color:#FF00FF">0</span>*<span style="color:#FF00FF">1024</span>kB <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">2048</span>kB (M) <span style="color:#FF00FF">0</span>*<span style="color:#FF00FF">4096</span>kB = <span style="color:#FF00FF">4128</span>kB +Feb  6 15:08:10 mail kernel: [   26.827439] Node 0 DMA: 26*4kB (M) 13*8kB (UM) 13*16kB (M) 10*32kB (UM) 1*64kB (M) 4*128kB (UM) 1*256kB (U) 1*512kB (M) 0*1024kB 1*2048kB (M) 0*4096kB = 4128kB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">828236</span>] Node <span style="color:#FF00FF">0</span> DMA32: <span style="color:#FF00FF">476</span>*<span style="color:#FF00FF">4</span>kB (UME) <span style="color:#FF00FF">201</span>*<span style="color:#FF00FF">8</span>kB (ME) <span style="color:#FF00FF">136</span>*<span style="color:#FF00FF">16</span>kB (ME) <span style="color:#FF00FF">87</span>*<span style="color:#FF00FF">32</span>kB (UME) <span style="color:#FF00FF">86</span>*<span style="color:#FF00FF">64</span>kB (UME) <span style="color:#FF00FF">48</span>*<span style="color:#FF00FF">128</span>kB (UME) <span style="color:#FF00FF">19</span>*<span style="color:#FF00FF">256</span>kB (UME) <span style="color:#FF00FF">3</span>*<span style="color:#FF00FF">512</span>kB (UM) <span style="color:#FF00FF">1</span>*<span style="color:#FF00FF">1024</span>kB (U) <span style="color:#FF00FF">8</span>*<span style="color:#FF00FF">2048</span>kB (UME) <span style="color:#FF00FF">0</span>*<span style="color:#FF00FF">4096</span>kB = <span style="color:#FF00FF">43928</span>kB +Feb  6 15:08:10 mail kernel: [   26.828236] Node 0 DMA32: 476*4kB (UME) 201*8kB (ME) 136*16kB (ME) 87*32kB (UME) 86*64kB (UME) 48*128kB (UME) 19*256kB (UME) 3*512kB (UM) 1*1024kB (U) 8*2048kB (UME) 0*4096kB = 43928kB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">829155</span>] Node <span style="color:#FF00FF">0</span> hugepages_total=<span style="color:#FF00FF">0</span> hugepages_free=<span style="color:#FF00FF">0</span> hugepages_surp=<span style="color:#FF00FF">0</span> hugepages_size=<span style="color:#FF00FF">1048576</span>kB +Feb  6 15:08:10 mail kernel: [   26.829155] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">829594</span>] Node <span style="color:#FF00FF">0</span> hugepages_total=<span style="color:#FF00FF">0</span> hugepages_free=<span style="color:#FF00FF">0</span> hugepages_surp=<span style="color:#FF00FF">0</span> hugepages_size=<span style="color:#FF00FF">2048</span>kB +Feb  6 15:08:10 mail kernel: [   26.829594] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">830021</span>] <span style="color:#FF00FF">913</span> total pagecache pages +Feb  6 15:08:10 mail kernel: [   26.830021] 913 total pagecache pages 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">830427</span>] <span style="color:#FF00FF">0</span> pages in swap cache +Feb  6 15:08:10 mail kernel: [   26.830427] 0 pages in swap cache 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">830779</span>] Swap cache stats: add <span style="color:#FF00FF">0</span>, delete <span style="color:#FF00FF">0</span>, find <span style="color:#FF00FF">0</span>/<span style="color:#FF00FF">0</span> +Feb  6 15:08:10 mail kernel: [   26.830779] Swap cache stats: add 0, delete 0, find 0/0 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">831302</span>] Free swap  = <span style="color:#FF00FF">0</span>kB +Feb  6 15:08:10 mail kernel: [   26.831302] Free swap  = 0kB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">831726</span>] Total swap = <span style="color:#FF00FF">0</span>kB +Feb  6 15:08:10 mail kernel: [   26.831726] Total swap = 0kB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">832224</span>] <span style="color:#FF00FF">262008</span> pages RAM +Feb  6 15:08:10 mail kernel: [   26.832224] 262008 pages RAM 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">832793</span>] <span style="color:#FF00FF">0</span> pages HighMem/MovableOnly +Feb  6 15:08:10 mail kernel: [   26.832793] 0 pages HighMem/MovableOnly 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">833272</span>] <span style="color:#FF00FF">97730</span> pages reserved +Feb  6 15:08:10 mail kernel: [   26.833272] 97730 pages reserved 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">833657</span>] <span style="color:#FF00FF">0</span> pages hwpoisoned +Feb  6 15:08:10 mail kernel: [   26.833657] 0 pages hwpoisoned 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">834067</span>] Unreclaimable slab info: +Feb  6 15:08:10 mail kernel: [   26.834067] Unreclaimable slab info: 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">834458</span>] Name                      Used          Total +Feb  6 15:08:10 mail kernel: [   26.834458] Name                      Used          Total 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">834929</span>] ext4_system_zone           <span style="color:#FF00FF">3</span>KB          <span style="color:#FF00FF">3</span>KB +Feb  6 15:08:10 mail kernel: [   26.834929] ext4_system_zone           3KB          3KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">835368</span>] scsi_sense_cache         <span style="color:#FF00FF">400</span>KB        <span style="color:#FF00FF">400</span>KB +Feb  6 15:08:10 mail kernel: [   26.835368] scsi_sense_cache         400KB        400KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">835764</span>] RAWv6                     <span style="color:#FF00FF">30</span>KB         <span style="color:#FF00FF">30</span>KB +Feb  6 15:08:10 mail kernel: [   26.835764] RAWv6                     30KB         30KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">836221</span>] UDPv6                     <span style="color:#FF00FF">94</span>KB         <span style="color:#FF00FF">94</span>KB +Feb  6 15:08:10 mail kernel: [   26.836221] UDPv6                     94KB         94KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">836631</span>] mqueue_inode_cache         <span style="color:#FF00FF">31</span>KB         <span style="color:#FF00FF">31</span>KB +Feb  6 15:08:10 mail kernel: [   26.836631] mqueue_inode_cache         31KB         31KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">837034</span>] UNIX                     <span style="color:#FF00FF">382</span>KB        <span style="color:#FF00FF">382</span>KB +Feb  6 15:08:10 mail kernel: [   26.837034] UNIX                     382KB        382KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">837435</span>] RAW                       <span style="color:#FF00FF">32</span>KB         <span style="color:#FF00FF">32</span>KB +Feb  6 15:08:10 mail kernel: [   26.837435] RAW                       32KB         32KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">837808</span>] hugetlbfs_inode_cache         <span style="color:#FF00FF">30</span>KB         <span style="color:#FF00FF">30</span>KB +Feb  6 15:08:10 mail kernel: [   26.837808] hugetlbfs_inode_cache         30KB         30KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">838234</span>] eventpoll_pwq             <span style="color:#FF00FF">47</span>KB         <span style="color:#FF00FF">47</span>KB +Feb  6 15:08:10 mail kernel: [   26.838234] eventpoll_pwq             47KB         47KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">838639</span>] request_queue            <span style="color:#FF00FF">411</span>KB        <span style="color:#FF00FF">506</span>KB +Feb  6 15:08:10 mail kernel: [   26.838639] request_queue            411KB        506KB 
-<span style="color:#FF00FF">Feb  6 </span><span style="color:#87FFAF">15:08:10 </span><span style="color:#00FFFF"><b>mail </b></span><span style="color:#FFD7D7">kernel:</span><span style="color:#FFFF00"> </span>[   <span style="color:#FF00FF">26</span>.<span style="color:#FF00FF">839075</span>] biovec-max               <span style="color:#FF00FF">480</span>KB        <span style="color:#FF00FF">480</span>KB +Feb  6 15:08:10 mail kernel: [   26.839075] biovec-max               480KB        480KB
-</pre> +
-</code> +
- +
-Going in to the Proxmox web interface and attach the console to see if there is any output on the terminal revealed some kind of memory issues: +
- +
-<code>+
  
 </code> </code>
  
 +Going in to the Proxmox web interface and attach the console to see if there is any output on the terminal revealed some kind of memory issues (oom, etc.)
 +Reboot did not help, VM crashed very soon after that.
  
 Fix: Disable the option for memory hotplug! Fix: Disable the option for memory hotplug!
Line 89: Line 88:
 hotplug: disk,network,usb,cpu # was additionally with memory hotplug: disk,network,usb,cpu # was additionally with memory
 </code> </code>
- 
  
  
Line 99: Line 97:
 pve-hans.cluster: ii  pve-qemu-kvm   8.1.5-5      amd64        Full virtualization on x86 hardware pve-hans.cluster: ii  pve-qemu-kvm   8.1.5-5      amd64        Full virtualization on x86 hardware
 </code> </code>
 +
 +Symptoms are similar to: https://forum.proxmox.com/threads/memory-hotplug-prevents-vm-boot.122599/
  
  
general/ipkm_it_post_mortem.1738918907.txt.gz · Last modified: 2025-02-07 10:01 by Markus Rosenstihl