cannot load a VM crash dump after failed kexec / kdump boot

[arno01]

Hello,

1. boot Ubuntu 16.04.1 LTS with Linux kernel 4.4.0-47-generic (buildd@lcy01-03) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2) ) #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016

2. load kdump using kexec --load-panic --reuse-cmdline --initrd=/boot/initrd.img-4.8.11-grsec /boot/vmlinuz-4.8.11-grsec

3. trigger crash echo c > /proc/sysrq-trigger

4. expected: system to load 4.8.11-grsec custom kernel
   reality: got 'out of memory' kernel oops, whole trace ->> http://pastebin.com/zjMVeekZ

5. did VM dump using virsh dump instance-## ubuntu1604-grsec-4811_2.dump

6. tried to load a dump using latest crash tool, but never succeeded. Please find the logs here http://pastebin.com/QfrGUd1W

Please let me know if there is anything I can do to load the dump?

Thanks in advance!

[crash-utility]

----- Original Message -----

Hello, 1. boot Ubuntu 16.04.1 LTS with Linux kernel `4.4.0-47-generic ***@***.***) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2) ) #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016` 2. load kdump using `kexec --load-panic --reuse-cmdline --initrd=/boot/initrd.img-4.8.11-grsec /boot/vmlinuz-4.8.11-grsec` 3. trigger crash `echo c > /proc/sysrq-trigger` 4. expected: system to load `4.8.11-grsec` custom kernel reality: got 'out of memory' kernel oops, whole trace ->> http://pastebin.com/zjMVeekZ 5. did VM dump using `virsh dump instance-## ubuntu1604-grsec-4811_2.dump` 6. tried to load a dump using latest crash tool, but never succeeded. Please find the logs here http://pastebin.com/QfrGUd1W Please let me know if there is anything I can do to load the dump?

In your case, it cannot calculate the kernel's phys_base value, and therefore all data reads are coming from the wrong location. But more importantly, you did not use the "virsh dump --memory-only" option when creating the dumpfile, which is the only QEMU/KVM dumpfile format supported at this time. The old "kvmdump" format that was created by your command, is really a file format that is used for guest migration, and is no longer supported by the crash utility, and is what is generating this message:

WARNING: IDT base contains: 0 cannot determine physical base address: defaulting to 0

Secondly, if your kernel is configured with CONFIG_RANDOMIZE_BASE, then dumpfiles created by "virsh dump --memory-only" will not work out of the box, because crash will need the kernel's "phys_base" value. There is a discussion upstream by the QEMU/virsh developers as to how to pass the phys_base value from the KVM guest to the host, and then for how the host should include it in the vmcore header. If CONFIG_RANDOMIZE_BASE is not configured, then "virsh dump --memory-only" may work, depending upon whether it can determine the phys_base. If it cannot, then you would have to supply its value on the crash command line, as in: $ crash --machdep phys_base=xxxx vmlinux vmcore where xxx is the hexadecimal phys_base address, although presumably you won't know it. Before crashing the system, you could run crash on the live system, and get the phys_base value like this: crash> help -m | grep phys_base And then apply that value as the "xxx" phys_base address. If the phys_base address is zero, however, a "virsh dump --memory-only" dumpfile may work, given that zero is the default value used if it cannot be determined otherwise. I can't help with the kexec/kdump issue, you might try the upstream kexec list at https://lists.infradead.org/mailman/listinfo/kexec. Dave

…

Thanks in advance! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #6

[arno01]

Thank you very much for this highly instructive reply, Dave!
I highly appreciate it.

I will try to follow your advice and in case of having any trouble, will reopen this issue.

Have a good one ;-)

[crash-utility]

----- Original Message -----

Thank you very much for this highly instructive reply, Dave! I highly appreciate it. I will try to follow your advice and in case of having any trouble, will reopen this issue. Have a good one ;-) -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #6 (comment)

It would be worth using the crash-utility mailing list instead of the github issue interface. There you will get far more visibility to your questions, and also potentially help others: https://www.redhat.com/mailman/listinfo/crash-utility It's a moderated list, so if you don't subscribe, I (as moderator) will manually let your mail go through when I see it. Thanks, Dave

[arno01]

Thank you for the heads up, Dave!
I have just registered there.