include sys/sysmacros.h for major/minor/makedev #4

vapier · 2016-04-21T04:52:26Z

These funcs are defined in the sys/sysmacros.h header, not sys/types.h.
Linux C libraries are updating to drop the implicit include, so we need
to include it explicitly.

These funcs are defined in the sys/sysmacros.h header, not sys/types.h. Linux C libraries are updating to drop the implicit include, so we need to include it explicitly.

crash-utility · 2016-04-21T12:52:53Z

----- Original Message -----

These funcs are defined in the sys/sysmacros.h header, not sys/types.h.
Linux C libraries are updating to drop the implicit include, so we need
to include it explicitly.
You can view, comment on, or merge this pull request online at:

#4

-- Commit Summary --

include sys/sysmacros.h for major/minor/makedev

-- File Changes --
M filesys.c (1)
-- Patch Links --

https://github.com/crash-utility/crash/pull/4.patch
https://github.com/crash-utility/crash/pull/4.diff

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#4

OK thanks Mike, I'll add the change.

BTW, I don't use the github pull request facility. Patches should
normally posted on the [email protected] mailing list for
general review. If accepted, they are checked in here and then
I push them to github.

Thanks again,
Dave

Without the patch, the backtrace displays the "cannot resolve stack trace" warning, dumps the backtrace, and then the text symbols: crash> bt PID: 0 TASK: f0962180 CPU: 6 COMMAND: "swapper/6" bt: cannot resolve stack trace: #0 [f095ff1c] __schedule at c0b6ef8d #1 [f095ff58] schedule at c0b6f4a9 #2 [f095ff64] schedule_preempt_disabled at c0b6f728 #3 [f095ff6c] cpu_startup_entry at c04b0310 #4 [f095ff94] start_secondary at c04468c0 bt: text symbols on stack: [f095ff1c] __schedule at c0b6ef8d [f095ff58] schedule at c0b6f4ae [f095ff64] schedule_preempt_disabled at c0b6f72d [f095ff6c] cpu_startup_entry at c04b0315 [f095ff94] start_secondary at c04468c5 crash> The backtrace shown is actually correct. ([email protected])

Signed-off-by: Leo Yan <[email protected]>

Fix for 'bt' command and options on Linux 5.8-rc1 or later kernels that contain merge commit 076f14be7fc942e112c94c841baec44124275cd0. The merged patches changed the name of exception functions that have been used by the crash utility to check the exception frame. Without the patch, the command and options cannot display it. Before: crash> bt PID: 8752 TASK: ffff8f80cb244380 CPU: 2 COMMAND: "insmod" #0 [ffffa3e40187f9f8] machine_kexec at ffffffffab25d267 crash-utility#1 [ffffa3e40187fa48] __crash_kexec at ffffffffab38e2ed crash-utility#2 [ffffa3e40187fb10] crash_kexec at ffffffffab38f1dd crash-utility#3 [ffffa3e40187fb28] oops_end at ffffffffab222cbd crash-utility#4 [ffffa3e40187fb48] do_trap at ffffffffab21fea1 crash-utility#5 [ffffa3e40187fb90] do_error_trap at ffffffffab21ff75 crash-utility#6 [ffffa3e40187fbd0] exc_invalid_op at ffffffffabb76a2c crash-utility#7 [ffffa3e40187fbf0] asm_exc_invalid_op at ffffffffabc00a72 crash-utility#8 [ffffa3e40187fc78] init_module at ffffffffc042b018 [invalid] crash-utility#9 [ffffa3e40187fca0] init_module at ffffffffc042b018 [invalid] crash-utility#10 [ffffa3e40187fca8] do_one_initcall at ffffffffab202806 crash-utility#11 [ffffa3e40187fd18] do_init_module at ffffffffab3888ba crash-utility#12 [ffffa3e40187fd38] load_module at ffffffffab38afde After: crash> bt PID: 8752 TASK: ffff8f80cb244380 CPU: 2 COMMAND: "insmod" #0 [ffffa3e40187f9f8] machine_kexec at ffffffffab25d267 crash-utility#1 [ffffa3e40187fa48] __crash_kexec at ffffffffab38e2ed crash-utility#2 [ffffa3e40187fb10] crash_kexec at ffffffffab38f1dd crash-utility#3 [ffffa3e40187fb28] oops_end at ffffffffab222cbd crash-utility#4 [ffffa3e40187fb48] do_trap at ffffffffab21fea1 crash-utility#5 [ffffa3e40187fb90] do_error_trap at ffffffffab21ff75 crash-utility#6 [ffffa3e40187fbd0] exc_invalid_op at ffffffffabb76a2c crash-utility#7 [ffffa3e40187fbf0] asm_exc_invalid_op at ffffffffabc00a72 [exception RIP: init_module+24] RIP: ffffffffc042b018 RSP: ffffa3e40187fca8 RFLAGS: 00010246 RAX: 000000000000001c RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff8f80fbd18000 RDI: ffff8f80fbd18000 RBP: ffffffffc042b000 R8: 000000000000029d R9: 000000000000002c R10: 0000000000000000 R11: ffffa3e40187fb58 R12: ffffffffc042d018 R13: ffffa3e40187fdf0 R14: ffffffffc042d000 R15: ffffa3e40187fe90 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 crash-utility#8 [ffffa3e40187fca0] init_module at ffffffffc042b018 [invalid] crash-utility#9 [ffffa3e40187fca8] do_one_initcall at ffffffffab202806 crash-utility#10 [ffffa3e40187fd18] do_init_module at ffffffffab3888ba crash-utility#11 [ffffa3e40187fd38] load_module at ffffffffab38afde Signed-off-by: Kazuhito Hagio <[email protected]>

Fix for 'bt' command and options on Linux 5.8-rc1 and later kernels that contain merge commit 076f14be7fc942e112c94c841baec44124275cd0. The merged patches changed the name of exception functions that have been used by the crash utility to check the exception frame. Without the patch, the command and options cannot display it. Before: crash> bt PID: 8752 TASK: ffff8f80cb244380 CPU: 2 COMMAND: "insmod" #0 [ffffa3e40187f9f8] machine_kexec at ffffffffab25d267 #1 [ffffa3e40187fa48] __crash_kexec at ffffffffab38e2ed #2 [ffffa3e40187fb10] crash_kexec at ffffffffab38f1dd #3 [ffffa3e40187fb28] oops_end at ffffffffab222cbd #4 [ffffa3e40187fb48] do_trap at ffffffffab21fea1 #5 [ffffa3e40187fb90] do_error_trap at ffffffffab21ff75 #6 [ffffa3e40187fbd0] exc_invalid_op at ffffffffabb76a2c #7 [ffffa3e40187fbf0] asm_exc_invalid_op at ffffffffabc00a72 #8 [ffffa3e40187fc78] init_module at ffffffffc042b018 [invalid] #9 [ffffa3e40187fca0] init_module at ffffffffc042b018 [invalid] #10 [ffffa3e40187fca8] do_one_initcall at ffffffffab202806 #11 [ffffa3e40187fd18] do_init_module at ffffffffab3888ba #12 [ffffa3e40187fd38] load_module at ffffffffab38afde After: crash> bt PID: 8752 TASK: ffff8f80cb244380 CPU: 2 COMMAND: "insmod" #0 [ffffa3e40187f9f8] machine_kexec at ffffffffab25d267 #1 [ffffa3e40187fa48] __crash_kexec at ffffffffab38e2ed #2 [ffffa3e40187fb10] crash_kexec at ffffffffab38f1dd #3 [ffffa3e40187fb28] oops_end at ffffffffab222cbd #4 [ffffa3e40187fb48] do_trap at ffffffffab21fea1 #5 [ffffa3e40187fb90] do_error_trap at ffffffffab21ff75 #6 [ffffa3e40187fbd0] exc_invalid_op at ffffffffabb76a2c #7 [ffffa3e40187fbf0] asm_exc_invalid_op at ffffffffabc00a72 [exception RIP: init_module+24] RIP: ffffffffc042b018 RSP: ffffa3e40187fca8 RFLAGS: 00010246 RAX: 000000000000001c RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff8f80fbd18000 RDI: ffff8f80fbd18000 RBP: ffffffffc042b000 R8: 000000000000029d R9: 000000000000002c R10: 0000000000000000 R11: ffffa3e40187fb58 R12: ffffffffc042d018 R13: ffffa3e40187fdf0 R14: ffffffffc042d000 R15: ffffa3e40187fe90 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #8 [ffffa3e40187fca0] init_module at ffffffffc042b018 [invalid] #9 [ffffa3e40187fca8] do_one_initcall at ffffffffab202806 #10 [ffffa3e40187fd18] do_init_module at ffffffffab3888ba #11 [ffffa3e40187fd38] load_module at ffffffffab38afde Signed-off-by: Kazuhito Hagio <[email protected]>

…'bt' command dumpfiles: (1) If the kernel's crash_notes are not available, read them from ELF notes. (2) If an online CPUs did not save its ELF notes, then adjust the mapping of each ELF note to its CPU accordingly. E.g. With this patch: crash> bt PID: 4768 TASK: 9800000243bcf200 CPU: 3 COMMAND: "bash" #0 [980000024291f930] __crash_kexec at ffffffff802fff84 crash-utility#1 [980000024291faa0] panic at ffffffff80248cac crash-utility#2 [980000024291fb40] die at ffffffff8021b338 crash-utility#3 [980000024291fb70] do_page_fault at ffffffff802315e0 crash-utility#4 [980000024291fbd0] tlb_do_page_fault_1 at ffffffff80239388 crash-utility#5 [980000024291fd00] sysrq_handle_crash at ffffffff8085d308 crash-utility#6 [980000024291fd10] __handle_sysrq at ffffffff8085d9e0 crash-utility#7 [980000024291fd60] write_sysrq_trigger at ffffffff8085e020 crash-utility#8 [980000024291fd80] proc_reg_write at ffffffff804762f0 crash-utility#9 [980000024291fda0] __vfs_write at ffffffff803f3138 Signed-off-by: Huacai Chen <[email protected]> Signed-off-by: Youling Tang <[email protected]>

…'bt' command dumpfiles: (1) If the kernel's crash_notes are not available, read them from ELF notes. (2) If an online CPUs did not save its ELF notes, then adjust the mapping of each ELF note to its CPU accordingly. E.g. With this patch: crash> bt PID: 4768 TASK: 9800000243bcf200 CPU: 3 COMMAND: "bash" #0 [980000024291f930] __crash_kexec at ffffffff802fff84 #1 [980000024291faa0] panic at ffffffff80248cac #2 [980000024291fb40] die at ffffffff8021b338 #3 [980000024291fb70] do_page_fault at ffffffff802315e0 #4 [980000024291fbd0] tlb_do_page_fault_1 at ffffffff80239388 #5 [980000024291fd00] sysrq_handle_crash at ffffffff8085d308 #6 [980000024291fd10] __handle_sysrq at ffffffff8085d9e0 #7 [980000024291fd60] write_sysrq_trigger at ffffffff8085e020 #8 [980000024291fd80] proc_reg_write at ffffffff804762f0 #9 [980000024291fda0] __vfs_write at ffffffff803f3138 Signed-off-by: Huacai Chen <[email protected]> Signed-off-by: Youling Tang <[email protected]>

Overflow stack supported since kernel 4.14 in commit 872d8327ce8, without this patch, bt command trigger a SIGSEGV fault due the SP pointed to the overflow stack which not yet loaded by crash. Before: KERNEL: ../vmlinux DUMPFILE: la_guestdump.gcore CPUS: 8 DATE: Tue Jul 13 19:59:44 CST 2021 UPTIME: 00:00:42 LOAD AVERAGE: 3.99, 1.13, 0.39 TASKS: 1925 NODENAME: localhost RELEASE: 4.14.156+ VERSION: crash-utility#1 SMP PREEMPT Tue Jul 13 10:37:23 UTC 2021 MACHINE: aarch64 (unknown Mhz) MEMORY: 8.7 GB PANIC: "Kernel panic - not syncing: kernel stack overflow" PID: 1969 COMMAND: "irq/139-0-0024" TASK: ffffffcc1a230000 [THREAD_INFO: ffffffcc1a230000] CPU: 0 STATE: TASK_RUNNING (PANIC) crash-7.3.0> bt PID: 1969 TASK: ffffffcc1a230000 CPU: 0 COMMAND: "irq/139-0-0024" Segmentation fault (core dumped) After: crash> bt PID: 1969 TASK: ffffffcc1a230000 CPU: 0 COMMAND: "irq/139-0-0024" #0 [ffffffcc7fd5cf50] __delay at ffffff8008c80774 crash-utility#1 [ffffffcc7fd5cf60] __const_udelay at ffffff8008c80864 crash-utility#2 [ffffffcc7fd5cf80] msm_trigger_wdog_bite at ffffff80084e9430 crash-utility#3 [ffffffcc7fd5cfa0] do_vm_restart at ffffff80087bc974 crash-utility#4 [ffffffcc7fd5cfc0] machine_restart at ffffff80080856fc crash-utility#5 [ffffffcc7fd5cfd0] emergency_restart at ffffff80080d49bc crash-utility#6 [ffffffcc7fd5d140] panic at ffffff80080af4c0 crash-utility#7 [ffffffcc7fd5d150] nmi_panic at ffffff80080af150 crash-utility#8 [ffffffcc7fd5d190] handle_bad_stack at ffffff800808b0b8 crash-utility#9 [ffffffcc7fd5d2d0] __bad_stack at ffffff800808285c --- <IRQ stack> --- crash-utility#10 [ffffff801187bc60] el1_error_invalid at ffffff8008082e7c crash-utility#11 [ffffff801187bcc0] cyttsp6_mt_attention at ffffff8000e8498c [cyttsp6] crash-utility#12 [ffffff801187bd20] call_atten_cb at ffffff8000e82030 [cyttsp6] crash-utility#13 [ffffff801187bdc0] cyttsp6_irq at ffffff8000e81e34 [cyttsp6] crash-utility#14 [ffffff801187bdf0] irq_thread_fn at ffffff8008128dd8 crash-utility#15 [ffffff801187be50] irq_thread at ffffff8008128ca4 crash-utility#16 [ffffff801187beb0] kthread at ffffff80080d2fc4 crash> Signed-off-by: Hong YANG <[email protected]>

When we use crash to troubleshoot softlockup and other problems, we often use the 'bt -a' command to print the stacks of running processes on all CPUs. But now some servers have hundreds of CPUs (such as AMD machines), which causes the 'bt -a' command to output a lot of process stacks. And many of these stacks are the stacks of the idle process, which are not needed by us. Therefore, in order to reduce this part of the interference information, this patch adds the -n option to the bt command. When we specify '-n idle' (meaning no idle), the stack of the idle process will be filtered out, thus speeding up our troubleshooting. And the option works only for crash dumps captured by kdump. The command output is as follows: crash> bt -a -n idle [...] PID: 0 TASK: ffff889ff8c34380 CPU: 8 COMMAND: "swapper/8" PID: 0 TASK: ffff889ff8c32d00 CPU: 9 COMMAND: "swapper/9" PID: 0 TASK: ffff889ff8c31680 CPU: 10 COMMAND: "swapper/10" PID: 0 TASK: ffff889ff8c35a00 CPU: 11 COMMAND: "swapper/11" PID: 0 TASK: ffff889ff8c3c380 CPU: 12 COMMAND: "swapper/12" PID: 150773 TASK: ffff889fe85a1680 CPU: 13 COMMAND: "bash" #0 [ffffc9000d35bcd0] machine_kexec at ffffffff8105a407 #1 [ffffc9000d35bd28] __crash_kexec at ffffffff8113033d #2 [ffffc9000d35bdf0] panic at ffffffff81081930 #3 [ffffc9000d35be70] sysrq_handle_crash at ffffffff814e38d1 #4 [ffffc9000d35be78] __handle_sysrq.cold.12 at ffffffff814e4175 #5 [ffffc9000d35bea8] write_sysrq_trigger at ffffffff814e404b #6 [ffffc9000d35beb8] proc_reg_write at ffffffff81330d86 #7 [ffffc9000d35bed0] vfs_write at ffffffff812a72d5 #8 [ffffc9000d35bf00] ksys_write at ffffffff812a7579 #9 [ffffc9000d35bf38] do_syscall_64 at ffffffff81004259 RIP: 00007fa7abcdc274 RSP: 00007fffa731f678 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fa7abcdc274 RDX: 0000000000000002 RSI: 0000563ca51ee6d0 RDI: 0000000000000001 RBP: 0000563ca51ee6d0 R8: 000000000000000a R9: 00007fa7abd6be80 R10: 000000000000000a R11: 0000000000000246 R12: 00007fa7abdad760 R13: 0000000000000002 R14: 00007fa7abda8760 R15: 0000000000000002 ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b [...] Signed-off-by: Qi Zheng <[email protected]> Acked-by: Kazuhito Hagio <[email protected]> Acked-by: Lianbo Jiang <[email protected]>

The previous implementation to locate the call instruction is to strstr "call", then check whether the previous char is ' ' or '\t'. The implementation is problematic. For example it cannot resolve the following disassembly string: "0xffffffffc0995378 <nfs41_callback_svc+344>:\tcall 0xffffffff8ecfa4c0 <schedule>\n" strstr will locate the "_call" and char check fails, as a result, extract_hex fails to get the calling address. NOTE: the issue is more likely to be reproduced when patch[1] applied. Because without patch[1], the disassembly string will be as follows, so the issue is no longer reproducible. "0xffffffffc0995378:\tcall 0xffffffff8ecfa4c0 <schedule>\n" Before the patch: crash> bt 1472 PID: 1472 TASK: ffff8c121fa72f70 CPU: 18 COMMAND: "nfsv4.1-svc" #0 [ffff8c16231a3db8] __schedule at ffffffff8ecf9ef3 #1 [ffff8c16231a3e40] schedule at ffffffff8ecfa4e9 After the patch: crash> bt 1472 PID: 1472 TASK: ffff8c121fa72f70 CPU: 18 COMMAND: "nfsv4.1-svc" #0 [ffff8c16231a3db8] __schedule at ffffffff8ecf9ef3 #1 [ffff8c16231a3e40] schedule at ffffffff8ecfa4e9 #2 [ffff8c16231a3e50] nfs41_callback_svc at ffffffffc099537d [nfsv4] #3 [ffff8c16231a3ec8] kthread at ffffffff8e6b966f #4 [ffff8c16231a3f50] ret_from_fork at ffffffff8ed07898 This patch fix the issue by strstr "\tcall" and " call", to locate the correct call instruction. [1]: https://listman.redhat.com/archives/crash-utility/2022-August/010085.html Signed-off-by: Tao Liu <[email protected]>

1, Add the implementation to get stack frame from active & inactive task's stack. 2, Add 'bt -l' command support get a line number associated with a current pc address. 3, Add 'bt -f' command support to display all stack data contained in a frame With the patch, we can get the backtrace, crash> bt PID: 113 TASK: ff6000000226c200 CPU: 0 COMMAND: "sh" #0 [ff20000010333b90] riscv_crash_save_regs at ffffffff800078f8 crash-utility#1 [ff20000010333cf0] panic at ffffffff806578c6 crash-utility#2 [ff20000010333d50] sysrq_reset_seq_param_set at ffffffff8038c03c crash-utility#3 [ff20000010333da0] __handle_sysrq at ffffffff8038c604 crash-utility#4 [ff20000010333e00] write_sysrq_trigger at ffffffff8038cae4 crash-utility#5 [ff20000010333e20] proc_reg_write at ffffffff801b7ee8 crash-utility#6 [ff20000010333e40] vfs_write at ffffffff80152bb2 crash-utility#7 [ff20000010333e80] ksys_write at ffffffff80152eda crash-utility#8 [ff20000010333ed0] sys_write at ffffffff80152f52 crash> bt -l PID: 113 TASK: ff6000000226c200 CPU: 0 COMMAND: "sh" #0 [ff20000010333b90] riscv_crash_save_regs at ffffffff800078f8 /buildroot/qemu_riscv64_virt_defconfig/build/linux-custom/arch/riscv/kernel/crash_save_regs.S: 47 crash-utility#1 [ff20000010333cf0] panic at ffffffff806578c6 /buildroot/qemu_riscv64_virt_defconfig/build/linux-custom/kernel/panic.c: 276 ... ... crash> bt -f PID: 113 TASK: ff6000000226c200 CPU: 0 COMMAND: "sh" #0 [ff20000010333b90] riscv_crash_save_regs at ffffffff800078f8 [PC: ffffffff800078f8 RA: ffffffff806578c6 SP: ff20000010333b90 SIZE: 352] ff20000010333b90: ff20000010333bb0 ffffffff800078f8 ff20000010333ba0: ffffffff8008862c ff20000010333b90 ff20000010333bb0: ffffffff810dde38 ff6000000226c200 ff20000010333bc0: ffffffff8032be68 0720072007200720 ... ... Signed-off-by: Xianting Tian <[email protected]>

We met "bt" command on KASAN kernel vmcore display truncated backtraces like this: crash> bt PID: 4131 TASK: ffff8001521df000 CPU: 3 COMMAND: "bash" #0 [ffff2000224b0cb0] machine_kexec_prepare at ffff2000200bff4c After digging the root cause, it turns out that arm64_in_kdump_text() found wrong bt->bptr at "machine_kexec" branch. Disassemble machine_kexec() of KASAN vmlinux (gcc 7.3.0): crash> dis -x machine_kexec 0xffff2000200bff50 <machine_kexec>: stp x29, x30, [sp,#-208]! 0xffff2000200bff54 <machine_kexec+0x4>: mov x29, sp 0xffff2000200bff58 <machine_kexec+0x8>: stp x19, x20, [sp,#16] 0xffff2000200bff5c <machine_kexec+0xc>: str x24, [sp,#56] 0xffff2000200bff60 <machine_kexec+0x10>: str x26, [sp,#72] 0xffff2000200bff64 <machine_kexec+0x14>: mov x2, #0x8ab3 0xffff2000200bff68 <machine_kexec+0x18>: add x1, x29, #0x70 0xffff2000200bff6c <machine_kexec+0x1c>: lsr x1, x1, #3 0xffff2000200bff70 <machine_kexec+0x20>: movk x2, #0x41b5, lsl #16 0xffff2000200bff74 <machine_kexec+0x24>: mov x19, #0x200000000000 0xffff2000200bff78 <machine_kexec+0x28>: adrp x3, 0xffff2000224b0000 0xffff2000200bff7c <machine_kexec+0x2c>: movk x19, #0xdfff, lsl #48 0xffff2000200bff80 <machine_kexec+0x30>: add x3, x3, #0xcb0 0xffff2000200bff84 <machine_kexec+0x34>: add x4, x1, x19 0xffff2000200bff88 <machine_kexec+0x38>: stp x2, x3, [x29,#112] 0xffff2000200bff8c <machine_kexec+0x3c>: adrp x2, 0xffff2000200bf000 <swsusp_arch_resume+0x1e8> 0xffff2000200bff90 <machine_kexec+0x40>: add x2, x2, #0xf50 0xffff2000200bff94 <machine_kexec+0x44>: str x2, [x29,#128] 0xffff2000200bff98 <machine_kexec+0x48>: mov w2, #0xf1f1f1f1 0xffff2000200bff9c <machine_kexec+0x4c>: str w2, [x1,x19] 0xffff2000200bffa0 <machine_kexec+0x50>: mov w2, #0xf200 0xffff2000200bffa4 <machine_kexec+0x54>: mov w1, #0xf3f3f3f3 0xffff2000200bffa8 <machine_kexec+0x58>: movk w2, #0xf2f2, lsl #16 0xffff2000200bffac <machine_kexec+0x5c>: stp w2, w1, [x4,#4] We notice that: 1. machine_kexec() start address is 0xffff2000200bff50 2. the instruction at machine_kexec+0x44 stores the same value 0xffff2000200bff50 (comes from 0xffff2000200bf000 + 0xf50) into stack postion [x29,#128]. When arm64_in_kdump_text() searches for LR from stack, it met 0xffff2000200bff50 firstly, so got wrong bt->bptr. We know that the real LR is always greater than the start address of a function, so let's fix it by changing the search conditon to (*ptr > xxx_start) && (*ptr < xxx_end). Signed-off-by: Ding Hui <[email protected]>

1, Add the implementation to get stack frame from active & inactive task's stack. 2, Add 'bt -l' command support get a line number associated with a current pc address. 3, Add 'bt -f' command support to display all stack data contained in a frame With the patch, we can get the backtrace, crash> bt PID: 113 TASK: ff6000000226c200 CPU: 0 COMMAND: "sh" #0 [ff20000010333b90] riscv_crash_save_regs at ffffffff800078f8 #1 [ff20000010333cf0] panic at ffffffff806578c6 #2 [ff20000010333d50] sysrq_reset_seq_param_set at ffffffff8038c03c #3 [ff20000010333da0] __handle_sysrq at ffffffff8038c604 #4 [ff20000010333e00] write_sysrq_trigger at ffffffff8038cae4 #5 [ff20000010333e20] proc_reg_write at ffffffff801b7ee8 #6 [ff20000010333e40] vfs_write at ffffffff80152bb2 #7 [ff20000010333e80] ksys_write at ffffffff80152eda #8 [ff20000010333ed0] sys_write at ffffffff80152f52 crash> bt -l PID: 113 TASK: ff6000000226c200 CPU: 0 COMMAND: "sh" #0 [ff20000010333b90] riscv_crash_save_regs at ffffffff800078f8 /buildroot/qemu_riscv64_virt_defconfig/build/linux-custom/arch/riscv/kernel/crash_save_regs.S: 47 #1 [ff20000010333cf0] panic at ffffffff806578c6 /buildroot/qemu_riscv64_virt_defconfig/build/linux-custom/kernel/panic.c: 276 ... ... crash> bt -f PID: 113 TASK: ff6000000226c200 CPU: 0 COMMAND: "sh" #0 [ff20000010333b90] riscv_crash_save_regs at ffffffff800078f8 [PC: ffffffff800078f8 RA: ffffffff806578c6 SP: ff20000010333b90 SIZE: 352] ff20000010333b90: ff20000010333bb0 ffffffff800078f8 ff20000010333ba0: ffffffff8008862c ff20000010333b90 ff20000010333bb0: ffffffff810dde38 ff6000000226c200 ff20000010333bc0: ffffffff8032be68 0720072007200720 ... ... Signed-off-by: Xianting Tian <[email protected]>

Kernel commit 7d65f4a65532 ("irq: Consolidate do_softirq() arch overriden implementations") renamed the call_softirq to do_softirq_own_stack, and there is no exception frame also when coming from do_softirq_own_stack. Without the patch, crash may unnecessarily output an exception frame with a warning as below: crash> foreach bt ... PID: 0 TASK: ffff914f820a8000 CPU: 25 COMMAND: "swapper/25" #0 [fffffe0000504e48] crash_nmi_callback at ffffffffa665d763 #1 [fffffe0000504e50] nmi_handle at ffffffffa662a423 #2 [fffffe0000504ea8] default_do_nmi at ffffffffa6fe7dc9 #3 [fffffe0000504ec8] do_nmi at ffffffffa662a97f #4 [fffffe0000504ef0] end_repeat_nmi at ffffffffa70015e8 [exception RIP: clone_endio+172] RIP: ffffffffc005c1ec RSP: ffffa1d403d08e98 RFLAGS: 00000246 RAX: 0000000000000000 RBX: ffff915326fba230 RCX: 0000000000000018 RDX: ffffffffc0075400 RSI: 0000000000000000 RDI: ffff915326fba230 RBP: ffff915326fba1c0 R8: 0000000000001000 R9: ffff915308d6d2a0 R10: 000000a97dfe5e10 R11: ffffa1d40038fe98 R12: ffff915302babc40 R13: ffff914f94360000 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> --- #5 [ffffa1d403d08e98] clone_endio at ffffffffc005c1ec [dm_mod] #6 [ffffa1d403d08ed0] blk_update_request at ffffffffa6a96954 #7 [ffffa1d403d08f10] scsi_end_request at ffffffffa6c9b968 #8 [ffffa1d403d08f48] scsi_io_completion at ffffffffa6c9bb3e #9 [ffffa1d403d08f90] blk_complete_reqs at ffffffffa6aa0e95 #10 [ffffa1d403d08fa0] __softirqentry_text_start at ffffffffa72000dc #11 [ffffa1d403d08ff0] do_softirq_own_stack at ffffffffa7000f9a --- <IRQ stack> --- #12 [ffffa1d40038fe70] do_softirq_own_stack at ffffffffa7000f9a [exception RIP: unknown or invalid address] RIP: 0000000000000000 RSP: 0000000000000000 RFLAGS: 00000000 RAX: ffffffffa672eae5 RBX: ffffffffa83b34e0 RCX: ffffffffa672eb12 RDX: 0000000000000010 RSI: 8b7d6c8869010c00 RDI: 0000000000000085 RBP: 0000000000000286 R8: ffff914f820a8000 R9: ffffffffa67a94e0 R10: 0000000000000286 R11: ffffffffa66fb4c5 R12: ffffffffa67a898b R13: 0000000000000000 R14: fffffffffffffff8 R15: ffffffffa67a1e68 ORIG_RAX: 0000000000000000 CS: 0000 SS: ffffffffa672edff bt: WARNING: possibly bogus exception frame #13 [ffffa1d40038ff30] start_secondary at ffffffffa665fa2c #14 [ffffa1d40038ff50] secondary_startup_64_no_verify at ffffffffa6600116 ... Reported-by: Marco Patalano <[email protected]> Signed-off-by: Lianbo Jiang <[email protected]>

On kernels configured with CONFIG_RANDOMIZE_KSTACK_OFFSET=y and random_kstack_offset=on, a random offset is added to the stack with __kstack_alloca() at the beginning of do_syscall_64() and other syscall entry functions. This eventually does the following instruction. <do_syscall_64+32>: sub %rax,%rsp On the other hand, crash uses only a part of data for ORC unwinder to unwind stacks and if an ip value doesn't have a usable ORC data, it caluculates the frame size with parsing the assembly of the function. However, crash cannot calculate the frame size correctly with the instruction above, and prints stale return addresses like this: crash> bt 1 PID: 1 TASK: ffff9c250023b880 CPU: 0 COMMAND: "systemd" #0 [ffffb7e5c001fc80] __schedule at ffffffff91ae2b16 crash-utility#1 [ffffb7e5c001fd00] schedule at ffffffff91ae2ed3 crash-utility#2 [ffffb7e5c001fd18] schedule_hrtimeout_range_clock at ffffffff91ae7ed8 crash-utility#3 [ffffb7e5c001fda8] ep_poll at ffffffff913ef828 crash-utility#4 [ffffb7e5c001fe48] do_epoll_wait at ffffffff913ef943 crash-utility#5 [ffffb7e5c001fe80] __x64_sys_epoll_wait at ffffffff913f0130 crash-utility#6 [ffffb7e5c001fed0] do_syscall_64 at ffffffff91ad7169 crash-utility#7 [ffffb7e5c001fef0] do_syscall_64 at ffffffff91ad7179 << crash-utility#8 [ffffb7e5c001ff10] syscall_exit_to_user_mode at ffffffff91adaab2 << stale entries crash-utility#9 [ffffb7e5c001ff20] do_syscall_64 at ffffffff91ad7179 << crash-utility#10 [ffffb7e5c001ff50] entry_SYSCALL_64_after_hwframe at ffffffff91c0009b RIP: 00007f258d9427ae RSP: 00007fffda631d60 RFLAGS: 00000293 ... To fix this, enhance the usage of ORC data. The ORC unwinder often uses %rbp value, so keep it from exception frames and inactive task stacks. Signed-off-by: Kazuhito Hagio <[email protected]>

On kernels configured with CONFIG_RANDOMIZE_KSTACK_OFFSET=y and random_kstack_offset=on, a random offset is added to task stacks with __kstack_alloca() at the beginning of do_syscall_64() and other syscall entry functions. This eventually does the following instruction. <do_syscall_64+32>: sub %rax,%rsp On the other hand, crash uses only a part of data for ORC unwinder to unwind stacks and if an ip value doesn't have a usable ORC data, it caluculates the frame size with parsing the assembly of the function. However, crash cannot calculate the frame size correctly with the instruction above, and prints stale return addresses like this: crash> bt 1 PID: 1 TASK: ffff9c250023b880 CPU: 0 COMMAND: "systemd" #0 [ffffb7e5c001fc80] __schedule at ffffffff91ae2b16 #1 [ffffb7e5c001fd00] schedule at ffffffff91ae2ed3 #2 [ffffb7e5c001fd18] schedule_hrtimeout_range_clock at ffffffff91ae7ed8 #3 [ffffb7e5c001fda8] ep_poll at ffffffff913ef828 #4 [ffffb7e5c001fe48] do_epoll_wait at ffffffff913ef943 #5 [ffffb7e5c001fe80] __x64_sys_epoll_wait at ffffffff913f0130 #6 [ffffb7e5c001fed0] do_syscall_64 at ffffffff91ad7169 #7 [ffffb7e5c001fef0] do_syscall_64 at ffffffff91ad7179 << #8 [ffffb7e5c001ff10] syscall_exit_to_user_mode at ffffffff91adaab2 << stale entries #9 [ffffb7e5c001ff20] do_syscall_64 at ffffffff91ad7179 << #10 [ffffb7e5c001ff50] entry_SYSCALL_64_after_hwframe at ffffffff91c0009b RIP: 00007f258d9427ae RSP: 00007fffda631d60 RFLAGS: 00000293 ... To fix this, enhance the use of ORC data. The ORC unwinder often uses %rbp value, so keep it from exception frames and inactive task stacks. Signed-off-by: Kazuhito Hagio <[email protected]>

Kernel commit fb799447ae29 ("x86,objtool: Split UNWIND_HINT_EMPTY in two"), which is contained in Linux 6.4 and later kernels, changed ORC_TYPE_CALL macro from 0 to 2. As a result, the "bt" command cannot use ORC entries and displays stale entries in a call trace. crash> bt 1 PID: 1 TASK: ffff93cd06294180 CPU: 51 COMMAND: "systemd" #0 [ffffb72bc00cbc98] __schedule at ffffffff86e52aae crash-utility#1 [ffffb72bc00cbd00] schedule at ffffffff86e52f6a crash-utility#2 [ffffb72bc00cbd18] schedule_hrtimeout_range_clock at ffffffff86e58ef5 crash-utility#3 [ffffb72bc00cbd88] ep_poll at ffffffff8669624d crash-utility#4 [ffffb72bc00cbe28] do_epoll_wait at ffffffff86696371 crash-utility#5 [ffffb72bc00cbe30] do_timerfd_settime at ffffffff8669902b << crash-utility#6 [ffffb72bc00cbe60] __x64_sys_epoll_wait at ffffffff86696bf0 crash-utility#7 [ffffb72bc00cbeb0] do_syscall_64 at ffffffff86e3feb9 crash-utility#8 [ffffb72bc00cbee0] __task_pid_nr_ns at ffffffff863330d7 << crash-utility#9 [ffffb72bc00cbf08] syscall_exit_to_user_mode at ffffffff86e466b2 << stale entries crash-utility#10 [ffffb72bc00cbf18] do_syscall_64 at ffffffff86e3fec9 << crash-utility#11 [ffffb72bc00cbf50] entry_SYSCALL_64_after_hwframe at ffffffff870000aa Also, struct orc_entry in kernel has changed, and debugging information for ORC unwinder can be displayed incorrectly. To fix these, (1) introduce "kernel_orc_entry_6_4" structure corresponding to 6.4 and abstruct structure "orc_entry" in crash, (2) switch ORC_TYPE_CALL to 2 or 0 with kernel's orc_entry structure. Signed-off-by: Kazuhito Hagio <[email protected]>

Kernel commit fb799447ae29 ("x86,objtool: Split UNWIND_HINT_EMPTY in two"), which is contained in Linux 6.4 and later kernels, changed ORC_TYPE_CALL macro from 0 to 2. As a result, the "bt" command cannot use ORC entries, and can display stale entries in a call trace. crash> bt 1 PID: 1 TASK: ffff93cd06294180 CPU: 51 COMMAND: "systemd" #0 [ffffb72bc00cbc98] __schedule at ffffffff86e52aae #1 [ffffb72bc00cbd00] schedule at ffffffff86e52f6a #2 [ffffb72bc00cbd18] schedule_hrtimeout_range_clock at ffffffff86e58ef5 #3 [ffffb72bc00cbd88] ep_poll at ffffffff8669624d #4 [ffffb72bc00cbe28] do_epoll_wait at ffffffff86696371 #5 [ffffb72bc00cbe30] do_timerfd_settime at ffffffff8669902b << #6 [ffffb72bc00cbe60] __x64_sys_epoll_wait at ffffffff86696bf0 #7 [ffffb72bc00cbeb0] do_syscall_64 at ffffffff86e3feb9 #8 [ffffb72bc00cbee0] __task_pid_nr_ns at ffffffff863330d7 << #9 [ffffb72bc00cbf08] syscall_exit_to_user_mode at ffffffff86e466b2 << stale entries #10 [ffffb72bc00cbf18] do_syscall_64 at ffffffff86e3fec9 << #11 [ffffb72bc00cbf50] entry_SYSCALL_64_after_hwframe at ffffffff870000aa Also, kernel commit ffb1b4a41016 added a member to struct orc_entry. Although this does not affect the crash's unwinder, its debugging information can be displayed incorrectly. To fix these, (1) introduce "kernel_orc_entry_6_4" structure corresponding to 6.4 and abstruction layer "orc_entry" structure in crash, (2) switch ORC_TYPE_CALL to 2 or 0 with kernel's orc_entry structure. Related orc_entry history: v4.14 39358a033b2e introduced struct orc_entry v4.19 d31a580266ee added orc_entry.end member v6.3 ffb1b4a41016 added orc_entry.signal member v6.4 fb799447ae29 removed end member and changed type member to 3 bits Signed-off-by: Kazuhito Hagio <[email protected]>

If we use crash to parse ramdump(Qcom phone device) rathen than vmcore. Start command should be like: crash vmlinux --kaslr=xxx DDRCS0_0.BIN@0x0000000080000000,... --machdep vabits_actual=39 Then We will see bt command show misleading backtrace information below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 crash-utility#2 [ffffffc034c438b0] __kvm_nvhe_$d.2314 at 86c54c6004ceff80 crash-utility#3 [ffffffc034c43950] __kvm_nvhe_$d.2314 at 55d6f96003a7b120 crash-utility#4 [ffffffc034c439f0] __kvm_nvhe_$d.2314 at 9ccec46003a80a64 crash-utility#5 [ffffffc034c43ac0] __kvm_nvhe_$d.2314 at 8cf41e6003a945c4 crash-utility#6 [ffffffc034c43b10] __kvm_nvhe_$d.2314 at a8f181e00372c818 crash-utility#7 [ffffffc034c43b40] __kvm_nvhe_$d.2314 at 6dedde600372c0d0 crash-utility#8 [ffffffc034c43b90] __kvm_nvhe_$d.2314 at 62cc07e00373d0ac crash-utility#9 [ffffffc034c43c00] __kvm_nvhe_$d.2314 at 72fb1de00373bedc ... PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 By checking the raw data below, will see the lr (fp+8) data show the pointer which already been replaced by PAC prefix. crash> bt -f PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 ffffffc034c437f0: ffffffc034c43850 6be732e004cf05a4 ffffffc034c43800: ffffffe006186108 a0ed07e004cf09c4 ffffffc034c43810: ffffff8a1a340000 ffffff8a8d343c00 ffffffc034c43820: ffffff89b3eada00 ffffff8b780db540 ffffffc034c43830: ffffff89b3eada00 0000000000000000 ffffffc034c43840: 0000000000000004 712b828118484a00 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 ffffffc034c43850: ffffffc034c438b0 86c54c6004ceff84 ffffffc034c43860: 000000708070f000 ffffffc034c43938 ffffffc034c43870: ffffff88bd822878 ffffff89b3eada00 ... So we check the CONFIG_ARM64_PTR_AUTH and CONFIG_ARM64_PTR_AUTH_KERNEL to double check if pac mechanism been enabled on this ramdump. Then we use vabits to figure it out. Fix then show the right backtrace below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __schedule at ffffffe004cf05a0 crash-utility#2 [ffffffc034c438b0] preempt_schedule_common at ffffffe004ceff80 crash-utility#3 [ffffffc034c43950] unmap_page_range at ffffffe003a7b120 crash-utility#4 [ffffffc034c439f0] unmap_vmas at ffffffe003a80a64 crash-utility#5 [ffffffc034c43ac0] exit_mmap at ffffffe003a945c4 crash-utility#6 [ffffffc034c43b10] __mmput at ffffffe00372c818 crash-utility#7 [ffffffc034c43b40] mmput at ffffffe00372c0d0 crash-utility#8 [ffffffc034c43b90] exit_mm at ffffffe00373d0ac crash-utility#9 [ffffffc034c43c00] do_exit at ffffffe00373bedc PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 Let's use GENMASK to replace the pac pointer to fix it. gki related commit url here: https://lore.kernel.org/all/[email protected]/

If we use crash to parse ramdump(Qcom phone device) rathen than vmcore. Start command should be like: crash vmlinux --kaslr=xxx DDRCS0_0.BIN@0x0000000080000000,... --machdep vabits_actual=39 Then We will see bt command show misleading backtrace information below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 crash-utility#2 [ffffffc034c438b0] __kvm_nvhe_$d.2314 at 86c54c6004ceff80 crash-utility#3 [ffffffc034c43950] __kvm_nvhe_$d.2314 at 55d6f96003a7b120 crash-utility#4 [ffffffc034c439f0] __kvm_nvhe_$d.2314 at 9ccec46003a80a64 crash-utility#5 [ffffffc034c43ac0] __kvm_nvhe_$d.2314 at 8cf41e6003a945c4 crash-utility#6 [ffffffc034c43b10] __kvm_nvhe_$d.2314 at a8f181e00372c818 crash-utility#7 [ffffffc034c43b40] __kvm_nvhe_$d.2314 at 6dedde600372c0d0 crash-utility#8 [ffffffc034c43b90] __kvm_nvhe_$d.2314 at 62cc07e00373d0ac crash-utility#9 [ffffffc034c43c00] __kvm_nvhe_$d.2314 at 72fb1de00373bedc ... PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 By checking the raw data below, will see the lr (fp+8) data show the pointer which already been replaced by PAC prefix. crash> bt -f PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 ffffffc034c437f0: ffffffc034c43850 6be732e004cf05a4 ffffffc034c43800: ffffffe006186108 a0ed07e004cf09c4 ffffffc034c43810: ffffff8a1a340000 ffffff8a8d343c00 ffffffc034c43820: ffffff89b3eada00 ffffff8b780db540 ffffffc034c43830: ffffff89b3eada00 0000000000000000 ffffffc034c43840: 0000000000000004 712b828118484a00 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 ffffffc034c43850: ffffffc034c438b0 86c54c6004ceff84 ffffffc034c43860: 000000708070f000 ffffffc034c43938 ffffffc034c43870: ffffff88bd822878 ffffff89b3eada00 ... So we check the CONFIG_ARM64_PTR_AUTH and CONFIG_ARM64_PTR_AUTH_KERNEL to double check if pac mechanism been enabled on this ramdump. Then we use vabits to figure it out. Fix then show the right backtrace below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __schedule at ffffffe004cf05a0 crash-utility#2 [ffffffc034c438b0] preempt_schedule_common at ffffffe004ceff80 crash-utility#3 [ffffffc034c43950] unmap_page_range at ffffffe003a7b120 crash-utility#4 [ffffffc034c439f0] unmap_vmas at ffffffe003a80a64 crash-utility#5 [ffffffc034c43ac0] exit_mmap at ffffffe003a945c4 crash-utility#6 [ffffffc034c43b10] __mmput at ffffffe00372c818 crash-utility#7 [ffffffc034c43b40] mmput at ffffffe00372c0d0 crash-utility#8 [ffffffc034c43b90] exit_mm at ffffffe00373d0ac crash-utility#9 [ffffffc034c43c00] do_exit at ffffffe00373bedc PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 Let's use GENMASK to replace the pac pointer to fix it. gki related commit url here: https://lore.kernel.org/all/[email protected]/ Signed-off-by: bevis_chen <[email protected]>

For ramdump(Qcom phone device) case with the kernel option CONFIG_ARM64_PTR_AUTH_KERNEL enabled, the bt command may print incorrect stacktrace as below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 #1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 #2 [ffffffc034c438b0] __kvm_nvhe_$d.2314 at 86c54c6004ceff80 #3 [ffffffc034c43950] __kvm_nvhe_$d.2314 at 55d6f96003a7b120 ... PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 Crash tool can not get the KERNELPACMASK value from the vmcoreinfo, need to calculate its value based on the vabits. With the patch: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 #1 [ffffffc034c43850] __schedule at ffffffe004cf05a0 #2 [ffffffc034c438b0] preempt_schedule_common at ffffffe004ceff80 #3 [ffffffc034c43950] unmap_page_range at ffffffe003a7b120 #4 [ffffffc034c439f0] unmap_vmas at ffffffe003a80a64 #5 [ffffffc034c43ac0] exit_mmap at ffffffe003a945c4 #6 [ffffffc034c43b10] __mmput at ffffffe00372c818 #7 [ffffffc034c43b40] mmput at ffffffe00372c0d0 #8 [ffffffc034c43b90] exit_mm at ffffffe00373d0ac #9 [ffffffc034c43c00] do_exit at ffffffe00373bedc PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 Related kernel commits: 689eae42afd7 ("arm64: mask PAC bits of __builtin_return_address") de1702f65feb ("arm64: move PAC masks to <asm/pointer_auth.h>") Signed-off-by: bevis_chen <[email protected]>

The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Signed-off-by: Tao Liu <[email protected]>

Currently, gdb passthroughs of 'bt', 'frame', 'up', 'down', 'info locals' don't work. This is due to gdb not knowing the register values to unwind the stack frames Every gdb passthrough goes through `gdb_interface`. And then, gdb expects `crash_target::fetch_registers` to give it the register values, which is dependent on `machdep->get_cpu_reg` to read the register values for specific architecture. ---------------------------- gdb passthrough (eg. "bt") | | crash -------------------------> | | | gdb_interface | | | | | | ---------------------- | fetch_registers | | | | crash_target<-------------------------+--| gdb | | --------------------------+->| | | Registers (SP,NIP, etc.)| | | | | | | | | ---------------------- | ---------------------------- Implement `machdep->get_cpu_reg` on PPC64, so that crash provides the register values to gdb to unwind stack frames properly With these changes, on powerpc, 'bt' command output in gdb mode, will look like this: gdb> bt #0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>, newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69 crash-utility#1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974 crash-utility#2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358 crash-utility#3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:155 crash-utility#4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99, check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602 crash-utility#5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1163 crash-utility#6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at fs/proc/inode.c:340 crash-utility#7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352 crash-utility#8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00, buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>, count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582 instead of earlier output without this patch: gdb> bt #0 <unavailable> in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Also, 'get_dumpfile_regs' has been introduced to get registers from multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS' has been introduced to tell helper functions to get registers, to not print registers with every call to backtrace in gdb. Note: This feature to support GDB unwinding doesn't support live debugging Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Improved-by: Tao Liu <[email protected]> Signed-off-by: Aditya Gupta <[email protected]>

See the following stack trace: (gdb) bt #0 0x00005635ac2b166b in arm64_unwind_frame (frame=0x7ffdaf35cb70, bt=0x7ffdaf35d430) at arm64.c:2821 #1 arm64_back_trace_cmd (bt=0x7ffdaf35d430) at arm64.c:3306 #2 0x00005635ac27b108 in back_trace (bt=bt@entry=0x7ffdaf35d430) at kernel.c:3239 #3 0x00005635ac2880ae in cmd_bt () at kernel.c:2863 #4 0x00005635ac1f16dc in exec_command () at main.c:893 #5 0x00005635ac1f192a in main_loop () at main.c:840 #6 0x00005635ac50df81 in captured_main (data=<optimized out>) at main.c:1284 #7 gdb_main (args=<optimized out>) at main.c:1313 #8 0x00005635ac50e000 in gdb_main_entry (argc=<optimized out>, argv=<optimized out>) at main.c:1338 #9 0x00005635ac1ea2a5 in main (argc=5, argv=0x7ffdaf35dde8) at main.c:721 The issue may be encountered when thread_union symbol not found in vmlinux due to compiling optimization. This patch will try the following 2 methods to get the irq_stack_size when thread_union symbol unavailable: 1. change the thread_shift when KASAN is enabled and with vmcoreinfo. In arm64/include/asm/memory.h: #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) ... #define IRQ_STACK_SIZE THREAD_SIZE Since enabling the KASAN will affect the final value, this patch reset IRQ_STACK_SIZE according to the calculation process in kernel code. 2. Try getting the value from kernel code disassembly, to get THREAD_SHIFT directly from tbnz instruction. In arch/arm64/kernel/entry.S: .macro kernel_ventry, el:req, ht:req, regsize:req, label:req ... add sp, sp, x0 sub x0, sp, x0 tbnz x0, #THREAD_SHIFT, 0f $ gdb vmlinux (gdb) disass vectors Dump of assembler code for function vectors: ... 0xffff800080010804 <+4>: add sp, sp, x0 0xffff800080010808 <+8>: sub x0, sp, x0 0xffff80008001080c <+12>: tbnz w0, #16, 0xffff80008001081c <vectors+28> Signed-off-by: yeping.zheng <[email protected]> Improved-by: Tao Liu <[email protected]>

Sometimes, in production environment, there are still some vmcores that are incomplete, such as partial header or the data is corrupted. When crash tool attempts to parse such vmcores, it may fail as below: $ ./crash --osrelease vmcore Bus error (core dumped) or $ crash vmlinux vmcore ... Bus error (core dumped) $ Gdb calltrace: $ gdb /home/lijiang/src/crash/crash /tmp/core.126301 Core was generated by `./crash --osrelease /home/lijiang/src/39317/vmcore'. Program terminated with signal SIGBUS, Bus error. #0 __memcpy_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:831 831 LOAD_ONE_SET((%rsi), PAGE_SIZE, %VMM(4), %VMM(5), %VMM(6), %VMM(7)) (gdb) bt #0 __memcpy_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:831 #1 0x0000000000651096 in read_dump_header (file=0x7ffc59ddff5f "/home/lijiang/src/39317/vmcore") at diskdump.c:820 #2 0x0000000000651cf3 in is_diskdump (file=0x7ffc59ddff5f "/home/lijiang/src/39317/vmcore") at diskdump.c:1042 #3 0x0000000000502ac9 in get_osrelease (dumpfile=0x7ffc59ddff5f "/home/lijiang/src/39317/vmcore") at main.c:1938 #4 0x00000000004fb2e8 in main (argc=3, argv=0x7ffc59dde3a8) at main.c:271 (gdb) frame 1 #1 0x0000000000651096 in read_dump_header (file=0x7ffc59ddff5f "/home/lijiang/src/39317/vmcore") at diskdump.c:820 820 memcpy(dd->dumpable_bitmap, dd->bitmap + bitmap_len/2, This may happen on attempting access to a page of the buffer that lies beyond the end of the mapped file(see the mmap() man page). Let's add a check to avoid such issues as much as possible, but still not guarantee that it can work well in any extreme situation. Fixes: a334423 ("diskdump: use mmap/madvise to improve the start-up") Reported-by: Buland Kumar Singh <[email protected]> Signed-off-by: Lianbo Jiang <[email protected]>

The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Signed-off-by: Tao Liu <[email protected]>

Currently, gdb passthroughs of 'bt', 'frame', 'up', 'down', 'info locals' don't work. This is due to gdb not knowing the register values to unwind the stack frames Every gdb passthrough goes through `gdb_interface`. And then, gdb expects `crash_target::fetch_registers` to give it the register values, which is dependent on `machdep->get_cpu_reg` to read the register values for specific architecture. ---------------------------- gdb passthrough (eg. "bt") | | crash -------------------------> | | | gdb_interface | | | | | | ---------------------- | fetch_registers | | | | crash_target<-------------------------+--| gdb | | --------------------------+->| | | Registers (SP,NIP, etc.)| | | | | | | | | ---------------------- | ---------------------------- Implement `machdep->get_cpu_reg` on PPC64, so that crash provides the register values to gdb to unwind stack frames properly With these changes, on powerpc, 'bt' command output in gdb mode, will look like this: gdb> bt #0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>, newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69 crash-utility#1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974 crash-utility#2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358 crash-utility#3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:155 crash-utility#4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99, check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602 crash-utility#5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1163 crash-utility#6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at fs/proc/inode.c:340 crash-utility#7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352 crash-utility#8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00, buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>, count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582 instead of earlier output without this patch: gdb> bt #0 <unavailable> in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Also, 'get_dumpfile_regs' has been introduced to get registers from multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS' has been introduced to tell helper functions to get registers, to not print registers with every call to backtrace in gdb. Note: This feature to support GDB unwinding doesn't support live debugging Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Improved-by: Tao Liu <[email protected]> Signed-off-by: Aditya Gupta <[email protected]>

Currently, gdb passthroughs of 'bt', 'frame', 'up', 'down', 'info locals' don't work. This is due to gdb not knowing the register values to unwind the stack frames Every gdb passthrough goes through `gdb_interface`. And then, gdb expects `crash_target::fetch_registers` to give it the register values, which is dependent on `machdep->get_current_task_reg` to read the register values for specific architecture. ---------------------------- gdb passthrough (eg. "bt") | | crash -------------------------> | | | gdb_interface | | | | | | ---------------------- | fetch_registers | | | | crash_target<-------------------------+--| gdb | | --------------------------+->| | | Registers (SP,NIP, etc.)| | | | | | | | | ---------------------- | ---------------------------- Implement `machdep->get_current_task_reg` on PPC64, so that crash provides the register values to gdb to unwind stack frames properly With these changes, on powerpc, 'bt' command output in gdb mode, will look like this: gdb> bt #0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>, newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69 #1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974 #2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358 #3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:155 #4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99, check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602 #5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1163 #6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at fs/proc/inode.c:340 #7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352 #8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00, buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>, count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582 instead of earlier output without this patch: gdb> bt #0 <unavailable> in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Also, 'get_dumpfile_regs' has been introduced to get registers from multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS' has been introduced to tell helper functions to get registers, to not print registers with every call to backtrace in gdb. Note: This feature to support GDB unwinding doesn't support live debugging [lijiang: squash these five patches(see the Link) into one patch] Link: https://www.mail-archive.com/[email protected]/msg01084.html Link: https://www.mail-archive.com/[email protected]/msg01083.html Link: https://www.mail-archive.com/[email protected]/msg01089.html Link: https://www.mail-archive.com/[email protected]/msg01090.html Link: https://www.mail-archive.com/[email protected]/msg01091.html Co-developed-by:: Tao Liu <[email protected]> Signed-off-by: Aditya Gupta <[email protected]>

…usly There is an issue that, for kernel modules, "dis -rl" fails to display modules code line number data after execute "bt" command in crash. Without the patch: crsah> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp 0xffffffffc0f60eb6 <lpfc_nlp_get+6>: push %rbx 0xffffffffc0f60eb7 <lpfc_nlp_get+7>: test %rdi,%rdi With the patch: crash> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6756 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6759 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp The root cause is, after kernel module been loaded by mod command, the symtable is not expanded in gdb side. crash bt or dis command will trigger such an expansion. However the symtable expansion is different for the 2 commands: The stack trace of "dis -rl" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... crash-utility#1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... crash-utility#2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... crash-utility#3 0x000000000077e8e9 in process_full_comp_unit ... crash-utility#4 process_queue ... crash-utility#5 dw2_do_instantiate_symtab ... crash-utility#6 0x000000000077ed67 in dw2_instantiate_symtab ... crash-utility#7 0x000000000077f75e in dw2_expand_all_symtabs ... crash-utility#8 0x00000000008f254d in gdb_get_line_number ... crash-utility#9 0x00000000008f22af in gdb_command_funnel_1 ... crash-utility#10 0x00000000008f2003 in gdb_command_funnel ... crash-utility#11 0x00000000005b7f02 in gdb_interface ... crash-utility#12 0x00000000005f5bd8 in get_line_number ... crash-utility#13 0x000000000059e574 in cmd_dis ... The stack trace of "bt" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... crash-utility#1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... crash-utility#2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... crash-utility#3 0x000000000077e8e9 in process_full_comp_unit ... crash-utility#4 process_queue ... crash-utility#5 dw2_do_instantiate_symtab ... crash-utility#6 0x000000000077ed67 in dw2_instantiate_symtab ... crash-utility#7 0x000000000077f8ed in dw2_lookup_symbol ... crash-utility#8 0x00000000008e6d03 in lookup_symbol_via_quick_fns ... crash-utility#9 0x00000000008e7153 in lookup_symbol_in_objfile ... crash-utility#10 0x00000000008e73c6 in lookup_symbol_global_or_static_iterator_cb ... crash-utility#11 0x00000000008b99c4 in svr4_iterate_over_objfiles_in_search_order ... crash-utility#12 0x00000000008e754e in lookup_global_or_static_symbol ... crash-utility#13 0x00000000008e75da in lookup_static_symbol ... crash-utility#14 0x00000000008e632c in lookup_symbol_aux ... crash-utility#15 0x00000000008e5a7a in lookup_symbol_in_language ... crash-utility#16 0x00000000008e5b30 in lookup_symbol ... crash-utility#17 0x00000000008f2a4a in gdb_get_datatype ... crash-utility#18 0x00000000008f22c0 in gdb_command_funnel_1 ... crash-utility#19 0x00000000008f2003 in gdb_command_funnel ... crash-utility#20 0x00000000005b7f02 in gdb_interface ... crash-utility#21 0x00000000005f8a9f in datatype_info ... crash-utility#22 0x0000000000599947 in cpu_map_size ... crash-utility#23 0x00000000005a975d in get_cpus_online ... crash-utility#24 0x0000000000637a8b in diskdump_get_prstatus_percpu ... crash-utility#25 0x000000000062f0e4 in get_netdump_regs_x86_64 ... crash-utility#26 0x000000000059fe68 in back_trace ... crash-utility#27 0x00000000005ab1cb in cmd_bt ... For the stacktrace of "dis -rl", it calls dw2_expand_all_symtabs() to expand all symtable of the objfile, or "*.ko.debug" in our case. However for the stacktrace of "bt", it doesn't expand all, but only a subset of symtable which is enough to find a symbol by dw2_lookup_symbol(). As a result, the objfile->compunit_symtabs, which is the head of a single linked list of struct compunit_symtab, is not NULL but didn't contain all symtables. It will not be reinitialized in gdb_get_line_number() by "dis -rl" because !objfile_has_full_symbols(objfile) check will fail, so it cannot display the proper code line number data. Since objfile_has_full_symbols(objfile) check cannot ensure all symbols been expanded, this patch add a new member as a flag for struct objfile to record if all symbols have been expanded. The flag will be set only ofter expand_all_symtabs been called. Signed-off-by: Tao Liu <[email protected]>

Same as the Linux commit f766f77a74f5 ("riscv/stacktrace: Fix stack output without ra on the stack top"). When a function doesn't have a callee, then it will not push ra into the stack, such as lkdtm functions, so correct the FP of the second frame and use pt_regs to get the right PC of the second frame. Before this patch, the `bt -f` outputs only the first frame with the wrong PC and FP of next frame: ``` crash> bt -f PID: 1 TASK: ff600000000e0000 CPU: 1 COMMAND: "sh" #0 [ff20000000013cf0] lkdtm_EXCEPTION at ffffffff805303c0 [PC: ffffffff805303c0 RA: ff20000000013d10 SP: ff20000000013cf0 SIZE: 16] <- wrong next PC ff20000000013cf0: 0000000000000001 ff20000000013d10 <- next FP ff20000000013d00: ff20000000013d40 crash> ``` After this patch, the `bt` outputs the full frames: ``` crash> bt PID: 1 TASK: ff600000000e0000 CPU: 1 COMMAND: "sh" #0 [ff20000000013cf0] lkdtm_EXCEPTION at ffffffff805303c0 crash-utility#1 [ff20000000013d00] lkdtm_do_action at ffffffff8052fe36 crash-utility#2 [ff20000000013d10] direct_entry at ffffffff80530018 crash-utility#3 [ff20000000013d40] full_proxy_write at ffffffff80305044 crash-utility#4 [ff20000000013d80] vfs_write at ffffffff801b68b4 crash-utility#5 [ff20000000013e30] ksys_write at ffffffff801b6c4a crash-utility#6 [ff20000000013e80] __riscv_sys_write at ffffffff801b6cc4 crash-utility#7 [ff20000000013e90] do_trap_ecall_u at ffffffff80836798 crash> ``` Acked-by: Kazuhito Hagio <[email protected]> Signed-off-by: Song Shuai <[email protected]>

This patch introduces per-cpu IRQ stacks for RISCV64 to let "bt" do backtrace on it and 'bt -E' search eframes on it, and the 'help -m' command displays the addresses of each per-cpu IRQ stack. TEST: a vmcore dumped via hacking the handle_irq_event_percpu() ( Why not using lkdtm INT_HW_IRQ_EN EXCEPTION ? There is a deadlock[1] in crash_kexec path if use that) crash> bt PID: 0 TASK: ffffffff8140db00 CPU: 0 COMMAND: "swapper/0" #0 [ff20000000003e60] __handle_irq_event_percpu at ffffffff8006462e crash-utility#1 [ff20000000003ed0] handle_irq_event_percpu at ffffffff80064702 crash-utility#2 [ff20000000003ef0] handle_irq_event at ffffffff8006477c crash-utility#3 [ff20000000003f20] handle_fasteoi_irq at ffffffff80068664 crash-utility#4 [ff20000000003f50] generic_handle_domain_irq at ffffffff80063988 crash-utility#5 [ff20000000003f60] plic_handle_irq at ffffffff8046633e crash-utility#6 [ff20000000003fb0] generic_handle_domain_irq at ffffffff80063988 crash-utility#7 [ff20000000003fc0] riscv_intc_irq at ffffffff80465f8e crash-utility#8 [ff20000000003fd0] handle_riscv_irq at ffffffff808361e8 PC: ffffffff80837314 [default_idle_call+50] RA: ffffffff80837310 [default_idle_call+46] SP: ffffffff81403da0 CAUSE: 8000000000000009 epc : ffffffff80837314 ra : ffffffff80837310 sp : ffffffff81403da0 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : ff2000000004bb18 t1 : 0000000000032c73 t2 : ffffffff81200a48 s0 : ffffffff81403db0 s1 : 0000000000000000 a0 : 0000000000000004 a1 : 0000000000000000 a2 : ff6000009f1e7000 a3 : 0000000000002304 a4 : ffffffff80c1c2d8 a5 : 0000000000000000 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ffffffff814f0220 s3 : 0000000000000001 s4 : 000000000000003f s5 : ffffffff814f03d8 s6 : 0000000000000000 s7 : ffffffff814f00d0 s8 : ffffffff81526f10 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000009 orig_a0: ffffffff80837310 --- <IRQ stack> --- crash-utility#9 [ffffffff81403da0] default_idle_call at ffffffff80837314 crash-utility#10 [ffffffff81403db0] do_idle at ffffffff8004d0a0 crash-utility#11 [ffffffff81403e40] cpu_startup_entry at ffffffff8004d21e crash-utility#12 [ffffffff81403e60] kernel_init at ffffffff8083746a crash-utility#13 [ffffffff81403e70] arch_post_acpi_subsys_init at ffffffff80a006d8 crash-utility#14 [ffffffff81403e80] console_on_rootfs at ffffffff80a00c92 crash> crash> bt -E CPU 0 IRQ STACK: KERNEL-MODE EXCEPTION FRAME AT: ff20000000003a48 PC: ffffffff8006462e [__handle_irq_event_percpu+30] RA: ffffffff80064702 [handle_irq_event_percpu+18] SP: ff20000000003e60 CAUSE: 000000000000000d epc : ffffffff8006462e ra : ffffffff80064702 sp : ff20000000003e60 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : 0000000000046600 t1 : ffffffff80836464 t2 : ffffffff81200a48 s0 : ff20000000003ed0 s1 : 0000000000000000 a0 : 0000000000000000 a1 : 0000000000000118 a2 : 0000000000000052 a3 : 0000000000000000 a4 : 0000000000000000 a5 : 0000000000010001 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ff60000000941ab0 s3 : ffffffff814a0658 s4 : ff60000000089230 s5 : ffffffff814a0518 s6 : ffffffff814a0620 s7 : ffffffff80e5f0f8 s8 : ffffffff80fc50b0 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000100 badaddr: 0000000000000078 cause: 000000000000000d orig_a0: ff20000000003ea0 CPU 1 IRQ STACK: (none found) crash> crash> help -m <snip> machspec: ced1e0 irq_stack_size: 16384 irq_stacks[0]: ff20000000000000 irq_stacks[1]: ff20000000008000 crash> [1]: https://lore.kernel.org/linux-riscv/[email protected]/ Signed-off-by: Song Shuai <[email protected]>

The patch introduces per-cpu overflow stacks for RISCV64 to let "bt" do backtrace on it and the 'help -m' command dispalys the addresss of each per-cpu overflow stack. TEST: a lkdtm DIRECT EXHAUST_STACK vmcore crash> bt PID: 1 TASK: ff600000000d8000 CPU: 1 COMMAND: "sh" #0 [ff6000001fc501c0] riscv_crash_save_regs at ffffffff8000a1dc crash-utility#1 [ff6000001fc50320] panic at ffffffff808773ec crash-utility#2 [ff6000001fc50380] walk_stackframe at ffffffff800056da PC: ffffffff80876a34 [memset+96] RA: ffffffff80563dc0 [recursive_loop+68] SP: ff2000000000fd50 CAUSE: 000000000000000f epc : ffffffff80876a34 ra : ffffffff80563dc0 sp : ff2000000000fd50 gp : ffffffff81515d38 tp : 0000000000000000 t0 : ff2000000000fd58 t1 : ff600000000d88c8 t2 : 6143203a6d74646b s0 : ff20000000010190 s1 : 0000000000000012 a0 : ff2000000000fd58 a1 : 1212121212121212 a2 : 0000000000000400 a3 : ff20000000010158 a4 : 0000000000000000 a5 : 725bedba92260900 a6 : 000000000130e0f0 a7 : 0000000000000000 s2 : ff2000000000fd58 s3 : ffffffff815170d8 s4 : ff20000000013e60 s5 : 000000000000000e s6 : ff20000000013e60 s7 : 0000000000000000 s8 : ff60000000861000 s9 : 00007fffc3641694 s10: 00007fffc3641690 s11: 00005555796ed240 t3 : 0000000000010297 t4 : ffffffff80c17810 t5 : ffffffff8195e7b8 t6 : ff20000000013b18 status: 0000000200000120 badaddr: ff2000000000fd58 cause: 000000000000000f orig_a0: 0000000000000000 --- <OVERFLOW stack> --- crash-utility#3 [ff2000000000fd50] memset at ffffffff80876a34 crash-utility#4 [ff20000000010190] recursive_loop at ffffffff80563e16 crash-utility#5 [ff200000000105d0] recursive_loop at ffffffff80563e16 < recursive_loop ...> crash-utility#16 [ff20000000013490] recursive_loop at ffffffff80563e16 crash-utility#17 [ff200000000138d0] recursive_loop at ffffffff80563e16 crash-utility#18 [ff20000000013d10] lkdtm_EXHAUST_STACK at ffffffff8088005e crash-utility#19 [ff20000000013d30] lkdtm_do_action at ffffffff80563292 crash-utility#20 [ff20000000013d40] direct_entry at ffffffff80563474 crash-utility#21 [ff20000000013d70] full_proxy_write at ffffffff8032fb3a crash-utility#22 [ff20000000013db0] vfs_write at ffffffff801d6414 crash-utility#23 [ff20000000013e60] ksys_write at ffffffff801d67b8 crash-utility#24 [ff20000000013eb0] __riscv_sys_write at ffffffff801d6832 crash-utility#25 [ff20000000013ec0] do_trap_ecall_u at ffffffff80884a20 crash> crash> help -m <snip> irq_stack_size: 16384 irq_stacks[0]: ff20000000000000 irq_stacks[1]: ff20000000008000 overflow_stack_size: 4096 overflow_stacks[0]: ff6000001fa7a510 overflow_stacks[1]: ff6000001fc4f510 crash> Signed-off-by: Song Shuai <[email protected]>

On recent x86_64 kernels, the check of caller function (BT_CHECK_CALLER) does not work correctly due to inappropriate direct_call_targets. As a result, the correct frame is ignored and the remaining frames will be truncated. Skip the caller check if ORC unwinder is available, as the check is not necessary with it. Without the patch: crash> bt 493113 PID: 493113 TASK: ff2e34ecbd3ca2c0 CPU: 27 COMMAND: "sriov_fec_daemo" #0 [ff77abc4e81cfb08] __schedule at ffffffff81b239cb crash-utility#1 [ff77abc4e81cfb70] schedule at ffffffff81b23e2d crash-utility#2 [ff77abc4e81cfb88] schedule_timeout at ffffffff81b2c9e8 RIP: 000000000047cdbb RSP: 000000c0000975a8 RFLAGS: 00000216 ... With the patch: crash> bt 493113 PID: 493113 TASK: ff2e34ecbd3ca2c0 CPU: 27 COMMAND: "sriov_fec_daemo" #0 [ff77abc4e81cfb08] __schedule at ffffffff81b239cb crash-utility#1 [ff77abc4e81cfb70] schedule at ffffffff81b23e2d crash-utility#2 [ff77abc4e81cfb88] schedule_timeout at ffffffff81b2c9e8 crash-utility#3 [ff77abc4e81cfbf0] __wait_for_common at ffffffff81b24abb crash-utility#4 [ff77abc4e81cfc68] vfio_unregister_group_dev at ffffffffc10e76ae [vfio] crash-utility#5 [ff77abc4e81cfca8] vfio_pci_core_unregister_device at ffffffffc11bb599 [vfio_pci_core] crash-utility#6 [ff77abc4e81cfcc0] vfio_pci_remove at ffffffffc103e045 [vfio_pci] crash-utility#7 [ff77abc4e81cfcd0] pci_device_remove at ffffffff815d7513 ... Reported-by: Crystal Wood <[email protected]> Signed-off-by: Kazuhito Hagio <[email protected]>

…ss range Previously, to find a module symbol and its offset by an arbitrary address, all symbols within the module will be iterated by address ascending order until the last symbol with a smaller address been noticed. However if the address is not within the module address range, e.g. the address is higher than the module's last symbol's address, then the module can be surely skipped, because its symbol iteration is unnecessary. This can speed up the kernel module symbols finding and improve the overall performance. Without the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 crash-utility#1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 crash-utility#2 [ffff927569cd7768] __mutex_lock_slowpath at ffffffffb3db6ca7 crash-utility#3 [ffff927569cd77c0] mutex_lock at ffffffffb3db602f crash-utility#4 [ffff927569cd77d8] ucache_retrieve at ffffffffc0cf4409 [secfs2] ...snip the stacktrace of the same module... crash-utility#11 [ffff927569cd7ba0] cskal_path_vfs_getattr_nosec at ffffffffc05cae76 [falcon_kal] ...snip... crash-utility#13 [ffff927569cd7c40] _ZdlPv at ffffffffc086e751 [falcon_lsm_serviceable] ...snip... crash-utility#20 [ffff927569cd7ef8] unload_network_ops_symbols at ffffffffc06f11c0 [falcon_lsm_pinned_14713] crash-utility#21 [ffff927569cd7f50] system_call_fastpath at ffffffffb3dc539a RIP: 00007f2b28ed4023 RSP: 00007f2a45fe7f80 RFLAGS: 00000206 RAX: 0000000000000012 RBX: 00007f2a68302e00 RCX: 00007f2a682546d8 RDX: 0000000000000826 RSI: 00007eb57ea6a000 RDI: 00000000000000e3 RBP: 00007eb57ea6a000 R8: 0000000000000826 R9: 00000002670bdfd2 R10: 00000002670bdfd2 R11: 0000000000000293 R12: 00000002670bdfd2 R13: 00007f29d501a480 R14: 0000000000000826 R15: 00000002670bdfd2 ORIG_RAX: 0000000000000012 CS: 0033 SS: 002b crash> real 7m14.826s user 7m12.502s sys 0m1.091s With the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 crash-utility#1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 ...snip the same output... crash> real 0m8.827s user 0m7.896s sys 0m0.938s Signed-off-by: Tao Liu <[email protected]>

- Add basic support for the 'bt' command. - LooongArch64: Add 'bt -f' command support - LoongArch64: Add 'bt -l' command support E.g. With this patch: crash> bt PID: 1832 TASK: 900000009a552100 CPU: 11 COMMAND: "bash" #0 [900000009beffb60] __cpu_possible_mask at 90000000014168f0 crash-utility#1 [900000009beffb60] __crash_kexec at 90000000002e7660 crash-utility#2 [900000009beffcd0] panic at 9000000000f0ec28 crash-utility#3 [900000009beffd60] sysrq_handle_crash at 9000000000a2c188 crash-utility#4 [900000009beffd70] __handle_sysrq at 9000000000a2c85c crash-utility#5 [900000009beffdc0] write_sysrq_trigger at 9000000000a2ce10 crash-utility#6 [900000009beffde0] proc_reg_write at 90000000004ce454 crash-utility#7 [900000009beffe00] vfs_write at 900000000043e838 crash-utility#8 [900000009beffe40] ksys_write at 900000000043eb58 crash-utility#9 [900000009beffe80] do_syscall at 9000000000f2da54 crash-utility#10 [900000009beffea0] handle_syscall at 9000000000221440 crash> ... Co-developed-by: Youling Tang <[email protected]> Signed-off-by: Youling Tang <[email protected]> Signed-off-by: Ming Wang <[email protected]>

…ame" warning The "bogus exception frame" warning was observed again on a specific vmcore, and the remaining frame was truncated on x86_64 machine, when executing the "bt" command as below: crash> bt 0 -c 8 PID: 0 TASK: ffff9948c08f5640 CPU: 8 COMMAND: "swapper/8" #0 [fffffe1788788e58] crash_nmi_callback at ffffffff972672bb crash-utility#1 [fffffe1788788e68] nmi_handle at ffffffff9722eb8e crash-utility#2 [fffffe1788788eb0] default_do_nmi at ffffffff97e51cd0 crash-utility#3 [fffffe1788788ed0] exc_nmi at ffffffff97e51ee1 crash-utility#4 [fffffe1788788ef0] end_repeat_nmi at ffffffff980015f9 [exception RIP: __update_load_avg_se+13] RIP: ffffffff9736b16d RSP: ffffbec3c08acc78 RFLAGS: 00000046 RAX: 0000000000000000 RBX: ffff994c2f2b1a40 RCX: ffffbec3c08acdc0 RDX: ffff9948e4fe1d80 RSI: ffff994c2f2b1a40 RDI: 0000001d7ad7d55d RBP: ffffbec3c08acc88 R8: 0000001d921fca6f R9: ffff994c2f2b1328 R10: 00000000fffd0010 R11: ffffffff98e060c0 R12: 0000001d7ad7d55d R13: 0000000000000005 R14: ffff994c2f2b19c0 R15: 0000000000000001 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> --- crash-utility#5 [ffffbec3c08acc78] __update_load_avg_se at ffffffff9736b16d crash-utility#6 [ffffbec3c08acce0] enqueue_entity at ffffffff9735c9ab crash-utility#7 [ffffbec3c08acd28] enqueue_task_fair at ffffffff9735cef8 ... crash-utility#18 [ffffbec3c08acf90] blk_complete_reqs at ffffffff977978d0 crash-utility#19 [ffffbec3c08acfa0] __do_softirq at ffffffff97e66f7a crash-utility#20 [ffffbec3c08acff0] do_softirq at ffffffff9730f6ef --- <IRQ stack> --- crash-utility#21 [ffffbec3c022ff18] do_idle at ffffffff97368288 [exception RIP: unknown or invalid address] RIP: 0000000000000000 RSP: 0000000000000000 RFLAGS: 00000000 RAX: 0000000000000000 RBX: 000000089726a2d0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: ffffffff9726a3dd R8: 0000000000000000 R9: 0000000000000000 R10: ffffffff9720015a R11: e48885e126bc1600 R12: 0000000000000000 R13: ffffffff973684a9 R14: 0000000000000094 R15: 0000000040000000 ORIG_RAX: 0000000000000000 CS: 0000 SS: 0000 bt: WARNING: possibly bogus exception frame crash> Actually there is no exception frame, when called from do_softirq(). With the patch: crash> bt 0 -c 8 ... crash-utility#18 [ffffbec3c08acf90] blk_complete_reqs at ffffffff977978d0 crash-utility#19 [ffffbec3c08acfa0] __do_softirq at ffffffff97e66f7a crash-utility#20 [ffffbec3c08acff0] do_softirq at ffffffff9730f6ef --- <IRQ stack> --- crash-utility#21 [ffffbec3c022ff28] cpu_startup_entry at ffffffff973684a9 crash-utility#22 [ffffbec3c022ff38] start_secondary at ffffffff9726a3dd crash-utility#23 [ffffbec3c022ff50] secondary_startup_64_no_verify at ffffffff9720015a crash> Reported-by: Jie Li <[email protected]> Signed-off-by: Lianbo Jiang <[email protected]>

The following segmentation fault occurred during session initialization: $ crash vmlinx vmcore ... please wait... (determining panic task)Segmentation fault Here is the backtrace of the crash-utility: (gdb) bt #0 value_search_module_6_4 (value=18446603338276298752, offset=0x7ffffffface0) at symbols.c:5564 crash-utility#1 0x0000555555812bd0 in value_to_symstr (value=18446603338276298752, buf=buf@entry=0x7fffffffb9c0 "", radix=10, radix@entry=0) at symbols.c:5872 crash-utility#2 0x00005555557694a2 in display_memory (addr=<optimized out>, count=2048, flag=208, memtype=memtype@entry=1, opt=opt@entry=0x0) at memory.c:1740 crash-utility#3 0x0000555555769e1f in raw_stack_dump (stackbase=<optimized out>, size=<optimized out>) at memory.c:2194 crash-utility#4 0x00005555557923ff in get_active_set_panic_task () at task.c:8639 crash-utility#5 0x00005555557930d2 in get_dumpfile_panic_task () at task.c:7628 crash-utility#6 0x00005555557a89d3 in panic_search () at task.c:7380 crash-utility#7 get_panic_context () at task.c:6267 crash-utility#8 task_init () at task.c:687 crash-utility#9 0x00005555557305b3 in main_loop () at main.c:787 ... This is due to lack of existence check on module symbol table. Not all mod_mem_type will be existent for a module, e.g. in the following module case: (gdb) p lm->symtable[0] $1 = (struct syment *) 0x4dcbad0 (gdb) p lm->symtable[1] $2 = (struct syment *) 0x4dcbb70 (gdb) p lm->symtable[2] $3 = (struct syment *) 0x4dcbc10 (gdb) p lm->symtable[3] $4 = (struct syment *) 0x0 (gdb) p lm->symtable[4] $5 = (struct syment *) 0x4dcbcb0 (gdb) p lm->symtable[5] $6 = (struct syment *) 0x4dcbd00 (gdb) p lm->symtable[6] $7 = (struct syment *) 0x0 MOD_RO_AFTER_INIT(3) and MOD_INIT_RODATA(6) do not exist, which should be skipped, otherwise the segmentation fault will happen. Fixes: 7750e61 ("Support module memory layout change on Linux 6.4") Closes: crash-utility#176 Reported-by: Naveen Chaudhary <[email protected]> Signed-off-by: Tao Liu <[email protected]>

With Kernel commit 65c9cc9e2c14 ("x86/fred: Reserve space for the FRED stack frame") in Linux 6.9-rc1 and later, x86_64 will add extra padding ('TOP_OF_KERNEL_STACK_PADDING (2 * 8)', see: arch/x86/include/asm\ /thread_info.h,) for kernel stack when the CONFIG_X86_FRED is enabled. As a result, the pt_regs will be moved downwards due to the offset of padding, and the values of registers read from pt_regs will be incorrect as below. Without the patch: crash> bt PID: 2040 TASK: ffff969136fc4180 CPU: 16 COMMAND: "bash" #0 [ffffa996409aba38] machine_kexec at ffffffff9f881eb7 crash-utility#1 [ffffa996409aba90] __crash_kexec at ffffffff9fa1e49e crash-utility#2 [ffffa996409abb48] panic at ffffffff9f91a6cd crash-utility#3 [ffffa996409abbc8] sysrq_handle_crash at ffffffffa0015076 crash-utility#4 [ffffa996409abbd0] __handle_sysrq at ffffffffa0015640 crash-utility#5 [ffffa996409abc00] write_sysrq_trigger at ffffffffa0015ce5 crash-utility#6 [ffffa996409abc28] proc_reg_write at ffffffff9fd35bf5 crash-utility#7 [ffffa996409abc40] vfs_write at ffffffff9fc8d462 crash-utility#8 [ffffa996409abcd0] ksys_write at ffffffff9fc8dadf crash-utility#9 [ffffa996409abd08] do_syscall_64 at ffffffffa0517429 crash-utility#10 [ffffa996409abf40] entry_SYSCALL_64_after_hwframe at ffffffffa060012b [exception RIP: unknown or invalid address] RIP: 0000000000000246 RSP: 0000000000000000 RFLAGS: 0000002b RAX: 0000000000000002 RBX: 00007f9b9f5b13e0 RCX: 000055cee7486fb0 RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00007f9b9f4fda57 RBP: 0000000000000246 R8: 00007f9b9f4fda57 R9: ffffffffffffffda R10: 0000000000000000 R11: 00007f9b9f5b14e0 R12: 0000000000000002 R13: 000055cee7486fb0 R14: 0000000000000002 R15: 00007f9b9f5fb780 ORIG_RAX: 0000000000000033 CS: 7ffe65327978 SS: 0000 bt: WARNING: possibly bogus exception frame crash> With the patch: crash> bt PID: 2040 TASK: ffff969136fc4180 CPU: 16 COMMAND: "bash" #0 [ffffa996409aba38] machine_kexec at ffffffff9f881eb7 crash-utility#1 [ffffa996409aba90] __crash_kexec at ffffffff9fa1e49e crash-utility#2 [ffffa996409abb48] panic at ffffffff9f91a6cd crash-utility#3 [ffffa996409abbc8] sysrq_handle_crash at ffffffffa0015076 crash-utility#4 [ffffa996409abbd0] __handle_sysrq at ffffffffa0015640 crash-utility#5 [ffffa996409abc00] write_sysrq_trigger at ffffffffa0015ce5 crash-utility#6 [ffffa996409abc28] proc_reg_write at ffffffff9fd35bf5 crash-utility#7 [ffffa996409abc40] vfs_write at ffffffff9fc8d462 crash-utility#8 [ffffa996409abcd0] ksys_write at ffffffff9fc8dadf crash-utility#9 [ffffa996409abd08] do_syscall_64 at ffffffffa0517429 crash-utility#10 [ffffa996409abf40] entry_SYSCALL_64_after_hwframe at ffffffffa060012b RIP: 00007f9b9f4fda57 RSP: 00007ffe65327978 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f9b9f4fda57 RDX: 0000000000000002 RSI: 000055cee7486fb0 RDI: 0000000000000001 RBP: 000055cee7486fb0 R8: 0000000000000000 R9: 00007f9b9f5b14e0 R10: 00007f9b9f5b13e0 R11: 0000000000000246 R12: 0000000000000002 R13: 00007f9b9f5fb780 R14: 0000000000000002 R15: 00007f9b9f5f69e0 ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b crash> Link: https://www.mail-archive.com/[email protected]/msg00754.html Signed-off-by: Lianbo Jiang <[email protected]> Signed-off-by: Tao Liu <[email protected]>

The commit 48764a1 may cause a regression issue when the CONFIG_X86_FRED is not enabled, this is because the SIZE(fred_frame) will call the SIZE_verify() to determine if the fred_frame is valid, otherwise it will emit an error: crash> bt 1 bt: invalid structure size: fred_frame FILE: x86_64.c LINE: 4089 FUNCTION: x86_64_low_budget_back_trace_cmd() [/home/k-hagio/bin/crash] error trace: 588df3 => 5cbc72 => 5eb3e1 => 5eb366 PID: 1 TASK: ffff9f94c024b980 CPU: 2 COMMAND: "systemd" #0 [ffffade44001bca8] __schedule at ffffffffb948ebbb crash-utility#1 [ffffade44001bd10] schedule at ffffffffb948f04d crash-utility#2 [ffffade44001bd20] schedule_hrtimeout_range_clock at ffffffffb9494fef crash-utility#3 [ffffade44001bda8] ep_poll at ffffffffb8c91be8 crash-utility#4 [ffffade44001be48] do_epoll_wait at ffffffffb8c91d11 crash-utility#5 [ffffade44001be80] __x64_sys_epoll_wait at ffffffffb8c92590 crash-utility#6 [ffffade44001bed0] do_syscall_64 at ffffffffb947f459 crash-utility#7 [ffffade44001bf50] entry_SYSCALL_64_after_hwframe at ffffffffb96000ea 5eb366: SIZE_verify.part.42+70 5eb3e1: SIZE_verify+49 5cbc72: x86_64_low_budget_back_trace_cmd+3010 588df3: back_trace+1523 bt: invalid structure size: fred_frame FILE: x86_64.c LINE: 4089 FUNCTION: x86_64_low_budget_back_trace_cmd() Let's replace the SIZE(fred_frame) with the VALID_SIZE(fred_frame) to fix it. Fixes: 48764a1 ("x86_64: fix for adding top_of_kernel_stack_padding for kernel stack") Reported-by: Kazuhito Hagio <[email protected]> Signed-off-by: Lianbo Jiang <[email protected]>

For ramdump(Qcom phone device) case with the kernel option CONFIG_ARM64_PTR_AUTH_KERNEL enabled, the bt command may print incorrect stacktrace as below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 crash-utility#2 [ffffffc034c438b0] __kvm_nvhe_$d.2314 at 86c54c6004ceff80 crash-utility#3 [ffffffc034c43950] __kvm_nvhe_$d.2314 at 55d6f96003a7b120 ... PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 Crash tool can not get the KERNELPACMASK value from the vmcoreinfo, need to calculate its value based on the vabits. With the patch: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __schedule at ffffffe004cf05a0 crash-utility#2 [ffffffc034c438b0] preempt_schedule_common at ffffffe004ceff80 crash-utility#3 [ffffffc034c43950] unmap_page_range at ffffffe003a7b120 crash-utility#4 [ffffffc034c439f0] unmap_vmas at ffffffe003a80a64 crash-utility#5 [ffffffc034c43ac0] exit_mmap at ffffffe003a945c4 crash-utility#6 [ffffffc034c43b10] __mmput at ffffffe00372c818 crash-utility#7 [ffffffc034c43b40] mmput at ffffffe00372c0d0 crash-utility#8 [ffffffc034c43b90] exit_mm at ffffffe00373d0ac crash-utility#9 [ffffffc034c43c00] do_exit at ffffffe00373bedc PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 Related kernel commits: 689eae42afd7 ("arm64: mask PAC bits of __builtin_return_address") de1702f65feb ("arm64: move PAC masks to <asm/pointer_auth.h>") Signed-off-by: bevis_chen <[email protected]>

See the following stack trace: (gdb) bt #0 0x00005635ac2b166b in arm64_unwind_frame (frame=0x7ffdaf35cb70, bt=0x7ffdaf35d430) at arm64.c:2821 crash-utility#1 arm64_back_trace_cmd (bt=0x7ffdaf35d430) at arm64.c:3306 crash-utility#2 0x00005635ac27b108 in back_trace (bt=bt@entry=0x7ffdaf35d430) at kernel.c:3239 crash-utility#3 0x00005635ac2880ae in cmd_bt () at kernel.c:2863 crash-utility#4 0x00005635ac1f16dc in exec_command () at main.c:893 crash-utility#5 0x00005635ac1f192a in main_loop () at main.c:840 crash-utility#6 0x00005635ac50df81 in captured_main (data=<optimized out>) at main.c:1284 crash-utility#7 gdb_main (args=<optimized out>) at main.c:1313 crash-utility#8 0x00005635ac50e000 in gdb_main_entry (argc=<optimized out>, argv=<optimized out>) at main.c:1338 crash-utility#9 0x00005635ac1ea2a5 in main (argc=5, argv=0x7ffdaf35dde8) at main.c:721 The issue may be encountered when thread_union symbol not found in vmlinux due to compiling optimization. This patch will try the following 2 methods to get the irq_stack_size when thread_union symbol unavailable: 1. change the thread_shift when KASAN is enabled and with vmcoreinfo. In arm64/include/asm/memory.h: #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) ... #define IRQ_STACK_SIZE THREAD_SIZE Since enabling the KASAN will affect the final value, this patch reset IRQ_STACK_SIZE according to the calculation process in kernel code. 2. Try getting the value from kernel code disassembly, to get THREAD_SHIFT directly from tbnz instruction. In arch/arm64/kernel/entry.S: .macro kernel_ventry, el:req, ht:req, regsize:req, label:req ... add sp, sp, x0 sub x0, sp, x0 tbnz x0, #THREAD_SHIFT, 0f $ gdb vmlinux (gdb) disass vectors Dump of assembler code for function vectors: ... 0xffff800080010804 <+4>: add sp, sp, x0 0xffff800080010808 <+8>: sub x0, sp, x0 0xffff80008001080c <+12>: tbnz w0, crash-utility#16, 0xffff80008001081c <vectors+28> Signed-off-by: yeping.zheng <[email protected]> Improved-by: Tao Liu <[email protected]>

Sometimes, in production environment, there are still some vmcores that are incomplete, such as partial header or the data is corrupted. When crash tool attempts to parse such vmcores, it may fail as below: $ ./crash --osrelease vmcore Bus error (core dumped) or $ crash vmlinux vmcore ... Bus error (core dumped) $ Gdb calltrace: $ gdb /home/lijiang/src/crash/crash /tmp/core.126301 Core was generated by `./crash --osrelease /home/lijiang/src/39317/vmcore'. Program terminated with signal SIGBUS, Bus error. #0 __memcpy_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:831 831 LOAD_ONE_SET((%rsi), PAGE_SIZE, %VMM(4), %VMM(5), %VMM(6), %VMM(7)) (gdb) bt #0 __memcpy_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:831 crash-utility#1 0x0000000000651096 in read_dump_header (file=0x7ffc59ddff5f "/home/lijiang/src/39317/vmcore") at diskdump.c:820 crash-utility#2 0x0000000000651cf3 in is_diskdump (file=0x7ffc59ddff5f "/home/lijiang/src/39317/vmcore") at diskdump.c:1042 crash-utility#3 0x0000000000502ac9 in get_osrelease (dumpfile=0x7ffc59ddff5f "/home/lijiang/src/39317/vmcore") at main.c:1938 crash-utility#4 0x00000000004fb2e8 in main (argc=3, argv=0x7ffc59dde3a8) at main.c:271 (gdb) frame 1 crash-utility#1 0x0000000000651096 in read_dump_header (file=0x7ffc59ddff5f "/home/lijiang/src/39317/vmcore") at diskdump.c:820 820 memcpy(dd->dumpable_bitmap, dd->bitmap + bitmap_len/2, This may happen on attempting access to a page of the buffer that lies beyond the end of the mapped file(see the mmap() man page). Let's add a check to avoid such issues as much as possible, but still not guarantee that it can work well in any extreme situation. Fixes: a334423 ("diskdump: use mmap/madvise to improve the start-up") Reported-by: Buland Kumar Singh <[email protected]> Signed-off-by: Lianbo Jiang <[email protected]>

Currently, gdb passthroughs of 'bt', 'frame', 'up', 'down', 'info locals' don't work. This is due to gdb not knowing the register values to unwind the stack frames Every gdb passthrough goes through `gdb_interface`. And then, gdb expects `crash_target::fetch_registers` to give it the register values, which is dependent on `machdep->get_current_task_reg` to read the register values for specific architecture. ---------------------------- gdb passthrough (eg. "bt") | | crash -------------------------> | | | gdb_interface | | | | | | ---------------------- | fetch_registers | | | | crash_target<-------------------------+--| gdb | | --------------------------+->| | | Registers (SP,NIP, etc.)| | | | | | | | | ---------------------- | ---------------------------- Implement `machdep->get_current_task_reg` on PPC64, so that crash provides the register values to gdb to unwind stack frames properly With these changes, on powerpc, 'bt' command output in gdb mode, will look like this: gdb> bt #0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>, newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69 crash-utility#1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974 crash-utility#2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358 crash-utility#3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:155 crash-utility#4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99, check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602 crash-utility#5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1163 crash-utility#6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at fs/proc/inode.c:340 crash-utility#7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352 crash-utility#8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00, buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>, count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582 instead of earlier output without this patch: gdb> bt #0 <unavailable> in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Also, 'get_dumpfile_regs' has been introduced to get registers from multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS' has been introduced to tell helper functions to get registers, to not print registers with every call to backtrace in gdb. Note: This feature to support GDB unwinding doesn't support live debugging [lijiang: squash these five patches(see the Link) into one patch] Link: https://www.mail-archive.com/[email protected]/msg01084.html Link: https://www.mail-archive.com/[email protected]/msg01083.html Link: https://www.mail-archive.com/[email protected]/msg01089.html Link: https://www.mail-archive.com/[email protected]/msg01090.html Link: https://www.mail-archive.com/[email protected]/msg01091.html Co-developed-by:: Tao Liu <[email protected]> Signed-off-by: Aditya Gupta <[email protected]>

include sys/sysmacros.h for major/minor/makedev

f22dabb

These funcs are defined in the sys/sysmacros.h header, not sys/types.h. Linux C libraries are updating to drop the implicit include, so we need to include it explicitly.

crash-utility closed this Apr 21, 2016

crash-utility mentioned this pull request May 26, 2016

BT_REGS_NOT_FOUND when bt a disk dump file #5

Closed

leilchen mentioned this pull request Aug 31, 2017

memory corruption #20

Closed

crash-utility mentioned this pull request Oct 25, 2017

fail to get stack frame on crash 7.2.0 #22

Open

Leo-Yan pushed a commit to Leo-Yan/crash that referenced this pull request Nov 20, 2017

Fixup crash-utility#4

09e9b1c

Signed-off-by: Leo Yan <[email protected]>

pandafromspace mentioned this pull request Jan 28, 2019

Unable to start 7.2.5 crash with SLES15 coredump #28

Open

crash-utility mentioned this pull request Nov 13, 2019

bt: cannot transition from exception stack to current process stack #43

Open

This was referenced Nov 28, 2019

Is crash tested with AddressSanitizer? #44

Open

Fix SIGSEGV when cur_req is null pointer #45

Closed

zhaoqianli0202 mentioned this pull request Mar 4, 2020

Trace32 has ability to show inline call in callstack #49

Open

cjl20062529 mentioned this pull request Jun 3, 2020

Error reading global variable value in module with p command #50

Open

yangh mentioned this pull request Nov 15, 2021

arm64: Support overflow stack panic #100

Closed

zslxiu mentioned this pull request Jun 29, 2024

bt command issue and don't know how to handle section `. relr. dyn' [0x13] #181

Open

happybevis mentioned this pull request Jul 3, 2024

arm64: Fix bt command show wrong stacktrace on ramdump source #183

Closed

happybevis mentioned this pull request Jul 3, 2024

arm64: Fix bt command show wrong stacktrace on ramdump source #184

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

include sys/sysmacros.h for major/minor/makedev #4

include sys/sysmacros.h for major/minor/makedev #4

vapier commented Apr 21, 2016

crash-utility commented Apr 21, 2016

include sys/sysmacros.h for major/minor/makedev #4

include sys/sysmacros.h for major/minor/makedev #4

Conversation

vapier commented Apr 21, 2016

crash-utility commented Apr 21, 2016