crash: fix build failure with mips #11

DengkeDu · 2017-05-04T07:20:02Z

When build crash with mips:

mips-linux-nat.c:157:1: error: conflicting types for 'ps_get_thread_area'
 ps_get_thread_area (const struct ps_prochandle *ph,
 ^~~~~~~~~~~~~~~~~~
In file included from gdb_proc_service.h:26:0,
                 from mips-linux-nat.c:32:

This problem have been fixed on intel and arm plantform, we can use the
similar approch to fix it on mips.

Signed-off-by: Dengke Du [email protected]

When build crash with mips: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ mips-linux-nat.c:157:1: error: conflicting types for 'ps_get_thread_area' ps_get_thread_area (const struct ps_prochandle *ph, ^~~~~~~~~~~~~~~~~~ In file included from gdb_proc_service.h:26:0, from mips-linux-nat.c:32: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This problem have been fixed on intel and arm plantform, we can use the similar approch to fix it on mips. Signed-off-by: Dengke Du <[email protected]>

crash-utility · 2017-05-04T13:50:33Z

----- Original Message -----

When build crash with mips: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ mips-linux-nat.c:157:1: error: conflicting types for 'ps_get_thread_area' ps_get_thread_area (const struct ps_prochandle *ph, ^~~~~~~~~~~~~~~~~~ In file included from gdb_proc_service.h:26:0, from mips-linux-nat.c:32: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This problem have been fixed on intel and arm plantform, we can use the similar approch to fix it on mips. Signed-off-by: Dengke Du ***@***.***> You can view, comment on, or merge this pull request online at: #11 -- Commit Summary -- * crash: fix build failure with mips -- File Changes -- M Makefile (1) A crash-fix-build-failure-with-mips.patch (39) -- Patch Links -- https://github.com/crash-utility/crash/pull/11.patch https://github.com/crash-utility/crash/pull/11.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #11

OK, but what I will do is to append your mips-linux-nat.c patch segment onto the end of the existing gdb-7.6-proc_service.h.patch. Also, please note that I don't do pull requests from the github repository. In the future, please post any patch requests to the [email protected] mailing list here: https://www.redhat.com/mailman/listinfo/crash-utility I sent you an invite email. It's a moderated list, but even if you don't subscribe, when I look through the daily spam messages, I will accept any legitimate emails. Thanks, Dave

crash-utility · 2017-05-04T14:27:24Z

The patch has been queued for crash-7.2.0: 3ef5191 Fix for a 32-bit MIPS compilation error if glibc-2.25 or later has been installed on the host build machine. Without the patch, the build fails with the error message "mips-linux-nat.c:157:1: error: conflicting types for 'ps_get_thread_area'". ([email protected]) Thanks, Dave

…

----- Original Message -----

When build crash with mips: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ mips-linux-nat.c:157:1: error: conflicting types for 'ps_get_thread_area' ps_get_thread_area (const struct ps_prochandle *ph, ^~~~~~~~~~~~~~~~~~ In file included from gdb_proc_service.h:26:0, from mips-linux-nat.c:32: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This problem have been fixed on intel and arm plantform, we can use the similar approch to fix it on mips. Signed-off-by: Dengke Du ***@***.***> You can view, comment on, or merge this pull request online at: #11 -- Commit Summary -- * crash: fix build failure with mips -- File Changes -- M Makefile (1) A crash-fix-build-failure-with-mips.patch (39) -- Patch Links -- https://github.com/crash-utility/crash/pull/11.patch https://github.com/crash-utility/crash/pull/11.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #11

Fix for 'bt' command and options on Linux 5.8-rc1 or later kernels that contain merge commit 076f14be7fc942e112c94c841baec44124275cd0. The merged patches changed the name of exception functions that have been used by the crash utility to check the exception frame. Without the patch, the command and options cannot display it. Before: crash> bt PID: 8752 TASK: ffff8f80cb244380 CPU: 2 COMMAND: "insmod" #0 [ffffa3e40187f9f8] machine_kexec at ffffffffab25d267 crash-utility#1 [ffffa3e40187fa48] __crash_kexec at ffffffffab38e2ed crash-utility#2 [ffffa3e40187fb10] crash_kexec at ffffffffab38f1dd crash-utility#3 [ffffa3e40187fb28] oops_end at ffffffffab222cbd crash-utility#4 [ffffa3e40187fb48] do_trap at ffffffffab21fea1 crash-utility#5 [ffffa3e40187fb90] do_error_trap at ffffffffab21ff75 crash-utility#6 [ffffa3e40187fbd0] exc_invalid_op at ffffffffabb76a2c crash-utility#7 [ffffa3e40187fbf0] asm_exc_invalid_op at ffffffffabc00a72 crash-utility#8 [ffffa3e40187fc78] init_module at ffffffffc042b018 [invalid] crash-utility#9 [ffffa3e40187fca0] init_module at ffffffffc042b018 [invalid] crash-utility#10 [ffffa3e40187fca8] do_one_initcall at ffffffffab202806 crash-utility#11 [ffffa3e40187fd18] do_init_module at ffffffffab3888ba crash-utility#12 [ffffa3e40187fd38] load_module at ffffffffab38afde After: crash> bt PID: 8752 TASK: ffff8f80cb244380 CPU: 2 COMMAND: "insmod" #0 [ffffa3e40187f9f8] machine_kexec at ffffffffab25d267 crash-utility#1 [ffffa3e40187fa48] __crash_kexec at ffffffffab38e2ed crash-utility#2 [ffffa3e40187fb10] crash_kexec at ffffffffab38f1dd crash-utility#3 [ffffa3e40187fb28] oops_end at ffffffffab222cbd crash-utility#4 [ffffa3e40187fb48] do_trap at ffffffffab21fea1 crash-utility#5 [ffffa3e40187fb90] do_error_trap at ffffffffab21ff75 crash-utility#6 [ffffa3e40187fbd0] exc_invalid_op at ffffffffabb76a2c crash-utility#7 [ffffa3e40187fbf0] asm_exc_invalid_op at ffffffffabc00a72 [exception RIP: init_module+24] RIP: ffffffffc042b018 RSP: ffffa3e40187fca8 RFLAGS: 00010246 RAX: 000000000000001c RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff8f80fbd18000 RDI: ffff8f80fbd18000 RBP: ffffffffc042b000 R8: 000000000000029d R9: 000000000000002c R10: 0000000000000000 R11: ffffa3e40187fb58 R12: ffffffffc042d018 R13: ffffa3e40187fdf0 R14: ffffffffc042d000 R15: ffffa3e40187fe90 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 crash-utility#8 [ffffa3e40187fca0] init_module at ffffffffc042b018 [invalid] crash-utility#9 [ffffa3e40187fca8] do_one_initcall at ffffffffab202806 crash-utility#10 [ffffa3e40187fd18] do_init_module at ffffffffab3888ba crash-utility#11 [ffffa3e40187fd38] load_module at ffffffffab38afde Signed-off-by: Kazuhito Hagio <[email protected]>

Fix for 'bt' command and options on Linux 5.8-rc1 and later kernels that contain merge commit 076f14be7fc942e112c94c841baec44124275cd0. The merged patches changed the name of exception functions that have been used by the crash utility to check the exception frame. Without the patch, the command and options cannot display it. Before: crash> bt PID: 8752 TASK: ffff8f80cb244380 CPU: 2 COMMAND: "insmod" #0 [ffffa3e40187f9f8] machine_kexec at ffffffffab25d267 #1 [ffffa3e40187fa48] __crash_kexec at ffffffffab38e2ed #2 [ffffa3e40187fb10] crash_kexec at ffffffffab38f1dd #3 [ffffa3e40187fb28] oops_end at ffffffffab222cbd #4 [ffffa3e40187fb48] do_trap at ffffffffab21fea1 #5 [ffffa3e40187fb90] do_error_trap at ffffffffab21ff75 #6 [ffffa3e40187fbd0] exc_invalid_op at ffffffffabb76a2c #7 [ffffa3e40187fbf0] asm_exc_invalid_op at ffffffffabc00a72 #8 [ffffa3e40187fc78] init_module at ffffffffc042b018 [invalid] #9 [ffffa3e40187fca0] init_module at ffffffffc042b018 [invalid] #10 [ffffa3e40187fca8] do_one_initcall at ffffffffab202806 #11 [ffffa3e40187fd18] do_init_module at ffffffffab3888ba #12 [ffffa3e40187fd38] load_module at ffffffffab38afde After: crash> bt PID: 8752 TASK: ffff8f80cb244380 CPU: 2 COMMAND: "insmod" #0 [ffffa3e40187f9f8] machine_kexec at ffffffffab25d267 #1 [ffffa3e40187fa48] __crash_kexec at ffffffffab38e2ed #2 [ffffa3e40187fb10] crash_kexec at ffffffffab38f1dd #3 [ffffa3e40187fb28] oops_end at ffffffffab222cbd #4 [ffffa3e40187fb48] do_trap at ffffffffab21fea1 #5 [ffffa3e40187fb90] do_error_trap at ffffffffab21ff75 #6 [ffffa3e40187fbd0] exc_invalid_op at ffffffffabb76a2c #7 [ffffa3e40187fbf0] asm_exc_invalid_op at ffffffffabc00a72 [exception RIP: init_module+24] RIP: ffffffffc042b018 RSP: ffffa3e40187fca8 RFLAGS: 00010246 RAX: 000000000000001c RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff8f80fbd18000 RDI: ffff8f80fbd18000 RBP: ffffffffc042b000 R8: 000000000000029d R9: 000000000000002c R10: 0000000000000000 R11: ffffa3e40187fb58 R12: ffffffffc042d018 R13: ffffa3e40187fdf0 R14: ffffffffc042d000 R15: ffffa3e40187fe90 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #8 [ffffa3e40187fca0] init_module at ffffffffc042b018 [invalid] #9 [ffffa3e40187fca8] do_one_initcall at ffffffffab202806 #10 [ffffa3e40187fd18] do_init_module at ffffffffab3888ba #11 [ffffa3e40187fd38] load_module at ffffffffab38afde Signed-off-by: Kazuhito Hagio <[email protected]>

Overflow stack supported since kernel 4.14 in commit 872d8327ce8, without this patch, bt command trigger a SIGSEGV fault due the SP pointed to the overflow stack which not yet loaded by crash. Before: KERNEL: ../vmlinux DUMPFILE: la_guestdump.gcore CPUS: 8 DATE: Tue Jul 13 19:59:44 CST 2021 UPTIME: 00:00:42 LOAD AVERAGE: 3.99, 1.13, 0.39 TASKS: 1925 NODENAME: localhost RELEASE: 4.14.156+ VERSION: crash-utility#1 SMP PREEMPT Tue Jul 13 10:37:23 UTC 2021 MACHINE: aarch64 (unknown Mhz) MEMORY: 8.7 GB PANIC: "Kernel panic - not syncing: kernel stack overflow" PID: 1969 COMMAND: "irq/139-0-0024" TASK: ffffffcc1a230000 [THREAD_INFO: ffffffcc1a230000] CPU: 0 STATE: TASK_RUNNING (PANIC) crash-7.3.0> bt PID: 1969 TASK: ffffffcc1a230000 CPU: 0 COMMAND: "irq/139-0-0024" Segmentation fault (core dumped) After: crash> bt PID: 1969 TASK: ffffffcc1a230000 CPU: 0 COMMAND: "irq/139-0-0024" #0 [ffffffcc7fd5cf50] __delay at ffffff8008c80774 crash-utility#1 [ffffffcc7fd5cf60] __const_udelay at ffffff8008c80864 crash-utility#2 [ffffffcc7fd5cf80] msm_trigger_wdog_bite at ffffff80084e9430 crash-utility#3 [ffffffcc7fd5cfa0] do_vm_restart at ffffff80087bc974 crash-utility#4 [ffffffcc7fd5cfc0] machine_restart at ffffff80080856fc crash-utility#5 [ffffffcc7fd5cfd0] emergency_restart at ffffff80080d49bc crash-utility#6 [ffffffcc7fd5d140] panic at ffffff80080af4c0 crash-utility#7 [ffffffcc7fd5d150] nmi_panic at ffffff80080af150 crash-utility#8 [ffffffcc7fd5d190] handle_bad_stack at ffffff800808b0b8 crash-utility#9 [ffffffcc7fd5d2d0] __bad_stack at ffffff800808285c --- <IRQ stack> --- crash-utility#10 [ffffff801187bc60] el1_error_invalid at ffffff8008082e7c crash-utility#11 [ffffff801187bcc0] cyttsp6_mt_attention at ffffff8000e8498c [cyttsp6] crash-utility#12 [ffffff801187bd20] call_atten_cb at ffffff8000e82030 [cyttsp6] crash-utility#13 [ffffff801187bdc0] cyttsp6_irq at ffffff8000e81e34 [cyttsp6] crash-utility#14 [ffffff801187bdf0] irq_thread_fn at ffffff8008128dd8 crash-utility#15 [ffffff801187be50] irq_thread at ffffff8008128ca4 crash-utility#16 [ffffff801187beb0] kthread at ffffff80080d2fc4 crash> Signed-off-by: Hong YANG <[email protected]>

Kernel commit 7d65f4a65532 ("irq: Consolidate do_softirq() arch overriden implementations") renamed the call_softirq to do_softirq_own_stack, and there is no exception frame also when coming from do_softirq_own_stack. Without the patch, crash may unnecessarily output an exception frame with a warning as below: crash> foreach bt ... PID: 0 TASK: ffff914f820a8000 CPU: 25 COMMAND: "swapper/25" #0 [fffffe0000504e48] crash_nmi_callback at ffffffffa665d763 #1 [fffffe0000504e50] nmi_handle at ffffffffa662a423 #2 [fffffe0000504ea8] default_do_nmi at ffffffffa6fe7dc9 #3 [fffffe0000504ec8] do_nmi at ffffffffa662a97f #4 [fffffe0000504ef0] end_repeat_nmi at ffffffffa70015e8 [exception RIP: clone_endio+172] RIP: ffffffffc005c1ec RSP: ffffa1d403d08e98 RFLAGS: 00000246 RAX: 0000000000000000 RBX: ffff915326fba230 RCX: 0000000000000018 RDX: ffffffffc0075400 RSI: 0000000000000000 RDI: ffff915326fba230 RBP: ffff915326fba1c0 R8: 0000000000001000 R9: ffff915308d6d2a0 R10: 000000a97dfe5e10 R11: ffffa1d40038fe98 R12: ffff915302babc40 R13: ffff914f94360000 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> --- #5 [ffffa1d403d08e98] clone_endio at ffffffffc005c1ec [dm_mod] #6 [ffffa1d403d08ed0] blk_update_request at ffffffffa6a96954 #7 [ffffa1d403d08f10] scsi_end_request at ffffffffa6c9b968 #8 [ffffa1d403d08f48] scsi_io_completion at ffffffffa6c9bb3e #9 [ffffa1d403d08f90] blk_complete_reqs at ffffffffa6aa0e95 #10 [ffffa1d403d08fa0] __softirqentry_text_start at ffffffffa72000dc #11 [ffffa1d403d08ff0] do_softirq_own_stack at ffffffffa7000f9a --- <IRQ stack> --- #12 [ffffa1d40038fe70] do_softirq_own_stack at ffffffffa7000f9a [exception RIP: unknown or invalid address] RIP: 0000000000000000 RSP: 0000000000000000 RFLAGS: 00000000 RAX: ffffffffa672eae5 RBX: ffffffffa83b34e0 RCX: ffffffffa672eb12 RDX: 0000000000000010 RSI: 8b7d6c8869010c00 RDI: 0000000000000085 RBP: 0000000000000286 R8: ffff914f820a8000 R9: ffffffffa67a94e0 R10: 0000000000000286 R11: ffffffffa66fb4c5 R12: ffffffffa67a898b R13: 0000000000000000 R14: fffffffffffffff8 R15: ffffffffa67a1e68 ORIG_RAX: 0000000000000000 CS: 0000 SS: ffffffffa672edff bt: WARNING: possibly bogus exception frame #13 [ffffa1d40038ff30] start_secondary at ffffffffa665fa2c #14 [ffffa1d40038ff50] secondary_startup_64_no_verify at ffffffffa6600116 ... Reported-by: Marco Patalano <[email protected]> Signed-off-by: Lianbo Jiang <[email protected]>

Kernel commit fb799447ae29 ("x86,objtool: Split UNWIND_HINT_EMPTY in two"), which is contained in Linux 6.4 and later kernels, changed ORC_TYPE_CALL macro from 0 to 2. As a result, the "bt" command cannot use ORC entries and displays stale entries in a call trace. crash> bt 1 PID: 1 TASK: ffff93cd06294180 CPU: 51 COMMAND: "systemd" #0 [ffffb72bc00cbc98] __schedule at ffffffff86e52aae crash-utility#1 [ffffb72bc00cbd00] schedule at ffffffff86e52f6a crash-utility#2 [ffffb72bc00cbd18] schedule_hrtimeout_range_clock at ffffffff86e58ef5 crash-utility#3 [ffffb72bc00cbd88] ep_poll at ffffffff8669624d crash-utility#4 [ffffb72bc00cbe28] do_epoll_wait at ffffffff86696371 crash-utility#5 [ffffb72bc00cbe30] do_timerfd_settime at ffffffff8669902b << crash-utility#6 [ffffb72bc00cbe60] __x64_sys_epoll_wait at ffffffff86696bf0 crash-utility#7 [ffffb72bc00cbeb0] do_syscall_64 at ffffffff86e3feb9 crash-utility#8 [ffffb72bc00cbee0] __task_pid_nr_ns at ffffffff863330d7 << crash-utility#9 [ffffb72bc00cbf08] syscall_exit_to_user_mode at ffffffff86e466b2 << stale entries crash-utility#10 [ffffb72bc00cbf18] do_syscall_64 at ffffffff86e3fec9 << crash-utility#11 [ffffb72bc00cbf50] entry_SYSCALL_64_after_hwframe at ffffffff870000aa Also, struct orc_entry in kernel has changed, and debugging information for ORC unwinder can be displayed incorrectly. To fix these, (1) introduce "kernel_orc_entry_6_4" structure corresponding to 6.4 and abstruct structure "orc_entry" in crash, (2) switch ORC_TYPE_CALL to 2 or 0 with kernel's orc_entry structure. Signed-off-by: Kazuhito Hagio <[email protected]>

Kernel commit fb799447ae29 ("x86,objtool: Split UNWIND_HINT_EMPTY in two"), which is contained in Linux 6.4 and later kernels, changed ORC_TYPE_CALL macro from 0 to 2. As a result, the "bt" command cannot use ORC entries, and can display stale entries in a call trace. crash> bt 1 PID: 1 TASK: ffff93cd06294180 CPU: 51 COMMAND: "systemd" #0 [ffffb72bc00cbc98] __schedule at ffffffff86e52aae #1 [ffffb72bc00cbd00] schedule at ffffffff86e52f6a #2 [ffffb72bc00cbd18] schedule_hrtimeout_range_clock at ffffffff86e58ef5 #3 [ffffb72bc00cbd88] ep_poll at ffffffff8669624d #4 [ffffb72bc00cbe28] do_epoll_wait at ffffffff86696371 #5 [ffffb72bc00cbe30] do_timerfd_settime at ffffffff8669902b << #6 [ffffb72bc00cbe60] __x64_sys_epoll_wait at ffffffff86696bf0 #7 [ffffb72bc00cbeb0] do_syscall_64 at ffffffff86e3feb9 #8 [ffffb72bc00cbee0] __task_pid_nr_ns at ffffffff863330d7 << #9 [ffffb72bc00cbf08] syscall_exit_to_user_mode at ffffffff86e466b2 << stale entries #10 [ffffb72bc00cbf18] do_syscall_64 at ffffffff86e3fec9 << #11 [ffffb72bc00cbf50] entry_SYSCALL_64_after_hwframe at ffffffff870000aa Also, kernel commit ffb1b4a41016 added a member to struct orc_entry. Although this does not affect the crash's unwinder, its debugging information can be displayed incorrectly. To fix these, (1) introduce "kernel_orc_entry_6_4" structure corresponding to 6.4 and abstruction layer "orc_entry" structure in crash, (2) switch ORC_TYPE_CALL to 2 or 0 with kernel's orc_entry structure. Related orc_entry history: v4.14 39358a033b2e introduced struct orc_entry v4.19 d31a580266ee added orc_entry.end member v6.3 ffb1b4a41016 added orc_entry.signal member v6.4 fb799447ae29 removed end member and changed type member to 3 bits Signed-off-by: Kazuhito Hagio <[email protected]>

…usly There is an issue that, for kernel modules, "dis -rl" fails to display modules code line number data after execute "bt" command in crash. Without the patch: crsah> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... #8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp 0xffffffffc0f60eb6 <lpfc_nlp_get+6>: push %rbx 0xffffffffc0f60eb7 <lpfc_nlp_get+7>: test %rdi,%rdi With the patch: crash> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... #8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6756 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6759 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp The root cause is, after kernel module been loaded by mod command, the symtable is not expanded in gdb side. crash bt or dis command will trigger such an expansion. However the symtable expansion is different for the 2 commands: The stack trace of "dis -rl" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... #1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... #2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... #3 0x000000000077e8e9 in process_full_comp_unit ... #4 process_queue ... #5 dw2_do_instantiate_symtab ... #6 0x000000000077ed67 in dw2_instantiate_symtab ... #7 0x000000000077f75e in dw2_expand_all_symtabs ... #8 0x00000000008f254d in gdb_get_line_number ... #9 0x00000000008f22af in gdb_command_funnel_1 ... #10 0x00000000008f2003 in gdb_command_funnel ... #11 0x00000000005b7f02 in gdb_interface ... #12 0x00000000005f5bd8 in get_line_number ... #13 0x000000000059e574 in cmd_dis ... The stack trace of "bt" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... #1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... #2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... #3 0x000000000077e8e9 in process_full_comp_unit ... #4 process_queue ... #5 dw2_do_instantiate_symtab ... #6 0x000000000077ed67 in dw2_instantiate_symtab ... #7 0x000000000077f8ed in dw2_lookup_symbol ... #8 0x00000000008e6d03 in lookup_symbol_via_quick_fns ... #9 0x00000000008e7153 in lookup_symbol_in_objfile ... #10 0x00000000008e73c6 in lookup_symbol_global_or_static_iterator_cb ... #11 0x00000000008b99c4 in svr4_iterate_over_objfiles_in_search_order ... #12 0x00000000008e754e in lookup_global_or_static_symbol ... #13 0x00000000008e75da in lookup_static_symbol ... #14 0x00000000008e632c in lookup_symbol_aux ... #15 0x00000000008e5a7a in lookup_symbol_in_language ... #16 0x00000000008e5b30 in lookup_symbol ... #17 0x00000000008f2a4a in gdb_get_datatype ... #18 0x00000000008f22c0 in gdb_command_funnel_1 ... #19 0x00000000008f2003 in gdb_command_funnel ... #20 0x00000000005b7f02 in gdb_interface ... #21 0x00000000005f8a9f in datatype_info ... #22 0x0000000000599947 in cpu_map_size ... #23 0x00000000005a975d in get_cpus_online ... #24 0x0000000000637a8b in diskdump_get_prstatus_percpu ... #25 0x000000000062f0e4 in get_netdump_regs_x86_64 ... #26 0x000000000059fe68 in back_trace ... #27 0x00000000005ab1cb in cmd_bt ... For the stacktrace of "dis -rl", it calls dw2_expand_all_symtabs() to expand all symtable of the objfile, or "*.ko.debug" in our case. However for the stacktrace of "bt", it doesn't expand all, but only a subset of symtable which is enough to find a symbol by dw2_lookup_symbol(). As a result, the objfile->compunit_symtabs, which is the head of a single linked list of struct compunit_symtab, is not NULL but didn't contain all symtables. It will not be reinitialized in gdb_get_line_number() by "dis -rl" because !objfile_has_full_symbols(objfile) check will fail, so it cannot display the proper code line number data. Since objfile_has_full_symbols(objfile) check cannot ensure all symbols been expanded, this patch add a new member as a flag for struct objfile to record if all symbols have been expanded. The flag will be set only ofter expand_all_symtabs been called. Signed-off-by: Tao Liu <[email protected]>

…usly There is an issue that, for kernel modules, "dis -rl" fails to display modules code line number data after execute "bt" command in crash. Without the patch: crsah> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp 0xffffffffc0f60eb6 <lpfc_nlp_get+6>: push %rbx 0xffffffffc0f60eb7 <lpfc_nlp_get+7>: test %rdi,%rdi With the patch: crash> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6756 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6759 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp The root cause is, after kernel module been loaded by mod command, the symtable is not expanded in gdb side. crash bt or dis command will trigger such an expansion. However the symtable expansion is different for the 2 commands: The stack trace of "dis -rl" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... crash-utility#1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... crash-utility#2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... crash-utility#3 0x000000000077e8e9 in process_full_comp_unit ... crash-utility#4 process_queue ... crash-utility#5 dw2_do_instantiate_symtab ... crash-utility#6 0x000000000077ed67 in dw2_instantiate_symtab ... crash-utility#7 0x000000000077f75e in dw2_expand_all_symtabs ... crash-utility#8 0x00000000008f254d in gdb_get_line_number ... crash-utility#9 0x00000000008f22af in gdb_command_funnel_1 ... crash-utility#10 0x00000000008f2003 in gdb_command_funnel ... crash-utility#11 0x00000000005b7f02 in gdb_interface ... crash-utility#12 0x00000000005f5bd8 in get_line_number ... crash-utility#13 0x000000000059e574 in cmd_dis ... The stack trace of "bt" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... crash-utility#1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... crash-utility#2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... crash-utility#3 0x000000000077e8e9 in process_full_comp_unit ... crash-utility#4 process_queue ... crash-utility#5 dw2_do_instantiate_symtab ... crash-utility#6 0x000000000077ed67 in dw2_instantiate_symtab ... crash-utility#7 0x000000000077f8ed in dw2_lookup_symbol ... crash-utility#8 0x00000000008e6d03 in lookup_symbol_via_quick_fns ... crash-utility#9 0x00000000008e7153 in lookup_symbol_in_objfile ... crash-utility#10 0x00000000008e73c6 in lookup_symbol_global_or_static_iterator_cb ... crash-utility#11 0x00000000008b99c4 in svr4_iterate_over_objfiles_in_search_order ... crash-utility#12 0x00000000008e754e in lookup_global_or_static_symbol ... crash-utility#13 0x00000000008e75da in lookup_static_symbol ... crash-utility#14 0x00000000008e632c in lookup_symbol_aux ... crash-utility#15 0x00000000008e5a7a in lookup_symbol_in_language ... crash-utility#16 0x00000000008e5b30 in lookup_symbol ... crash-utility#17 0x00000000008f2a4a in gdb_get_datatype ... crash-utility#18 0x00000000008f22c0 in gdb_command_funnel_1 ... crash-utility#19 0x00000000008f2003 in gdb_command_funnel ... crash-utility#20 0x00000000005b7f02 in gdb_interface ... crash-utility#21 0x00000000005f8a9f in datatype_info ... crash-utility#22 0x0000000000599947 in cpu_map_size ... crash-utility#23 0x00000000005a975d in get_cpus_online ... crash-utility#24 0x0000000000637a8b in diskdump_get_prstatus_percpu ... crash-utility#25 0x000000000062f0e4 in get_netdump_regs_x86_64 ... crash-utility#26 0x000000000059fe68 in back_trace ... crash-utility#27 0x00000000005ab1cb in cmd_bt ... For the stacktrace of "dis -rl", it calls dw2_expand_all_symtabs() to expand all symtable of the objfile, or "*.ko.debug" in our case. However for the stacktrace of "bt", it doesn't expand all, but only a subset of symtable which is enough to find a symbol by dw2_lookup_symbol(). As a result, the objfile->compunit_symtabs, which is the head of a single linked list of struct compunit_symtab, is not NULL but didn't contain all symtables. It will not be reinitialized in gdb_get_line_number() by "dis -rl" because !objfile_has_full_symbols(objfile) check will fail, so it cannot display the proper code line number data. Since objfile_has_full_symbols(objfile) check cannot ensure all symbols been expanded, this patch add a new member as a flag for struct objfile to record if all symbols have been expanded. The flag will be set only ofter expand_all_symtabs been called. Signed-off-by: Tao Liu <[email protected]>

…eviously There is an issue that, for kernel modules, "dis -rl" fails to display modules code line number data after execute "bt" command in crash. Without the patch: crsah> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp 0xffffffffc0f60eb6 <lpfc_nlp_get+6>: push %rbx 0xffffffffc0f60eb7 <lpfc_nlp_get+7>: test %rdi,%rdi With the patch: crash> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6756 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6759 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp The root cause is, after kernel module been loaded by mod command, the symtable is not expanded in gdb side. crash bt or dis command will trigger such an expansion. However the symtable expansion is different for the 2 commands: The stack trace of "dis -rl" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... crash-utility#1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... crash-utility#2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... crash-utility#3 0x000000000077e8e9 in process_full_comp_unit ... crash-utility#4 process_queue ... crash-utility#5 dw2_do_instantiate_symtab ... crash-utility#6 0x000000000077ed67 in dw2_instantiate_symtab ... crash-utility#7 0x000000000077f75e in dw2_expand_all_symtabs ... crash-utility#8 0x00000000008f254d in gdb_get_line_number ... crash-utility#9 0x00000000008f22af in gdb_command_funnel_1 ... crash-utility#10 0x00000000008f2003 in gdb_command_funnel ... crash-utility#11 0x00000000005b7f02 in gdb_interface ... crash-utility#12 0x00000000005f5bd8 in get_line_number ... crash-utility#13 0x000000000059e574 in cmd_dis ... The stack trace of "bt" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... crash-utility#1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... crash-utility#2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... crash-utility#3 0x000000000077e8e9 in process_full_comp_unit ... crash-utility#4 process_queue ... crash-utility#5 dw2_do_instantiate_symtab ... crash-utility#6 0x000000000077ed67 in dw2_instantiate_symtab ... crash-utility#7 0x000000000077f8ed in dw2_lookup_symbol ... crash-utility#8 0x00000000008e6d03 in lookup_symbol_via_quick_fns ... crash-utility#9 0x00000000008e7153 in lookup_symbol_in_objfile ... crash-utility#10 0x00000000008e73c6 in lookup_symbol_global_or_static_iterator_cb ... crash-utility#11 0x00000000008b99c4 in svr4_iterate_over_objfiles_in_search_order ... crash-utility#12 0x00000000008e754e in lookup_global_or_static_symbol ... crash-utility#13 0x00000000008e75da in lookup_static_symbol ... crash-utility#14 0x00000000008e632c in lookup_symbol_aux ... crash-utility#15 0x00000000008e5a7a in lookup_symbol_in_language ... crash-utility#16 0x00000000008e5b30 in lookup_symbol ... crash-utility#17 0x00000000008f2a4a in gdb_get_datatype ... crash-utility#18 0x00000000008f22c0 in gdb_command_funnel_1 ... crash-utility#19 0x00000000008f2003 in gdb_command_funnel ... crash-utility#20 0x00000000005b7f02 in gdb_interface ... crash-utility#21 0x00000000005f8a9f in datatype_info ... crash-utility#22 0x0000000000599947 in cpu_map_size ... crash-utility#23 0x00000000005a975d in get_cpus_online ... crash-utility#24 0x0000000000637a8b in diskdump_get_prstatus_percpu ... crash-utility#25 0x000000000062f0e4 in get_netdump_regs_x86_64 ... crash-utility#26 0x000000000059fe68 in back_trace ... crash-utility#27 0x00000000005ab1cb in cmd_bt ... For the stacktrace of "dis -rl", it calls dw2_expand_all_symtabs() to expand all symtable of the objfile, or "*.ko.debug" in our case. However for the stacktrace of "bt", it doesn't expand all, but only a subset of symtable which is enough to find a symbol by dw2_lookup_symbol(). As a result, the objfile->compunit_symtabs, which is the head of a single linked list of struct compunit_symtab, is not NULL but didn't contain all symtables. It will not be reinitialized in gdb_get_line_number() by "dis -rl" because !objfile_has_full_symbols(objfile) check will fail, so it cannot display the proper code line number data. Since objfile_has_full_symbols(objfile) check cannot ensure all symbols been expanded, this patch add a new member as a flag for struct objfile to record if all symbols have been expanded. The flag will be set only ofter expand_all_symtabs been called. Signed-off-by: Tao Liu <[email protected]>

This patch introduces per-cpu IRQ stacks for RISCV64 to let "bt" do backtrace on it and 'bt -E' search eframes on it, and the 'help -m' command displays the addresses of each per-cpu IRQ stack. TEST: a vmcore dumped via hacking the handle_irq_event_percpu() ( Why not using lkdtm INT_HW_IRQ_EN EXCEPTION ? There is a deadlock[1] in crash_kexec path if use that) ``` crash> bt PID: 0 TASK: ffffffff8140db00 CPU: 0 COMMAND: "swapper/0" #0 [ff20000000003e60] __handle_irq_event_percpu at ffffffff8006462e crash-utility#1 [ff20000000003ed0] handle_irq_event_percpu at ffffffff80064702 crash-utility#2 [ff20000000003ef0] handle_irq_event at ffffffff8006477c crash-utility#3 [ff20000000003f20] handle_fasteoi_irq at ffffffff80068664 crash-utility#4 [ff20000000003f50] generic_handle_domain_irq at ffffffff80063988 crash-utility#5 [ff20000000003f60] plic_handle_irq at ffffffff8046633e crash-utility#6 [ff20000000003fb0] generic_handle_domain_irq at ffffffff80063988 crash-utility#7 [ff20000000003fc0] riscv_intc_irq at ffffffff80465f8e crash-utility#8 [ff20000000003fd0] handle_riscv_irq at ffffffff808361e8 PC: ffffffff80837314 [default_idle_call+50] RA: ffffffff80837310 [default_idle_call+46] SP: ffffffff81403da0 CAUSE: 8000000000000009 epc : ffffffff80837314 ra : ffffffff80837310 sp : ffffffff81403da0 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : ff2000000004bb18 t1 : 0000000000032c73 t2 : ffffffff81200a48 s0 : ffffffff81403db0 s1 : 0000000000000000 a0 : 0000000000000004 a1 : 0000000000000000 a2 : ff6000009f1e7000 a3 : 0000000000002304 a4 : ffffffff80c1c2d8 a5 : 0000000000000000 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ffffffff814f0220 s3 : 0000000000000001 s4 : 000000000000003f s5 : ffffffff814f03d8 s6 : 0000000000000000 s7 : ffffffff814f00d0 s8 : ffffffff81526f10 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000009 orig_a0: ffffffff80837310 --- <IRQ stack> --- crash-utility#9 [ffffffff81403da0] default_idle_call at ffffffff80837314 crash-utility#10 [ffffffff81403db0] do_idle at ffffffff8004d0a0 crash-utility#11 [ffffffff81403e40] cpu_startup_entry at ffffffff8004d21e crash-utility#12 [ffffffff81403e60] kernel_init at ffffffff8083746a crash-utility#13 [ffffffff81403e70] arch_post_acpi_subsys_init at ffffffff80a006d8 crash-utility#14 [ffffffff81403e80] console_on_rootfs at ffffffff80a00c92 crash> crash> bt -E CPU 0 IRQ STACK: KERNEL-MODE EXCEPTION FRAME AT: ff20000000003a48 PC: ffffffff8006462e [__handle_irq_event_percpu+30] RA: ffffffff80064702 [handle_irq_event_percpu+18] SP: ff20000000003e60 CAUSE: 000000000000000d epc : ffffffff8006462e ra : ffffffff80064702 sp : ff20000000003e60 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : 0000000000046600 t1 : ffffffff80836464 t2 : ffffffff81200a48 s0 : ff20000000003ed0 s1 : 0000000000000000 a0 : 0000000000000000 a1 : 0000000000000118 a2 : 0000000000000052 a3 : 0000000000000000 a4 : 0000000000000000 a5 : 0000000000010001 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ff60000000941ab0 s3 : ffffffff814a0658 s4 : ff60000000089230 s5 : ffffffff814a0518 s6 : ffffffff814a0620 s7 : ffffffff80e5f0f8 s8 : ffffffff80fc50b0 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000100 badaddr: 0000000000000078 cause: 000000000000000d orig_a0: ff20000000003ea0 CPU 1 IRQ STACK:(none found) crash> crash> help -m <snip> machspec: ced1e0 irq_stack_size: 16384 irq_stacks[0]: ff20000000000000 irq_stacks[1]: ff20000000008000 crash> ``` [1]: https://lore.kernel.org/linux-riscv/[email protected]/ Signed-off-by: Song Shuai <[email protected]>

This patch introduces per-cpu IRQ stacks for RISCV64 to let "bt" do backtrace on it and 'bt -E' search eframes on it, and the 'help -m' command displays the addresses of each per-cpu IRQ stack. TEST: a vmcore dumped via hacking the handle_irq_event_percpu() ( Why not using lkdtm INT_HW_IRQ_EN EXCEPTION ? There is a deadlock[1] in crash_kexec path if use that) crash> bt PID: 0 TASK: ffffffff8140db00 CPU: 0 COMMAND: "swapper/0" #0 [ff20000000003e60] __handle_irq_event_percpu at ffffffff8006462e #1 [ff20000000003ed0] handle_irq_event_percpu at ffffffff80064702 #2 [ff20000000003ef0] handle_irq_event at ffffffff8006477c #3 [ff20000000003f20] handle_fasteoi_irq at ffffffff80068664 #4 [ff20000000003f50] generic_handle_domain_irq at ffffffff80063988 #5 [ff20000000003f60] plic_handle_irq at ffffffff8046633e #6 [ff20000000003fb0] generic_handle_domain_irq at ffffffff80063988 #7 [ff20000000003fc0] riscv_intc_irq at ffffffff80465f8e #8 [ff20000000003fd0] handle_riscv_irq at ffffffff808361e8 PC: ffffffff80837314 [default_idle_call+50] RA: ffffffff80837310 [default_idle_call+46] SP: ffffffff81403da0 CAUSE: 8000000000000009 epc : ffffffff80837314 ra : ffffffff80837310 sp : ffffffff81403da0 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : ff2000000004bb18 t1 : 0000000000032c73 t2 : ffffffff81200a48 s0 : ffffffff81403db0 s1 : 0000000000000000 a0 : 0000000000000004 a1 : 0000000000000000 a2 : ff6000009f1e7000 a3 : 0000000000002304 a4 : ffffffff80c1c2d8 a5 : 0000000000000000 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ffffffff814f0220 s3 : 0000000000000001 s4 : 000000000000003f s5 : ffffffff814f03d8 s6 : 0000000000000000 s7 : ffffffff814f00d0 s8 : ffffffff81526f10 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000009 orig_a0: ffffffff80837310 --- <IRQ stack> --- #9 [ffffffff81403da0] default_idle_call at ffffffff80837314 #10 [ffffffff81403db0] do_idle at ffffffff8004d0a0 #11 [ffffffff81403e40] cpu_startup_entry at ffffffff8004d21e #12 [ffffffff81403e60] kernel_init at ffffffff8083746a #13 [ffffffff81403e70] arch_post_acpi_subsys_init at ffffffff80a006d8 #14 [ffffffff81403e80] console_on_rootfs at ffffffff80a00c92 crash> crash> bt -E CPU 0 IRQ STACK: KERNEL-MODE EXCEPTION FRAME AT: ff20000000003a48 PC: ffffffff8006462e [__handle_irq_event_percpu+30] RA: ffffffff80064702 [handle_irq_event_percpu+18] SP: ff20000000003e60 CAUSE: 000000000000000d epc : ffffffff8006462e ra : ffffffff80064702 sp : ff20000000003e60 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : 0000000000046600 t1 : ffffffff80836464 t2 : ffffffff81200a48 s0 : ff20000000003ed0 s1 : 0000000000000000 a0 : 0000000000000000 a1 : 0000000000000118 a2 : 0000000000000052 a3 : 0000000000000000 a4 : 0000000000000000 a5 : 0000000000010001 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ff60000000941ab0 s3 : ffffffff814a0658 s4 : ff60000000089230 s5 : ffffffff814a0518 s6 : ffffffff814a0620 s7 : ffffffff80e5f0f8 s8 : ffffffff80fc50b0 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000100 badaddr: 0000000000000078 cause: 000000000000000d orig_a0: ff20000000003ea0 CPU 1 IRQ STACK: (none found) crash> crash> help -m <snip> machspec: ced1e0 irq_stack_size: 16384 irq_stacks[0]: ff20000000000000 irq_stacks[1]: ff20000000008000 crash> [1]: https://lore.kernel.org/linux-riscv/[email protected]/ Signed-off-by: Song Shuai <[email protected]>

…ss range Previously, to find a module symbol and its offset by an arbitrary address, all symbols within the module will be iterated by address ascending order until the last symbol with a smaller address been noticed. However if the address is not within the module address range, e.g. the address is higher than the module's last symbol's address, then the module can be surely skipped, because its symbol iteration is unnecessary. This can speed up the kernel module symbols finding and improve the overall performance. Without the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 #1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 #2 [ffff927569cd7768] __mutex_lock_slowpath at ffffffffb3db6ca7 #3 [ffff927569cd77c0] mutex_lock at ffffffffb3db602f #4 [ffff927569cd77d8] ucache_retrieve at ffffffffc0cf4409 [secfs2] ...snip the stacktrace of the same module... #11 [ffff927569cd7ba0] cskal_path_vfs_getattr_nosec at ffffffffc05cae76 [falcon_kal] ...snip... #13 [ffff927569cd7c40] _ZdlPv at ffffffffc086e751 [falcon_lsm_serviceable] ...snip... #20 [ffff927569cd7ef8] unload_network_ops_symbols at ffffffffc06f11c0 [falcon_lsm_pinned_14713] #21 [ffff927569cd7f50] system_call_fastpath at ffffffffb3dc539a RIP: 00007f2b28ed4023 RSP: 00007f2a45fe7f80 RFLAGS: 00000206 RAX: 0000000000000012 RBX: 00007f2a68302e00 RCX: 00007f2a682546d8 RDX: 0000000000000826 RSI: 00007eb57ea6a000 RDI: 00000000000000e3 RBP: 00007eb57ea6a000 R8: 0000000000000826 R9: 00000002670bdfd2 R10: 00000002670bdfd2 R11: 0000000000000293 R12: 00000002670bdfd2 R13: 00007f29d501a480 R14: 0000000000000826 R15: 00000002670bdfd2 ORIG_RAX: 0000000000000012 CS: 0033 SS: 002b crash> real 7m14.826s user 7m12.502s sys 0m1.091s With the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 #1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 ...snip the same output... crash> real 0m8.827s user 0m7.896s sys 0m0.938s Signed-off-by: Tao Liu <[email protected]>

…ss range Previously, to find a module symbol and its offset by an arbitrary address, all symbols within the module will be iterated by address ascending order until the last symbol with a smaller address been noticed. However if the address is not within the module address range, e.g. the address is higher than the module's last symbol's address, then the module can be surely skipped, because its symbol iteration is unnecessary. This can speed up the kernel module symbols finding and improve the overall performance. Without the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 crash-utility#1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 crash-utility#2 [ffff927569cd7768] __mutex_lock_slowpath at ffffffffb3db6ca7 crash-utility#3 [ffff927569cd77c0] mutex_lock at ffffffffb3db602f crash-utility#4 [ffff927569cd77d8] ucache_retrieve at ffffffffc0cf4409 [secfs2] ...snip the stacktrace of the same module... crash-utility#11 [ffff927569cd7ba0] cskal_path_vfs_getattr_nosec at ffffffffc05cae76 [falcon_kal] ...snip... crash-utility#13 [ffff927569cd7c40] _ZdlPv at ffffffffc086e751 [falcon_lsm_serviceable] ...snip... crash-utility#20 [ffff927569cd7ef8] unload_network_ops_symbols at ffffffffc06f11c0 [falcon_lsm_pinned_14713] crash-utility#21 [ffff927569cd7f50] system_call_fastpath at ffffffffb3dc539a RIP: 00007f2b28ed4023 RSP: 00007f2a45fe7f80 RFLAGS: 00000206 RAX: 0000000000000012 RBX: 00007f2a68302e00 RCX: 00007f2a682546d8 RDX: 0000000000000826 RSI: 00007eb57ea6a000 RDI: 00000000000000e3 RBP: 00007eb57ea6a000 R8: 0000000000000826 R9: 00000002670bdfd2 R10: 00000002670bdfd2 R11: 0000000000000293 R12: 00000002670bdfd2 R13: 00007f29d501a480 R14: 0000000000000826 R15: 00000002670bdfd2 ORIG_RAX: 0000000000000012 CS: 0033 SS: 002b crash> real 7m14.826s user 7m12.502s sys 0m1.091s With the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 crash-utility#1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 ...snip the same output... crash> real 0m8.827s user 0m7.896s sys 0m0.938s Signed-off-by: Tao Liu <[email protected]>

This patch introduces per-cpu IRQ stacks for RISCV64 to let "bt" do backtrace on it and 'bt -E' search eframes on it, and the 'help -m' command displays the addresses of each per-cpu IRQ stack. TEST: a vmcore dumped via hacking the handle_irq_event_percpu() ( Why not using lkdtm INT_HW_IRQ_EN EXCEPTION ? There is a deadlock[1] in crash_kexec path if use that) crash> bt PID: 0 TASK: ffffffff8140db00 CPU: 0 COMMAND: "swapper/0" #0 [ff20000000003e60] __handle_irq_event_percpu at ffffffff8006462e crash-utility#1 [ff20000000003ed0] handle_irq_event_percpu at ffffffff80064702 crash-utility#2 [ff20000000003ef0] handle_irq_event at ffffffff8006477c crash-utility#3 [ff20000000003f20] handle_fasteoi_irq at ffffffff80068664 crash-utility#4 [ff20000000003f50] generic_handle_domain_irq at ffffffff80063988 crash-utility#5 [ff20000000003f60] plic_handle_irq at ffffffff8046633e crash-utility#6 [ff20000000003fb0] generic_handle_domain_irq at ffffffff80063988 crash-utility#7 [ff20000000003fc0] riscv_intc_irq at ffffffff80465f8e crash-utility#8 [ff20000000003fd0] handle_riscv_irq at ffffffff808361e8 PC: ffffffff80837314 [default_idle_call+50] RA: ffffffff80837310 [default_idle_call+46] SP: ffffffff81403da0 CAUSE: 8000000000000009 epc : ffffffff80837314 ra : ffffffff80837310 sp : ffffffff81403da0 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : ff2000000004bb18 t1 : 0000000000032c73 t2 : ffffffff81200a48 s0 : ffffffff81403db0 s1 : 0000000000000000 a0 : 0000000000000004 a1 : 0000000000000000 a2 : ff6000009f1e7000 a3 : 0000000000002304 a4 : ffffffff80c1c2d8 a5 : 0000000000000000 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ffffffff814f0220 s3 : 0000000000000001 s4 : 000000000000003f s5 : ffffffff814f03d8 s6 : 0000000000000000 s7 : ffffffff814f00d0 s8 : ffffffff81526f10 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000009 orig_a0: ffffffff80837310 --- <IRQ stack> --- crash-utility#9 [ffffffff81403da0] default_idle_call at ffffffff80837314 crash-utility#10 [ffffffff81403db0] do_idle at ffffffff8004d0a0 crash-utility#11 [ffffffff81403e40] cpu_startup_entry at ffffffff8004d21e crash-utility#12 [ffffffff81403e60] kernel_init at ffffffff8083746a crash-utility#13 [ffffffff81403e70] arch_post_acpi_subsys_init at ffffffff80a006d8 crash-utility#14 [ffffffff81403e80] console_on_rootfs at ffffffff80a00c92 crash> crash> bt -E CPU 0 IRQ STACK: KERNEL-MODE EXCEPTION FRAME AT: ff20000000003a48 PC: ffffffff8006462e [__handle_irq_event_percpu+30] RA: ffffffff80064702 [handle_irq_event_percpu+18] SP: ff20000000003e60 CAUSE: 000000000000000d epc : ffffffff8006462e ra : ffffffff80064702 sp : ff20000000003e60 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : 0000000000046600 t1 : ffffffff80836464 t2 : ffffffff81200a48 s0 : ff20000000003ed0 s1 : 0000000000000000 a0 : 0000000000000000 a1 : 0000000000000118 a2 : 0000000000000052 a3 : 0000000000000000 a4 : 0000000000000000 a5 : 0000000000010001 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ff60000000941ab0 s3 : ffffffff814a0658 s4 : ff60000000089230 s5 : ffffffff814a0518 s6 : ffffffff814a0620 s7 : ffffffff80e5f0f8 s8 : ffffffff80fc50b0 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000100 badaddr: 0000000000000078 cause: 000000000000000d orig_a0: ff20000000003ea0 CPU 1 IRQ STACK: (none found) crash> crash> help -m <snip> machspec: ced1e0 irq_stack_size: 16384 irq_stacks[0]: ff20000000000000 irq_stacks[1]: ff20000000008000 crash> [1]: https://lore.kernel.org/linux-riscv/[email protected]/ Signed-off-by: Song Shuai <[email protected]>

…ss range Previously, to find a module symbol and its offset by an arbitrary address, all symbols within the module will be iterated by address ascending order until the last symbol with a smaller address been noticed. However if the address is not within the module address range, e.g. the address is higher than the module's last symbol's address, then the module can be surely skipped, because its symbol iteration is unnecessary. This can speed up the kernel module symbols finding and improve the overall performance. Without the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 crash-utility#1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 crash-utility#2 [ffff927569cd7768] __mutex_lock_slowpath at ffffffffb3db6ca7 crash-utility#3 [ffff927569cd77c0] mutex_lock at ffffffffb3db602f crash-utility#4 [ffff927569cd77d8] ucache_retrieve at ffffffffc0cf4409 [secfs2] ...snip the stacktrace of the same module... crash-utility#11 [ffff927569cd7ba0] cskal_path_vfs_getattr_nosec at ffffffffc05cae76 [falcon_kal] ...snip... crash-utility#13 [ffff927569cd7c40] _ZdlPv at ffffffffc086e751 [falcon_lsm_serviceable] ...snip... crash-utility#20 [ffff927569cd7ef8] unload_network_ops_symbols at ffffffffc06f11c0 [falcon_lsm_pinned_14713] crash-utility#21 [ffff927569cd7f50] system_call_fastpath at ffffffffb3dc539a RIP: 00007f2b28ed4023 RSP: 00007f2a45fe7f80 RFLAGS: 00000206 RAX: 0000000000000012 RBX: 00007f2a68302e00 RCX: 00007f2a682546d8 RDX: 0000000000000826 RSI: 00007eb57ea6a000 RDI: 00000000000000e3 RBP: 00007eb57ea6a000 R8: 0000000000000826 R9: 00000002670bdfd2 R10: 00000002670bdfd2 R11: 0000000000000293 R12: 00000002670bdfd2 R13: 00007f29d501a480 R14: 0000000000000826 R15: 00000002670bdfd2 ORIG_RAX: 0000000000000012 CS: 0033 SS: 002b crash> real 7m14.826s user 7m12.502s sys 0m1.091s With the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 crash-utility#1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 ...snip the same output... crash> real 0m8.827s user 0m7.896s sys 0m0.938s Signed-off-by: Tao Liu <[email protected]>

The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao(a)redhat.com>

The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <[email protected]>

The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Signed-off-by: Tao Liu <[email protected]>

There may be extra "=>" prefix before gdb disassembly, as a result, parse_line() will return string "=>" as arglist[0], which will be converted to number by htol() and fails. E.g.: crash> gdb x/40i __list_del_entry ... 0xffffffff8133c384 <__list_del_entry+36>: cmp %rcx,%rax 0xffffffff8133c387 <__list_del_entry+39>: je 0xffffffff8133c403 <__list_del_entry+163> => 0xffffffff8133c389 <__list_del_entry+41>: mov (%rax),%r8 0xffffffff8133c38c <__list_del_entry+44>: cmp %r8,%rdi 0xffffffff8133c38f <__list_del_entry+47>: jne 0xffffffff8133c3e4 <__list_del_entry+132> 0xffffffff8133c391 <__list_del_entry+49>: mov 0x8(%rdx),%r8 Before the patch: crash> bt ... crash-utility#10 [ffff880095647c00] async_page_fault at ffffffff816a8638 [exception RIP: __list_del_entry+41] RIP: ffffffff8133c389 RSP: ffff880095647cb0 RFLAGS: 00010207 RAX: 0000000000000000 RBX: ffffea0400408020 RCX: dead000000000200 RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffffea0400408020 RBP: ffff880095647cb0 R8: 0000000080000431 R9: ffffffff81e835c0 R10: 0000000000000000 R11: 0000000000000400 R12: ffff880138795b58 R13: 0000000010010201 R14: ffff880095647d70 R15: 0000000400408040 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 bt: invalid input: "=>" crash-utility#11 [ffff880095647cb8] list_del at ffffffff8133c43d crash-utility#12 [ffff880095647cd0] devm_memremap_pages at ffffffff81180c53 After the patch: No string as 'bt: invalid input: "=>"' of output. Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Signed-off-by: Tao Liu <[email protected]>

The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Signed-off-by: Tao Liu <[email protected]>

There may be extra "=>" prefix before gdb disassembly, as a result, parse_line() will return string "=>" as arglist[0], which will be converted to number by htol() and fails. E.g.: crash> gdb x/40i __list_del_entry ... 0xffffffff8133c384 <__list_del_entry+36>: cmp %rcx,%rax 0xffffffff8133c387 <__list_del_entry+39>: je 0xffffffff8133c403 <__list_del_entry+163> => 0xffffffff8133c389 <__list_del_entry+41>: mov (%rax),%r8 0xffffffff8133c38c <__list_del_entry+44>: cmp %r8,%rdi 0xffffffff8133c38f <__list_del_entry+47>: jne 0xffffffff8133c3e4 <__list_del_entry+132> 0xffffffff8133c391 <__list_del_entry+49>: mov 0x8(%rdx),%r8 Before the patch: crash> bt ... crash-utility#10 [ffff880095647c00] async_page_fault at ffffffff816a8638 [exception RIP: __list_del_entry+41] RIP: ffffffff8133c389 RSP: ffff880095647cb0 RFLAGS: 00010207 RAX: 0000000000000000 RBX: ffffea0400408020 RCX: dead000000000200 RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffffea0400408020 RBP: ffff880095647cb0 R8: 0000000080000431 R9: ffffffff81e835c0 R10: 0000000000000000 R11: 0000000000000400 R12: ffff880138795b58 R13: 0000000010010201 R14: ffff880095647d70 R15: 0000000400408040 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 bt: invalid input: "=>" crash-utility#11 [ffff880095647cb8] list_del at ffffffff8133c43d crash-utility#12 [ffff880095647cd0] devm_memremap_pages at ffffffff81180c53 After the patch: No string as 'bt: invalid input: "=>"' of output. Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Signed-off-by: Tao Liu <[email protected]>

The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Signed-off-by: Tao Liu <[email protected]>

There may be extra "=>" prefix before gdb disassembly, as a result, parse_line() will return string "=>" as arglist[0], which will be converted to number by htol() and fails. E.g.: crash> gdb x/40i __list_del_entry ... 0xffffffff8133c384 <__list_del_entry+36>: cmp %rcx,%rax 0xffffffff8133c387 <__list_del_entry+39>: je 0xffffffff8133c403 <__list_del_entry+163> => 0xffffffff8133c389 <__list_del_entry+41>: mov (%rax),%r8 0xffffffff8133c38c <__list_del_entry+44>: cmp %r8,%rdi 0xffffffff8133c38f <__list_del_entry+47>: jne 0xffffffff8133c3e4 <__list_del_entry+132> 0xffffffff8133c391 <__list_del_entry+49>: mov 0x8(%rdx),%r8 Before the patch: crash> bt ... crash-utility#10 [ffff880095647c00] async_page_fault at ffffffff816a8638 [exception RIP: __list_del_entry+41] RIP: ffffffff8133c389 RSP: ffff880095647cb0 RFLAGS: 00010207 RAX: 0000000000000000 RBX: ffffea0400408020 RCX: dead000000000200 RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffffea0400408020 RBP: ffff880095647cb0 R8: 0000000080000431 R9: ffffffff81e835c0 R10: 0000000000000000 R11: 0000000000000400 R12: ffff880138795b58 R13: 0000000010010201 R14: ffff880095647d70 R15: 0000000400408040 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 bt: invalid input: "=>" crash-utility#11 [ffff880095647cb8] list_del at ffffffff8133c43d crash-utility#12 [ffff880095647cd0] devm_memremap_pages at ffffffff81180c53 After the patch: No string as 'bt: invalid input: "=>"' of output. Cc: Sourabh Jain <[email protected]> Cc: Hari Bathini <[email protected]> Cc: Mahesh J Salgaonkar <[email protected]> Cc: Naveen N. Rao <[email protected]> Cc: Lianbo Jiang <[email protected]> Cc: HAGIO KAZUHITO(萩尾一仁) <[email protected]> Cc: Tao Liu <[email protected]> Cc: Alexey Makhalov <[email protected]> Signed-off-by: Tao Liu <[email protected]>

Previously, "retq" is used to determine the end of a function, so the end of framesize calculation. However "ret" might be outputted by gdb rather than "retq", as a result, the framesize is returned incorrectly, and bogus stack trace will be outputted. Without the patch: $ crash -d 3 vmcore vmlinux crash> bt 0xffffffff92da7545 <copy_process+5>: push %rbp [framesize: 8] ... 0xffffffff92da7561 <copy_process+33>: sub $0x238,%rsp [framesize: 624] ... 0xffffffff92da776a <copy_process+554>: pop %r15 [framesize: 8] 0xffffffff92da776c <copy_process+556>: pop %rbp [framesize: 0] 0xffffffff92da776d <copy_process+557>: ret crash> bt -D dump framesize_cache_entries: ... [ 3]: ffffffff92dadcbd 0 CF (copy_process+26493) crash> bt ... #9 [ffff888263157bc0] copy_process at ffffffff92dadcbd #10 [ffff888263157d20] __mutex_init at ffffffff92ed8dd5 #11 [ffff888263157d38] __alloc_file at ffffffff93458397 #12 [ffff888263157d60] alloc_empty_file at ffffffff934585d2 #13 [ffff888263157da8] __alloc_fd at ffffffff934b5ead #14 [ffff888263157e38] _do_fork at ffffffff92dae7a1 #15 [ffff888263157f28] do_syscall_64 at ffffffff92c085f4 Stack #10 ~ #13 are bogus and misleading. With the patch: ... 0xffffffff92da776d <copy_process+557>: ret [framesize restored to: 624] crash> bt -D dump ... [ 3]: ffffffff92dadcbd 624 CF (copy_process+26493) crash> bt ... #9 [ffff888263157bc0] copy_process at ffffffff92dadcbd #10 [ffff888263157e38] _do_fork at ffffffff92dae7a1 #11 [ffff888263157f28] do_syscall_64 at ffffffff92c085f4 Signed-off-by: Tao Liu <[email protected]>

There may be extra "=>" prefix before gdb disassembly, as a result, parse_line() will return string "=>" as arglist[0], which will be converted to number by htol() and fails. E.g.: crash> gdb x/40i __list_del_entry ... 0xffffffff8133c384 <__list_del_entry+36>: cmp %rcx,%rax 0xffffffff8133c387 <__list_del_entry+39>: je 0xffffffff8133c403 <__list_del_entry+163> => 0xffffffff8133c389 <__list_del_entry+41>: mov (%rax),%r8 0xffffffff8133c38c <__list_del_entry+44>: cmp %r8,%rdi 0xffffffff8133c38f <__list_del_entry+47>: jne 0xffffffff8133c3e4 <__list_del_entry+132> 0xffffffff8133c391 <__list_del_entry+49>: mov 0x8(%rdx),%r8 Before the patch: crash> bt ... #10 [ffff880095647c00] async_page_fault at ffffffff816a8638 [exception RIP: __list_del_entry+41] RIP: ffffffff8133c389 RSP: ffff880095647cb0 RFLAGS: 00010207 RAX: 0000000000000000 RBX: ffffea0400408020 RCX: dead000000000200 RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffffea0400408020 RBP: ffff880095647cb0 R8: 0000000080000431 R9: ffffffff81e835c0 R10: 0000000000000000 R11: 0000000000000400 R12: ffff880138795b58 R13: 0000000010010201 R14: ffff880095647d70 R15: 0000000400408040 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 bt: invalid input: "=>" #11 [ffff880095647cb8] list_del at ffffffff8133c43d #12 [ffff880095647cd0] devm_memremap_pages at ffffffff81180c53 After the patch: No string as 'bt: invalid input: "=>"' of output. Signed-off-by: Tao Liu <[email protected]>

…usly There is an issue that, for kernel modules, "dis -rl" fails to display modules code line number data after execute "bt" command in crash. Without the patch: crsah> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp 0xffffffffc0f60eb6 <lpfc_nlp_get+6>: push %rbx 0xffffffffc0f60eb7 <lpfc_nlp_get+7>: test %rdi,%rdi With the patch: crash> mod -S crash> bt PID: 1500 TASK: ff2bd8b093524000 CPU: 16 COMMAND: "lpfc_worker_0" #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3 ...snip... crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc] ...snip... crash> dis -rl ffffffffc0f60f82 /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6756 0xffffffffc0f60eb0 <lpfc_nlp_get>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6759 0xffffffffc0f60eb5 <lpfc_nlp_get+5>: push %rbp The root cause is, after kernel module been loaded by mod command, the symtable is not expanded in gdb side. crash bt or dis command will trigger such an expansion. However the symtable expansion is different for the 2 commands: The stack trace of "dis -rl" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... crash-utility#1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... crash-utility#2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... crash-utility#3 0x000000000077e8e9 in process_full_comp_unit ... crash-utility#4 process_queue ... crash-utility#5 dw2_do_instantiate_symtab ... crash-utility#6 0x000000000077ed67 in dw2_instantiate_symtab ... crash-utility#7 0x000000000077f75e in dw2_expand_all_symtabs ... crash-utility#8 0x00000000008f254d in gdb_get_line_number ... crash-utility#9 0x00000000008f22af in gdb_command_funnel_1 ... crash-utility#10 0x00000000008f2003 in gdb_command_funnel ... crash-utility#11 0x00000000005b7f02 in gdb_interface ... crash-utility#12 0x00000000005f5bd8 in get_line_number ... crash-utility#13 0x000000000059e574 in cmd_dis ... The stack trace of "bt" for symtable expanding: #0 0x00000000008d8d9f in add_compunit_symtab_to_objfile ... crash-utility#1 0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ... crash-utility#2 0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ... crash-utility#3 0x000000000077e8e9 in process_full_comp_unit ... crash-utility#4 process_queue ... crash-utility#5 dw2_do_instantiate_symtab ... crash-utility#6 0x000000000077ed67 in dw2_instantiate_symtab ... crash-utility#7 0x000000000077f8ed in dw2_lookup_symbol ... crash-utility#8 0x00000000008e6d03 in lookup_symbol_via_quick_fns ... crash-utility#9 0x00000000008e7153 in lookup_symbol_in_objfile ... crash-utility#10 0x00000000008e73c6 in lookup_symbol_global_or_static_iterator_cb ... crash-utility#11 0x00000000008b99c4 in svr4_iterate_over_objfiles_in_search_order ... crash-utility#12 0x00000000008e754e in lookup_global_or_static_symbol ... crash-utility#13 0x00000000008e75da in lookup_static_symbol ... crash-utility#14 0x00000000008e632c in lookup_symbol_aux ... crash-utility#15 0x00000000008e5a7a in lookup_symbol_in_language ... crash-utility#16 0x00000000008e5b30 in lookup_symbol ... crash-utility#17 0x00000000008f2a4a in gdb_get_datatype ... crash-utility#18 0x00000000008f22c0 in gdb_command_funnel_1 ... crash-utility#19 0x00000000008f2003 in gdb_command_funnel ... crash-utility#20 0x00000000005b7f02 in gdb_interface ... crash-utility#21 0x00000000005f8a9f in datatype_info ... crash-utility#22 0x0000000000599947 in cpu_map_size ... crash-utility#23 0x00000000005a975d in get_cpus_online ... crash-utility#24 0x0000000000637a8b in diskdump_get_prstatus_percpu ... crash-utility#25 0x000000000062f0e4 in get_netdump_regs_x86_64 ... crash-utility#26 0x000000000059fe68 in back_trace ... crash-utility#27 0x00000000005ab1cb in cmd_bt ... For the stacktrace of "dis -rl", it calls dw2_expand_all_symtabs() to expand all symtable of the objfile, or "*.ko.debug" in our case. However for the stacktrace of "bt", it doesn't expand all, but only a subset of symtable which is enough to find a symbol by dw2_lookup_symbol(). As a result, the objfile->compunit_symtabs, which is the head of a single linked list of struct compunit_symtab, is not NULL but didn't contain all symtables. It will not be reinitialized in gdb_get_line_number() by "dis -rl" because !objfile_has_full_symbols(objfile) check will fail, so it cannot display the proper code line number data. Since objfile_has_full_symbols(objfile) check cannot ensure all symbols been expanded, this patch add a new member as a flag for struct objfile to record if all symbols have been expanded. The flag will be set only ofter expand_all_symtabs been called. Signed-off-by: Tao Liu <[email protected]>

This patch introduces per-cpu IRQ stacks for RISCV64 to let "bt" do backtrace on it and 'bt -E' search eframes on it, and the 'help -m' command displays the addresses of each per-cpu IRQ stack. TEST: a vmcore dumped via hacking the handle_irq_event_percpu() ( Why not using lkdtm INT_HW_IRQ_EN EXCEPTION ? There is a deadlock[1] in crash_kexec path if use that) crash> bt PID: 0 TASK: ffffffff8140db00 CPU: 0 COMMAND: "swapper/0" #0 [ff20000000003e60] __handle_irq_event_percpu at ffffffff8006462e crash-utility#1 [ff20000000003ed0] handle_irq_event_percpu at ffffffff80064702 crash-utility#2 [ff20000000003ef0] handle_irq_event at ffffffff8006477c crash-utility#3 [ff20000000003f20] handle_fasteoi_irq at ffffffff80068664 crash-utility#4 [ff20000000003f50] generic_handle_domain_irq at ffffffff80063988 crash-utility#5 [ff20000000003f60] plic_handle_irq at ffffffff8046633e crash-utility#6 [ff20000000003fb0] generic_handle_domain_irq at ffffffff80063988 crash-utility#7 [ff20000000003fc0] riscv_intc_irq at ffffffff80465f8e crash-utility#8 [ff20000000003fd0] handle_riscv_irq at ffffffff808361e8 PC: ffffffff80837314 [default_idle_call+50] RA: ffffffff80837310 [default_idle_call+46] SP: ffffffff81403da0 CAUSE: 8000000000000009 epc : ffffffff80837314 ra : ffffffff80837310 sp : ffffffff81403da0 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : ff2000000004bb18 t1 : 0000000000032c73 t2 : ffffffff81200a48 s0 : ffffffff81403db0 s1 : 0000000000000000 a0 : 0000000000000004 a1 : 0000000000000000 a2 : ff6000009f1e7000 a3 : 0000000000002304 a4 : ffffffff80c1c2d8 a5 : 0000000000000000 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ffffffff814f0220 s3 : 0000000000000001 s4 : 000000000000003f s5 : ffffffff814f03d8 s6 : 0000000000000000 s7 : ffffffff814f00d0 s8 : ffffffff81526f10 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000009 orig_a0: ffffffff80837310 --- <IRQ stack> --- crash-utility#9 [ffffffff81403da0] default_idle_call at ffffffff80837314 crash-utility#10 [ffffffff81403db0] do_idle at ffffffff8004d0a0 crash-utility#11 [ffffffff81403e40] cpu_startup_entry at ffffffff8004d21e crash-utility#12 [ffffffff81403e60] kernel_init at ffffffff8083746a crash-utility#13 [ffffffff81403e70] arch_post_acpi_subsys_init at ffffffff80a006d8 crash-utility#14 [ffffffff81403e80] console_on_rootfs at ffffffff80a00c92 crash> crash> bt -E CPU 0 IRQ STACK: KERNEL-MODE EXCEPTION FRAME AT: ff20000000003a48 PC: ffffffff8006462e [__handle_irq_event_percpu+30] RA: ffffffff80064702 [handle_irq_event_percpu+18] SP: ff20000000003e60 CAUSE: 000000000000000d epc : ffffffff8006462e ra : ffffffff80064702 sp : ff20000000003e60 gp : ffffffff814ef848 tp : ffffffff8140db00 t0 : 0000000000046600 t1 : ffffffff80836464 t2 : ffffffff81200a48 s0 : ff20000000003ed0 s1 : 0000000000000000 a0 : 0000000000000000 a1 : 0000000000000118 a2 : 0000000000000052 a3 : 0000000000000000 a4 : 0000000000000000 a5 : 0000000000010001 a6 : ff6000001fe01958 a7 : 00002496ea89dbf1 s2 : ff60000000941ab0 s3 : ffffffff814a0658 s4 : ff60000000089230 s5 : ffffffff814a0518 s6 : ffffffff814a0620 s7 : ffffffff80e5f0f8 s8 : ffffffff80fc50b0 s9 : ffffffff80c1d880 s10: 0000000000000000 s11: 0000000000000001 t3 : 0000000000003392 t4 : 0000000000000000 t5 : 0000000000000000 t6 : 0000000000000040 status: 0000000200000100 badaddr: 0000000000000078 cause: 000000000000000d orig_a0: ff20000000003ea0 CPU 1 IRQ STACK: (none found) crash> crash> help -m <snip> machspec: ced1e0 irq_stack_size: 16384 irq_stacks[0]: ff20000000000000 irq_stacks[1]: ff20000000008000 crash> [1]: https://lore.kernel.org/linux-riscv/[email protected]/ Signed-off-by: Song Shuai <[email protected]>

…ss range Previously, to find a module symbol and its offset by an arbitrary address, all symbols within the module will be iterated by address ascending order until the last symbol with a smaller address been noticed. However if the address is not within the module address range, e.g. the address is higher than the module's last symbol's address, then the module can be surely skipped, because its symbol iteration is unnecessary. This can speed up the kernel module symbols finding and improve the overall performance. Without the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 crash-utility#1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 crash-utility#2 [ffff927569cd7768] __mutex_lock_slowpath at ffffffffb3db6ca7 crash-utility#3 [ffff927569cd77c0] mutex_lock at ffffffffb3db602f crash-utility#4 [ffff927569cd77d8] ucache_retrieve at ffffffffc0cf4409 [secfs2] ...snip the stacktrace of the same module... crash-utility#11 [ffff927569cd7ba0] cskal_path_vfs_getattr_nosec at ffffffffc05cae76 [falcon_kal] ...snip... crash-utility#13 [ffff927569cd7c40] _ZdlPv at ffffffffc086e751 [falcon_lsm_serviceable] ...snip... crash-utility#20 [ffff927569cd7ef8] unload_network_ops_symbols at ffffffffc06f11c0 [falcon_lsm_pinned_14713] crash-utility#21 [ffff927569cd7f50] system_call_fastpath at ffffffffb3dc539a RIP: 00007f2b28ed4023 RSP: 00007f2a45fe7f80 RFLAGS: 00000206 RAX: 0000000000000012 RBX: 00007f2a68302e00 RCX: 00007f2a682546d8 RDX: 0000000000000826 RSI: 00007eb57ea6a000 RDI: 00000000000000e3 RBP: 00007eb57ea6a000 R8: 0000000000000826 R9: 00000002670bdfd2 R10: 00000002670bdfd2 R11: 0000000000000293 R12: 00000002670bdfd2 R13: 00007f29d501a480 R14: 0000000000000826 R15: 00000002670bdfd2 ORIG_RAX: 0000000000000012 CS: 0033 SS: 002b crash> real 7m14.826s user 7m12.502s sys 0m1.091s With the patch: $ time echo "bt 8993" | ~/crash-dev/crash vmcore vmlinux crash> bt 8993 PID: 8993 TASK: ffff927569cc2100 CPU: 2 COMMAND: "WriterPool0" #0 [ffff927569cd76f0] __schedule at ffffffffb3db78d8 crash-utility#1 [ffff927569cd7758] schedule_preempt_disabled at ffffffffb3db8bf9 ...snip the same output... crash> real 0m8.827s user 0m7.896s sys 0m0.938s Signed-off-by: Tao Liu <[email protected]>

Previously, "retq" is used to determine the end of a function, so the end of framesize calculation. However "ret" might be outputted by gdb rather than "retq", as a result, the framesize is returned incorrectly, and bogus stack trace will be outputted. Without the patch: $ crash -d 3 vmcore vmlinux crash> bt 0xffffffff92da7545 <copy_process+5>: push %rbp [framesize: 8] ... 0xffffffff92da7561 <copy_process+33>: sub $0x238,%rsp [framesize: 624] ... 0xffffffff92da776a <copy_process+554>: pop %r15 [framesize: 8] 0xffffffff92da776c <copy_process+556>: pop %rbp [framesize: 0] 0xffffffff92da776d <copy_process+557>: ret crash> bt -D dump framesize_cache_entries: ... [ 3]: ffffffff92dadcbd 0 CF (copy_process+26493) crash> bt ... crash-utility#9 [ffff888263157bc0] copy_process at ffffffff92dadcbd crash-utility#10 [ffff888263157d20] __mutex_init at ffffffff92ed8dd5 crash-utility#11 [ffff888263157d38] __alloc_file at ffffffff93458397 crash-utility#12 [ffff888263157d60] alloc_empty_file at ffffffff934585d2 crash-utility#13 [ffff888263157da8] __alloc_fd at ffffffff934b5ead crash-utility#14 [ffff888263157e38] _do_fork at ffffffff92dae7a1 crash-utility#15 [ffff888263157f28] do_syscall_64 at ffffffff92c085f4 Stack crash-utility#10 ~ crash-utility#13 are bogus and misleading. With the patch: ... 0xffffffff92da776d <copy_process+557>: ret [framesize restored to: 624] crash> bt -D dump ... [ 3]: ffffffff92dadcbd 624 CF (copy_process+26493) crash> bt ... crash-utility#9 [ffff888263157bc0] copy_process at ffffffff92dadcbd crash-utility#10 [ffff888263157e38] _do_fork at ffffffff92dae7a1 crash-utility#11 [ffff888263157f28] do_syscall_64 at ffffffff92c085f4 Signed-off-by: Tao Liu <[email protected]>

There may be extra "=>" prefix before gdb disassembly, as a result, parse_line() will return string "=>" as arglist[0], which will be converted to number by htol() and fails. E.g.: crash> gdb x/40i __list_del_entry ... 0xffffffff8133c384 <__list_del_entry+36>: cmp %rcx,%rax 0xffffffff8133c387 <__list_del_entry+39>: je 0xffffffff8133c403 <__list_del_entry+163> => 0xffffffff8133c389 <__list_del_entry+41>: mov (%rax),%r8 0xffffffff8133c38c <__list_del_entry+44>: cmp %r8,%rdi 0xffffffff8133c38f <__list_del_entry+47>: jne 0xffffffff8133c3e4 <__list_del_entry+132> 0xffffffff8133c391 <__list_del_entry+49>: mov 0x8(%rdx),%r8 Before the patch: crash> bt ... crash-utility#10 [ffff880095647c00] async_page_fault at ffffffff816a8638 [exception RIP: __list_del_entry+41] RIP: ffffffff8133c389 RSP: ffff880095647cb0 RFLAGS: 00010207 RAX: 0000000000000000 RBX: ffffea0400408020 RCX: dead000000000200 RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffffea0400408020 RBP: ffff880095647cb0 R8: 0000000080000431 R9: ffffffff81e835c0 R10: 0000000000000000 R11: 0000000000000400 R12: ffff880138795b58 R13: 0000000010010201 R14: ffff880095647d70 R15: 0000000400408040 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 bt: invalid input: "=>" crash-utility#11 [ffff880095647cb8] list_del at ffffffff8133c43d crash-utility#12 [ffff880095647cd0] devm_memremap_pages at ffffffff81180c53 After the patch: No string as 'bt: invalid input: "=>"' of output. Signed-off-by: Tao Liu <[email protected]>

crash-utility closed this May 4, 2017

pandafromspace mentioned this pull request Jan 28, 2019

Unable to start 7.2.5 crash with SLES15 coredump #28

Open

This was referenced Nov 28, 2019

Is crash tested with AddressSanitizer? #44

Open

Fix SIGSEGV when cur_req is null pointer #45

Closed

zhaoqianli0202 mentioned this pull request Mar 4, 2020

Trace32 has ability to show inline call in callstack #49

Open

cjl20062529 mentioned this pull request Jun 3, 2020

Error reading global variable value in module with p command #50

Open

yangh mentioned this pull request Nov 15, 2021

arm64: Support overflow stack panic #100

Closed

liutgnu mentioned this pull request Apr 2, 2024

Segmentation fault seen while decoding ARM64 kdump. #176

Closed

zslxiu mentioned this pull request Jun 29, 2024

bt command issue and don't know how to handle section `. relr. dyn' [0x13] #181

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crash: fix build failure with mips #11

crash: fix build failure with mips #11

DengkeDu commented May 4, 2017

crash-utility commented May 4, 2017 via email

crash-utility commented May 4, 2017 via email

crash: fix build failure with mips #11

crash: fix build failure with mips #11

Conversation

DengkeDu commented May 4, 2017

crash-utility commented May 4, 2017 via email

crash-utility commented May 4, 2017 via email