Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MIPS issue with crash utility #26

Open
rwiner opened this issue Jul 4, 2018 · 3 comments
Open

MIPS issue with crash utility #26

rwiner opened this issue Jul 4, 2018 · 3 comments

Comments

@rwiner
Copy link

rwiner commented Jul 4, 2018

Hi,
I'm Intel employee, having a MIPS Platform running Linux OS.
I enabled the kexec\kdump flow in Linux version we use and now I have vmcore memory dump, and now I'm getting into troubles....
I'm running the crash utility compiled with "target=MIPS" and getting the output:

crash: read error: kernel virtual address: xxxx30fc type: "possible"
WARNING: cannot read cpu_possible_map
crash: read error: kernel virtual address: xxxx30f4 type: "present"
WARNING: cannot read cpu_present_map
crash: read error: kernel virtual address: xxxx30f8 type: "online"
WARNING: cannot read cpu_online_map
crash: read error: kernel virtual address: xxxx30f0 type: "active"
WARNING: cannot read cpu_active_map
crash: read error: kernel virtual address:xxxx0a48 type: "shadow_timekeeper xtime_sec"
crash: read error: kernel virtual address: xxxxa444 type: "init_uts_ns"
crash: linux/vmlinux and vmcore do not match!

As you see the tool claims that the vmlinux doesn't match to the vmcore although the vmcore captured from the given vmlinux.
When I open the vmcore with gdb from our toolchain provided by MIPS, it's successfully opened and I see that the init_uts_ns is in offset xxxxa440 instead of xxxxa444 it looks for in the crash utility.

I thought that maybe the issue is we use a propriety toolchain provided by Codescape (the next link) and tried to replace the gdb packet in the Makefile - but then the tool wasn't compile, it fails in the gdb_merge step:
http://codescape-mips-sdk.imgtec.com/components/toolchain/2017.10-05/downloads.html

Can you help us to understand how to overcome this issue? is there any easy way to replace the gdb binary the tool uses?

we'll very appreciate your help.

@crash-utility
Copy link
Collaborator

Hi,
I'm Intel employee, having a MIPS Platform running Linux OS.

Unfortunately I'm a Red Hat employee, and I know pretty much nothing about MIPS.

The patches for the support of the 32-MIPS architecture were posted externally on
the crash-utility mailing list, and I accepted it, and all subsequent patches posted
there:

https://www.redhat.com/mailman/listinfo/crash-utility

You may be able to get some help by posting your query there.

What I can say, though, is that replacing the embedded gdb-7.6 version built
into the crash utility is a major task, and certainly not anything that can be
done by tinkering with the Makefile. So absolutely no, there is no easy way.

It would be interesting if you were to take the gdb-7.6.tar.gz package and
build it standalone, and then run "gdb vmlinux". Are you saying that the
symbol values shown by gdb-7.6 are off-by-4 from what you see when you
run "nm -Bn vmlinux"? Seems strange to me.

Anyway, sorry, I can be of much help.

@shengduiliang
Copy link

Hello,I got the same problem on MIPS platform running linux-5.4.May I ask you that if you have solved this problem?

@crash-utility
Copy link
Collaborator

Sorry no. As I mentioned above, I have no way to investigate issues on that architecture,
and nobody has brought it up on the mailing list.

k-hagio pushed a commit that referenced this issue Nov 29, 2023
…usly

There is an issue that, for kernel modules, "dis -rl" fails to display
modules code line number data after execute "bt" command in crash.

Without the patch:
  crsah> mod -S
  crash> bt
  PID: 1500     TASK: ff2bd8b093524000  CPU: 16   COMMAND: "lpfc_worker_0"
   #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3
   ...snip...
   #8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc]
   ...snip...
  crash> dis -rl ffffffffc0f60f82
  0xffffffffc0f60eb0 <lpfc_nlp_get>:      nopl   0x0(%rax,%rax,1) [FTRACE NOP]
  0xffffffffc0f60eb5 <lpfc_nlp_get+5>:    push   %rbp
  0xffffffffc0f60eb6 <lpfc_nlp_get+6>:    push   %rbx
  0xffffffffc0f60eb7 <lpfc_nlp_get+7>:    test   %rdi,%rdi

With the patch:
  crash> mod -S
  crash> bt
  PID: 1500     TASK: ff2bd8b093524000  CPU: 16   COMMAND: "lpfc_worker_0"
   #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3
   ...snip...
   #8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc]
   ...snip...
  crash> dis -rl ffffffffc0f60f82
  /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6756
  0xffffffffc0f60eb0 <lpfc_nlp_get>:      nopl   0x0(%rax,%rax,1) [FTRACE NOP]
  /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6759
  0xffffffffc0f60eb5 <lpfc_nlp_get+5>:    push   %rbp

The root cause is, after kernel module been loaded by mod command, the symtable
is not expanded in gdb side. crash bt or dis command will trigger such an
expansion. However the symtable expansion is different for the 2 commands:

The stack trace of "dis -rl" for symtable expanding:

  #0  0x00000000008d8d9f in add_compunit_symtab_to_objfile ...
  #1  0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ...
  #2  0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ...
  #3  0x000000000077e8e9 in process_full_comp_unit ...
  #4  process_queue ...
  #5  dw2_do_instantiate_symtab ...
  #6  0x000000000077ed67 in dw2_instantiate_symtab ...
  #7  0x000000000077f75e in dw2_expand_all_symtabs ...
  #8  0x00000000008f254d in gdb_get_line_number ...
  #9  0x00000000008f22af in gdb_command_funnel_1 ...
  #10 0x00000000008f2003 in gdb_command_funnel ...
  #11 0x00000000005b7f02 in gdb_interface ...
  #12 0x00000000005f5bd8 in get_line_number ...
  #13 0x000000000059e574 in cmd_dis ...

The stack trace of "bt" for symtable expanding:

  #0  0x00000000008d8d9f in add_compunit_symtab_to_objfile ...
  #1  0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ...
  #2  0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ...
  #3  0x000000000077e8e9 in process_full_comp_unit ...
  #4  process_queue ...
  #5  dw2_do_instantiate_symtab ...
  #6  0x000000000077ed67 in dw2_instantiate_symtab ...
  #7  0x000000000077f8ed in dw2_lookup_symbol ...
  #8  0x00000000008e6d03 in lookup_symbol_via_quick_fns ...
  #9  0x00000000008e7153 in lookup_symbol_in_objfile ...
  #10 0x00000000008e73c6 in lookup_symbol_global_or_static_iterator_cb ...
  #11 0x00000000008b99c4 in svr4_iterate_over_objfiles_in_search_order ...
  #12 0x00000000008e754e in lookup_global_or_static_symbol ...
  #13 0x00000000008e75da in lookup_static_symbol ...
  #14 0x00000000008e632c in lookup_symbol_aux ...
  #15 0x00000000008e5a7a in lookup_symbol_in_language ...
  #16 0x00000000008e5b30 in lookup_symbol ...
  #17 0x00000000008f2a4a in gdb_get_datatype ...
  #18 0x00000000008f22c0 in gdb_command_funnel_1 ...
  #19 0x00000000008f2003 in gdb_command_funnel ...
  #20 0x00000000005b7f02 in gdb_interface ...
  #21 0x00000000005f8a9f in datatype_info ...
  #22 0x0000000000599947 in cpu_map_size ...
  #23 0x00000000005a975d in get_cpus_online ...
  #24 0x0000000000637a8b in diskdump_get_prstatus_percpu ...
  #25 0x000000000062f0e4 in get_netdump_regs_x86_64 ...
  #26 0x000000000059fe68 in back_trace ...
  #27 0x00000000005ab1cb in cmd_bt ...

For the stacktrace of "dis -rl", it calls dw2_expand_all_symtabs() to expand
all symtable of the objfile, or "*.ko.debug" in our case. However for
the stacktrace of "bt", it doesn't expand all, but only a subset of symtable
which is enough to find a symbol by dw2_lookup_symbol(). As a result, the
objfile->compunit_symtabs, which is the head of a single linked list of
struct compunit_symtab, is not NULL but didn't contain all symtables. It
will not be reinitialized in gdb_get_line_number() by "dis -rl" because
!objfile_has_full_symbols(objfile) check will fail, so it cannot display
the proper code line number data.

Since objfile_has_full_symbols(objfile) check cannot ensure all symbols
been expanded, this patch add a new member as a flag for struct objfile
to record if all symbols have been expanded. The flag will be set only ofter
expand_all_symtabs been called.

Signed-off-by: Tao Liu <[email protected]>
liutgnu added a commit to liutgnu/crash-preview that referenced this issue Nov 30, 2023
…usly

There is an issue that, for kernel modules, "dis -rl" fails to display
modules code line number data after execute "bt" command in crash.

Without the patch:
  crsah> mod -S
  crash> bt
  PID: 1500     TASK: ff2bd8b093524000  CPU: 16   COMMAND: "lpfc_worker_0"
   #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3
   ...snip...
   crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc]
   ...snip...
  crash> dis -rl ffffffffc0f60f82
  0xffffffffc0f60eb0 <lpfc_nlp_get>:      nopl   0x0(%rax,%rax,1) [FTRACE NOP]
  0xffffffffc0f60eb5 <lpfc_nlp_get+5>:    push   %rbp
  0xffffffffc0f60eb6 <lpfc_nlp_get+6>:    push   %rbx
  0xffffffffc0f60eb7 <lpfc_nlp_get+7>:    test   %rdi,%rdi

With the patch:
  crash> mod -S
  crash> bt
  PID: 1500     TASK: ff2bd8b093524000  CPU: 16   COMMAND: "lpfc_worker_0"
   #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3
   ...snip...
   crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc]
   ...snip...
  crash> dis -rl ffffffffc0f60f82
  /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6756
  0xffffffffc0f60eb0 <lpfc_nlp_get>:      nopl   0x0(%rax,%rax,1) [FTRACE NOP]
  /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6759
  0xffffffffc0f60eb5 <lpfc_nlp_get+5>:    push   %rbp

The root cause is, after kernel module been loaded by mod command, the symtable
is not expanded in gdb side. crash bt or dis command will trigger such an
expansion. However the symtable expansion is different for the 2 commands:

The stack trace of "dis -rl" for symtable expanding:

  #0  0x00000000008d8d9f in add_compunit_symtab_to_objfile ...
  crash-utility#1  0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ...
  crash-utility#2  0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ...
  crash-utility#3  0x000000000077e8e9 in process_full_comp_unit ...
  crash-utility#4  process_queue ...
  crash-utility#5  dw2_do_instantiate_symtab ...
  crash-utility#6  0x000000000077ed67 in dw2_instantiate_symtab ...
  crash-utility#7  0x000000000077f75e in dw2_expand_all_symtabs ...
  crash-utility#8  0x00000000008f254d in gdb_get_line_number ...
  crash-utility#9  0x00000000008f22af in gdb_command_funnel_1 ...
  crash-utility#10 0x00000000008f2003 in gdb_command_funnel ...
  crash-utility#11 0x00000000005b7f02 in gdb_interface ...
  crash-utility#12 0x00000000005f5bd8 in get_line_number ...
  crash-utility#13 0x000000000059e574 in cmd_dis ...

The stack trace of "bt" for symtable expanding:

  #0  0x00000000008d8d9f in add_compunit_symtab_to_objfile ...
  crash-utility#1  0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ...
  crash-utility#2  0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ...
  crash-utility#3  0x000000000077e8e9 in process_full_comp_unit ...
  crash-utility#4  process_queue ...
  crash-utility#5  dw2_do_instantiate_symtab ...
  crash-utility#6  0x000000000077ed67 in dw2_instantiate_symtab ...
  crash-utility#7  0x000000000077f8ed in dw2_lookup_symbol ...
  crash-utility#8  0x00000000008e6d03 in lookup_symbol_via_quick_fns ...
  crash-utility#9  0x00000000008e7153 in lookup_symbol_in_objfile ...
  crash-utility#10 0x00000000008e73c6 in lookup_symbol_global_or_static_iterator_cb ...
  crash-utility#11 0x00000000008b99c4 in svr4_iterate_over_objfiles_in_search_order ...
  crash-utility#12 0x00000000008e754e in lookup_global_or_static_symbol ...
  crash-utility#13 0x00000000008e75da in lookup_static_symbol ...
  crash-utility#14 0x00000000008e632c in lookup_symbol_aux ...
  crash-utility#15 0x00000000008e5a7a in lookup_symbol_in_language ...
  crash-utility#16 0x00000000008e5b30 in lookup_symbol ...
  crash-utility#17 0x00000000008f2a4a in gdb_get_datatype ...
  crash-utility#18 0x00000000008f22c0 in gdb_command_funnel_1 ...
  crash-utility#19 0x00000000008f2003 in gdb_command_funnel ...
  crash-utility#20 0x00000000005b7f02 in gdb_interface ...
  crash-utility#21 0x00000000005f8a9f in datatype_info ...
  crash-utility#22 0x0000000000599947 in cpu_map_size ...
  crash-utility#23 0x00000000005a975d in get_cpus_online ...
  crash-utility#24 0x0000000000637a8b in diskdump_get_prstatus_percpu ...
  crash-utility#25 0x000000000062f0e4 in get_netdump_regs_x86_64 ...
  crash-utility#26 0x000000000059fe68 in back_trace ...
  crash-utility#27 0x00000000005ab1cb in cmd_bt ...

For the stacktrace of "dis -rl", it calls dw2_expand_all_symtabs() to expand
all symtable of the objfile, or "*.ko.debug" in our case. However for
the stacktrace of "bt", it doesn't expand all, but only a subset of symtable
which is enough to find a symbol by dw2_lookup_symbol(). As a result, the
objfile->compunit_symtabs, which is the head of a single linked list of
struct compunit_symtab, is not NULL but didn't contain all symtables. It
will not be reinitialized in gdb_get_line_number() by "dis -rl" because
!objfile_has_full_symbols(objfile) check will fail, so it cannot display
the proper code line number data.

Since objfile_has_full_symbols(objfile) check cannot ensure all symbols
been expanded, this patch add a new member as a flag for struct objfile
to record if all symbols have been expanded. The flag will be set only ofter
expand_all_symtabs been called.

Signed-off-by: Tao Liu <[email protected]>
liutgnu added a commit to liutgnu/crash-preview that referenced this issue Dec 1, 2023
…eviously

There is an issue that, for kernel modules, "dis -rl" fails to display
modules code line number data after execute "bt" command in crash.

Without the patch:
  crsah> mod -S
  crash> bt
  PID: 1500     TASK: ff2bd8b093524000  CPU: 16   COMMAND: "lpfc_worker_0"
   #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3
   ...snip...
   crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc]
   ...snip...
  crash> dis -rl ffffffffc0f60f82
  0xffffffffc0f60eb0 <lpfc_nlp_get>:      nopl   0x0(%rax,%rax,1) [FTRACE NOP]
  0xffffffffc0f60eb5 <lpfc_nlp_get+5>:    push   %rbp
  0xffffffffc0f60eb6 <lpfc_nlp_get+6>:    push   %rbx
  0xffffffffc0f60eb7 <lpfc_nlp_get+7>:    test   %rdi,%rdi

With the patch:
  crash> mod -S
  crash> bt
  PID: 1500     TASK: ff2bd8b093524000  CPU: 16   COMMAND: "lpfc_worker_0"
   #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3
   ...snip...
   crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc]
   ...snip...
  crash> dis -rl ffffffffc0f60f82
  /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6756
  0xffffffffc0f60eb0 <lpfc_nlp_get>:      nopl   0x0(%rax,%rax,1) [FTRACE NOP]
  /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6759
  0xffffffffc0f60eb5 <lpfc_nlp_get+5>:    push   %rbp

The root cause is, after kernel module been loaded by mod command, the symtable
is not expanded in gdb side. crash bt or dis command will trigger such an
expansion. However the symtable expansion is different for the 2 commands:

The stack trace of "dis -rl" for symtable expanding:

  #0  0x00000000008d8d9f in add_compunit_symtab_to_objfile ...
  crash-utility#1  0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ...
  crash-utility#2  0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ...
  crash-utility#3  0x000000000077e8e9 in process_full_comp_unit ...
  crash-utility#4  process_queue ...
  crash-utility#5  dw2_do_instantiate_symtab ...
  crash-utility#6  0x000000000077ed67 in dw2_instantiate_symtab ...
  crash-utility#7  0x000000000077f75e in dw2_expand_all_symtabs ...
  crash-utility#8  0x00000000008f254d in gdb_get_line_number ...
  crash-utility#9  0x00000000008f22af in gdb_command_funnel_1 ...
  crash-utility#10 0x00000000008f2003 in gdb_command_funnel ...
  crash-utility#11 0x00000000005b7f02 in gdb_interface ...
  crash-utility#12 0x00000000005f5bd8 in get_line_number ...
  crash-utility#13 0x000000000059e574 in cmd_dis ...

The stack trace of "bt" for symtable expanding:

  #0  0x00000000008d8d9f in add_compunit_symtab_to_objfile ...
  crash-utility#1  0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ...
  crash-utility#2  0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ...
  crash-utility#3  0x000000000077e8e9 in process_full_comp_unit ...
  crash-utility#4  process_queue ...
  crash-utility#5  dw2_do_instantiate_symtab ...
  crash-utility#6  0x000000000077ed67 in dw2_instantiate_symtab ...
  crash-utility#7  0x000000000077f8ed in dw2_lookup_symbol ...
  crash-utility#8  0x00000000008e6d03 in lookup_symbol_via_quick_fns ...
  crash-utility#9  0x00000000008e7153 in lookup_symbol_in_objfile ...
  crash-utility#10 0x00000000008e73c6 in lookup_symbol_global_or_static_iterator_cb ...
  crash-utility#11 0x00000000008b99c4 in svr4_iterate_over_objfiles_in_search_order ...
  crash-utility#12 0x00000000008e754e in lookup_global_or_static_symbol ...
  crash-utility#13 0x00000000008e75da in lookup_static_symbol ...
  crash-utility#14 0x00000000008e632c in lookup_symbol_aux ...
  crash-utility#15 0x00000000008e5a7a in lookup_symbol_in_language ...
  crash-utility#16 0x00000000008e5b30 in lookup_symbol ...
  crash-utility#17 0x00000000008f2a4a in gdb_get_datatype ...
  crash-utility#18 0x00000000008f22c0 in gdb_command_funnel_1 ...
  crash-utility#19 0x00000000008f2003 in gdb_command_funnel ...
  crash-utility#20 0x00000000005b7f02 in gdb_interface ...
  crash-utility#21 0x00000000005f8a9f in datatype_info ...
  crash-utility#22 0x0000000000599947 in cpu_map_size ...
  crash-utility#23 0x00000000005a975d in get_cpus_online ...
  crash-utility#24 0x0000000000637a8b in diskdump_get_prstatus_percpu ...
  crash-utility#25 0x000000000062f0e4 in get_netdump_regs_x86_64 ...
  crash-utility#26 0x000000000059fe68 in back_trace ...
  crash-utility#27 0x00000000005ab1cb in cmd_bt ...

For the stacktrace of "dis -rl", it calls dw2_expand_all_symtabs() to expand
all symtable of the objfile, or "*.ko.debug" in our case. However for
the stacktrace of "bt", it doesn't expand all, but only a subset of symtable
which is enough to find a symbol by dw2_lookup_symbol(). As a result, the
objfile->compunit_symtabs, which is the head of a single linked list of
struct compunit_symtab, is not NULL but didn't contain all symtables. It
will not be reinitialized in gdb_get_line_number() by "dis -rl" because
!objfile_has_full_symbols(objfile) check will fail, so it cannot display
the proper code line number data.

Since objfile_has_full_symbols(objfile) check cannot ensure all symbols
been expanded, this patch add a new member as a flag for struct objfile
to record if all symbols have been expanded. The flag will be set only ofter
expand_all_symtabs been called.

Signed-off-by: Tao Liu <[email protected]>
liutgnu added a commit to liutgnu/crash-preview that referenced this issue Dec 5, 2024
…usly

There is an issue that, for kernel modules, "dis -rl" fails to display
modules code line number data after execute "bt" command in crash.

Without the patch:
  crsah> mod -S
  crash> bt
  PID: 1500     TASK: ff2bd8b093524000  CPU: 16   COMMAND: "lpfc_worker_0"
   #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3
   ...snip...
   crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc]
   ...snip...
  crash> dis -rl ffffffffc0f60f82
  0xffffffffc0f60eb0 <lpfc_nlp_get>:      nopl   0x0(%rax,%rax,1) [FTRACE NOP]
  0xffffffffc0f60eb5 <lpfc_nlp_get+5>:    push   %rbp
  0xffffffffc0f60eb6 <lpfc_nlp_get+6>:    push   %rbx
  0xffffffffc0f60eb7 <lpfc_nlp_get+7>:    test   %rdi,%rdi

With the patch:
  crash> mod -S
  crash> bt
  PID: 1500     TASK: ff2bd8b093524000  CPU: 16   COMMAND: "lpfc_worker_0"
   #0 [ff2c9f725c39f9e0] machine_kexec at ffffffff8e0686d3
   ...snip...
   crash-utility#8 [ff2c9f725c39fcc0] __lpfc_sli_release_iocbq_s4 at ffffffffc0f2f425 [lpfc]
   ...snip...
  crash> dis -rl ffffffffc0f60f82
  /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6756
  0xffffffffc0f60eb0 <lpfc_nlp_get>:      nopl   0x0(%rax,%rax,1) [FTRACE NOP]
  /usr/src/debug/kernel-4.18.0-425.13.1.el8_7/linux-4.18.0-425.13.1.el8_7.x86_64/drivers/scsi/lpfc/lpfc_hbadisc.c: 6759
  0xffffffffc0f60eb5 <lpfc_nlp_get+5>:    push   %rbp

The root cause is, after kernel module been loaded by mod command, the symtable
is not expanded in gdb side. crash bt or dis command will trigger such an
expansion. However the symtable expansion is different for the 2 commands:

The stack trace of "dis -rl" for symtable expanding:

  #0  0x00000000008d8d9f in add_compunit_symtab_to_objfile ...
  crash-utility#1  0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ...
  crash-utility#2  0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ...
  crash-utility#3  0x000000000077e8e9 in process_full_comp_unit ...
  crash-utility#4  process_queue ...
  crash-utility#5  dw2_do_instantiate_symtab ...
  crash-utility#6  0x000000000077ed67 in dw2_instantiate_symtab ...
  crash-utility#7  0x000000000077f75e in dw2_expand_all_symtabs ...
  crash-utility#8  0x00000000008f254d in gdb_get_line_number ...
  crash-utility#9  0x00000000008f22af in gdb_command_funnel_1 ...
  crash-utility#10 0x00000000008f2003 in gdb_command_funnel ...
  crash-utility#11 0x00000000005b7f02 in gdb_interface ...
  crash-utility#12 0x00000000005f5bd8 in get_line_number ...
  crash-utility#13 0x000000000059e574 in cmd_dis ...

The stack trace of "bt" for symtable expanding:

  #0  0x00000000008d8d9f in add_compunit_symtab_to_objfile ...
  crash-utility#1  0x00000000006d3293 in buildsym_compunit::end_symtab_with_blockvector ...
  crash-utility#2  0x00000000006d336a in buildsym_compunit::end_symtab_from_static_block ...
  crash-utility#3  0x000000000077e8e9 in process_full_comp_unit ...
  crash-utility#4  process_queue ...
  crash-utility#5  dw2_do_instantiate_symtab ...
  crash-utility#6  0x000000000077ed67 in dw2_instantiate_symtab ...
  crash-utility#7  0x000000000077f8ed in dw2_lookup_symbol ...
  crash-utility#8  0x00000000008e6d03 in lookup_symbol_via_quick_fns ...
  crash-utility#9  0x00000000008e7153 in lookup_symbol_in_objfile ...
  crash-utility#10 0x00000000008e73c6 in lookup_symbol_global_or_static_iterator_cb ...
  crash-utility#11 0x00000000008b99c4 in svr4_iterate_over_objfiles_in_search_order ...
  crash-utility#12 0x00000000008e754e in lookup_global_or_static_symbol ...
  crash-utility#13 0x00000000008e75da in lookup_static_symbol ...
  crash-utility#14 0x00000000008e632c in lookup_symbol_aux ...
  crash-utility#15 0x00000000008e5a7a in lookup_symbol_in_language ...
  crash-utility#16 0x00000000008e5b30 in lookup_symbol ...
  crash-utility#17 0x00000000008f2a4a in gdb_get_datatype ...
  crash-utility#18 0x00000000008f22c0 in gdb_command_funnel_1 ...
  crash-utility#19 0x00000000008f2003 in gdb_command_funnel ...
  crash-utility#20 0x00000000005b7f02 in gdb_interface ...
  crash-utility#21 0x00000000005f8a9f in datatype_info ...
  crash-utility#22 0x0000000000599947 in cpu_map_size ...
  crash-utility#23 0x00000000005a975d in get_cpus_online ...
  crash-utility#24 0x0000000000637a8b in diskdump_get_prstatus_percpu ...
  crash-utility#25 0x000000000062f0e4 in get_netdump_regs_x86_64 ...
  crash-utility#26 0x000000000059fe68 in back_trace ...
  crash-utility#27 0x00000000005ab1cb in cmd_bt ...

For the stacktrace of "dis -rl", it calls dw2_expand_all_symtabs() to expand
all symtable of the objfile, or "*.ko.debug" in our case. However for
the stacktrace of "bt", it doesn't expand all, but only a subset of symtable
which is enough to find a symbol by dw2_lookup_symbol(). As a result, the
objfile->compunit_symtabs, which is the head of a single linked list of
struct compunit_symtab, is not NULL but didn't contain all symtables. It
will not be reinitialized in gdb_get_line_number() by "dis -rl" because
!objfile_has_full_symbols(objfile) check will fail, so it cannot display
the proper code line number data.

Since objfile_has_full_symbols(objfile) check cannot ensure all symbols
been expanded, this patch add a new member as a flag for struct objfile
to record if all symbols have been expanded. The flag will be set only ofter
expand_all_symtabs been called.

Signed-off-by: Tao Liu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants