Skip to content

Conversation

@mchoo7
Copy link
Contributor

@mchoo7 mchoo7 commented Jan 26, 2026

Kernel panic is a special case, and there is no signal or exception for that so we need to rely on special workaround called dumptid. FreeBSDKernel plugin is supposed to find this thread and set it manually through SetStopInfo() in CalculateStopInfo() like Mach core plugin does.

Before (We had to find and select crashed thread list otherwise thread 1 was selected by default):

➜ sudo lldb /boot/panic/kernel -c /var/crash/vmcore.last
(lldb) target create "/boot/panic/kernel" --core "/var/crash/vmcore.last"
Core file '/var/crash/vmcore.last' (x86_64) was loaded.
(lldb) bt
* thread #1, name = '(pid 12991) dtrace'
  * frame #0: 0xffffffff80bf9322 kernel`sched_switch(td=0xfffff8015882f780, flags=259) at sched_ule.c:2448:26
    frame #1: 0xffffffff80bd38d2 kernel`mi_switch(flags=259) at kern_synch.c:530:2
    frame #2: 0xffffffff80c29799 kernel`sleepq_switch(wchan=0xfffff8014edff300, pri=0) at subr_sleepqueue.c:608:2
    frame #3: 0xffffffff80c29b76 kernel`sleepq_catch_signals(wchan=0xfffff8014edff300, pri=0) at subr_sleepqueue.c:523:3
    frame #4: 0xffffffff80c29d32 kernel`sleepq_timedwait_sig(wchan=<unavailable>, pri=<unavailable>) at subr_sleepqueue.c:704:11
    frame #5: 0xffffffff80bd2e2d kernel`_sleep(ident=0xfffff8014edff300, lock=0xffffffff81df2880, priority=768, wmesg="uwait", sbt=2573804118162, pr=0, flags=512) at kern_synch.c:215:10
    frame #6: 0xffffffff80be8622 kernel`umtxq_sleep(uq=0xfffff8014edff300, wmesg="uwait", timo=0xfffffe0279cb3d20) at kern_umtx.c:843:11
    frame #7: 0xffffffff80bef87a kernel`do_wait(td=0xfffff8015882f780, addr=<unavailable>, id=0, timeout=0xfffffe0279cb3d90, compat32=1, is_private=1) at kern_umtx.c:1316:12
    frame #8: 0xffffffff80bed264 kernel`__umtx_op_wait_uint_private(td=0xfffff8015882f780, uap=0xfffffe0279cb3dd8, ops=<unavailable>) at kern_umtx.c:3990:10
    frame #9: 0xffffffff80beaabe kernel`sys__umtx_op [inlined] kern__umtx_op(td=<unavailable>, obj=<unavailable>, op=<unavailable>, val=<unavailable>, uaddr1=<unavailable>, uaddr2=<unavailable>, ops=<unavailable>) at kern_umtx.c:4999:10
    frame #10: 0xffffffff80beaa89 kernel`sys__umtx_op(td=<unavailable>, uap=<unavailable>) at kern_umtx.c:5024:10
    frame #11: 0xffffffff81122cd1 kernel`amd64_syscall [inlined] syscallenter(td=0xfffff8015882f780) at subr_syscall.c:165:11
    frame #12: 0xffffffff81122c19 kernel`amd64_syscall(td=0xfffff8015882f780, traced=0) at trap.c:1208:2
    frame #13: 0xffffffff810f1dbb kernel`fast_syscall_common at exception.S:570

After:

➜ sudo ./build/bin/lldb /boot/panic/kernel -c /var/crash/vmcore.last
(lldb) target create "/boot/panic/kernel" --core "/var/crash/vmcore.last"
Core file '/var/crash/vmcore.last' (x86_64) was loaded.
(lldb) bt
* thread #18, name = '(pid 5409) powerd (crashed)', stop reason = kernel panic
  * frame #0: 0xffffffff80bc6c91 kernel`__curthread at pcpu_aux.h:57:2 [inlined]
    frame #1: 0xffffffff80bc6c91 kernel`doadump(textdump=0) at kern_shutdown.c:399:2
    frame #2: 0xffffffff804b3b7a kernel`db_dump(dummy=<unavailable>, dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>) at db_command.c:596:10
    frame #3: 0xffffffff804b396d kernel`db_command(last_cmdp=<unavailable>, cmd_table=<unavailable>, dopager=true) at db_command.c:508:3
    frame #4: 0xffffffff804b362d kernel`db_command_loop at db_command.c:555:3
    frame #5: 0xffffffff804b7026 kernel`db_trap(type=<unavailable>, code=<unavailable>) at db_main.c:267:3
    frame #6: 0xffffffff80c16aaf kernel`kdb_trap(type=3, code=0, tf=0xfffffe01b605b930) at subr_kdb.c:790:13
    frame #7: 0xffffffff8112154e kernel`trap(frame=<unavailable>) at trap.c:614:8
    frame #8: 0xffffffff810f14c8 kernel`calltrap at exception.S:285
    frame #9: 0xffffffff81da2290 kernel`cn_devtab + 64
    frame #10: 0xfffffe01b605b8b0
    frame #11: 0xffffffff84001c43 dtrace.ko`dtrace_panic(format=<unavailable>) at dtrace.c:652:2
    frame #12: 0xffffffff84005524 dtrace.ko`dtrace_action_panic(ecb=0xfffff80539cad580) at dtrace.c:7022:2 [inlined]
    frame #13: 0xffffffff840054de dtrace.ko`dtrace_probe(id=88998, arg0=14343377283488, arg1=<unavailable>, arg2=<unavailable>, arg3=<unavailable>, arg4=<unavailable>) at dtrace.c:7665:6
    frame #14: 0xffffffff83e5213d systrace.ko`systrace_probe(sa=<unavailable>, type=<unavailable>, retval=<unavailable>) at systrace.c:226:2
    frame #15: 0xffffffff8112318d kernel`syscallenter(td=0xfffff801318d5780) at subr_syscall.c:160:4 [inlined]
    frame #16: 0xffffffff81123112 kernel`amd64_syscall(td=0xfffff801318d5780, traced=0) at trap.c:1208:2
    frame #17: 0xffffffff810f1dbb kernel`fast_syscall_common at exception.S:570

@mchoo7 mchoo7 requested a review from JDevlieghere as a code owner January 26, 2026 22:28
@mchoo7
Copy link
Contributor Author

mchoo7 commented Jan 26, 2026

Ping @aokblast @emaste

@github-actions
Copy link

Thank you for submitting a Pull Request (PR) to the LLVM Project!

This PR will be automatically labeled and the relevant teams will be notified.

If you wish to, you can add reviewers by using the "Reviewers" section on this page.

If this is not working for you, it is probably because you do not have write permissions for the repository. In which case you can instead tag reviewers by name in a comment by using @ followed by their GitHub username.

If you have received no comments on your PR for a week, you can request a review by "ping"ing the PR by adding a comment “Ping”. The common courtesy "ping" rate is once a week. Please remember that you are asking for valuable time from other developers.

If you have further questions, they may be answered by the LLVM GitHub User Guide.

You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums.

@llvmbot llvmbot added the lldb label Jan 26, 2026
@llvmbot
Copy link
Member

llvmbot commented Jan 26, 2026

@llvm/pr-subscribers-lldb

Author: Minsoo Choo (mchoo7)

Changes

Example:

➜ sudo ./build/bin/lldb /boot/kernel/kernel -c /var/crash/vmcore.last
(lldb) target create "/boot/kernel/kernel" --core "/var/crash/vmcore.last"
Core file '/var/crash/vmcore.last' (x86_64) was loaded.
(lldb) bt
* thread #<!-- -->1, name = '(pid 67404) sysctl (crashed)'
  * frame #<!-- -->0: 0xffffffff80bc6461 kernel`__curthread at pcpu_aux.h:57:2 [inlined]
    frame #<!-- -->1: 0xffffffff80bc6461 kernel`doadump(textdump=0) at kern_shutdown.c:399:2
    frame #<!-- -->2: 0xffffffff804b3b7a kernel`db_dump(dummy=&lt;unavailable&gt;, dummy2=&lt;unavailable&gt;, dummy3=&lt;unavailable&gt;, dummy4=&lt;unavailable&gt;) at db_command.c:596:10
    frame #<!-- -->3: 0xffffffff804b396d kernel`db_command(last_cmdp=&lt;unavailable&gt;, cmd_table=&lt;unavailable&gt;, dopager=true) at db_command.c:508:3
    frame #<!-- -->4: 0xffffffff804b362d kernel`db_command_loop at db_command.c:555:3
    frame #<!-- -->5: 0xffffffff804b7026 kernel`db_trap(type=&lt;unavailable&gt;, code=&lt;unavailable&gt;) at db_main.c:267:3
    frame #<!-- -->6: 0xffffffff80c1603f kernel`kdb_trap(type=3, code=0, tf=0xfffffe0279718960) at subr_kdb.c:790:13
    frame #<!-- -->7: 0xffffffff8112054e kernel`trap(frame=&lt;unavailable&gt;) at trap.c:614:8
    frame #<!-- -->8: 0xffffffff810f0528 kernel`calltrap at exception.S:285
    frame #<!-- -->9: 0xffffffff81da1290 kernel`cn_devtab + 64
    frame #<!-- -->10: 0xfffffe02797188e0
    frame #<!-- -->11: 0xffffffff80bc6533 kernel`panic(fmt=&lt;unavailable&gt;) at kern_shutdown.c:887:2
    frame #<!-- -->12: 0xffffffff80c16493 kernel`kdb_sysctl_panic(oidp=&lt;unavailable&gt;, arg1=&lt;unavailable&gt;, arg2=&lt;unavailable&gt;, req=&lt;unavailable&gt;) at subr_kdb.c:219:2
    frame #<!-- -->13: 0xffffffff80bd7aec kernel`sysctl_root_handler_locked(oid=0xffffffff81ade300, arg1=&lt;unavailable&gt;, arg2=&lt;unavailable&gt;, req=0xfffffe0279718cc0, tracker=0xfffffe0279718c40) at kern_sysctl.c:199:10
    frame #<!-- -->14: 0xffffffff80bd6e4f kernel`sysctl_root(oidp=&lt;unavailable&gt;, arg1=0x0000000000000000, arg2=0, req=0xfffffe0279718cc0) at kern_sysctl.c:2403:10
    frame #<!-- -->15: 0xffffffff80bd7536 kernel`userland_sysctl(td=0xfffff80166aa3780, name=0xfffffe0279718d80, namelen=&lt;unavailable&gt;, old=&lt;unavailable&gt;, oldlenp=&lt;unavailable&gt;, inkernel=0, new=0x0000354208609000, newlen=4, retval=0xfffffe0279718de8, flags=0) at kern_sysctl.c:2562:11
    frame #<!-- -->16: 0xffffffff80bd7365 kernel`sys___sysctl(td=0xfffff80166aa3780, uap=0xfffff80166aa3ba8) at kern_sysctl.c:2433:10
    frame #<!-- -->17: 0xffffffff811219e9 kernel`syscallenter(td=0xfffff80166aa3780) at subr_syscall.c:193:11 [inlined]
    frame #<!-- -->18: 0xffffffff811218ae kernel`amd64_syscall(td=0xfffff80166aa3780, traced=0) at trap.c:1208:2
    frame #<!-- -->19: 0xffffffff810f0e1b kernel`fast_syscall_common at exception.S:570
(lldb) quit
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y

Full diff: https://github.com/llvm/llvm-project/pull/178069.diff

2 Files Affected:

  • (modified) lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp (+7-1)
  • (modified) lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.h (+2)
diff --git a/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp b/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp
index d209e5b5384f3..ced8fa6f28c84 100644
--- a/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp
+++ b/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp
@@ -118,7 +118,12 @@ bool ProcessFreeBSDKernel::CanDebug(lldb::TargetSP target_sp,
   return true;
 }
 
-void ProcessFreeBSDKernel::RefreshStateAfterStop() {}
+void ProcessFreeBSDKernel::RefreshStateAfterStop() {
+  if (m_crashed_thread_id != LLDB_INVALID_THREAD_ID) {
+    GetThreadList().SetSelectedThreadByID(m_crashed_thread_id);
+    m_crashed_thread_id = LLDB_INVALID_THREAD_ID;
+  }
+}
 
 bool ProcessFreeBSDKernel::DoUpdateThreadList(ThreadList &old_thread_list,
                                               ThreadList &new_thread_list) {
@@ -232,6 +237,7 @@ bool ProcessFreeBSDKernel::DoUpdateThreadList(ThreadList &old_thread_list,
           // NB: dumppcb can be LLDB_INVALID_ADDRESS if reading it failed
           pcb_addr = dumppcb;
           thread_desc += " (crashed)";
+          m_crashed_thread_id = tid;
         } else if (oncpu != -1) {
           // if we managed to read stoppcbs and pcb_size, use them to find
           // the correct PCB
diff --git a/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.h b/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.h
index 06c9d062441e5..c36e31839db19 100644
--- a/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.h
+++ b/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.h
@@ -49,6 +49,8 @@ class ProcessFreeBSDKernel : public lldb_private::PostMortemProcess {
                           lldb_private::ThreadList &new_thread_list) override;
 
   lldb::addr_t FindSymbol(const char* name);
+private:
+  lldb::tid_t m_crashed_thread_id = LLDB_INVALID_THREAD_ID;
 };
 
 #endif // LLDB_SOURCE_PLUGINS_PROCESS_FREEBSDKERNEL_PROCESSFREEBSDKERNEL_H

@github-actions
Copy link

github-actions bot commented Jan 27, 2026

✅ With the latest revision this PR passed the C/C++ code formatter.

@DavidSpickett
Copy link
Collaborator

Looks fine to me. I checked other plugins and they do not do this, but I see why it's useful. I wonder if the other formats should do this, or don't need to for some reason, but no need to look into that yourself.

Is there a test case in tree for this kernel plugin? If there is, please update that, if not, you don't need to add one. Even if this somehow has problems, the impact is limited.

Source changes look fine to me, let's give some time to the FreeBSD specialists to respond.

@DavidSpickett
Copy link
Collaborator

Please update the PR description to show before and after. You can edit the lldb output to remove unimportant bits, that's fine.

I assume lldb previously selected thread 0, or whatever was the running thread at the time of the dump?

@mchoo7
Copy link
Contributor Author

mchoo7 commented Jan 27, 2026

Is there a test case in tree for this kernel plugin? If there is, please update that, if not, you don't need to add one. Even if this somehow has problems, the impact is limited.

We have test cases for FreeBSDKernel but I've never used it. It'll take some time for me to understand how it works. A FreeBSD developer will review this on Saturday, so we have time.

@mchoo7
Copy link
Contributor Author

mchoo7 commented Jan 27, 2026

Please update the PR description to show before and after. You can edit the lldb output to remove unimportant bits, that's fine.

I assume lldb previously selected thread 0, or whatever was the running thread at the time of the dump?

It's selecting thread 1 by default. Just realized that I need to test this on other threads as well since kernel panic occurs on thread 1.

@mchoo7
Copy link
Contributor Author

mchoo7 commented Jan 27, 2026

My first approach was nonsense. Updating to correct approach with correct before/after.

@mchoo7 mchoo7 force-pushed the thread branch 4 times, most recently from 70b0e8d to 329a0ef Compare January 27, 2026 22:01
@github-actions
Copy link

github-actions bot commented Jan 28, 2026

🐧 Linux x64 Test Results

  • 33289 tests passed
  • 509 tests skipped

✅ The build succeeded and all tests passed.

@DavidSpickett
Copy link
Collaborator

lldb-api.tools/lldb-dap/progress/TestDAP_Progress.py

Probably obvious already, but ignore this failure. Your changes are unrelated.

Copy link
Collaborator

@DavidSpickett DavidSpickett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating the PR description, very clear now what this changes. Code is fine just an optional style comment on that.

Please look into the test cases that exist for this. I know some of them require a FreeBSD core parsing library to be installed, so watch out for that. If you have trouble getting them to work, I can look at least the corefile based ones (getting a FreeBSD system is extra steps for me).

It's likely they're not being run in many (if any) places.

At a glance, the core based tests check that expected threads exist but not which one is selected.

@mchoo7
Copy link
Contributor Author

mchoo7 commented Jan 29, 2026

Please look into the test cases that exist for this. I know some of them require a FreeBSD core parsing library to be installed, so watch out for that. If you have trouble getting them to work, I can look at least the corefile based ones (getting a FreeBSD system is extra steps for me).

This will pass the test case since the test core dump was created by sysctl debug.kernel.panic=1. To test this exact behaviour, I need to create core dump again, which requires a dedicated machine (My machine has 96GB ram so minidump will be fine but full dump will be ridiculously large). There will be other PRs that would require updating test cases anyways, so I prefer writing a complete test case once everything is done.

@mchoo7
Copy link
Contributor Author

mchoo7 commented Jan 29, 2026

For testing, see #178714

@DavidSpickett
Copy link
Collaborator

To test this exact behaviour, I need to create core dump again, which requires a dedicated machine (My machine has 96GB ram so minidump will be fine but full dump will be ridiculously large).

Yeah I didn't realise there were all those scripts to minify the dump.

There will be other PRs that would require updating test cases anyways, so I prefer writing a complete test case once everything is done.

That sounds good to me.

I suppose the existing tests might break in the meantime, but I think the only folks running them would be over in FreeBSD itself. So I trust you to deal with that if it happens.

Copy link
Collaborator

@DavidSpickett DavidSpickett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with the braces fixed.

In theory we could wait for @emaste to comment but this seems like a clear usability improvement to me so I'm approving it myself. We've discussed the existing tests and worst case, if someone's CI fails, these patches will be easy to revert.

(plus we are just past the llvm release branching, so there's no danger of being included in a release right now)

Copy link
Contributor

@aokblast aokblast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@aokblast aokblast merged commit b6e5e8e into llvm:main Feb 1, 2026
10 checks passed
@github-actions
Copy link

github-actions bot commented Feb 1, 2026

@mchoo7 Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our build bots. If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail here.

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of LLVM development. You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!

@aokblast
Copy link
Contributor

aokblast commented Feb 1, 2026

I just merge it right now. @emaste Please revert it if you have any issue on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants