bun test --parallel: abort the run when a Windows worker dies with a fatal NTSTATUS - #37129
Merged
Jarred-Sumner merged 4 commits intoAug 7, 2026
Merged
Conversation
…Windows A parallel test worker dying of a native fault on Windows (unhandled exception or __fastfail) terminates with the NTSTATUS as its exit code. The coordinator stored exit codes as u8, so 0xC0000409 (STATUS_STACK_BUFFER_OVERRUN, what UCRT abort() and Rust aborts raise) was reported as 'worker crashed: exit code 9', indistinguishable from process.exit(9), and the run carried on in a fresh worker instead of aborting the way the POSIX fatal-signal path does. Capture the untruncated GetExitCodeProcess DWORD on Process, carry it into the parallel Worker, classify a fixed list of fatal NTSTATUS codes as worker panics (mirroring the POSIX signal list; TerminateProcess and Ctrl+C stay per-file failures like SIGKILL), and print NTSTATUS exit codes in hex. Adds a crash_handler.fastfail test binding that dies via __fastfail so tests can exercise a worker death the crash handler cannot intercept.
Collaborator
Author
|
Status: verified on windows-x64. The fastfail fixture reproduces the exact "worker crashed: exit code 9" signature from build 89973 when the classification is disabled, and aborts with "worker crashed: exit code 0xC0000409" plus the crash banner with this change. Full parallel suite on Windows matches the main-built baseline (no new failures); Linux is behaviorally unchanged. Note for the record: the proof is Windows-only (POSIX exit codes are 8-bit, the new test skips there). |
Contributor
WalkthroughChangesCross-platform crash handling
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
… terminates them libuv's Windows uv__kill only maps SIGQUIT/SIGTERM/SIGKILL/SIGINT to TerminateProcess and returns ENOSYS for anything else, so the kill(1) in abort_on_worker_panic (and abort_all) was a discarded error. That was dead code while Windows crashes were never classified as panics; now that fatal NTSTATUS exits take this path, a busy sibling would have kept the run alive past the banner. The Windows test now hangs its sibling file to exercise the termination, matching the POSIX test.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Collaborator
Author
|
Updated 6:30 AM PT - Aug 7th, 2026
✅ @robobun, your commit d442075649121272271c9fb6b50ebf6588855cf8 passed in 🧪 To try this PR locally: bunx bun-pr 37129That installs a local version of the PR into your bun-37129 --bun |
Collaborator
Author
|
Addressed the review finding about sibling termination in 7b9d545: the abort path killed siblings with signal 1, which libuv-win rejects with ENOSYS (it only maps SIGQUIT/SIGTERM/SIGKILL/SIGINT to TerminateProcess), so on Windows the newly reachable abort would have waited on a busy sibling instead of stopping it. Both abort paths now use SIGKILL, and the Windows test hangs its sibling file (like the POSIX test) so the termination is exercised: the run ends immediately with the banner last and the sibling accounted as "aborted: sibling worker panicked". Verified on windows-x64. |
…orrect the allowlist rationale Review feedback, three items: - The untruncated Windows exit code now lives in the exit status itself (cfg(windows) Exited.raw, set at the single Windows construction site) instead of a parallel side-channel through Process and Worker fields plus widened signatures. The status was already threaded end-to-end; this deletes the duplicate state, and the eventual Exited::code widening (#33720) will now conflict at the struct, forcing the two fields to be reconciled rather than coexisting silently. - crash_handler.fastfail promised the crash handler never runs, but on non-ASAN POSIX std::process::abort() raises SIGABRT into Bun's own handler. The POSIX arm now uses raise_ignoring_panic_handler(SIGABRT), which resets dispositions first, making the documented contract true on both platforms; a POSIX twin of the Windows test pins it (raw SIGABRT death -> fatal-signal classification -> abort). - The allowlist comment justified itself with 'process.exit(-1) exits with 0xFFFFFFFF', which is false for Bun workers (process.exit truncates to u8 and exits 255). The true grounds: 0xC000013A is Ctrl+C, and foreign code can exit with arbitrary DWORDs (CRT exit(-1) is 0xFFFFFFFF).
Collaborator
Author
|
FYI: the underlying fastfail behind the 0xC0000409 worker deaths this PR classifies is identified and fixed in #37140 (worker_threads argv/execArgv cross-thread AtomString abort). The triage improvement here is still valuable on its own. |
springmin
pushed a commit
to springmin/bun
that referenced
this pull request
Aug 7, 2026
…fatal NTSTATUS (oven-sh#37129) ## Problem In [Buildkite build 89973](https://buildkite.com/bun/bun/builds/89973), both Windows test lanes (x64 and aarch64) printed, for one file of the parallel bucket: ``` ✗ test\js\node\zlib\zlib-estimated-size-gc.test.ts (worker crashed: exit code 9) ``` with zero per-test output under the file header. The file then passed 6/6 when retried alone, so the event was recorded as a flaky test and forgotten. "Exit code 9" was really NTSTATUS `0xC0000409` (STATUS_STACK_BUFFER_OVERRUN, i.e. `__fastfail`: what UCRT `abort()`, Rust aborts in native addons, and /GS stack checks raise). The libuv exit callback narrows the 32-bit `GetExitCodeProcess` value to `u8`, and `0xC0000409 & 0xFF == 9`. Two gaps compound into the masking, and the second is the one this PR is about. The coordinator classifies a worker death as a panic (which aborts the whole run, the behavior added in oven-sh#30216) only by fatal signal. Windows never delivers a signal, so a native fault there was indistinguishable from `process.exit(N)`: the file got marked failed, a fresh worker was spawned, and the run carried on, which in CI turns a genuine native crash into a "flaky" retry. The `is_panic_status` comment described exactly this limitation. And since the printed exit code was the truncated one, the log did not even identify the crash. (oven-sh#33720 separately widens the JS-visible `Exited::code` to match Node. This PR keeps `code` untouched, so it has no JS-visible effect and no dependency on that change; it stores the raw value alongside, and the two fields meet at the same struct, so whichever lands second reconciles them there.) ## Fix - `Exited` gains a `#[cfg(windows)] raw: u32` carrying the untruncated `GetExitCodeProcess` DWORD, set at its single Windows construction site (the libuv exit callback). - `is_panic_status` also treats a fixed allowlist of fatal NTSTATUS exit codes as panics, mirroring the POSIX fatal-signal list: access violation, in-page error, illegal/privileged instruction, the float and integer faults, stack overflow, heap corruption, `__fastfail`, CRT fatals, assertion failure. An allowlist rather than `code >= 0xC0000000` because high exit codes are not all faults: `0xC000013A` is Ctrl+C (the SIGINT analog), and foreign code in the worker can exit with arbitrary DWORDs (CRT `exit(-1)` is `0xFFFFFFFF`). `TerminateProcess` (taskkill, job limits) intentionally stays a per-file failure, like SIGKILL on POSIX. - `describe_status` prints the untruncated code, in hex for NTSTATUS-range values: `worker crashed: exit code 0xC0000409`. - The abort path's sibling termination used `kill(1)`, which libuv-win rejects with ENOSYS (it only maps SIGQUIT/SIGTERM/SIGKILL/SIGINT to `TerminateProcess`), so the newly reachable Windows abort would have waited on a busy sibling instead of stopping it. Both abort paths now use SIGKILL. - New `crash_handler.fastfail` binding in `bun:internal-for-testing`: dies like foreign native code with the crash handler provably out of the way on both platforms — `__fastfail` on Windows (kernel-terminated with `0xC0000409`, no VEH/SEH dispatch) and a raw SIGABRT on POSIX (dispositions reset first). Every existing crash-trigger binding routes through the crash handler, which exits with code 3 on Windows. A crash that does reach Bun's Windows crash handler still exits with code 3, which genuinely cannot be told apart from `process.exit(3)`; that case keeps the old behavior and is recognizable by its banner in the captured worker stderr. ## Verification The fix caught the real crash live: in this PR's own CI ([build 90096](https://buildkite.com/bun/bun/builds/90096), windows 2019 x64), the same zlib file's worker died again and the batch log now shows ``` ✗ test\js\node\zlib\zlib-estimated-size-gc.test.ts (worker crashed: exit code 0xC0000409) error: a test worker process crashed with exit code 0xC0000409 while running test\js\node\zlib\zlib-estimated-size-gc.test.ts. This indicates a bug in Bun or in a native addon, not in the test itself. Aborting. ``` instead of `exit code 9` and a silent continue. (The underlying zlib crash is reported separately.) Tests, mirroring the existing POSIX segfault test (hung sibling + crashing file): - Windows (`skipIf(!isWindows)`): a fastfail worker must produce the hex report, the banner, the abort, and the terminated sibling accounted as `aborted: sibling worker panicked`. Verified on windows-x64: fails on the released canary, passes with this build; with the classification short-circuited, the run reproduces the CI logs' exact `worker crashed: exit code 9` signature and carries on. - POSIX (`skipIf(isWindows)`): the same fixture dies by raw SIGABRT and must abort identically via the fatal-signal list. Fails on the released canary, passes with this build on linux x64 (debug+ASAN). `bun bd test test/cli/test/parallel.test.ts` matches the main-built baseline on both platforms (the pre-existing failures on Windows debug builds are assertion noise reported separately; Windows CI lanes run release builds and are unaffected), and the existing `process.exit(7)` mid-run test still reports `exit code 7` and continues on both. <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/cli/test/parallel.test.ts <!-- robobun:evidence:end -->
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In Buildkite build 89973, both Windows test lanes (x64 and aarch64) printed, for one file of the parallel bucket:
with zero per-test output under the file header. The file then passed 6/6 when retried alone, so the event was recorded as a flaky test and forgotten.
"Exit code 9" was really NTSTATUS
0xC0000409(STATUS_STACK_BUFFER_OVERRUN, i.e.__fastfail: what UCRTabort(), Rust aborts in native addons, and /GS stack checks raise). The libuv exit callback narrows the 32-bitGetExitCodeProcessvalue tou8, and0xC0000409 & 0xFF == 9.Two gaps compound into the masking, and the second is the one this PR is about. The coordinator classifies a worker death as a panic (which aborts the whole run, the behavior added in #30216) only by fatal signal. Windows never delivers a signal, so a native fault there was indistinguishable from
process.exit(N): the file got marked failed, a fresh worker was spawned, and the run carried on, which in CI turns a genuine native crash into a "flaky" retry. Theis_panic_statuscomment described exactly this limitation. And since the printed exit code was the truncated one, the log did not even identify the crash.(#33720 separately widens the JS-visible
Exited::codeto match Node. This PR keepscodeuntouched, so it has no JS-visible effect and no dependency on that change; it stores the raw value alongside, and the two fields meet at the same struct, so whichever lands second reconciles them there.)Fix
Exitedgains a#[cfg(windows)] raw: u32carrying the untruncatedGetExitCodeProcessDWORD, set at its single Windows construction site (the libuv exit callback).is_panic_statusalso treats a fixed allowlist of fatal NTSTATUS exit codes as panics, mirroring the POSIX fatal-signal list: access violation, in-page error, illegal/privileged instruction, the float and integer faults, stack overflow, heap corruption,__fastfail, CRT fatals, assertion failure. An allowlist rather thancode >= 0xC0000000because high exit codes are not all faults:0xC000013Ais Ctrl+C (the SIGINT analog), and foreign code in the worker can exit with arbitrary DWORDs (CRTexit(-1)is0xFFFFFFFF).TerminateProcess(taskkill, job limits) intentionally stays a per-file failure, like SIGKILL on POSIX.describe_statusprints the untruncated code, in hex for NTSTATUS-range values:worker crashed: exit code 0xC0000409.kill(1), which libuv-win rejects with ENOSYS (it only maps SIGQUIT/SIGTERM/SIGKILL/SIGINT toTerminateProcess), so the newly reachable Windows abort would have waited on a busy sibling instead of stopping it. Both abort paths now use SIGKILL.crash_handler.fastfailbinding inbun:internal-for-testing: dies like foreign native code with the crash handler provably out of the way on both platforms —__fastfailon Windows (kernel-terminated with0xC0000409, no VEH/SEH dispatch) and a raw SIGABRT on POSIX (dispositions reset first). Every existing crash-trigger binding routes through the crash handler, which exits with code 3 on Windows.A crash that does reach Bun's Windows crash handler still exits with code 3, which genuinely cannot be told apart from
process.exit(3); that case keeps the old behavior and is recognizable by its banner in the captured worker stderr.Verification
The fix caught the real crash live: in this PR's own CI (build 90096, windows 2019 x64), the same zlib file's worker died again and the batch log now shows
instead of
exit code 9and a silent continue. (The underlying zlib crash is reported separately.)Tests, mirroring the existing POSIX segfault test (hung sibling + crashing file):
skipIf(!isWindows)): a fastfail worker must produce the hex report, the banner, the abort, and the terminated sibling accounted asaborted: sibling worker panicked. Verified on windows-x64: fails on the released canary, passes with this build; with the classification short-circuited, the run reproduces the CI logs' exactworker crashed: exit code 9signature and carries on.skipIf(isWindows)): the same fixture dies by raw SIGABRT and must abort identically via the fatal-signal list. Fails on the released canary, passes with this build on linux x64 (debug+ASAN).bun bd test test/cli/test/parallel.test.tsmatches the main-built baseline on both platforms (the pre-existing failures on Windows debug builds are assertion noise reported separately; Windows CI lanes run release builds and are unaffected), and the existingprocess.exit(7)mid-run test still reportsexit code 7and continues on both.no test proof · iteration 0 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/cli/test/parallel.test.ts