Skip to content

worker: survive terminate() while Bun.serve is streaming a type:"direct" body - #36342

Open
robobun wants to merge 7 commits into
mainfrom
farm/8dfbc553/worker-terminate-direct-stream-defertermination
Open

robobun wants to merge 7 commits into
mainfrom
farm/8dfbc553/worker-terminate-direct-stream-defertermination

Conversation

@robobun

@robobun robobun commented Jul 29, 2026 •

Copy link
Copy Markdown
Collaborator

What

A Worker running Bun.serve() that streams a type:"direct" ReadableStream body, terminated mid-stream, SIGABRTs debug/ASAN builds with:

ASSERTION FAILED: vm.hasTerminationRequest()
vendor/WebKit/Source/JavaScriptCore/runtime/VMTraps.cpp(539) : void JSC::VMTraps::deferTerminationSlow(DeferAction)

Release builds are unaffected (the ASSERT is compiled out). Deterministic on main, first iteration every run. The same assert also fires for an async fetch() handler that spins on await when terminated mid-drain.

Repro

const { Worker } = require("node:worker_threads");
const src = "const server = Bun.serve({ port: 0, fetch() {" +
  "return new Response(new ReadableStream({ type: 'direct', async pull(ctrl) {" +
  "  for (let i = 0; i < 20000; i++) { ctrl.write(new Uint8Array(8192).fill(i)); await ctrl.flush(); }" +
  "  ctrl.close();" +
  "}, cancel() {} }));" +
  "} });" +
  "require('node:worker_threads').parentPort.postMessage(server.port);";
for (let i = 0; i < 4; i++) {
  const w = new Worker(src, { eval: true });
  const port = await new Promise(r => w.once("message", r));
  const res = await fetch("http://127.0.0.1:" + port + "/");
  const rd = res.body.getReader();
  await rd.read();
  const done = new Promise(r => w.once("exit", r));
  w.terminate();
  await done;
  await rd.cancel().catch(() => {});
}
console.log("survived");

Cause

do_render_stream calls assign_to_stream (which kicks off the direct stream's async pull), then drain_microtasks() to drive the await ctrl.flush() continuations. When terminate() lands cross-thread during that drain:

  1. VMTraps::handleTraps sets hasTerminationRequest and throws the TerminationException.
  2. The microtask's outermost VMEntryScope destructor runs executeEntryScopeServicesOnExit(), which clears hasTerminationRequest (the NeedTermination trap bit has been consumed) but leaves m_exception pointing at the termination exception.
  3. GlobalObject::drainMicrotasks observes the pending termination and returns 1; RequestContext::drain_microtasks discards that.
  4. do_render_stream continues into the Pending branch and calls NativePromiseContext::create. That is the first touch of m_NativePromiseContextStructure, so LazyProperty::callFunc runs and opens a DeferTerminationForAWhile scope. VMTraps::deferTerminationSlow sees hasPendingTerminationException() true and asserts hasTerminationRequest(), which is now false.

With the lazy-init assert avoided, the very next step (then_with_value → JSC::profiledCall) trips executeCallImpl's scope.assertNoException() for the same reason. on_response has the identical drain_microtasks() → NativePromiseContext::create → then_with_value sequence for an async fetch() handler.

Separately, WebWorker::shutdown called clear_has_termination_request() before on_exit(), which was a no-op in practice (the dispatchExitInternal termination guard still tripped on the pending exception, matching Node's "skip process.on('exit') on terminate"), and left the sticky exception in place for close_all_socket_groups' on_close callbacks to re-enter JS against.

Fix

  • RequestContext::do_render_stream, RequestContext::on_response, and on_node_http_request_with_upgrade_ctx (the node:http-compat twin): after drain_microtasks(), if an exception is pending (only the TerminationException survives the drain), return instead of continuing into paths that re-enter JS. Worker shutdown's close_all_socket_groups aborts the response and tears the sink down.
  • WebWorker::shutdown: drop the no-op clear_has_termination_request() before on_exit() and instead call clear_termination_exception() (clears both the flag and the pending exception) after on_exit(), so dispatchExitInternal still skips process.on('exit') on terminate (Node semantics) while close_all_socket_groups' on_close callbacks can re-enter JS.
  • Delete the now-unused VM::clear_has_termination_request wrapper and its JSC__VM__clearHasTerminationRequest FFI shim.

#36331 fixes the sibling assertNoException face for the non-direct stream path at the HTTP(S)ResponseSink__onClose/__onReady/detach sites; this PR covers the type:"direct", async-handler, and node:http-compat paths at do_render_stream/on_response/on_node_http_request_with_upgrade_ctx and at shutdown.

Verification

  • New type:"direct" cell in worker-terminate-lifetime.test.ts: fails on main with the deferTerminationSlow assert (first iteration, 5/5), passes with the fix (5/5).
  • New async-fetch() cell: same assert on main, passes with the fix; also asserts process.on('exit') is skipped (sab[1] stays 0), which fails if clear_termination_exception() is moved back before on_exit().
  • New idle-terminate cell: process.on('exit') is skipped when terminate() lands while the worker is parked (Node compat; the trap-bit path).
  • test/js/bun/http/serve-direct-readable-stream.test.ts: 13 pass.
  • serve-stream-body-error.test.ts, serve-error-handler-stream.test.ts, async-iterator-stream.test.ts, serve-async-stream-client-abort.test.ts: 121 pass.
  • test/js/node/worker_threads/worker_threads.test.ts: 91 pass.

no test proof · iteration 4 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/js/web/workers/worker-terminate-lifetime.test.ts

@coderabbitai

coderabbitai Bot commented Jul 29, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

Worker shutdown now preserves termination exceptions, request handlers stop after termination-related exceptions, and regression tests cover streaming, asynchronous, and idle worker termination.

Worker termination handling

Layer / File(s) Summary
Termination cleanup and VM bindings
src/jsc/VM.rs, src/jsc/bindings/bindings.cpp, src/jsc/web_worker.rs
Removes termination-request clearing APIs, tracks callers for throw_error, and clears the global termination exception after worker exit handling.
Request execution exception guards
src/runtime/server/RequestContext.rs
Streaming and non-streaming response paths return early when microtask draining leaves a global JavaScript exception.
Worker termination regression coverage
test/js/web/workers/worker-terminate-lifetime.test.ts
Adds coverage for termination during streaming, asynchronous fetch execution, and idle workers.

Possibly related PRs

  • oven-sh/bun#35976: Related to preserving uncatchable termination exceptions during worker VM evaluation.
  • oven-sh/bun#36097: Also adjusts the WebWorker::shutdown() teardown sequence.
  • oven-sh/bun#36331: Also changes stream handling around pending termination exceptions.

Suggested reviewers: jarred-sumner

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly captures the main change: surviving worker termination during streaming Bun.serve direct responses.
Description check ✅ Passed The description is detailed and covers the bug, repro, fix, and verification, though it uses different section headings than the template.

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Jul 29, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 12:59 AM PT - Jul 30th, 2026

✅ @robobun, your commit 2183b9d3d41e5bab34305cbf9c914add2f50610c passed in Build #85645! 🎉


🧪   To try this PR locally:

bunx bun-pr 36342

That installs a local version of the PR into your bun-36342 executable, so you can run:

bun-36342 --bun

Comment thread src/jsc/bindings/ZigGlobalObject.cpp Outdated
Comment thread src/jsc/web_worker.rs Outdated
Comment thread src/runtime/server/RequestContext.rs Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Found 2 issues this PR may fix:

  1. ASAN CI: ExceptionScope::assertNoException during worker terminate (worker-transfer-terminate-stress, separate from #34095) #34690 - ASAN CI: ExceptionScope::assertNoException assertion failure during worker terminate — this PR fixes the exact termination-vs-exception-scope invariant violation by re-setting hasTerminationRequest in drainMicrotasks() and using clear_termination_exception() in WebWorker::shutdown
  2. Worker create+terminate cycle aborts process after ~100k–900k iterations on macOS arm64 #30421 - Worker create+terminate cycle SIGABRTs after many iterations — the PR's changes to termination exception clearing and early-return-on-pending-exception in do_render_stream address the termination-time abort described here

If this is helpful, copy the block below into the PR description to auto-close these issues on merge.

Fixes #34690
Fixes #30421

🤖 Generated with Claude Code

Comment thread src/jsc/web_worker.rs Outdated
Comment thread test/js/web/workers/worker-terminate-lifetime.test.ts Outdated
Comment thread src/runtime/server/RequestContext.rs
Comment thread src/jsc/web_worker.rs Outdated
Comment thread src/jsc/web_worker.rs Outdated
Comment thread src/jsc/web_worker.rs Outdated
Comment thread src/runtime/server/RequestContext.rs Outdated
Comment thread src/runtime/server/RequestContext.rs Outdated

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the inline nit, I verified the four earlier findings are addressed (clear_termination_exception now runs after on_exit; both new tests gate on isDebug || isASAN; on_response has the sibling guard; the dead clear_has_termination_request wrapper + FFI shim are deleted). I also checked ref balance on the two new early returns: in on_response the bare return matches the adjacent is_aborted_or_ended() path and ctx.ref_() hasn't run yet; in do_render_stream this.ref_() is at line 2178 (after the guard), and set_has_marked_pending only suppresses the caller's auto-render (RequestContext.rs:793/974) — no extra ref is expected.

Extended reasoning...

All four issues from the previous review pass were addressed in 4dafa97/97822300. This run's only new finding is a nit about the third test's comment overstating what ordering it guards. I additionally traced the two new has_exception() early-return sites for ref-count imbalance and confirmed neither runs after a ref_(); the has_marked_pending flag set in the do_render_stream guard is read only as a suppress-auto-render gate, not as a ref-held indicator. Deferring rather than approving because the change sits at the intersection of JSC VMTraps/TerminationException state and Bun.serve request-context teardown, which warrants a maintainer look.

Comment thread test/js/web/workers/worker-terminate-lifetime.test.ts

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All four earlier concerns are addressed (clear-after-on_exit ordering, on_response sibling guard, dead FFI shim removed, load-bearing mid-JS-terminate exit-handler check via sab[1]). No new issues found.

Checked: the on_response early return sits alongside the existing is_aborted_or_ended() bare-return, so no cleanup is skipped; do_render_stream's guard mirrors the Pending arm's on_first_write/ctx/has_marked_pending handling; clear_termination_exception() is placed after on_exit() so dispatchExitInternal's guard still trips, and before close_all_socket_groups so on_close callbacks can re-enter JS.

Deferring to a human given this touches the JSC termination-exception state machine and worker-shutdown ordering.

Extended reasoning...

Overview

Fixes a debug/ASAN-only SIGABRT when worker.terminate() lands mid-drain_microtasks() while Bun.serve is streaming a type:"direct" body. Two has_exception() early-returns are added in RequestContext (do_render_stream, on_response) after their drain_microtasks() calls; WebWorker::shutdown now calls clear_termination_exception() after on_exit() (was clear_has_termination_request() before it, which was a no-op on the flag and left the sticky exception for close_all_socket_groups to trip on). The dead VM::clear_has_termination_request wrapper and its JSC__VM__clearHasTerminationRequest FFI shim are removed. Three regression tests added.

Security risks

None. No user-controlled input parsing, auth, or crypto is touched. The change only affects control flow when a worker VM is already terminating.

Level of scrutiny

High. The change is small in line count but sits at the intersection of the JSC VMTraps/TerminationException state machine, cross-thread worker shutdown ordering, and HTTP request-context lifecycle. Getting the clear_termination_exception() placement wrong changes observable Node-compat behavior (whether process.on('exit') fires on terminate()), and getting the RequestContext early-return cleanup wrong could leak per-request state. Both were the subject of earlier review rounds and are now covered by load-bearing tests, but a maintainer familiar with the VMEntryScope/executeEntryScopeServicesOnExit/DeferTermination interaction should confirm the state-machine reasoning.

Other factors

All four inline findings from my earlier passes were addressed across commits 4dafa97, 9782230, f56cd1c, and 6717d6b. The author verified the mid-JS-terminate process.on('exit') test is load-bearing by locally swapping the clear before on_exit() and observing the test fail. The PR description's verification section reports the new test failing 5/5 on main and passing 5/5 with the fix, plus 200+ passing tests across the affected serve/stream/worker suites. The related PR #36331 covers the non-direct stream sibling. Comment-cop flags are all resolved.

@robobun

robobun commented Jul 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

CI on 65d6b7d (build 85636, post-rebase): debian-13 x64-asan lane, where the new skipIf(!(isDebug || isASAN)) tests run, passed. The one non-flaky failure is test/napi/napi.test.ts "napi_wrap has the right lifetime" on windows-aarch64 (GC-timing, also failing on main build 85630 and unrelated to this diff; filed to main-break triage). Everything else is tagged flaky (passed alone or on retry). Diff is ready for review.

robobun and others added 6 commits July 30, 2026 06:40
…ct" body

A Worker running Bun.serve() that streams a type:"direct" ReadableStream
body, terminated mid-stream, SIGABRTed debug/ASAN builds with

  ASSERTION FAILED: vm.hasTerminationRequest()
  VMTraps.cpp(539) : void JSC::VMTraps::deferTerminationSlow(DeferAction)

do_render_stream drives the async pull() loop via drain_microtasks(). When
the termination trap fires inside it, the outermost VMEntryScope's
executeEntryScopeServicesOnExit clears hasTerminationRequest while the
TerminationException stays pending. do_render_stream then kept going and
lazy-initialized NativePromiseContextStructure; LazyProperty::callFunc
opens a DeferTerminationForAWhile scope unconditionally, whose
deferTerminationSlow asserts hasTerminationRequest() whenever a
termination exception is pending.

WebWorker::shutdown had the same inconsistency: it cleared only the
request flag before on_exit / close_all_socket_groups, leaving the sticky
exception in place.

Fix:
- do_render_stream: bail out after drain_microtasks() when an exception is
  pending instead of re-entering JS; worker shutdown tears down the sink.
- drainMicrotasks: re-set hasTerminationRequest before returning the
  termination signal so the DeferTermination invariant holds for any
  caller that proceeds to touch a LazyProperty.
- WebWorker::shutdown: clear the pending TerminationException along with
  the request flag so on_exit handlers and close_all_socket_groups can
  re-enter JS.
…drainMicrotasks change, cover on_response sibling, delete dead clear_has_termination_request
@robobun
robobun force-pushed the farm/8dfbc553/worker-terminate-direct-stream-defertermination branch from 6717d6b to 65d6b7d Compare July 30, 2026 06:50
Comment thread src/runtime/server/RequestContext.rs

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No new findings on 2183b9d — all four prior review threads (on_response/on_node_http_request siblings, dead clear_has_termination_request wrapper, clear_termination_exception ordering vs on_exit, load-bearing exit-handler test) are addressed. Deferring to a human because the change sits at the intersection of JSC termination invariants (VMTraps/DeferTermination), worker shutdown ordering, and HTTP request-context resource release across three early-return sites.

What was reviewed:

  • The three new has_exception() guards return before any then2/NativePromiseContext::create, and each releases what was acquired above (strong_promise.deinit(), sink on_first_write/ctx cleared).
  • WebWorker::shutdown now runs on_exit() before clearing the termination exception, so dispatchExitInternal still skips process.on('exit') on both the mid-JS and idle terminate paths; the async-fetch test's sab[1] counter locks that ordering.
  • clear_has_termination_request and its FFI shim have no remaining callers.
Extended reasoning...

Overview

Fixes a debug/ASAN-only SIGABRT (ASSERT(vm.hasTerminationRequest()) in VMTraps::deferTerminationSlow, and assertNoException in executeCallImpl) when a Worker running Bun.serve is terminated mid-stream/mid-drain. Touches src/jsc/web_worker.rs (shutdown ordering), src/runtime/server/RequestContext.rs (do_render_stream, on_response), src/runtime/server/mod.rs (on_node_http_request_with_upgrade_ctx), deletes the now-dead VM::clear_has_termination_request + its FFI shim, and adds four test cells to worker-terminate-lifetime.test.ts.

Security risks

None identified. This is a lifecycle/assertion fix; no user-input parsing, auth, or crypto is touched.

Level of scrutiny

High. The change reasons about JSC's termination state machine (trap bit vs hasTerminationRequest flag vs the sticky TerminationException), reorders worker shutdown so dispatchExitInternal still trips on terminate (Node compat), and inserts three early-return paths into HTTP request handling — each of which must not leak refs or leave the request context in a state that survives close_all_socket_groups. The do_render_stream guard clears on_first_write/ctx and sets has_marked_pending, mirroring the Pending arm's shape; the on_response guard sits before any state mutation (same position as the existing is_aborted_or_ended() bail); the mod.rs guard deinit()s strong_promise and returns Success, which falls through to the abort-handler wiring without re-entering JS (needs_to_drain was already flipped false). These look consistent, but a maintainer familiar with the request-context refcount model should confirm the do_render_stream early return leaves the sink in a state that shutdown's socket-group teardown fully reclaims.

Other factors

This PR went through three review iterations; every prior finding (sibling coverage in on_response and on_node_http_request_with_upgrade_ctx, dead-code deletion, skipIf(!(isDebug || isASAN)) gate, the clear_termination_exception-before-on_exit regression, and the non-load-bearing idle test) has been addressed with a corresponding commit and test. CI on the prior commit was green on the release-asan lane. The remaining reason not to auto-approve is domain complexity, not any concrete unresolved concern.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants