Skip to content

thread pool: report a pool that cannot spawn any worker instead of hanging - #37738

Open
robobun wants to merge 5 commits into
mainfrom
farm/5faeb243/thread-pool-spawn-failure
Open

robobun wants to merge 5 commits into
mainfrom
farm/5faeb243/thread-pool-spawn-failure

Conversation

@robobun

@robobun robobun commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • At the process or thread limit (EAGAIN from pthread_create under a cgroup pids limit or RLIMIT_NPROC), bun build, Bun.build(), bun install and the first fs.promises / zlib / dns call hang forever with no message. Reproduced on 1.4.0 and on main.
  • Cause: when a pool's first worker fails to spawn, the tasks stay queued and the pool's "a spawn was requested" latch stays set, so no later schedule() tries to spawn again and nothing ever drains the queue.
  • A pool that already has one worker is unaffected: workers never exit before shutdown, so the next idle one drains the queue. The bug is exactly "the pool never got a worker".

Fix

  • Warming a pool now returns the errno if it leaves the pool with no worker. The bundler is the only consumer that warms before scheduling, so bun build (with or without --watch) and the dev server print Failed to create a worker thread for the bundler: EAGAIN. The process or thread limit may have been reached (...) and exit 1, and Bun.build() rejects with AggregateError: Bundle failed carrying that message.
  • Every other pool is fed by fire-and-forget schedule() with no error channel, so a failed spawn that leaves no worker while tasks are already queued exits the process with EAGAIN: failed to create a worker thread for the thread pool; the work already queued on it could never run. Nothing else can ever run those tasks. The ulimit -u / pids hint is added only for EAGAIN.
  • "No worker left" is the count returned when the failed slot is released, so a worker another thread spawned meanwhile counts. A pool's first worker is retried once after 10ms; later failures only cost parallelism and are not retried.
  • Verification, per the original: a new Linux test uses an LD_PRELOAD shim that fails pthread_create for pool-sized threads; nine of its ten cases hang unfixed, all ten pass fixed. The errno mapping helper has a unit test, its Windows half run under miri only. The evidence block below records that the test did not run in the final check here and is left to CI.

Background

  • bun_threading::ThreadPool backs the runtime's WorkPool, the bundler's pools and bun install's pool. Workers are spawned lazily: schedule() publishes the tasks first, then sets a latch and asks for a thread; once the latch is set, later schedule() calls assume someone will drain the queue.
  • warm(n) pre-spawns up to n workers before any task exists. Only the bundler uses it, which is why the bundler can get a returned error and every other consumer gets a process exit.
  • The pool's wait group counts tasks published but not finished; non-zero at the moment a spawn fails means work is already stranded.
  • On Windows a std::io::Error carries a Win32 code, not an errno, so it needs a different mapper; the file watcher had this mapping inline and now shares the new SystemErrno::from_io_error with the pool.
  • bun build --watch normally reacts to a build error by parking until the file watcher triggers a rebuild. The watcher is installed after the pool is started, so this error has to skip that path or the watch build would hang.

no test proof · iteration 0 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/js/bun/sys/thread-pool-spawn-failure.test.ts

Original description

What

When the OS refuses to create a thread (EAGAIN from pthread_create at the cgroup pids limit or RLIMIT_NPROC), a bun_threading::ThreadPool that has no worker yet hangs whatever is waiting on it, forever and silently:

# as root in a container with the cgroup v2 pids controller: fill pids.current to
# pids.max - 1 with sleeping threads, then (as a non-root user `ulimit -u` does the same)
bun build entry.js        # hangs: Threads: 1, main thread in epoll_pwait2 with no timeout

The same happens to Bun.build(), to bun install (tarball extraction runs on its pool) and to the first use of the runtime's WorkPool (fs.promises.readFile, zlib, dns, ...). Reproduced on 1.4.0 and on a debug build of main with an LD_PRELOAD shim that fails pthread_create for threads asking for the pool's worker stack size; that shim is what the new test uses.

Cause

ThreadPool::warm() and notify_slow() (src/threading/ThreadPool.rs) handled a failed std::thread::Builder::spawn by undoing the spawned increment and returning. The batch that triggered the spawn stays in the run queue, and because notify_slow had already set the notified latch, every later schedule() takes the fast path and never tries to spawn again. With at least one worker in the pool this is harmless (workers never exit before shutdown, and the next one to go idle consumes the latch and drains the queue), so the bug is exactly "the pool could not get its first worker". The upstream zap code this is derived from has the same shape.

Fix

  • warm() returns the errno when a failed spawn leaves the pool with no worker at all, judged by the count the slot release itself returns (so a worker another thread managed to spawn meanwhile counts; failing to add more workers to a working pool still just ends the warm-up). The bundler is the one consumer that warms before scheduling, so BundleV2::init checks it, adds the reason to the log and fails the build with a new bun_bundler::Error::ThreadSpawnFailed (same name as the existing variants in bun_runtime and bun_jsc):
    • bun build: error: Failed to create a worker thread for the bundler: EAGAIN. The process or thread limit may have been reached (ulimit -u, or the container's pids limit); raise it or reduce concurrency, exit code 1. The second sentence comes from thread_limit_hint() and is only added for EAGAIN; ENOMEM (what a failed CreateThread maps to on Windows), EPERM or EINVAL just name the errno, since the ulimit / pids advice does not apply to them.
    • bun build --watch: the same. This failure happens before init installs the watcher, so build_command must not take its usual "park and let the watcher restart us" path for it; under a real thread limit that would turn the baseline's clean watcher-spawn failure (Failed to start File Watcher: EAGAIN, exit 1) into a hang. The dedicated variant is what lets build_command tell this apart from the build errors that should keep a watch session alive.
    • Bun.build(): rejects with the usual AggregateError: Bundle failed carrying that message
    • dev server: its callers treat every start_async_bundle error as OOM (.expect("oom")), so DevServer prints the log message and exits 1 for this variant instead of producing an oom crash report with the reason never shown.
  • Both spawn sites now share spawn_worker(). When a failure leaves the pool with no worker (the count comes from the same fetch_sub that releases the slot, so a spawn that succeeded concurrently on another thread is respected) while tasks are already queued (wait_group is non-zero), the process exits:
    EAGAIN: failed to create a worker thread for the thread pool; the work already queued on it could never run
    note: The process or thread limit may have been reached (ulimit -u, or the container's pids limit); raise it or reduce concurrency
    
    (the note: line again only for EAGAIN). This covers bun install, WorkPool and each() / wait_for_all() callers.
  • The first worker of a pool is retried once after 10ms, the same remedy Watcher::start already uses for the same error; later spawns are not retried since their failure only costs parallelism.
  • SystemErrno::from_io_error holds the io::Error -> errno mapping (Win32 code on Windows) that Watcher::start had inline, so both users share it. The only behavior difference for the watcher is the fallback for an OS code without a SystemErrno mapping (EIO before, EAGAIN now); pthread_create only returns EAGAIN, EINVAL and EPERM, so it is not reachable. It gets a unit test next to the existing bun_errno ones (a non-EAGAIN errno, a non-OS error, and the Win32 codes on Windows; the Windows half was run with cargo miri test -p bun_errno --target x86_64-pc-windows-msvc, since no CI lane executes this crate's cfg(windows) tests).
  • unregister() is split into unspawn() (the accounting, now also used by the failed-spawn path and returning the remaining worker count) and the worker-exit wait chain; its thread argument becomes NonNull since nothing passes null anymore.

Why exit instead of returning an error from schedule()

schedule() is fire-and-forget: the tasks are already published when the spawn is attempted, the 40-odd WorkPool call sites have no error channel, and the waiters (the bundler's and install's event loops, wait_for_all(), a JS promise) are on the other side of a queue nobody will ever drain. Exiting with the errno and the limit named is the same treatment the HTTP client thread and bun install's startup failures already get, and it is what a process in this state does anyway on its next fetch() or GC (#36985 covers the crash-report presentation of those). The bundler gets the recoverable path because it is the one consumer that can find out before it schedules anything, and Bun.build() in a running process is where a clean rejection matters.

Tests

test/js/bun/sys/thread-pool-spawn-failure.test.ts (Linux, needs a C compiler like the existing LD_PRELOAD tests) compiles a shim that fails pthread_create (with EAGAIN, or an errno the test picks) for threads requesting DEFAULT_THREAD_STACK_SIZE, following a per-attempt plan:

  • persistent failure: bun build and bun build --watch (exit 1 + message), bun build and bun install with EPERM injected (the messages name EPERM, so the mapping and not the fallback is what is shown, and neither carries the limit hint), Bun.build() (rejection carrying the message, process exits 0 on its own), dev server (HTML route requested from the test process: message + exit 1), fs.promises.readFile and bun install of a local tarball (both: the two stderr lines above, exit 1)
  • transient failure: the first spawn fails, the retry succeeds, readFile completes
  • a pool that got one worker and then loses every further spawn still bundles (guards the "no worker left" condition of the exit)

On the unfixed build the first nine hang and are killed; with the fix all ten pass. The existing watcher spawn-failure test in test/cli/watch/watch.test.ts still passes; cargo check of the touched crates also passes for the x86_64-pc-windows-msvc and aarch64-apple-darwin targets.

…nging

When std::thread::Builder::spawn failed (EAGAIN at a pids / RLIMIT_NPROC
limit), ThreadPool::warm() and notify_slow() only undid the `spawned`
increment. With no worker in the pool the queued tasks were never run and
nothing retried, so `bun build`, Bun.build(), `bun install` and every
WorkPool user (node:fs promises, zlib, dns) waited forever.

- warm() returns the spawn error when the pool could not get its first
  worker. The bundler checks it in BundleV2::init and fails the build with
  the reason in the log, so `bun build` exits 1 with an error message,
  Bun.build() rejects, and the dev server prints the reason and exits
  instead of tripping its callers' OOM expects.
- A failed spawn that leaves the pool without a worker while tasks are
  already queued exits the process with the errno and a note about the
  limit: schedule() is fire-and-forget, so there is nobody to hand the
  error to, and whoever waits on those tasks would otherwise hang.
- The pool's first worker is retried once after 10ms, as Watcher::start
  already does, since EAGAIN from pthread_create is often momentary.
- SystemErrno::from_io_error replaces the io::Error -> errno mapping that
  was inlined in Watcher::start, so both spawn sites share it.
@coderabbitai

coderabbitai Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c75be0f4-31f1-444a-9c93-80966d31973e

📥 Commits

Reviewing files that changed from the base of the PR and between f59f705 and 0ae50e1.

📒 Files selected for processing (10)
  • src/bundler/ThreadPool.rs
  • src/bundler/bundle_v2.rs
  • src/bundler/error.rs
  • src/errno/lib.rs
  • src/runtime/bake/DevServer.rs
  • src/runtime/cli/build_command.rs
  • src/threading/ThreadPool.rs
  • src/threading/WaitGroup.rs
  • src/watcher/Watcher.rs
  • test/js/bun/sys/thread-pool-spawn-failure.test.ts

Walkthrough

Changes

Thread creation failures now propagate from worker pools through bundler initialization and runtime commands. The change adds errno conversion, cleanup, diagnostics, retry handling, stranded-task termination, and Linux regression tests.

Thread spawn failure handling

Layer / File(s) Summary
Pool failure detection and errno conversion
src/threading/ThreadPool.rs, src/threading/WaitGroup.rs, src/errno/lib.rs, src/watcher/Watcher.rs
Worker creation reports SystemErrno values, retries initial failures, handles stranded tasks, and uses non-null worker registration. I/O error conversion and pending-task tracking were added.
Bundler startup error propagation
src/bundler/ThreadPool.rs, src/bundler/bundle_v2.rs, src/bundler/error.rs
Bundler startup now returns warm-up errors, deinitializes failed pools, logs thread-limit hints, and returns Error::ThreadSpawnFailed.
Runtime handling of startup failures
src/runtime/bake/DevServer.rs, src/runtime/cli/build_command.rs
The development server exits on ThreadSpawnFailed. Watch mode does not retain this failure path.
Spawn failure regression coverage
test/js/bun/sys/thread-pool-spawn-failure.test.ts
Linux tests inject pthread failures and verify diagnostics, exit statuses, retry behavior, and operation across build, server, filesystem, and install flows.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly describes the primary fix: reporting thread-pool worker-spawn failures instead of hanging.
Description check ✅ Passed The description explains the problem, fix, affected behavior, design rationale, and verification coverage in detail.

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: fixed and green; ready for a maintainer.

Reproduced the hang on bun 1.4.0 and on a debug build of main with an LD_PRELOAD shim that fails pthread_create for threads requesting the pool's worker stack size (the same mechanism the new test uses): bun build entry.js, Bun.build(), bun install of a local tarball and fs.promises.readFile all sat in the event loop forever. In test/js/bun/sys/thread-pool-spawn-failure.test.ts nine cases hang on the unfixed build (children killed on timeout) and all ten pass with the fix.

CI for the current head 0ae50e1 (build 93238): 192 jobs passed, no test failures (the seven annotations are flakes that passed on retry or alone, none in this area). The build shows as failed only because the two darwin 26 aarch64 - test-bun jobs expired without ever starting; that lane has been doing the same on every build in the pipeline for the last several hours, including main, so re-triggering would not change anything. The first run (92819) ended the same way, and the darwin 14 aarch64 lanes did run and pass this time.

All review threads are addressed and resolved (--watch exit, warm() deciding on the remaining worker count, the EAGAIN-only hint, from_io_error unit test). #37753 adds the same from_io_error and Watcher.rs hunks byte for byte, so the two merge cleanly in either order.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. Because it reworks the lock-free spawn/unspawn accounting in bun_threading::ThreadPool (including a Release→AcqRel ordering change and a new safety contract on unspawn) and introduces a process-exit path reachable from every WorkPool/bun install schedule, a human look at the concurrency reasoning and the exit-vs-propagate design choice would be worthwhile.

Checked: warm()'s only callers are the two updated bundler sites; tp.deinit() on the new early-return path releases the io-pool ref and (when owned) the boxed worker pool before this drops; the unregister→unspawn split preserves the "don't touch *pool past join_event.notify()" invariant; the Watcher refactor's only behavior change is the unreachable unmapped-code fallback (EIO→EAGAIN) as noted in the description.

Extended reasoning...

Overview

This PR fixes a hang when pthread_create fails (e.g. cgroup pids limit) before a bun_threading::ThreadPool has any worker. It touches the core lock-free thread pool (src/threading/ThreadPool.rs: new spawn_worker/unspawn/exit_with_stranded_tasks, warm() now returns Result, unregister split and its thread argument tightened to NonNull), WaitGroup (new pending() accessor), the bundler's pool wrapper and BundleV2::init (propagate the warm-up error into the log and fail cleanly), DevServer (exit 1 with the logged reason instead of the callers' .expect("oom") crash report), a new shared SystemErrno::from_io_error in src/errno/lib.rs (deduplicating what Watcher::start had inline), and a Linux-only LD_PRELOAD test that drives seven scenarios.

Security risks

None identified. No untrusted input parsing; the change is thread-creation error handling and process-exit messaging.

Level of scrutiny

High. src/threading/ThreadPool.rs is derived-from-zap lock-free code with a bespoke packed Sync word and careful memory-ordering comments. This PR changes an ordering (fetch_sub in unspawn from Release to AcqRel, with a stated pairing rationale), splits a function with a documented "pool may be deallocated after this returns" safety contract, and adds a new condition (workers_left == 0 && wait_group.pending() > 0) that races against concurrent spawns and schedules by design. The author's reasoning (using the same fetch_sub that releases the slot so a concurrently-succeeding spawn is respected) reads correctly to me, but this is exactly the kind of change a maintainer familiar with the pool's invariants should confirm.

The PR also makes a deliberate architectural choice — schedule() failures with no worker exit the process rather than propagate — and defends it (fire-and-forget callers, no error channel, matches the HTTP thread's treatment). That is reasonable but is a design decision a human should sign off on.

Other factors

  • warm() signature change: grepped for all callers; only the two bundler sites, both updated.
  • Cleanup on the new BundleV2::init early return: tp.deinit() frees the owned worker_pool box and releases the io-pool refcount; this (a Box<BundleV2>) drops normally on Err, and the watcher is installed only after this point per the updated comment.
  • unregister/unspawn split: the "── *pool may be invalid past this point ──" boundary is preserved; spawn_worker holds &self so its post-unspawn wait_group.pending() read is covered by the documented caller exception.
  • Watcher refactor: pure dedup into from_io_error; the only semantic difference (unmapped-OS-code fallback EIO→EAGAIN) is called out and unreachable for pthread_create.
  • Tests cover the failure, retry, and one-worker-survives paths across five entry points, are Linux+cc-gated, sequential (hang regression), and use a spawn-plan shim keyed on DEFAULT_THREAD_STACK_SIZE — a coupling the test file documents. One assertion matches bun install v1. in stdout, which will need updating on a major-version bump; flagged by a finder and judged acceptable.

@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

For whoever looks at the concurrency side, the argument behind the workers_left == 0 && wait_group.pending() > 0 check in spawn_worker, since it is not spelled out in the description:

  • spawned counts live workers plus spawns in flight, and a worker only leaves the count at shutdown. So the value returned by the fetch_sub that releases our slot is exact at that instant: 0 means no worker exists and nobody else is in the middle of creating one.
  • Two threads both failing on a fresh pool (one in warm(), one in notify_slow()): whichever releases its slot first sees 1 and returns quietly; the second sees 0. If the first one had queued tasks, they are counted in wait_group (the add happens before the push in schedule_impl), so the second one exits instead of returning an error that would leave those tasks stranded. The AcqRel on the fetch_sub is what makes the first thread's wait_group.add() visible to the second one; that is the only reason the ordering changed.
  • If the other thread's spawn succeeds instead, the failing thread sees 1 and returns quietly, and the new worker drains everything. The only spurious outcome left is a thread whose own two attempts failed exiting just as a third party's attempt would have succeeded, which still only happens while the limit is actually being hit.
  • A notify_slow caller always has work queued (it is only reached from schedule_impl or from a worker, and a worker holds a slot, so it never sees 0), so for it the check reduces to "no worker left". A warm() caller with nothing queued gets the error back instead.

On the exit itself: with the pool at zero workers the alternative is not an error somewhere, it is the existing behavior, a wait that never ends. The bundler gets the recoverable version because it is the one caller that asks before it has queued anything.

@robobun

robobun commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 12:05 PM PT - Aug 12th, 2026

❌ @robobun, your commit 0ae50e1 has some failures in Build #93238 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 37738

That installs a local version of the PR into your bun-37738 executable, so you can run:

bun-37738 --bun

…e errno mapping

BundleV2::init now fails with its own Error::ThreadSpawnFailed (the errno
stays in the log). That failure happens before the CLI watcher is
installed, so build_command must exit for it even under --watch instead
of parking until a watcher that does not exist restarts the process; the
dev server matches the same variant instead of the broad Sys(_).

Add a bun_errno unit test for SystemErrno::from_io_error (a non-EAGAIN
errno, a non-OS error, and the Win32 codes on Windows) and let the test
shim inject a chosen errno so one end-to-end case shows the mapped value
rather than the EAGAIN fallback in the message.
Comment thread src/bundler/ThreadPool.rs Outdated
Comment thread src/bundler/bundle_v2.rs Outdated
Comment thread src/bundler/error.rs Outdated
Comment thread src/errno/lib.rs Outdated
Comment thread src/errno/lib.rs Outdated
Comment thread src/errno/lib.rs Outdated
Comment thread src/runtime/bake/DevServer.rs Outdated
Comment thread src/runtime/cli/build_command.rs Outdated
Comment thread src/threading/ThreadPool.rs Outdated
Comment thread src/threading/ThreadPool.rs Outdated
Comment thread src/threading/ThreadPool.rs Outdated
Comment thread src/threading/ThreadPool.rs Outdated
Comment thread src/threading/ThreadPool.rs Outdated
Comment thread src/threading/ThreadPool.rs Outdated
Comment thread src/threading/ThreadPool.rs Outdated
Comment thread src/threading/ThreadPool.rs Outdated
robobun added a commit that referenced this pull request Aug 12, 2026
…awns

The bundle thread's spawn failure and Watcher::start() each mapped a
std::io::Error to a SystemErrno by hand, with the Win32 special case
duplicated. One helper in bun_errno does it; both callers pick EAGAIN for an
error without a usable code. (Same helper as #37738, so either PR can land
first.)

singleton::enqueue no longer copies the bundle thread's dequeue protocol for a
build that never reached the queue: nothing else can have claimed such a build,
so try_start() succeeding is asserted, and the free_released_unstarted branch,
which would have freed a task the caller is still setting up, is gone. The
thread-limit hint is appended for every errno, matching the pool's message.
Comment thread src/threading/ThreadPool.rs Outdated
@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

FYI from #37753 (the bundle thread's spawn failure): it adds the same SystemErrno::from_io_error and the same Watcher.rs change as this PR, byte-identical, so whichever lands second should merge cleanly. One difference worth considering here: #37753 only appends the "process or thread limit may have been reached (ulimit -u, ...)" sentence for EAGAIN, because on Windows a failed thread creation maps to ENOMEM (ERROR_NOT_ENOUGH_MEMORY), where that advice does not apply; the pool message in bundle_v2.rs here appends it for every errno.

…led spawn

warm() returned the error whenever the slot it took had been the pool's
first, but by the time the spawn failed (including its retry sleep) another
thread may have spawned a worker. Use the count unspawn() returned, the
same value the stranded-tasks exit already uses, so a pool that did get a
worker is reported as working.
Comment thread src/bundler/bundle_v2.rs Outdated
Only EAGAIN means a process or thread limit was hit; ENOMEM (what a failed
CreateThread maps to on Windows), EPERM and EINVAL would get advice about
ulimit and pids limits that does not apply. One helper in bun_threading now
supplies the sentence for both the stranded-tasks exit and the bundler's
log message, and the EPERM test cases assert that it is left out.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR again after 0ae50e1 and found no bugs. Both earlier findings (the stale pool_was_empty check in warm(), and the unconditional ulimit hint) are addressed — warm() now decides on SpawnFailed::workers_left from unspawn(), and thread_limit_hint() gates the remedy on EAGAIN at both call sites with test coverage for the EPERM case. Because this touches the lock-free pool's atomic ordering (Release→AcqRel on unspawn's fetch_sub) and introduces a process-exit path inside the schedule flow, a maintainer look at the design would still be worthwhile.

Also checked: bundler::ThreadPool::deinit() on the new early-return path in bundle_v2.rs — it only frees the owned worker pool and releases the io_pool refcount, so calling it before graph.pool is set is fine; and the Watcher::start errno-mapping refactor is behavior-preserving apart from the documented unreachable fallback.

Extended reasoning...

Overview

The PR fixes a hang when pthread_create fails (typically EAGAIN at RLIMIT_NPROC/cgroup pids limit) before a bun_threading::ThreadPool has any worker: previously the batch stayed queued forever with the notified latch set. The fix consolidates the two spawn sites (warm, notify_slow) into spawn_worker(), which on failure releases the slot via a new unspawn() (split from unregister), and either returns SpawnFailed { errno, workers_left } or, when workers_left == 0 and tasks are already queued (wait_group.pending() > 0), exits the process with a diagnostic. warm() propagates the error only when workers_left == 0, letting the bundler (BundleV2::init) fail cleanly with a new Error::ThreadSpawnFailed before scheduling anything. Downstream: build_command skips the watch-park for this variant, DevServer prints the log and exits instead of .expect("oom")-crashing, and SystemErrno::from_io_error is factored out of Watcher::start for reuse. A Linux-only LD_PRELOAD-shim test exercises nine scenarios across bun build, --watch, Bun.build(), dev server, fs.promises.readFile, bun install, EPERM mapping, transient-failure retry, and single-worker degradation.

Security risks

None identified. The change is error-path handling for thread-creation failure; no untrusted input parsing, no auth/crypto, no new attack surface. The LD_PRELOAD test shim is Linux-only test infrastructure that the child process opts into via its own env.

Level of scrutiny

High. src/threading/ThreadPool.rs is a lock-free work-stealing pool derived from zap; the diff changes an atomic ordering (fetch_sub on sync from Release to AcqRel, justified by needing to observe another failing spawner's wait_group.add()), splits unregister into unspawn+shutdown-chain, tightens unregister's thread argument to NonNull, and adds a Global::exit(1) reachable from any schedule() caller. The concurrency argument for workers_left == 0 && pending() > 0 is subtle enough that the author posted a separate PR comment walking through the interleavings. My earlier review caught a real race in warm() (stale pre-CAS check), which was fixed correctly by threading workers_left through SpawnFailed.

Other factors

  • Design decision needing sign-off: exiting the process from inside spawn_worker (rather than surfacing an error from schedule()) is deliberate and well-argued in the description — the ~40 WorkPool call sites have no error channel and the alternative is the current infinite hang — but it is the kind of choice a maintainer should confirm.
  • Prior review addressed: both my earlier inline findings are resolved in 0ae50e1/5ea19a5 and covered by the updated test (EPERM cases assert the hint is absent; the sf case exercises the workers_left > 0 branch).
  • Test quality: the LD_PRELOAD shim keys on DEFAULT_THREAD_STACK_SIZE to target only pool workers (with a documented exception for the HTTP client thread handled via the plan string), asserts full {stdout, stderr, exitCode, signalCode} objects with signalCode: null proving no timeout kill, and the negative case (sf build succeeds with one worker) guards against over-eager exits.
  • Cross-PR overlap: #37753 shares SystemErrno::from_io_error and the Watcher.rs change byte-for-byte, noted in the timeline; whichever lands second should merge cleanly.

robobun added a commit that referenced this pull request Aug 12, 2026
get() hands the io::Error back instead of collapsing an OS code bun has no
errno name for into EAGAIN, which on Windows is most failed thread creations
and which re-enabled the ulimit hint that is only meant for Unix's EAGAIN.
The log now names the errno when there is one (with the hint on Unix for
EAGAIN) and otherwise carries the OS's own description of the code.

src/errno/lib.rs is brought back in line with the same change in #37738,
which now also has a unit test for from_io_error.

Tests: an unnamed code, and a VM torn down (BUN_DESTRUCT_VM_ON_EXIT) while a
failed build's completion is still queued, which is the case the try_start()
call in enqueue exists for; without it that fixture hangs in teardown.
@robobun

robobun commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

One more data point for this PR, seen in the wild rather than with the shim: during bun bd in a container with a cgroup pids.max of 512, the node-fallbacks codegen step (src/node-fallbacks/build-fallbacks.ts, which runs about 24 bun build children concurrently while ninja and cargo are also spawning threads) never finished. One bun build ... os.js ... child (1.4.0 release build of main, linux x64) was left behind with exactly the state described here:

  • utime and stime of 0, RSS 11 MB, output file never written
  • 2 threads total (main plus mi-scavenger), no pool workers
  • main thread parked in ep_poll with only the loop's eventfd left to wake it; it stayed there after the parent died and the stdio socket peers were closed
  • the cgroup's pids.events showed max 9 for that window, and re-running the same ninja edge alone completed in 0.6s

So the path hit there is the bundler's start() (warm(8) on the worker pool) losing its first worker to EAGAIN, followed by notify_slow leaving the notified latch set, which is the warm() -> error -> exit 1 case this PR adds. With that in place the child exits non-zero and Bun.$ in build-fallbacks.ts turns it into a build failure instead of a silent hang, so no separate timeout there seems necessary.

@robobun

robobun commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

#39864 adds SystemErrno::from_io_error in the same shape as this PR's version, and moves Watcher.rs onto bun_threading::spawn_with_retry, which uses it. A rebase of this PR can drop the src/errno/lib.rs and src/watcher/Watcher.rs hunks. The thread pool change itself still applies.

@robobun

robobun commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Another occurrence of this hang, from stress runs of test/js/node/test/parallel/test-https-timeout.js. The runs used 48 debug processes in parallel, in a container with a cgroup pids.max of 512.

  • Some runs hang in dns.lookup("localhost"). The native log shows [getaddrinforequest] init, but never [sys] getaddrinfo(localhost, 0). Passing runs log that line about 0.2 ms later.
  • In a hung process, all sockets are closed. The main thread sits in epoll_wait, and the pending lookup keeps the loop alive.
  • The lookup runs on WorkPool (src/runtime/dns_jsc/dns.rs:168, then src/jsc/job.rs:334). Other processes in the same run failed pthread_create at the same time. This matches the notify_slow case that this PR handles.

This PR now conflicts with main. #38604 removed ThreadPool::wait_group, and spawn_worker reads self.wait_group.pending(). So a rebase also needs another way to know that tasks are queued. One option is the notified bit in the Sync value that the fetch_sub in unspawn returns. Before shutdown, with no worker left, only schedule() sets that bit, and only a worker clears it.

@robobun

robobun commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator Author

Issue #43354 reports this hang again, for bun install --lockfile-only at a cgroup v2 pids limit. I confirmed it on main (367d939) with the cgroup repro from the issue and with an LD_PRELOAD shim that fails pthread_create.

A smaller version of the same fix is on branch robobun/d5718ee8/thread-pool-spawn-fail (commit 1762dbf): ThreadPool::unregister returns the previous Sync state, and notify_slow exits with the spawn error when the failed spawn leaves the pool with no worker. The test covers bun build and bun install. This PR already covers that case, so I am not opening a second PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant