Skip to content

fix(install): optimize isolated linker to avoid O(N²) complexity in resumeUnblockedTasks - #25983

Closed
robobun wants to merge 1 commit into
mainfrom
claude/fix-isolated-install-performance
Closed

robobun wants to merge 1 commit into
mainfrom
claude/fix-isolated-install-performance

Conversation

@robobun

@robobun robobun commented Jan 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Fixes bun install hanging on Windows when using the isolated (symlink) linker with large monorepos
  • Adds a blocked_entries hashmap to track which entries are currently blocked
  • Reduces complexity from O(N²) to O(B * D) where B is the number of blocked entries

Problem

When using the isolated linker (linker = "symlink" or auto mode with workspaces), bun install would hang indefinitely on large monorepos (~1500+ packages) on Windows, consuming 5-6GB+ of RAM with CPU usage dropping to near zero.

The root cause was in resumeUnblockedTasks() which was called after every task completion and iterated through ALL entries (O(N)) to find blocked tasks. With ~1500 packages, this resulted in approximately N² iterations (1500 task completions × 1500 entries = 2.25 million iterations), plus additional O(D * P) work for cycle detection per blocked task.

Solution

Instead of iterating through all entries on each task completion, we now:

  1. Track blocked entries in a hashmap (blocked_entries) when they become blocked
  2. Only iterate through this much smaller list in resumeUnblockedTasks()
  3. Remove entries from the hashmap when they are unblocked

This reduces the complexity from O(N² * D * P) to O(B * D * P) where B is the number of currently blocked entries (typically much smaller than N).

Test plan

  • Build succeeds
  • Existing tests pass (semver.test.ts, architecture-match.test.ts)
  • Manual testing with large monorepo on Windows (I don't have access to Windows)

Closes #25970

🤖 Generated with Claude Code

…esumeUnblockedTasks

The isolated (symlink) linker was hanging on Windows with large monorepos due to
O(N²) complexity in resumeUnblockedTasks. When a task completed, the function
iterated through ALL entries to find blocked tasks. With ~1500 packages, this
resulted in billions of operations.

The fix adds a blocked_entries hashmap to track only the blocked entries,
reducing the complexity from O(N²) to O(B * D) where B is the number of blocked
entries (much smaller than N) and D is the number of dependencies per entry.

Closes #25970

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@robobun

robobun commented Jan 12, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 3:44 AM PT - Jan 12th, 2026

❌ Your commit 386de73b has 2 failures in Build #34548 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 25983

That installs a local version of the PR into your bun-25983 executable, so you can run:

bun-25983 --bun

@coderabbitai

coderabbitai Bot commented Jan 12, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

Optimizes task resumption logic in the Installer by introducing a blocked_entries map to track currently blocked entries. Replaces inefficient full-scan approach in resumeUnblockedTasks with targeted iteration over only blocked entries, reducing complexity in large dependency graphs.

Changes

Cohort / File(s) Summary
Installer optimization
src/install/isolated_install/Installer.zig
Adds public blocked_entries field to track blocked entries; extends deinit for cleanup; updates onTaskBlocked to record blocked entries; replaces O(N) full-scan approach in resumeUnblockedTasks with targeted iteration over blocked entries subset, collecting unblockable entries and updating their step to symlink_dependency_binaries before restarting tasks.

Suggested reviewers

  • dylan-conway
  • taylordotfish
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The PR title directly describes the main change: optimizing isolated linker by avoiding O(N²) complexity in resumeUnblockedTasks, which matches the primary objective of fixing the performance hang.
Description check ✅ Passed The PR description addresses both template sections: clearly explains what the PR does (fixes hang, adds blocked_entries hashmap) and covers verification (build succeeds, existing tests pass), though manual Windows testing is marked as TODO.
Linked Issues check ✅ Passed The PR directly addresses issue #25970's objective of resolving the indefinite hang in bun install with the symlink linker on large monorepos by reducing algorithmic complexity from O(N²) to O(B*D).
Out of Scope Changes check ✅ Passed All changes are scoped to addressing the identified performance issue: blocked_entries hashmap addition, deinit extension, onTaskBlocked recording, and resumeUnblockedTasks optimization—all directly serve the complexity reduction objective.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In @src/install/isolated_install/Installer.zig:
- Around line 24-26: The field blocked_entries should be made private by
renaming it to #blocked_entries; update its declaration in Installer and replace
all internal references (e.g., in onTaskBlocked, resumeUnblockedTasks, deinit)
to use this.#blocked_entries so accesses compile and follow the repository's
private-field convention.
📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Disabled knowledge base sources:

  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between beccd01 and 386de73.

📒 Files selected for processing (1)
  • src/install/isolated_install/Installer.zig
🧰 Additional context used
📓 Path-based instructions (2)
**/*.zig

📄 CodeRabbit inference engine (CLAUDE.md)

In Zig code, be careful with allocators and use defer for cleanup

Files:

  • src/install/isolated_install/Installer.zig
src/**/*.zig

📄 CodeRabbit inference engine (src/CLAUDE.md)

src/**/*.zig: Use the # prefix for private fields in Zig structs, e.g., struct { #foo: u32 };
Use Decl literals in Zig, e.g., const decl: Decl = .{ .binding = 0, .value = 0 };
Place @import statements at the bottom of the file in Zig (auto formatter will handle positioning)
Never use @import() inline inside functions in Zig; always place imports at the bottom of the file or containing struct

Files:

  • src/install/isolated_install/Installer.zig
🧠 Learnings (1)
📚 Learning: 2026-01-05T16:32:07.551Z
Learnt from: alii
Repo: oven-sh/bun PR: 25474
File: src/bun.js/event_loop/Sigusr1Handler.zig:0-0
Timestamp: 2026-01-05T16:32:07.551Z
Learning: In Zig codebases (e.g., Bun), treat std.posix.sigaction as returning void and do not perform runtime error handling for its failure. The Zig standard library views sigaction failures as programmer errors (unreachable) because they only occur with invalid signals like SIGKILL/SIGSTOP. Apply this pattern across Zig files that call sigaction (e.g., crash_handler.zig, main.zig, filter_run.zig, process.zig) and ensure failures are not handled as recoverable errors; prefer reaching an explicit unreachable/compile-time assumption when such failures are detected.

Applied to files:

  • src/install/isolated_install/Installer.zig
🔇 Additional comments (3)
src/install/isolated_install/Installer.zig (3)

28-31: LGTM!

The blocked_entries hashmap is properly deallocated using the same allocator (this.lockfile.allocator) that is used for insertions in onTaskBlocked. Memory management is correct.


223-224: LGTM!

The blocked entry is correctly tracked after setting the step to .blocked. The put operation handles the case if the entry was somehow already present. OOM handling is consistent with the codebase patterns.


326-343: LGTM! Clean optimization with correct two-phase approach.

The implementation correctly:

  1. Collects unblocked entries first to avoid mutating blocked_entries during iteration
  2. Uses swapRemove for O(1) removal from the hashmap
  3. Properly deallocates the temporary to_unblock list via defer

This effectively reduces complexity from O(N) to O(B) per task completion, addressing the performance issue in large monorepos.

Comment on lines +24 to +26
/// Tracks which entries are currently blocked, avoiding O(N) iteration over all entries
/// in resumeUnblockedTasks. Only modified from the main thread.
blocked_entries: std.AutoArrayHashMapUnmanaged(Store.Entry.Id, void) = .empty,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider marking blocked_entries as private with # prefix.

The field is only accessed within internal Installer methods (onTaskBlocked, resumeUnblockedTasks, deinit). As per coding guidelines for this repository, private fields should use the # prefix.

♻️ Suggested change
     /// Tracks which entries are currently blocked, avoiding O(N) iteration over all entries
     /// in resumeUnblockedTasks. Only modified from the main thread.
-    blocked_entries: std.AutoArrayHashMapUnmanaged(Store.Entry.Id, void) = .empty,
+    #blocked_entries: std.AutoArrayHashMapUnmanaged(Store.Entry.Id, void) = .empty,

Note: This would require updating all references to use this.#blocked_entries.

🤖 Prompt for AI Agents
In @src/install/isolated_install/Installer.zig around lines 24 - 26, The field
blocked_entries should be made private by renaming it to #blocked_entries;
update its declaration in Installer and replace all internal references (e.g.,
in onTaskBlocked, resumeUnblockedTasks, deinit) to use this.#blocked_entries so
accesses compile and follow the repository's private-field convention.

@github-actions

Copy link
Copy Markdown
Contributor

Closing this PR because it has been inactive for more than 90 days.

@github-actions github-actions Bot closed this Apr 13, 2026
Jarred-Sumner added a commit that referenced this pull request Aug 14, 2026
…ilter/--catalog, nested overrides, transitive update, and workspace fixes (#38333)

Brings Bun's package manager to parity with pnpm for monorepo workflows,
and fixes the bugs found while checking every command against pnpm's
implementation, pnpm's test suites, pnpm's open issue tracker, pnpm's
docs, npm's arborist fixtures, and — for `bun update` — running real
pnpm and Bun side by side on the same projects.

### What does this PR do?

#### New commands

- **`bun dedupe [--check]`** — collapses duplicate versions in
`bun.lock` onto the smallest set that still satisfies every dependent's
range, using only versions already in the lockfile, then installs. Never
downgrades a direct dependency unless that is the only way to drop a
version; keeps patched versions (and anything needed to reach them, and
says so); refuses to run on a lockfile that is behind `package.json`.
`--check` exits 1 without writing.
- **`bun prune [--production | --omit=…] [--dry-run] [--filter <ws>]`**
— removes everything in `node_modules` that the lockfile does not put
there; `--production` leaves exactly what `bun install --production`
would. Hoisted and isolated layouts, Windows junctions and shims,
workspace links, bundled deps; refuses when `package.json` and
`bun.lock` disagree or when `node_modules` was laid out by a different
linker; understands turbo-pruned checkouts.
- **`bun pm licenses [--json] [--prod|--dev] [--long] [--filter <ws>]`**
— installed packages grouped by license, with a `(dev)` marker and
`paths`/`license`/`description` in `--json`.
- **`bun audit fix [--latest] [--dry-run] [--json]`** — moves each
vulnerable package to the lowest safe version its dependents accept, per
installed instance; rewrites exact pins when that is the only way;
`--latest` also rewrites your own declared ranges (root, workspace,
catalog) so a semver-major fix can be taken, and every blocked or
unfixable item is followed by the command that resolves it (`bun audit
fix --latest`, `bun audit --ignore GHSA-…`); re-audits the tree it
actually installed and reports/exits from that second response (npm's
`_submitQuickAudit`), so an advisory that starts at the version it moved
to is not missed; works across registries; security fixes bypass
`minimumReleaseAge` with an annotation. `bun audit --json` honors
`--audit-level`/`--ignore` for its exit code; `--omit` is honored by
`audit` and `licenses`.

#### `bun update` semantics (pnpm's model)

- A bare `bun update` re-resolves **transitive** packages too — every
edge moves to the newest version its own range (or dist-tag) allows, per
dependent, so `bun.lock` no longer stays stale after an update;
overrides/catalogs changed since the last install are honored. From a
workspace member or `--filter`, only what the selected workspaces reach
is re-resolved; from the root, everything.
- `bun update <name>` reaches any depth, matches `npm:` aliases by real
name, updates in place, never adds to `package.json`, and errors on a
name nothing selected depends on. `-r`/`--filter` fan a named update out
across workspaces. `--latest` never downgrades a locked version that is
ahead of the tag, and `update <name> --latest` also refreshes that
package's own dependencies.
- Plain updates keep dist-tag literals and non-caret ranges (`*`, `1.x`,
`^1 || ^2`) exactly as written and only move the lockfile; `--latest`
rewrites them to the resolved version as before. `bun update -i` applies
only what you selected. New: positional patterns (`bun update
'@types/*'`), `--dev`/`--prod`/`--no-optional`, `-L`, `bun up`.
- `package.json` is written after resolution and `bun.lock`'s declared
ranges, overrides and catalogs are re-derived from the final
`package.json`, replacing the per-command literal rewriting; a no-op
update leaves the file byte-identical.

#### Overrides

- **Nested overrides** (#6608): npm's nested objects, yarn's `a/b` paths
and pnpm's `a>b` selectors, applied to the direct parent→child edge;
**version-scoped targets** (`"lodash@<4.17.21": "4.17.21"`, the shape
`pnpm audit --fix` writes), matched against the dependent's declared
range as pnpm does. Rules persist inside the `overrides` section and the
file is stamped `lockfileVersion: 3` **only when such rules exist** —
existing lockfiles are byte-identical. Flat overrides additionally fix
`$ref` to workspace-member deps, catalog-valued rules going stale, and
warn on pnpm's `-` / `pkg@` forms.

#### Workspaces and filters

- `bun add|remove|update … --filter <ws>` (also `-F`, also `bun install
<pkg> --filter`) edits the selected workspaces' `package.json` files and
**links only those workspaces**, like `bun install --filter`. Filters
gain pnpm's relation selectors (`foo...`, `...foo`, `foo^...`,
`...^foo`) and `{dir}` subtrees, for the install family **and** `bun run
--filter`; `--filter` may precede the subcommand; every command warns
about patterns that match nothing; `add`/`remove` no longer select the
root implicitly.
- `bun add <pkg> --catalog[=name]` reuses an existing catalog entry,
keeps a range an explicit version fits, catalogs the range a package
already declares, decides per target, and refuses workspace names and
local paths; a plain `bun add` uses a default-catalog entry when one
exists. A package defined in both `catalog` and `catalogs.default` is an
error. `catalog:` peers of registry packages bind to the importer's copy
instead of the root catalog.
- `--frozen-lockfile` / `bun ci` on turbo-pruned monorepos: pruned-away
workspaces are tolerated, a survivor depending on a pruned workspace is
an error, catalog subsets are accepted, and an overrides/catalogs change
is a frozen failure.

#### One output vocabulary

Every command here prints the install family's shapes: header, glyph
rows (`+`/`-`/`↑`, dedupe's `↳ name old → new`), exactly one noun-first
summary line with counts and a duration (`2 duplicate versions removed,
3 packages installed (checked 5 packages) [12ms]`, `N packages removed
(checked C) [t]`), no-ops that say what was checked, remedies printed as
copy-pasteable command lines, warnings as `warn:`, `--silent` printing
nothing, and errors with their remedy together on stderr. Transitive and
named updates render as the summary's `↑` rows (once per package;
`--dry-run` prints the same rows plus `N packages would be updated`);
dedupe reports after the install it triggers, so lifecycle-script output
never splits it. A lockfile whose bytes did not change is no longer
rewritten (`Saved lockfile` only prints on a real write;
`--lockfile-only` no-ops print `Done! Checked N packages (no changes)`).
This came out of running every command against fixtures and comparing
with `install`/`add`/`remove` (95 findings, all fixed).

#### Config precedence

A project's `bunfig.toml` now beats any `.npmrc` (project or user-level)
for the same key (npmrc files → bunfig's set fields → CLI); npmrc-only
settings such as `//host/:_authToken` still attach to bunfig-declared
registries, matched by host and path regardless of how either file
spells the trailing slash.

#### Lockfile migration

- `package-lock.json`: rebuilt around a reachability walk that derives
each resolution from the entry itself. Fixes `git+https://github.com/…`
resolutions being written unparseably (the next install threw the
lockfile away), root `bundleDependencies` migrating to an **empty**
lockfile, lockfileVersion 1 (and npm's upcoming 4) making `bun install`
exit 1 instead of resolving fresh, dependency-level bundles,
`dependencies`+`optionalDependencies` double edges, unreferenced entries
aborting the migration, duplicate packages for identical `name@version`
at nested paths, lost `optionalPeers`, lost integrity when a bundled
copy was seen first, and `overrides` not being carried over. All 57 of
arborist's v2/v3 fixture projects are vendored and migrated under
snapshot.
- `pnpm-lock.yaml` v9: bare-hash `patchedDependencies`, snapshot
aliases, `catalog:default`, recorded tarball URLs, git `path:`,
multi-document files, `runtime:` entries, named registries,
peer-suffixed keys chosen per importer, injected workspaces,
manifest-only importer deps.

#### Isolated linker

- An existing store entry whose dependencies re-resolved (override,
dedupe, update) now has its links refreshed on the next install
(measured cost below). `bun prune` builds the same store the installer
builds, so stale `name@version+<peerhash>` variants left by peer bumps
are removed and a kept package's real entry never is (whether it was
installed with full or `--production` features); on the hoisted linker,
dedupe / audit fix / update delete the nested copies whose rows they
collapsed instead of leaving the old copy loadable.
- Blocked entries resume through per-entry intrusive waiter lists
instead of a scan of every store entry after each completion (robobun's
#25983/#28425 attempted this). Measured on the reporter's repro from
#25799 (2,259 store entries) and a synthetic 6,425-entry monorepo, PR
build vs merge-base build: main-thread CPU in the link phase drops 0.85
→ 0.33 s and 5.5 → 0.85 s (the removed work grows quadratically); wall
time is unchanged with spare cores and 16% / 26% faster pinned to one
CPU, the CI/Docker shape in those reports. (The minute-long installs
originally reported were peer resolution, fixed before this PR's base.)
- Two pre-existing leaks surfaced by the new LSan-enabled tests are
fixed: the header buffer of every authenticated registry request, and
the per-entry lifecycle-script lists.

#### Other bug fixes

`bun add x@npm:pkg` writes a range; `bun add --trust a b` no longer
drops `b` when `a` was already trusted; `bun add x --dev` no longer
rewrote every group in `bun.lock`; `catalog:` peer hoisting;
`dependency::Version::eql` treated all `catalog:` specifiers as equal; a
`file:` package whose dependencies reach itself (its own name, an `npm:`
alias under its own name, two link targets depending on each other — the
shapes a `package-lock.json` migration produces, and #25202's
`workspace:.` self-reference) hung `bun install` forever in the hoisting
tree — the migration shapes now install, and #25202's literal shape now
terminates with `Workspace dependency "foo" not found` rather than
installing as npm does; a peer of a `file:` package that was only placed
nested was also written to `optionalPeers` in a migrated bun.lock, so
the next `--frozen-lockfile` failed and a plain install rewrote the
lockfile; `catalog:` literals in `bun update`; alias output in the
install summary; help/completions for everything above.

#### Behavior changes to note in the release notes

- `bun update` moves transitive packages; `bun update <name>` no longer
adds an undeclared package (exit 1); `--production`/`--prod` on update
means "only update `dependencies` and `optionalDependencies`" (a group
filter like `--dev`, not the install flag) and `-r`+names with no match
is an error; `-i` updates only the selection.
- Project `bunfig.toml` overrides any `.npmrc`.
- `bun install <pkg> --filter x` edits `x` (not the root); `bun add y
--filter x` no longer installs a package named `x`; `add`/`remove
--filter '*'` no longer includes the root.
- A plain `bun add x` in a workspace whose default catalog lists `x`
writes `catalog:`; `audit fix` may rewrite exact pins;
`--frozen-lockfile --lockfile-only` writes nothing; overrides/catalog
changes fail frozen installs.
- One-time lockfile churn after upgrading for projects with `catalog:`
peers or dead `pkg@range` override rows; lockfiles that use
nested/scoped overrides are v3 and unreadable by older Bun (only when
opted in). Turborepo, Nx and Dependabot have been checked; the needed
upstream changes are open (nrwl/nx#36666 covers v2 and v3;
vercel/turborepo#13740 accepts v3 and preserves the object rows through
prune — turborepo main today parses v2 and rejects v3; dependabot needs
nothing). Note v2 itself only exists on the 1.4 line.
- `bun audit --json` keeps npm's contract: `--audit-level`/`--ignore`
decide the exit code, the JSON document is the full registry report
(closes #31013 as won't-change). Automatic removal of stale
`node_modules` entries on plain `bun install` (#32974) is separate from
this PR: `bun prune` is the manual form, and dedupe / audit fix / update
now clean up the nested copies they collapse on the hoisted linker;
#32974 should reuse prune's planner, and #29512 (sbom) is sequenced
after this so it can build on `reachable.rs` instead of carrying its own
walk.
- Deliberately kept where we differ from pnpm: root `bun update` covers
the whole workspace; dependents whose ranges allow follow a moved
version (one copy, not two); `--latest` works on transitive names;
`--no-save` touches neither file; prune deletion failures exit 1; audit
requests stay per-registry.

#### Performance

Measured on a 1,113-package Next/Prisma/MUI app (PR build vs a PR build
of the merge base, interleaved, plus canary and 1.3.14): every hoisted
cell is within noise except no-op install, +0.8 ms (+2%, identical
syscalls); isolated no-op is +1.9 ms (+3.9%) — the deliberate cost of
re-checking existing entries' links every install rather than persisting
a stamp file. Everything added is otherwise off the plain-install path
(gated on the feature being used or on a diff), and id-indexed sets are
bitsets.

### How did you verify your code works?

~1,100 new or ported test cases across the install suites (designed
behavior, cases ported from pnpm's suites, pinning tests for pnpm bugs
this implementation is immune to, arborist's fixtures, and the CI review
findings), all `toStrictEqual`; the whole `test/cli/install` directory
passes locally and the existing suites are unchanged except where a
pre-existing expectation was deliberately changed (each listed above).
`bun update` was additionally verified with a rerunnable differential
harness that runs pnpm 11 and this branch on 26 scenario families
against one registry and diffs the resulting resolutions edge by edge —
after this PR only the deliberate differences above remain. Ecosystem:
Turborepo, Nx and Dependabot were checked against the new lockfile
output.

Co-authored work absorbed with credit: @kjanat's #38190 (alias handling,
co-author on the commit), @charpeni's #31143 and @crystalin's #34407
(both superseded), and the tests of the earlier `bun update` PRs (#31752
by @zlotnika, #33127, #36381, #36729, #38224). robobun's #34688
(folder-dependency cycles; its tests are lifted, co-author on the
commit) and #37289 (migrated optionalPeers; its test is lifted,
co-author on the commit) were fixed independently here and are closed by
this PR. #28422's quadratic scan is fixed here as well (already closed).

Related but not closed — `bun prune` gives these a manual fix while the
automatic-cleanup asks stay open: #8662, #26305, #29793, #21216, #16176.
Also related: #10930, #26970, #26751.

Fixes #1343
Fixes #3605
Fixes #14719
Fixes #24122
Fixes #18612
Fixes #20238
Fixes #25826
Fixes #23615
Fixes #26973
Fixes #20593
Closes #31013
Fixes #28959
Fixes #28402
Fixes #27897
Fixes #26675
Fixes #10949
Fixes #18504
Fixes #13388
Fixes #24523
Fixes #6608
Fixes #19059
Fixes #16569
Fixes #8262
Fixes #11901
Fixes #13469
Fixes #25202
Closes #29664
Closes #31143
Closes #34407
Closes #34688
Closes #37289
Closes #38190

---------

Co-authored-by: Kaj Kowalski <info@kajkowalski.nl>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: robobun <117481402+robobun@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: bun install hangs indefinitely with symlink linker in large monorepo

1 participant