compile: mirror embedded shared libraries into one temp directory before dlopen

[robobun]

Fixes #44063

Problem

• A --compile executable whose .node addon links a library next to it fails with ERR_DLOPEN_FAILED (libfoo.so: cannot open shared object file: No such file or directory), even with the library embedded. sharp fails the same way (Library not loaded: @rpath/libvips-cpp...dylib).
• resolve_embedded_file_to_buf (src/runtime/jsc_hooks.rs) wrote the requested file alone into the temp directory. The dependency is not there, and another user can put one there.

Fix

• The writer records every embedded shared library (NativeLibrarySet). The runtime writes the set once into a private directory, {tmpdir}/.bun-{euid}-{set hash}/, with the embedded layout.
• The layout sits eight levels below the top: a path that climbs with .. stays inside.
• When another user holds that name, or the filesystem shows another owner than the euid, the process loads from one copy of its own and removes it at exit.
• Verified: test/bundler/compile-asset-bunfs.test.ts, 29585.test.ts, 30717.test.ts, napi.test.ts.

Background

• dlopen(2) cannot read /$bunfs/, a compiled executable's virtual filesystem, so a library goes to disk first.
• $ORIGIN (@loader_path on macOS) is the directory of the library that names it.
• Considered a depth read from each library's search paths: it needs an ELF and Mach-O reader and misses paths built in code.

Downsides

• A path that climbs more than eight levels reaches the temp directory.
• Cold extraction writes every embedded library (about 18 MiB for sharp) plus 16 mkdirat calls and one fstat.
• A process that a signal kills leaves its own copy.

Notes

Own copies (bbea975)

• Review on 1e173bf found this: a temp directory can show another owner than the euid for what the process itself creates (NFS root_squash, CIFS uid=, drvfs). The process then did not recognise its own copy either, so each dlopen call wrote the set again. Nothing removed an own copy, there or when another user held the name.

• A first fix (7d9d924) took the shown owner for the canonical name too. It is dropped: another principal with that shown owner could create the directory first.

• This commit keeps the rule for the canonical name: this user's only by the euid. It reads the owner that the filesystem shows for the scratch directory from its handle. The process knows its own copy by that owner and by the euid it had. When that owner is not the euid, the scratch directory does not take the canonical name. An exit callback removes the own copies: one pass inside the copy over the members and their directories, with no retry.

• Measured on release builds, linux-x64, 3 runs of 3 dlopen calls each. setfsuid(12345) through bun:ffi stands for such a filesystem: the owner of a new file is then not the euid.

  	1e173bf	bbea975
  other owner shown: entries in the temp directory during run 1, 2, 3	3, 6, 9	1, 1, 1
  other owner shown: entries after run 3	9	0
  name held by a file: entries after run 1, 2, 3	2, 3, 4	1, 1, 1
  file syscalls of the mirror, cold, normal filesystem	31	32 (one more fstat)
  file syscalls of the mirror, warm	5	5
  release text (size)	80,679,484	80,681,788

• Limits. A process that a signal kills leaves its copy (SIGTERM measured). NFS and SMB keep a library that is still loaded until the process is gone, and Windows does not remove it, so directories of the copy can stay. Not measured: no such mount in the build container.

• Fail before: at 1e173bf the reworked test finds the own copy in the temp directory after the process exits, and 3 entries where 1 is expected for the other owner.

• Self-review of this commit: 13 concerns raised, 10 addressed. Not addressed: a signal skips the exit callback (the limit above). A relative BUN_TMPDIR with a later chdir makes the exit pass look in another directory (each call already has that). On a filesystem that shows one owner for every user, a temp sweeper can remove the own copy of a running process and another user can then create its name again (the process checks the name by owner, as before).

After the eight levels (1e173bf)

Three things that main does and 46d44da did not. Each is checked on release builds of main a4f1429 and of 46d44da, and on a debug build of this head (linux-x64).

• An embedded addon named better_sqlite3.node. process.dlopen refuses a path that ends in that name, and it looks at the path after extraction. On main that path is a hash, so the check never matched an embedded addon: it loads. The mirror keeps the embedded name, so 46d44da threw 'better-sqlite3' is not yet supported in Bun. The check now skips a path that came from the executable. It still applies to every other path.
• The mirror's name held by another user (a non-empty directory of uid 65534, the process runs as uid 1000). After 3 starts of 10 dlopen calls and 5 Workers: 45 copies of the whole set at 46d44da, one for each call, and 3 now, one for each process. main leaves 45 single files. The process remembers the copy it wrote and checks it like the mirror on each call.
• A directory where a library file goes, in the process's own mirror. A rename does not replace it, so the repair failed and the single-file fallback had the library without its dependency: the load failed at 46d44da. The process now loads from a copy under its own name. main loads.
• One lock covers every write of a mirror in the process, so Workers that start together write the set once.
• Fail before: at 46d44da the two new tests fail, one with ERR_DLOPEN_FAILED and one with 3 copies where 1 is expected.
• Same fixtures on main and on 46d44da, with a ptrace syscall counter (strace is not installed):
  • 18 shapes (a climb of 1 to 6 levels, hoisted or with --asset), the other user's library at seven levels from the temp directory up: main loads it in 18, 46d44da in 0. A climb of 9 or 10 levels from a hoisted addon loads it at 46d44da.
  • self-contained addon, cold: mkdirat 0 to 16, openat 23 to 26, renameat 1 to 2. Warm: one more newfstatat.
  • the same addon beside 4 MB of other embedded libraries, bytes written cold: 15,280 on main, 4,269,808 at 46d44da.
  • real sharp 0.34.5 with --asset node_modules: ERR_DLOPEN_FAILED on main; loads at 46d44da, 17,055,960 bytes in 2 files, mkdirat 27.
  • release text (size): 80,660,492 on main, 80,676,108 at 46d44da.
• Self-review of this commit: 3 concerns raised, 3 addressed (a field that only the Unix arms read broke the Windows build; the fit check of the returned path had moved after the writes; a match that clippy::manual_map rejects).
• Not in this PR: writing only the libraries one dlopen needs. It needs the reader that the depth from search paths needs. A version is on robobun/b76cf9cb/mirror-pad-closure.

The eight levels (46d44da)

• At 8bd6f2a the mirror directory was private, but one component below the temp directory. A path that climbed out of a library's directory landed in the temp directory, and a library that another user put there was loaded. main has this for $ORIGIN itself: the released 1.4.3-canary prints {"declared":666,"built":-1,"ffi":667} for the new test a library that another user put in the temp directory is not loaded. main does not have it for sharp, whose entries resolve to / there.
• Two users on a root-owned 1777 temp directory, this head. Nothing of the other user loads for: $ORIGIN, $ORIGIN/deps, $ORIGIN/../lib from a hoisted addon, $ORIGIN/../../lib with --asset lib, a five-level entry from node_modules/@img/x/lib, an eight-level entry from a hoisted addon, and a path built in code (dlopen("$ORIGIN/../lib/x.so"), and dladdr plus ../lib). A nine-level entry from a hoisted addon loads the other user's library: that is the limit in Downsides. At 8bd6f2a that entry resolved to /lib with the default /tmp, so the levels move the first exposed height from one level to nine.
• Fail before: at 8bd6f2a the two new tests print {"addon":666,"scoped":706} and {"declared":666,"built":666,"ffi":667}. With this head all 13 tests of the file pass.
• Why eight: the deepest climb in the 52 prebuilt ELF libraries of this machine's package caches is five (sharp, the only package there that climbs).
• Measured on debug builds of 8bd6f2a and of this head, linux-x64, glibc 2.41. strace and perf are not installed in the build container, so the syscall counts come from a ptrace counter.
  • loader probes in the temp directory outside the mirror (LD_DEBUG=libs): 4, 4, 4 before and 0, 0, 0 after, for a one-level, a two-level and a five-level climb
  • warm dlopen: the same syscalls before and after (openat, geteuid, one newfstatat per member plus one, close)
  • cold extraction: mkdirat 2 to 18 with --asset lib, and 1 to 16 for a hoisted addon, which also gets 2 more openat and 1 more close for its new parent directory
  • returned path: 16 more bytes
  • release binary: 80,844,360 B before and after (stat and size on release builds of 8bd6f2a and of this head)
• Considered and not taken: a depth computed from the search paths of the embedded libraries (DT_RPATH, DT_RUNPATH, LC_RPATH) and recorded at build time. It is exact for declared paths at any height. A library with no such path stays at the top of the mirror, so a path built in code still loaded the other user's library in the two-user run. It also needs a hand-written ELF and Mach-O reader. That work is on the branch robobun/b008d3fc/recorded-depth.
• Open question for a maintainer: a per-user base directory with the temp directory as fallback, the shape of bunx: store the package cache under the per-user bun cache directory #31447 for the bunx cache, closes a climb of any height. It moves where a compiled executable writes, and no temp sweeper collects it. It is not in this PR.
• The same shape outside this PR: the bunx cache is {tmpdir}/bunx-{uid}-{pkg}/node_modules/..., and sharp's five-level entry from there is the temp directory. bunx: store the package cache under the per-user bun cache directory #31447 moves that cache.
• This came from an internal fuzzing pass. No public issue reports it.
• Run locally on linux-x64 glibc. The file also passes on the macOS (arm64, x64) and Alpine lanes, 13 of 13. Windows skips the library tests (no C toolchain). bun_standalone_graph joins the Miri set, so CI runs the unit tests of native_libs.

The mirror (first five commits)

Fix, in full:

• The standalone writer records every embedded shared library in the module graph (NativeLibrarySet, flag HAS_NATIVE_LIBRARY_SET): each file's index, a hash over the set, and for the bundler's hoisted addon-[hash].node the index of the --asset copy read from the same source file (both paths go through realpath). The --asset copy is always the one loaded. Its bytes are stored once.
• At runtime the helper writes the recorded set once into {tmpdir}/.bun-{euid}-{set hash}/ with the embedded layout and returns the requested file's path inside it. process.dlopen, require("x.node") and bun:ffi all go through it.
• Reuse needs the directory to be ours (owned by the euid, not a symlink) and every member to be ours with the right size: one lstat each, no content hash. A missing member is written back in place through a temp name and a rename. The directory itself is never removed, so a process loading from it keeps what it sees. A new set is written into a scratch directory and renamed into place, so Workers and concurrent processes never see a half-written set. If the set cannot be written in full, the requested file is mirrored on its own, as before.
• Verified: test/bundler/compile-asset-bunfs.test.ts (two new tests, the released bun fails both with the error above), test/regression/issue/29585.test.ts (one copy across 10 dlopens, 5 Workers, restart), 30717.test.ts, test/napi/napi.test.ts --compile, bun-build-compile.test.ts.

Background, in full:

• /$bunfs/ is the virtual filesystem of a compiled executable. dlopen(2) reads real paths only, so Dedupe extracted embedded native modules in compiled binaries #29587 wrote an embedded library to a content-hashed, euid-scoped, 0600 file in the temp dir. This keeps those properties at directory level.
• StandaloneModuleGraph chains optional records after the module table in Flags bit order. An older bun ignores a bit it does not know, and an executable without the record is mirrored one file at a time, as before.
• Considered a runtime-only mirror with no record: it reads and hashes the whole set on the first dlopen of every process (19 to 25 ms for an 18.9 MB sharp-sized set in a release bun, JS probe through Bun.embeddedFiles) and keeps the addon embedded twice. Rpath parsing needs ELF, Mach-O and PE readers and still cannot reach sharp's $ORIGIN/../../sharp-libvips-*/lib without the whole tree.

Costs, in full:

• Cold extraction writes every embedded shared library, not only the requested one: 46,216 B for the three-library repro (du -b), about 18 MiB for sharp. Paid once per set per machine, and again after a temp sweep. A user who embeds many libraries and loads one pays for all of them.
• Warm dlopen costs one lstat per member of the set plus one for the directory, instead of one lstat of the file: 4 syscalls before (geteuid, openat, fstatat, close) and 3 + N after. The per-call content hash is gone. Counted from the code: strace and perf are not installed in the build container.
• Embedded graph size: +15 B for a one-addon build without --asset (the record), -15,538 B with --asset lib (the addon's second copy is stored once).

More:

• Mirror layout for the repro: .bun-0-e7c1e2b45a40e6c2/_/_/_/_/_/_/_/_/lib/{addon.node,libfoo.so}. The hoisted addon-8hekbvxp.node is not written: it aliases lib/addon.node. A symlinked --asset directory, a single-file --asset lib/addon.node, and --asset-naming assets/native/[name]-[hash].[ext] all produce the alias and load.
• The temp dir is opened once, before the reuse check, so Refuse a temp directory that another user owns for the bunx cache, embedded addons, and bun upgrade #41915's open_trusted_temp_dir has one call to replace and its check covers both paths.
• Windows: LoadLibraryExW with LOAD_WITH_ALTERED_SEARCH_PATH (already used by process.dlopen and bun:ffi) finds a dependent DLL next to the loaded one. Renaming the scratch directory onto an existing one fails with EPERM there, the same fallback applies. Not run on Windows here (the test needs a C toolchain and skips there); bun run rust:check-all is green.
• Names with .. segments (custom --asset-naming [dir]/...) are rewritten to _.._ the way bun build does, so every member stays inside the mirror.
• Graph byte counts come from the Offsets trailer of bun build --compile app.js [--asset lib] built by the released 1.4.3 and by this branch (5 B of the delta is baseline drift between the two builds).
• Keep the addon name in the temp file an embedded native library extracts to #37102 (open) wanted the addon's basename visible in the extracted path for crash reports. The mirrored path ends in lib/addon.node, so that goal is met here.
• Not in scope: symlinks inside an --asset tree are skipped today (versioned soname links such as libfoo.so.1 -> libfoo.so.1.0.0), bare-specifier resolution from /$bunfs/root/node_modules ("bun build" does not embed binaries from node_modules correctly #15374), bun:sqlite's setCustomSQLite and cc({library}), which never call this helper. Data files next to an addon are not mirrored.
• Workaround for older versions: dlopen the dependency by path through bun:ffi first so the loader matches it by soname, then process.dlopen the addon.
• Ownership of the mirror directory is by owner, not mode bits, the rule Refuse a temp directory that another user owns for the bunx cache, embedded addons, and bun upgrade #41915 uses for bun's other temp directories: another user cannot create a directory owned by the euid, and some filesystems (drvfs, vfat, ntfs-3g, CIFS) make mode bits up.
• A mount that maps the caller to a shared uid (NFS with root_squash, CIFS with uid=) never reports the euid as owner, so the canonical directory is never trusted there and each process keeps a copy of its own. That fails closed on purpose: the mapped uid is shared with other principals. BUN_TMPDIR on a private filesystem is the way out.
• Self-reviewed: 4 concerns raised, 4 addressed (twin link by source path instead of bytes, one temp dir handle for both paths, the two PRs folded into one, standalone: record the embedded shared-library set at build time #44082 closed). Review findings on the first push (directory ownership, swept members, scratch leak, alias chains, provenance, single-file assets) are addressed in 099f933; the repair race, a double close, the record bounds and the name filter in a0e6f51.
• cargo clippy clean on the touched crates.

---

no test proof · iteration 2 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/napi/napi.test.ts, test/bundler/compile-asset-bunfs.test.ts

[robobun]

^{Updated 1:37 AM PT - Oct 1st, 2026}

❌ @robobun, your commit a383dfd has 1 failures in Build #122269 (All Failures):

• test/js/bun/spawn/spawn.test.ts - code 1: stdout reader of an unref'd child and process lifetime > an idle reader stopped at the highwater … on 🐧 13 x64-asan

---

🧪   To try this PR locally:

bunx bun-pr 44083

That installs a local version of the PR into your bun-44083 executable, so you can run:

bun-44083 --bun

[claude]

Beyond the inline finding, I also checked the Windows build against the new owner field on MirrorDir (src/runtime/jsc_hooks.rs:4593): the #[cfg(windows)] arm of is_ours at line 4608 reads it, so the workspace's dead_code = "deny" is not tripped there.

Extended reasoning...

The latest two commits (7d9d924, f7f1e2d) change how the mirror owner is determined and add a Windows read of the field; the Windows dead-code compile-failure claim was ruled out by reading the cfg arm. The remaining posted finding concerns the owner-from-filesystem fallback on squashing mounts, and the hunt exited on the max-bugs bound, so this is not an approval.

[claude]

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

[claude]

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

[robobun]

CI at a383dfd: 181 of 182 jobs pass. The one red job is test/js/bun/spawn/spawn.test.ts on debian x64-asan, which fails the same way on main and is reported separately. Every test this PR touches passes on every lane, including the macOS lanes. Ready for review.