Skip to content

standalone: record the embedded shared-library set at build time - #44082

Closed
robobun wants to merge 1 commit into
mainfrom
robobun/fa73e908/native-library-set-record
Closed

robobun wants to merge 1 commit into
mainfrom
robobun/fa73e908/native-library-set-record

Conversation

@robobun

@robobun robobun commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #44063

Behaviour change: none

Problem

  • A compiled executable extracts an embedded .node addon alone before dlopen, so a shared library next to it is not found (bun build --compile: an embedded native addon is extracted alone, so a shared library next to it is not found (Library not loaded: @rpath/...) #44063). To extract the addon with its dependencies, the runtime needs to know which embedded files are shared libraries, and that the bundler's hoisted addon-[hash].node is the same addon as the --asset copy that sits next to those dependencies.
  • Today the runtime could only find that out by paging in and hashing every embedded file on the first dlopen of every process (about 5.6 ms and 265 page faults for an 18 MiB sharp-sized set). The standalone writer already knows it at build time.

Fix

  • to_bytes collects every output file whose name is a shared library (.node, .so, .so.N, .dylib, .dll) into a NativeLibrarySet: the file index of each member, and for a member whose bytes equal another member's at a deeper path, that member's index (alias_index). The twin's bytes are stored once.
  • The record is chained after the linked-payload record under a new flag, Flags::HAS_NATIVE_LIBRARY_SET: u64 set_hash, u32 count, then count x {u32 file_index, u32 alias_index}. from_bytes reads it with the same bounds checks as the other records and ignores a malformed one.
  • Nothing reads the set yet. compile: mirror embedded shared libraries into one temp directory before dlopen #44083 mirrors it into one temp directory before dlopen.
  • Verified: test/bundler/bun-build-compile.test.ts, compile-asset-bunfs.test.ts, test/regression/issue/29585.test.ts, 30717.test.ts, test/napi/napi.test.ts --compile. Bun.embeddedFiles still lists every file.

Background

  • StandaloneModuleGraph is the serialized list of files a compiled executable carries. After the module table, optional records chain in Flags bit order. An older bun ignores a bit it does not know.
  • The bundler hoists require("./lib/addon.node") to /$bunfs/root/addon-[hash].node (ParseTask.rs, Loader::Napi). --asset lib embeds a second copy at lib/addon.node next to lib/libfoo.so. The alias links the two by content, the way HAS_SOURCE_HASHES precomputes what the runtime would otherwise hash.
  • A runtime-only design was considered: hash the set in the helper on first use. It pays the hash per process and keeps the addon embedded twice.

Downsides

  • The embedded graph grows by 12 + 8 x N bytes for N shared libraries: 16,210 B to 16,225 B for a one-addon build (20 B record, 5 B of baseline drift between the two bun builds compared). With --asset lib it shrinks 15,538 B (the addon's 15,576 B second copy is stored once).
  • None found at runtime. Checked Bun.embeddedFiles, source_text_pages, SOURCE_TEXT_CONTIGUOUS, and the startup prefetch span: a shared contents pointer breaks none of them.
Notes
  • Record-chain walkers in bun-build-compile.test.ts and bundler_compile_splitting.test.ts stop at bit 13, so the new record after it leaves them untouched.
  • Graph byte counts come from the Offsets trailer of bun build --compile app.js [--asset lib] built by the released 1.4.3 and by this branch.
  • The set hash covers each member's relative name and content hash, so two executables with the same libraries at different paths never share a mirror directory.
  • cargo clippy clean on bun_standalone_graph, bun run rust:check-all green.

A compiled executable's module graph now carries a record of every
embedded shared library (.node, .so, .so.N, .dylib, .dll): the file
index of each, a hash over the whole set, and for a library whose bytes
match another at a deeper path (the bundler's hoisted [name]-[hash].node
next to its --asset copy) the index of that copy. The twin's bytes are
stored once.

Nothing reads the record yet. The runtime will use it to mirror the set
into one temp directory before dlopen, so an addon's $ORIGIN and
@loader_path dependencies resolve.
@robobun

robobun commented Sep 27, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: folded into #44083. The record is only reviewable next to the code that reads it, and the twin dedup does change the executable's bytes, so the no-behaviour-change label did not hold.

@robobun robobun closed this Sep 27, 2026
@robobun
robobun deleted the robobun/fa73e908/native-library-set-record branch September 27, 2026 04:16
@robobun
robobun restored the robobun/fa73e908/native-library-set-record branch September 27, 2026 04:18
@robobun
robobun deleted the robobun/fa73e908/native-library-set-record branch September 27, 2026 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant