Repository navigation
Conversation
A compiled executable's module graph now carries a record of every embedded shared library (.node, .so, .so.N, .dylib, .dll): the file index of each, a hash over the whole set, and for a library whose bytes match another at a deeper path (the bundler's hoisted [name]-[hash].node next to its --asset copy) the index of that copy. The twin's bytes are stored once. Nothing reads the record yet. The runtime will use it to mirror the set into one temp directory before dlopen, so an addon's $ORIGIN and @loader_path dependencies resolve.
Collaborator
Author
|
Closing: folded into #44083. The record is only reviewable next to the code that reads it, and the twin dedup does change the executable's bytes, so the no-behaviour-change label did not hold. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #44063
Behaviour change: none
Problem
.nodeaddon alone beforedlopen, so a shared library next to it is not found (bun build --compile: an embedded native addon is extracted alone, so a shared library next to it is not found (Library not loaded: @rpath/...) #44063). To extract the addon with its dependencies, the runtime needs to know which embedded files are shared libraries, and that the bundler's hoistedaddon-[hash].nodeis the same addon as the--assetcopy that sits next to those dependencies.dlopenof every process (about 5.6 ms and 265 page faults for an 18 MiB sharp-sized set). The standalone writer already knows it at build time.Fix
to_bytescollects every output file whose name is a shared library (.node,.so,.so.N,.dylib,.dll) into aNativeLibrarySet: the file index of each member, and for a member whose bytes equal another member's at a deeper path, that member's index (alias_index). The twin's bytes are stored once.Flags::HAS_NATIVE_LIBRARY_SET:u64 set_hash,u32 count, thencountx{u32 file_index, u32 alias_index}.from_bytesreads it with the same bounds checks as the other records and ignores a malformed one.dlopen.test/bundler/bun-build-compile.test.ts,compile-asset-bunfs.test.ts,test/regression/issue/29585.test.ts,30717.test.ts,test/napi/napi.test.ts --compile.Bun.embeddedFilesstill lists every file.Background
StandaloneModuleGraphis the serialized list of files a compiled executable carries. After the module table, optional records chain inFlagsbit order. An older bun ignores a bit it does not know.require("./lib/addon.node")to/$bunfs/root/addon-[hash].node(ParseTask.rs,Loader::Napi).--asset libembeds a second copy atlib/addon.nodenext tolib/libfoo.so. The alias links the two by content, the wayHAS_SOURCE_HASHESprecomputes what the runtime would otherwise hash.Downsides
--asset libit shrinks 15,538 B (the addon's 15,576 B second copy is stored once).Bun.embeddedFiles,source_text_pages,SOURCE_TEXT_CONTIGUOUS, and the startup prefetch span: a shared contents pointer breaks none of them.Notes
bun-build-compile.test.tsandbundler_compile_splitting.test.tsstop at bit 13, so the new record after it leaves them untouched.Offsetstrailer ofbun build --compile app.js [--asset lib]built by the released 1.4.3 and by this branch.cargo clippyclean onbun_standalone_graph,bun run rust:check-allgreen.