Skip to content

bundler: mark a metafile import external only when it has no source index - #47161

Open
robobun wants to merge 2 commits into
mainfrom
robobun/c3256148/metafile-external-source-index
Open

robobun wants to merge 2 commits into
mainfrom
robobun/c3256148/metafile-external-source-index

Conversation

@robobun

@robobun robobun commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • bun build --metafile marks import db from "./app.db" with { type: "sqlite", embed: "true" } as "external": true in inputs[..].imports. The same metafile gives app.db 91 bytes of the chunk, and --compile --aot --metafile prints the entry without external.
  • Cause: the external test in generate (src/bundler/linker_context/MetafileBuilder.rs:364) is flags.contains(IS_EXTERNAL_WITHOUT_SIDE_EFFECTS) || !source_index.is_valid(). resolve_import_records (src/bundler/bundle_v2.rs:6822) sets that flag on an embedded import before it resolves to a bundled file.

Fix

  • The test is now !record.source_index.is_valid(), which esbuild uses for inputs (internal/bundler/bundler.go:2359, v0.25.1). A record with no source index keeps "external": true.
  • Verified: test/bundler/metafile.test.ts (10 of 14 new tests fail without the change). Also test/bundler/esbuild/metafile.test.ts, bundler_bun.test.ts.
  • Self-reviewed: 18 concerns raised, 17 addressed. Notes names the one not done.

Background

  • A record's source_index names the bundled file an import resolves to.
  • IS_EXTERNAL_WITHOUT_SIDE_EFFECTS is a tree shaking hint: "this import can be removed if it's unused" (src/ast/import_record.rs:76). The linker reads it only when the record has no source index.
  • Considered removing the flag from embedded imports. The linker needs it to drop an unused one that a plugin marks external.

Downsides

Notes

Repro

printf x > app.db
printf 'import db from "./app.db" with { type: "sqlite", embed: "true" };\nconsole.log(db);\n' > index.js
bun build index.js --target=bun --outdir=dist --metafile=meta.json --metafile-md=meta.md

Before, meta.json has all of these:

  • inputs["index.js"].imports: [{"path":"app.db","kind":"import-statement","original":"./app.db","external":true}]
  • inputs["app.db"] is a key.
  • outputs["./index.js"].inputs["app.db"].bytesInOutput: 91

Before, meta.md lists app.db at 91 bytes | 65.0% of the output. It also prints | External imports | 1 |, [EXTERNAL: index.js imports app.db] and Imported by: (entry point or orphan) for it. With the change it prints [IMPORT: index.js -> app.db] and [IMPORTED_BY: app.db <- index.js].

History

What changes

  • An embedded database that a file uses. The entry is now the one that a file import prints. Each route has a test: a static import, type: "sqlite_embedded", import() without splitting, an onResolve plugin that returns a path, an onResolve plugin that returns nothing, the files map, two importers of one database, and the CLI (--metafile, --metafile-md). --compile calls the same generate (checked by hand). With a plugin path, the loader comes from the extension of that path, so the file loader copies the database. The record still has the flag from the attribute.
  • An embedded database that nothing uses. Tree shaking drops the import, and inputs does not list a dropped file (bundler: don't emit chunks for import() targets referenced only from dead code #39591, test/bundler/bundler_splitting.test.ts:383). The database is still copied to the output directory. The entry was external: true. It is now an entry whose path is not a key of inputs, which is what an unused file import prints on main. A reader that follows such an entry into inputs without a check fails, on main too. bundler: list tree-shaken files in the metafile inputs so import edges resolve #38624 proposes to list such files. The test for this case compares the whole entry and does not check inputs.
  • A record that patch_import_record_source_indices binds by path text (src/bundler/bundle_v2.rs:7563). That lookup can give a source index to a record that did not resolve to a module, for example import b from "/app.db" with { type: "sqlite" } next to an embedded import of the same file. The output then reads the embedded module for both imports. The entry was external: true. It now has no external, in agreement with the output. For two sibling files, the lookup result depends on the parse order, so the output differs between two builds of one input, on main too. In 80 builds of such a case, main printed external: true in all 80, and the output kept the import in 60. This branch printed external: true in 66 of 80, the same 66 whose output kept the import. bundler: link an import record through the path map only when resolution pointed it at a module #45248 removes the lookup for such records. The test for this case checks that external agrees with the output.

What keeps external: true, each with a test (these 4 pass with and without the change): with { type: "sqlite" } without embed, an embedded import under external: ["*.db"], an embedded import that an onResolve plugin marks external, and a split import() (the entry keeps entryPoint and external: true, as for each split import).

Not changed here

  • The flag write in resolve_import_records.
  • generateCompileResultForHtmlChunk.rs:151 reads the flag with no source index test. HTML tags carry no import attributes, so an embedded sqlite record does not reach it.
  • outputs[..].imports lists imports to other chunks only (packages/bun-types/bun.d.ts:4805). esbuild also lists external imports there. That is a separate feature. The self-review asked for a tracking issue for it. None is filed: no user asked for it.
  • A TypeScript import that is only used as a type is listed with external: true. esbuild prints the same entry (bundler.go:505 leaves it with no resolve result).
  • In the same repro, inputs["app.db"].bytes is 0 and the copied database is not in outputs. bundler: read the metafile size of a copied input from its asset file #38341 and fix(bundler): include emitted files in metafile outputs #43823 are open for those fields.

Measurements

Base is the same commit with the hunk reverted. Both are release builds (ThinLTO, linux x64).

  • release binary: -256 bytes .text, metafile_builder::generate 9,246 -> 9,230 bytes (nm -S). No other symbol of 182,037 changes size. The stripped binary stays at 138,197,144 bytes.
  • metafile loop: 2 -> 1 compare-and-branch pairs per import record before the external push (objdump). The test/jne on flag bit 11 is gone.
  • no-metafile build: 0 entries into generate (gdb breakpoint, 403 modules). resolve_import_records (13,725 bytes) and patch_import_record_source_indices (1,613 bytes) symbol sizes equal: yes.
  • instructions in the body of metafile_builder::generate on an 801-import graph: 251,462 -> 249,859. Counted with gdb single-step, callees skipped. valgrind (cachegrind, dhat) cannot be installed in the build container.
  • emitted files that differ across 24 matrix builds: 0 of 44 (sha256). The matrix: resolver, second importer, plugin path, plugin nothing, plugin external, external option, files map, import() with and without splitting, plain sqlite, sqlite_embedded, and a control with builtins, each with the binding used and unused.
  • metafile edges that lose external: 16 across 24 builds. Edges with no source index changed: 0 of 19. No other metafile field differs.
  • repro build: 36 syscall kinds have equal counts in both builds (strace -fc, 3 runs each). 8 kinds such as futex vary between runs of one build. Allocator calls inside generate: 12 malloc, 18 realloc, 26 free in both (gdb breakpoint counts).

no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/bundler/metafile.test.ts

…ndex

The metafile wrote `"external": true` for an import when its record had
IS_EXTERNAL_WITHOUT_SIDE_EFFECTS or no source index. The flag is a tree
shaking hint ("this import can be removed if it's unused"). An embedded
sqlite import (`with { type: "sqlite", embed: "true" }`) gets the flag
before resolution and then resolves to a bundled file, so the metafile
marked a bundled input as external.

The test is now the source index alone. esbuild uses that test for
`inputs`, and the `--compile --aot` metafile already follows that rule.
A record with no source index keeps `"external": true`.
@robobun

robobun commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

@robobun

robobun commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. The tests of this change pass in CI. The red steps are tests that this PR cannot reach.

How I reproduced it.

printf x > app.db
printf 'import db from "./app.db" with { type: "sqlite", embed: "true" };\nconsole.log(db);\n' > index.js
bun build index.js --target=bun --outdir=dist --metafile=meta.json
jq -c '.inputs["index.js"].imports, (.inputs | keys)' meta.json

On main (11c3671) and on bun 1.4.3-canary.1 the import entry is {"path":"app.db","kind":"import-statement","original":"./app.db","external":true}, and app.db is also a key of inputs. With this branch the entry has no external.

test/bundler/metafile.test.ts has 14 new tests. 10 of them fail without the change in src/.

CI. Build 134382 (fa2d86c):

  • test/bundler/metafile.test.ts ran and passed on Linux (79 pass) and on Windows (78 pass, and 1 test that Windows already skipped).
  • The only failed jobs are 5 test-aot-drawn jobs. That step is a soft failure. It runs a part of the --aot compiler tests, drawn for each build. 7 test files of it are red here: test/aot/programs-that-ask-the-shell-1 and -3, test/aot/perf/generated-code-of-idioms-2, -4 and -14, test/aot/minifying-first-saves-no-code-10, and test/aot/generated-by-python.
  • The same step fails on main. Build 134092 of 11c3671 (the base of this branch) has 4 failed test-aot-drawn jobs, and build 134067 has 8. Three of the 7 files above also failed in build 134067.
  • Those tests measure the code that bun build --compile --aot generates. They do not ask for a metafile. The code of this PR (metafile_builder::generate) runs only with --metafile or metafile: true.
  • When this was written, the 4 macOS test jobs were still in the queue.

PR: #47161

@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 69cd96f and fa2d86c.

Files selected for processing: 1
  • test/bundler/metafile.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

Walkthrough

The linker now marks imports as external only when their source index is invalid. New tests cover embedded SQLite imports in bundler and CLI JSON and Markdown metafiles.

Changes

Metafile import records

Layer / File(s) Summary
External-import classification and coverage
src/bundler/linker_context/MetafileBuilder.rs, test/bundler/metafile.test.ts
The linker no longer marks imports with a valid source index as external based on IS_EXTERNAL_WITHOUT_SIDE_EFFECTS. Tests cover embedded SQLite imports, excluded and tree-shaken databases, split dynamic imports, and CLI metafile output.

Suggested reviewers: alii

Priority: Normal

Merge Risk: Minimal · up to fa2d8

The change is consistent with the intended metafile behavior and adds coverage for the relevant embedded and external import cases.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly and concisely identifies the main change: metafile imports are marked external only when they lack a valid source index.
Description check Passed The description is detailed and covers the change, cause, fix, verification, tests, scope, and known limitations. It does not use the template headings, but it includes the required information and is…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description is detailed and covers the change, cause, fix, verification, tests, scope, and known limitations. It does not use the template headings, but it includes the required information and is substantially complete.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline findings, I also checked the sibling consumers of IS_EXTERNAL_WITHOUT_SIDE_EFFECTS and the other metafile writers: the markdown writer derives external from the JSON external key (import_target in MetafileBuilder.rs), so the [EXTERNAL:]/[IMPORT:] lines and the "External imports" count follow the new condition without a separate change; the --compile --aot writer in src/bundler/aot/metafile.rs already keys external on ImportTarget::External only, so the two writers now agree on importer entries; and the remaining flag readers (mergeSmallChunks.rs, generateCompileResultForHtmlChunk.rs, LinkerContext.rs) are tree-shaking/HTML consumers, not metafile output.

Extended reasoning...

The change is a one-condition narrowing in src/bundler/linker_context/MetafileBuilder.rs plus new tests in test/bundler/metafile.test.ts; it touches no security-sensitive surface. The inline findings (parse-order-dependent external for a mixed embed/non-embed absolute import, two weak assertions, discarded stdout in the CLI test) are the reasons a human should look; the note above records only the sibling-site checks that were ruled out.

Comment thread test/bundler/metafile.test.ts Resolved
Comment thread test/bundler/metafile.test.ts Resolved
Comment thread test/bundler/metafile.test.ts Resolved
The test for an unused embedded import compares the whole entry. The test
for a plain sqlite import beside an embedded one compares `external` of
both entries with the output. The CLI test checks that the build lists
the copied database on stdout.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the follow-up commit pins the exact metafile entries and the CLI stdout, and the one-line condition change checks out.

What was reviewed:

  • Traced every bundler site that sets IS_EXTERNAL_WITHOUT_SIDE_EFFECTS (bundle_v2.rs 6745/6792/6807/6818/7178): each either sets source_index = INVALID or continues before resolution, so builtins, bun: specifiers, plain type: "sqlite" and resolver-externals still emit external: true; only resolved SqliteEmbedded records change.
  • Confirmed the linker (LinkerContext.rs:3188) reads the flag only when source_index is invalid, so leaving the flag on embedded records is harmless and the metafile was the sole consumer that read it on a resolved record.
  • Checked bundler_bun.test.ts (UnknownBunSpecifier metafile assertion) still holds under the new condition; ImportRecordFlags remains used at MetafileBuilder.rs:352 so no dead import.
Extended reasoning...

The change narrows the external condition in src/bundler/linker_context/MetafileBuilder.rs:364 from a flag-or-no-source-index test to just no-source-index, plus about 200 lines of new tests in test/bundler/metafile.test.ts covering the embedded sqlite routes (static, sqlite_embedded, dynamic with and without splitting, onResolve variants, files map, shared importers, unused, external option, plain sqlite, CLI --metafile and --metafile-md). It touches no security-sensitive surface; it only affects a diagnostic JSON field. The second commit addressed three of the four test-shape nits from the prior review (exact entry for the unused case, stdout assertion in the CLI test, a comment on the plugin-path fixture), and the fourth was flagged as pre-existing and non-blocking. The Rust change is one line, verified behavior-preserving for every other producer of the flag, no CODEOWNER covers the paths, and the hunt ran dry with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant