Skip to content

test(aot): build the layers once for seven cases of require-cache-in-test-runs-3 and state what each case prints - #47758

Open
robobun wants to merge 2 commits into
mainfrom
robobun/e93aac3c/require-cache-3-share-layers
Open

test(aot): build the layers once for seven cases of require-cache-in-test-runs-3 and state what each case prints#47758
robobun wants to merge 2 commits into
mainfrom
robobun/e93aac3c/require-cache-3-share-layers

Conversation

@robobun

@robobun robobun commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Problem

  • test/aot/require-cache-in-test-runs-3.test.ts takes 307 s on debian 13 x64-asan (build 136399). Each of its 8 cases runs bun test --aot over two layers in its own project, and each such run compiles all of Bun's own code: 32 of 40 s locally.
  • The file does not state what a case prints.

Fix

  • Seven cases share one run in layers, as folders of one project. The other runs do not change.
  • One case keeps its own run: its build lets every file load again, which changes the lowest layer. The test asserts that.
  • Each case states the lines it prints and the tests it passes, and every run must tell that.
  • Verified: bun bd test takes 309 s before, 148 s and 160 s after (CPU time, user + sys: 2622 s before, 1077 s and 1140 s after). In CI on x64-asan: 36 s.

Background

  • Over layers, bun test --aot compiles Bun's own code as the lowest program, what the test files import above it, and each test file on top.
  • Considered one project for the runs as one program too (about 280 s of CPU time less). Three cases are then compiled to other code.

Downsides

  • If the shared run fails to build or times out, seven tests fail together. Before, one.
  • The runs in layers have the stage log on. It changes no byte of a program.
  • A -t filter for one case in layers builds the shared run: about 260 s of CPU time, before 230 s.
Notes

All numbers are from a debug build with ASAN on Linux x64, 12 processors, 16 GB. The host shares the processors with other machines, so the runs vary. user is the steadiest number.

Where the time goes. One command at a time, on a quiet host.

Command Wall CPU (user + sys) Peak memory of all its processes
bun test --isolate, one case 0.4 s 0.4 s 0.4 GB
bun test --isolate --aot, one case, TestRunLayers:0 18 s to 21 s 46 s to 50 s 1.2 GB
The same, TestRunLayers:2,TestRunPartFiles:1 39.8 s 231 s 1.8 GB
The same in layers, seven cases as folders of one project 48.6 s to 50.7 s 250 s to 262 s 2.0 GB to 2.2 GB
  • In the run of one case in layers, BUN_JSC_logAOTPhaseTimes=1 gives the steps. The lowest layer (the packages that test files share) is built from 0.2 s to 32.5 s. The layer above takes 4.4 s, the program of the test file 2.4 s, the worker 0.3 s.
  • Main's file starts 24 commands: 8 plain, 8 as one program, 8 in layers. This PR starts 19: 9 plain, 8 as one program, 2 in layers.

Timings of bun bd test <file>. Each row is one run. Rows of one block ran one after the other.

Host File Wall user sys Peak memory
quiet main's 215.6 s 2158 s 104 s 9.0 GB
quiet main's 230.9 s 2202 s 131 s 9.2 GB
quiet this PR, tests of one program before those of layers 106.1 s 906 s 88 s 6.9 GB
busy this PR 147.7 s 937 s 141 s 6.2 GB
busy main's 309.2 s 2356 s 267 s 9.1 GB
busy this PR 159.8 s 946 s 194 s 6.2 GB
very busy main's 497.3 s 2470 s 558 s 9.1 GB
very busy main's 517.6 s (1 of 8 tests timed out) 2141 s 490 s 9.0 GB
very busy this PR 275.4 s 1017 s 334 s 6.1 GB
very busy this PR 247.2 s 992 s 357 s 6.1 GB
nearly quiet this PR, head 129.5 s 910 s 98 s 6.2 GB
  • "Busy" and "very busy": the host took processor time away. /proc/stat counts 350 s to 800 s of steal in the busy runs and 1300 s to 3100 s in the very busy runs.
  • On the very busy host main's file is close to its limit of 270 s a test. Seven tests took 242 s to 261 s and the eighth timed out. The longest test of this PR took 173 s and 191 s there.
  • With the environment that scripts/runner.node.ts sets on the ASAN lane, and --max-concurrency=2 as in a bucket of 7 files on 8 processors: main's file 478.6 s (user 2283 s), this PR 241.3 s (user 955 s). Busy host.
  • No release build of this commit was at hand. CI gives those numbers.

Timings in CI. Each cell is one run. Main's file is from build 136399, this PR from build 136925 at the first commit and from build 136938 at the head.

Lane Main's file This PR, first commit This PR, head
debian 13 x64-asan 307 s 40.8 s 35.8 s
debian 13 x64 60 s 19.4 s 24.1 s
debian 13 aarch64 44 s 18.4 s 14.1 s
  • Main's build and the builds of this PR did not run the file alike. In build 136399 it ran in a bucket of 38 files, 7 files at a time, with 2 tests under way. In the builds of this PR the runner put it first in the group of files that build many executables, because the PR changes it: 3 files at a time on x64-asan and 2 on the other lanes, with 8 tests under way.
  • The file passes on all three lanes in both builds. The red tests of those builds are in files that this PR does not touch, and the CI report marks them as failing on main too.

Why seven cases can share a run in layers. Bytes of each program, from AOT test build | program bytes. "Alone" is a run inside the folder of the case, so every path is the same.

Program Each case alone Seven cases in one run All eight in one run
Lowest layer 34,357,612 34,357,600 36,733,280
Layer above 2,212,391 to 2,311,687 2,429,786 2,479,475
one.test.ts of case 1, of case 2 1,900,800 1,900,800 1,917,184
one.test.ts of case 3, of case 4 1,851,648 1,851,648 1,868,032
call.test.ts of case 5 1,851,666 1,851,666 1,868,050
statement.test.ts of case 5 1,884,419 1,884,419 1,900,803
one.test.ts of case 6 1,868,032 1,868,032 1,884,416
one.test.ts of case 8 1,884,416 1,884,416 1,900,800
  • The lowest layer of the shared run is 12 bytes smaller. The path of its folder is 4 bytes shorter and is stored 3 times.
  • The program of a test file has the same size alone and in the shared run. For cases 3 and 8 I also built it over both pairs of layers with --aot-build-test-part and read the compiler's map (BUN_JSC_aotMapFilePath): every function has the same bytes for every operation.
  • The layer above is one program for the seven cases. Of the 110 functions that the cases have there alone, 67 have the same bytes for every operation in the shared layer. 21 differ in operand sizes, or in the bytes that belong to no operation: the function of a module begins with 104 such bytes in place of 8. In 22 a call that is made in place alone is a direct call to shared code: Array.prototype.join, includes, and the string concatenation of look, of which the run holds seven copies. No operation of the module loader comes or goes.
  • The bytes were measured on the files as main's test writes them. In the shared run of this PR a test file also hands its own name to console.log ("c5/call.test.ts|"), so that a line of stdout says which test file it is of.

Why the eighth case is alone. src/file.cjs of "a file that takes itself out of require.cache while it runs" has delete require.cache[__filename]. The build cannot read that key. The stage log then says every file may load again, and every program of the run is compiled for that: the lowest layer has 2.4 MB more and each test file 16,384 bytes more (last column). The run of that case takes about 1.4 times the CPU time of a run without it.

  • The test holds both runs to this. The shared run must give no reason why every file may load again. The run of the eighth case must give its two reasons. Without the mark on the eighth case, all 8 tests in layers fail with the reasons in the message.

Why the runs as one program stay alone. I built the seven cases as one program and compared the compiler's map with the seven programs of main's test. Of 155 functions, 85 have the same bytes for every operation. In 5 functions of cases 3, 5 and 8 GetGlobalBinding appears, and in 2 CheckInitialized: their files look up globals such as console at run time and check a binding before they read it. Alone they do neither. One build for the seven takes 54 s of CPU time. Seven builds take about 335 s.

The order of the test files of a case. bun test sorts the test files of the folder that it starts in. It runs those of a folder below in the order in which the file system lists them (src/runtime/cli/test/Scanner.rs), which on ext4 is the order of a hash. The case with two test files lies in a folder below in the shared run. So what a run tells of a case lists its test files by name, each with its lines of stdout and of the report. With the two files named e.test.ts and a.test.ts, which this file system lists in that order, the first commit of this PR fails the test of that case in layers, and the head passes.

Faults. A debug build with a switch in src/aot that is not in this PR. Each row is one run of each file.

Fault Main's file This PR
No module has the flag IS_LISTED_IN_CACHE Cases 1 and 8 fail Cases 1 and 8 fail as one program. All 8 pass in layers
The build compiles another text than the disk holds: one source file of case 3, one of case 8, one test file of case 5 Cases 3, 5 and 8 fail Cases 3, 5 and 8 fail in both forms. The 18 other tests pass
  • In main's file a case that fails as one program never comes to its run in layers. In this PR each form has its own test.
  • In the shared run the four cases without a fault pass while three fail.
  • A test file of one case that imports a missing file (no switch needed): plain bun test and the run in layers end with 1. The two tests of that case fail, and the 14 other tests of plain and of layers pass.

Assertions.

Subject Before After
Plain bun test Exit code 0 The lines that the case prints, the lines of the reporter for its test files without the times, the exit code: equal to literals in the file
A compiled run stdout and the exit code equal those of plain bun test The same three things, for each case, equal those of plain bun test over the same project. Every line of stdout but the first (bun test v..) counts, blank lines too
Form of a run as one program Not read stderr has AOT test run | one program
Form of a run in layers layers 2 (run() of the harness) The same
Why every file may load again Not read See above. Only where the run has layers
Tests 8: one for each case 24: one for each case and each of plain, one program, two layers. The 8 names are the first part of the new names
  • On the canary of 18 September (367d939), where bun test --aot compiles nothing, main's file passes its comparisons of one program and fails at the form of the layers. In this PR the 8 tests of one program fail too.
  • The literals are arrays of strings and not inline snapshots: 7 of the 34 lines end in a space, and one begins with one.
  • The eight projects are the same files as before. git diff -w shows no changed line inside them.

Order of the tests. A run begins when the first test asks for it. So no more builds are under way than tests, and --max-concurrency of the runner still holds. The tests of the layers come first, and the case with its own run first of all. Then both builds in layers begin together, also with two tests at a time.

Limits.

  • The file runs on three lanes: debian 13 x64, aarch64 and x64-asan. A Bun on macOS or Windows, and a hardened build, load one program a realm and build no layers. CI does not run the file there, and I did not either. run() of the harness lets such a run pass without layers, and the test then leaves out the assertion on the stage log.
  • If a test times out, afterAll waits for the runs that are still under way before it removes their folders.
  • test/expected-durations.json comes from CI runs. This PR does not edit it.

The stage log. BUN_JSC_logAOTPhaseTimes=1 is on for the two runs in layers, so that a build says why every file may load again. The two layers of case 3, built with and without it, have the same SHA-256. It adds about 1 MB to the stderr that the test reads.

Also run. prettier --check and oxlint on the file. tsc with the file alone in include reports nothing for it.

[auto-merge] gate passed · iteration 1 · 1 files touched

passes on PR (with fix) 
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/aot/require-cache-in-test-runs-3.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "test/aot/require-cache-in-test-runs-3.test.ts"
bun test v1.4.4 (367d939d9)

test/aot/require-cache-in-test-runs-3.test.ts:
(pass) a CommonJS file finds its own module in require.cache while it runs, whoever has begun it: plain bun test [646.27ms]
(pass) files that require each other are all listed while the innermost runs: plain bun test [543.05ms]
(pass) what a required ES module imports sees that module listed, and not itself: plain bun test [566.19ms]
(pass) an ES module is listed while it runs if a require() has begun it, and not if an import has: plain bun test [588.88ms]
(pass) an ES module that an import has begun is listed from the first require() of it: plain bun test [574.67ms]
(pass) an ES module whose import fails behind a require() of it that has returned stays listed, and require() answers: plain bun test [360.81ms]
(pass) a file that takes itself out of require.cache while it runs: a CommonJS file runs again, an ES module does not: plain bun test [390.59ms]
(pass) a file that throws while it is required was listed until then, and is not afterwards: plain bun test [415.88ms]
(pass) a CommonJS file finds its own module in require.cache while it runs, whoever has begun it: two layers [132118.57ms]
(pass) an ES module that an import has begun is listed from the first require() of it: two layers [12.31ms]
(pass) a file that throws while it is required was listed until then, and is not afterwards: two layers [3.98ms]
(pass) an ES module whose import fails behind a require() of it that has returned stays listed, and require() answers: two layers [3.09ms]
(pass) an ES module is listed while it runs if a require() has begun it, and not if an import has: two layers [131824.04ms]
(pass) what a required ES module imports sees that module listed, and not itself: two layers [131819.52ms]
(pass) files that require each other are all listed while the innermost runs: two layers
... (truncated)
Exit: 0
diff hotspot 
test/aot/require-cache-in-test-runs-3.test.ts | 292 ++++++++++++++++++++++----
 1 file changed, 251 insertions(+), 41 deletions(-)

gate history · 2 passed · 0 rejected · iteration 1

evidence per changed file 
file                                           reads  edits  tests
test/aot/require-cache-in-test-runs-3.test.ts      4      1     38

@robobun
test(aot): build the layers once for seven cases of require-cache-in-…
1dc8493 
…test-runs-3

Each of the eight cases ran `bun test --aot` over two layers in a project
of its own. Each such run compiles all of Bun's own code for the lowest
layer.

Seven cases now share one run in layers, as the folders of one project.
`TestRunPartFiles:1` keeps each test file a program of its own. The case
whose file deletes `require.cache[__filename]` keeps its own run: its
build lets every file load again, which changes the lowest layer. The
test asserts what the stage log of both runs says about that.

The plain runs and the runs as one program do not change.

Each case states the lines that it prints and the tests that it passes.
Plain `bun test` and each compiled run must tell that of the case. A run
as one program must say its form. There is one test for each case and
each form: 24 tests, before 8.
@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 271b65e and 914205b.

Files selected for processing: 1
  • test/aot/require-cache-in-test-runs-3.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 0 remain after this review.

Walkthrough

The test file replaces separate require-cache test callbacks with a case-driven harness. It runs each case with plain bun test, one-program AOT, and two-layer AOT, then checks output, reports, and applicable stage-log entries against expectations.

Changes

Require-cache test runs

Layer / File(s) Summary
Require-cache scenario cases
test/aot/require-cache-in-test-runs-3.test.ts		 Defines cases for CommonJS and ES-module cache visibility, circular loads, thrown modules, and cache deletion. Each case records fixture files and expected output and reports.
Test mode execution and result parsing
test/aot/require-cache-in-test-runs-3.test.ts		 Defines plain, one-program AOT, and two-layer AOT modes. The run helper executes cases in shared or separate projects, parses per-case results and stage-log data, and the cleanup hook disposes temporary folders.
Plain and compiled run assertions
test/aot/require-cache-in-test-runs-3.test.ts		 Checks plain-run expectations for every case, compares compiled output with plain output, and checks two-layer stage-log entries when available.

Possibly related PRs

  • oven-sh/bun#47755: Updates adjacent require-cache-in-test-runs-4 AOT cases with harnesses that check printed output and require-cache behavior.

Suggested reviewers: jarred-sumner

Merge Risk: Minimal · up to 91420

The change consolidates require-cache tests and adds explicit output expectations. No actionable merge-blocking risk is established; merge after normal checks pass.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly identifies the main changes: sharing AOT layers across seven cases and stating each case's output.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is detailed and follows the repository template in substance. It covers the problem, the fix, and verification with timings and CI results. The template asks for "What does this PR do?…
Full details: Description check

Explanation

The description is detailed and follows the repository template in substance. It covers the problem, the fix, and verification with timings and CI results. The template asks for "What does this PR do?" and "How did you verify your code works?" Both are present: the Problem and Fix sections describe the change, and the Notes section and the verification bullets describe how the author tested it. The description uses headings such as Problem, Fix, Background, Downsides, and Notes rather than the exact template headings. The author should rename or map the content to the two template headings so reviewers can find them. The description also contains a large amount of detail in a collapsed block, which makes the core summary harder to scan. The description has no separate statement of the exact commands run for the verification section beyond the timing runs and the lint and type checks, which is adequate. Overall the description is mostly complete.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions github-actions Bot added the claude label Nov 2, 2026
@robobun

robobun commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author
Updated 11:37 AM PT - Nov 2nd, 2026

❌ @robobun, your commit 914205b has 40 failures in Build #136938 (All Failures):

🧪   To try this PR locally:

bunx bun-pr 47758

That installs a local version of the PR into your bun-47758 executable, so you can run:

bun-47758 --bun

@robobun

robobun commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Status: ready for review.

At the head commit the file passes on its three lanes (build 136938). The red tests of that build are in files that this PR does not touch, and the CI report marks them as failing on main too. The review findings are answered in 914205b.

How the time was measured.

Where Main's file This PR
bun bd test of the file, debug build with ASAN, Linux x64, 12 processors, runs one after the other 309 s, CPU (user + sys) 2622 s, 8 pass 148 s and 160 s, CPU 1077 s and 1140 s, 24 pass
CI, debian 13 x64-asan 307 s (build 136399) 40.8 s (build 136925), 35.8 s (build 136938)
CI, debian 13 x64 60 s 19.4 s, 24.1 s
CI, debian 13 aarch64 44 s 18.4 s, 14.1 s

Main's build and the builds of this PR did not run the file alike: the builds of this PR ran it first in its group because the PR changes it. The description has the details, every local run, and what was compared to show that the seven cases get the same programs in the shared run.

claude[bot]
claude Bot reviewed Nov 2, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread test/aot/require-cache-in-test-runs-3.test.ts Resolved
Comment thread test/aot/require-cache-in-test-runs-3.test.ts Resolved
Comment thread test/aot/require-cache-in-test-runs-3.test.ts Resolved
Comment thread test/aot/require-cache-in-test-runs-3.test.ts Resolved
Comment thread test/aot/require-cache-in-test-runs-3.test.ts Resolved
@robobun
test(aot): list the test files of a case by name in require-cache-in-…
914205b 
…test-runs-3

`bun test` runs the test files of a subfolder in the order in which the
file system lists them. In the shared run the case with two test files
lies in a subfolder, so the order of its report depended on the file
system. A line of stdout now says which test file it is of, and what a
run tells of a case lists its test files by name.

Also:
- A blank line of stdout counts again. Only the banner is left out.
- A test file that does not load fails its own case alone in the shared
  run.
- The assertion on the stage log holds only where the run has layers.
- afterAll waits for the runs before it removes their folders.
- The output of a failed run leaves out the warnings that the stage log
  adds.
claude[bot]
claude Bot reviewed Nov 2, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

This was referenced Nov 2, 2026
Open
Open

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

claude

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@robobun