Skip to content

test(aot): share one run between two tests of layers-of-a-test-run-share-one-realm.test.ts and state its output - #47855

Open
robobun wants to merge 3 commits into
mainfrom
robobun/123e589c/faster-layers-share-one-realm-test
Open

robobun wants to merge 3 commits into
mainfrom
robobun/123e589c/faster-layers-share-one-realm-test

Conversation

@robobun

@robobun robobun commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/aot/layers-of-a-test-run-share-one-realm.test.ts takes 307 s on debian 13 x64-asan (build #137158). Each of its three tests builds Bun's own library again: 70 to 90% of a run's CPU.
  • Its checks are floors: at least 115, 234 and 100 lines, and no exit code.

Fix

  • The first two tests share one project and one pair of runs. Three library builds become two.
  • Each test states a run's lines with its summary and exit code: every statement that comes through, the 3 values that differ in the second test, the 57 lines of the third.
  • Two probes of the third project could not fail (Error.prepareStackTrace, Error.stackTraceLimit). Each now can.
  • Verified: bun bd test takes 273 s before, 208 s after. Release build, 4 CPUs: 23 s before, 18 s after.

Background

  • bun test --aot in layers compiles the packages with Bun's library, then the project's files, then the test files, a few to a program.
  • The lowest layer of the first two tests is the same program. 58 of 59 programs of test files hold the same files as on main.
  • Considered one project for all three: one expression reads Bun.jest, so the third test's lowest layer would be built for mocks. Considered concurrent tests: AOT: hangs, wrong answers, failed builds; bun test --aot follows #44833's mocks #47605 keeps this file one by one.

Downsides

  • The first test's files now stand above the second's project file (671 functions, 7 on main).
  • On a machine without a program named true, the first two tests are skipped and say why. main runs them there.
Notes

Where the time goes. One run in layers of each project of main, release build, 12 CPUs, with BUN_JSC_logAOTPhaseTimes=1. "CPU" is user plus system time.

Project The whole run Lowest layer: Bun's library, and the package in the third Layer of the project Programs of test files
First test: 118 test files 7.3 s wall, 40.6 s CPU 5.3 s wall, 30.0 s CPU 0.2 s 30
Second test: 117 test files, src/all.js 8.8 s wall, 44.7 s CPU 5.8 s wall, 31.2 s CPU 0.9 s 30
Third test: 2 test files, a package, src/shared.js 9.2 s wall, 37.6 s CPU 7.6 s wall, 33.8 s CPU 0.9 s 2

With the debug build the lowest layer takes 75 s of the 125 s of the shared run, and 85 s of the 108 s of the third project's run. The plain runs take 0.1 to 0.4 s (release) and 2 to 7 s (debug).

What the shared project builds. The build of a layer prints its size with BUN_JSC_logAOTPhaseTimes=1. Release build.

Program On main In the shared project
Lowest layer of the first and of the second test 11,024 functions, 5,025,752 code bytes, 1,438 shapes, 90,374 call sites, for both the same numbers
Layer of the project First test: 7 functions. Second test: 671 functions, 755,536 code bytes, 4,159 call sites the numbers of the second test
Programs of test files 30 and 30 59
  • The 235 test files and src/all.js have the same bytes as on main. Only the names of the test files differ.
  • A program above the layers holds four test files that follow one another by name. A name begins with the number of its four (000-statement-000.test.js, 000-value-000.test.js), so a program holds files of one test.
  • 58 programs have the source bytes of a program of main and compile the same number of library files themselves (0, or 10 to 13). The last program holds 029-statement-116, 029-statement-117 and 029-value-117. On main those three files are two programs.
  • The run in layers of the shared project prints the 351 lines that plain Bun prints for the two projects of main. --isolate gives each test file a new global object, a new process.stdout and new performance entries, so a file sees nothing of the file before it.

Why the third test keeps its project. The build scans the test files of a run before it builds a layer. One expression of the first two tests is typeof Bun.jest. The scan then says mocks any module true and spies on objects from somewhere true for the whole run. The third project says false for both.

Lowest layer of the third project Functions Code bytes Call sites
As on main 11,128 5,026,880 91,287
With the one test file that reads Bun.jest beside its two 11,122 5,010,472 91,212

The first project without that expression says false for both too, and its lowest layer has 90,379 call sites, not 90,374. So one project for all three tests changes what each test builds. This PR changes no expression.

Why the tests stay one after another. With --concurrent the new file takes 10.6 to 12.4 s where it takes 14.5 to 17.4 s one by one (release, 12 idle CPUs, 3 rounds in turn). A lane of CI has no idle CPUs: the file runs in a batch, seven files at a time on 8 CPUs. #47605 imports testOneByOne for this file, with the reason that a case beside others passes its limit under a sanitizer. The hunk of #47605 applies on top of this PR, and the tests of this PR need nothing else from it.

What runs at once. The plain run and the run in layers of a project start together. Debug build, in turn: 247 s and 251 s one after the other, 228 s and 239 s together.

Local timing. Linux x64, 12 CPUs, 16 GB. The host takes a changing part of the machine, so wall time moves between rounds. User time is steady.

Debug build with ASAN. "The binary" is build/debug/bun-debug test, which bun bd test starts. The rows are in the order in which they ran. The last five ran one after another, hours after the others.

Run Wall User System
Before, bun bd test 357.7 s 2148 s 215 s
Before, the binary 388.6 s 2099 s 329 s
After, bun bd test, a draft with the same runs 250.7 s 1494 s 179 s
After, the binary, the same draft 227.7 s 1452 s 93 s
After, the binary, the same draft 238.9 s 1471 s 150 s
After, the binary, the first commit 256.1 s 1514 s 164 s
After, bun bd test, the first commit 225.4 s 1476 s 107 s
After, bun bd test, the second commit 199.9 s 1296 s 69 s
Before, the binary 278.6 s 1885 s 83 s
Before, bun bd test 272.6 s 1829 s 83 s
After, bun bd test, the second commit 207.6 s 1377 s 70 s
After, bun bd test, the third commit 205.5 s 1363 s 71 s

bun bd test runs the file as it is on main: 3 pass. In the last pair of "Before" and "After" the tests take 91 s, 98 s and 82 s before, and 115 s, 0.05 s and 91 s after.

Release build (build/release/bun-profile test), rounds in turn:

CPUs Before: wall, user After: wall, user
12 13.8 s, 76.9 s 11.4 s, 56.6 s
12 16.9 s, 81.8 s 13.3 s, 57.5 s
12 17.8 s, 80.3 s 14.4 s, 57.1 s
4 (taskset -c 0-3) 22.7 s, 71.4 s 16.8 s, 51.6 s
4 24.1 s, 73.9 s 19.0 s, 53.4 s

Six more runs of the new file with the release build passed: 12.5 to 15.3 s.

Assertions, before and after.

Before After
How a run ends Not checked. A line worker crashed fails the run in layers For the plain run and for the run in layers: 351 pass, 1 fail, Ran 352 tests across 235 files. and exit code 1. For the third project: 12 pass, 0 fail, Ran 12 tests across 2 files. and exit code 0. No line worker crashed in either. A test compares this in one object with the lines of the run, so a failure shows both
How the run in layers is built layers 2, by run() of the harness Also test files 235 | to a program 4, and test files 2 | to a program 1
First test At least 115 lines under plain Bun The 117 statements that come through, by number, for both runs
Second test At least 234 lines under plain Bun One own and one theirs line for each of 117 expressions, none says threw, and the three that differ are stated with both values
Third test At least 100 lines under plain Bun The 57 lines of answers.txt, each twice, for both runs
Layers against plain Bun All lines equal All lines equal

The one statement that throws makes a console.Console over two objects that can only write. Node v26.3.0 throws there too: TypeError: stream.removeListener is not a function.

One more statement starts a program named true. Where Bun.which("true") finds nothing, the first two tests are skipped, and the file says why on stderr. With an empty PATH the file ends with 1 pass, 2 skip, 0 fail. With the skip taken out there, the first test fails and shows - "SEEN 027 came through" beside 350 pass and 2 fail.

The three values that differ under plain Bun show one realm. The file of the project answers second. It counts the listener that the test file added to process.stdout (1, then 2), and the performance mark of the test file (1, then 2). import.meta.main is true in the test file and false in the file of the project.

Two faults, each put into a copy of both versions of the file and run with the release build:

Fault main's file This PR
Every run in layers gets 16 test files to a program, not 4 and 1 3 pass 3 fail: Expected: "test files 235 | to a program 4", Received: "test files 235 | to a program 16"
The file of the project always answers for the first expression The second test passes The second test fails: 99 of 117 values differ

Two more, in this PR's file alone. With one answer taken out of answers.txt, the third test fails and shows the line twice. With the test file of the second test kept for the expression that throws, and the counts raised for it, the second test fails on the two lines that the run in layers words otherwise.

One run for two tests. If the shared run fails, the first two tests fail together, with the same message.

Two probes of the third project. Both are in the fixtures of main, and neither could fail.

Probe On main Now
Error.prepareStackTrace set by the test file, a stack made in each place shared.txt printed text for every stack that is a string, prepared or not It prints the first line of the stack: prepared n in all three places
Error.stackTraceLimit set by the test file, a stack made in each place The limit was 3. A stack has two frames there, so it had three lines with or without the limit: 3 3 The limit is 1: 1 2

Plain Bun and the run in layers print the new lines, with the release build and with the debug build.

Removed. The word TestRunPartsGoOnAfterCrash in BUN_DEBUG_TEST_AOT_WORDS. No code in src/ reads it, and the release binary does not hold the text.

Not changed. The three titles, the expressions, the flags of the inner runs (--isolate --parallel=4 --timeout=60000), the limit of 600 s a test, and the files under test/aot/harness/. asks.txt and shared.txt change in the two probes above and nowhere else.

Line endings. listed splits the fixtures that are lists at /\r?\n/. With a copy of the four fixtures that ends every line with \r\n, the three tests pass. The second commit alone fails the third test there, because each stated answer keeps a \r.

Not run. Windows and macOS: the file runs on the three Debian lanes only. A release build with ASAN: a cold build of it did not fit the 16 GB of this machine, so the debug build stands in for the lane with a sanitizer.

CI. The file passed on its three lanes in build #137272 (the first commit) and in build #137284 (the third commit).

Lane main, build #137158 First commit, build #137272 Third commit, build #137284
debian 13 x64-asan 307 s 140.9 s 144.7 s
debian 13 x64 144 s 50.7 s 50.2 s
debian 13 aarch64 81 s 22.7 s 25.7 s

The column of main and the columns of this PR are not like for like. On main the file runs in a batch, seven files at a time on x64-asan. CI runs a file that a PR changes in a process of its own, beside one or two other files.


[auto-merge] gate passed · iteration 0 · 4 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/aot/layers-of-a-test-run-share-one-realm.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/aot/layers-of-a-test-run-share-one-realm.test.ts
bun test v1.4.4 (367d939d9)

test/aot/layers-of-a-test-run-share-one-realm.test.ts:
(pass) a value that the library or native code makes, used by one statement of a test file [112937.43ms]
(pass) the same values, in a test file and in a file of the project [42.67ms]
(pass) one of each thing, one queue of each kind, and what one layer sets the others see [89697.46ms]

 3 pass
 0 fail
 14 expect() calls
Ran 3 tests across 1 file. [204.63s]
Exit: 0
diff hotspot
.../layers-of-a-test-run-share-one-realm.test.ts   | 168 +++++++++++++++------
 .../answers.txt                                    |  57 +++++++
 .../layers-of-a-test-run-share-one-realm/asks.txt  |   2 +-
 .../shared.txt                                     |   2 +-
 4 files changed, 182 insertions(+), 47 deletions(-)

gate history · 2 passed · 0 rejected · iteration 0

evidence per changed file
file                                                      reads  edits  tests
test/aot/layers-of-a-test-run-share-one-realm.test.ts         4      3     48
…st/aot/layers-of-a-test-run-share-one-realm/answers.txt      0      0     50
test/aot/layers-of-a-test-run-share-one-realm/asks.txt        0      0     51
test/aot/layers-of-a-test-run-share-one-realm/shared.txt      0      0     51

…are-one-realm.test.ts

The first two tests each built Bun's own library as the lowest layer of
a run. That layer is the same program for both, so their test files are
now one project, which runs once under plain Bun and once in layers.
The third test keeps its own project.

Each run now states how it ends: the summary, the exit code, no dead
worker, and how many test files go to a program. The tests state which
statements come through, where a value differs between a test file and
a file of the project, and every line of the third project.
@github-actions github-actions Bot added the claude label Nov 3, 2026
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The test suite compares plain Bun and AOT layered runs for generated statements and values. It also checks shared-realm observations against expected answers and verifies run summaries.

Changes

Shared-realm test coverage

Layer / File(s) Summary
Compare plain and AOT expression runs
test/aot/layers-of-a-test-run-share-one-realm.test.ts
The test groups generated cases into files of four and compares statement and value outputs across plain and layered runs. It checks run summaries, the AOT files-per-program report, and specified differences between test-file and project-file evaluations.
Check shared-realm observations
test/aot/layers-of-a-test-run-share-one-realm.test.ts, test/aot/layers-of-a-test-run-share-one-realm/*
The test checks fixture answers and successful run summaries. The fixtures record observations across the helper, own, and pkg realms, including values, callbacks, prototypes, stack limits, store results, and identity.

Possibly related issues

  • oven-sh/bun#41357 — The tests check for worker-crash output and run summaries, but they do not fix or reproduce the reported SIGSEGV.

Suggested reviewers: jarred-sumner

Merge Risk: Minimal · up to d7e7e

The test-only changes share expensive runs and strengthen output assertions. No concrete merge-blocking risk is established; normal checks should pass before merging.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly identifies the main change: sharing one AOT test run between two tests and recording its output.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is complete and relevant. It explains the problem, fix, design decisions, limitations, verification steps, test results, and untested platforms. It does not use the exact template head…
Full details: Description check

Explanation

The description is complete and relevant. It explains the problem, fix, design decisions, limitations, verification steps, test results, and untested platforms. It does not use the exact template headings, but it provides the required information under equivalent sections such as Problem, Fix, and Verified.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 22bd7d1 and d7e7edc.

Files selected for processing: 1
  • test/aot/layers-of-a-test-run-share-one-realm.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 0 remain after this review.

@robobun

robobun commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 8:12 AM PT - Nov 3rd, 2026

❌ @robobun, your commit d7e7edc has 42 failures in Build #137284 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 47855

That installs a local version of the PR into your bun-47855 executable, so you can run:

bun-47855 --bun

@robobun

robobun commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. This is a test-only change to test/aot/layers-of-a-test-run-share-one-realm.test.ts and to the fixtures that only this file reads.

The file passes on its three lanes at the head commit, in build #137284. That build is red for 28 other test files. All of them also fail in the builds of recently merged PRs. This PR changes none of them.

How this was checked:

  • CI build #137284 (the head commit): 144.7 s on debian 13 x64-asan, 50.2 s on x64 and 25.7 s on aarch64. On main the file takes 307 s, 144 s and 81 s there, inside a batch. CI runs a changed file in a process of its own, so the two are not like for like.
  • bun bd test test/aot/layers-of-a-test-run-share-one-realm.test.ts (debug build with ASAN, 12 CPUs), one run after the other: 3 pass in 273 s on main, 3 pass in 208 s with this PR.
  • Release build on 4 CPUs: 23 s on main, 18 s with this PR.
  • The shared project builds the lowest layer that both projects of main build (11,024 functions, 5,025,752 code bytes, 90,374 call sites). Its runs print the 351 lines of those two projects.
  • The four review comments are answered in their threads: each has a fix in the second or the third commit.
  • Not run: Windows, macOS, and a release build with ASAN.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline findings, I checked the hardcoded summary at test/aot/layers-of-a-test-run-share-one-realm.test.ts:95 against the fixture: made-by-the-library.txt has 118 expressions, one of which throws, so 118 statement files + 117 value files = 235 files and 352 tests, 351 pass / 1 fail — the arithmetic matches. The 600000 ms per-test timeouts predate this change rather than being introduced by it.

Extended reasoning...

The change rewrites one AOT test file so two tests share a single pair of plain/--aot runs and pins exact summaries, exit codes, SEEN line sets and a new 57-line answers.txt fixture; it touches no production or security-sensitive code. Three inline findings about environment-dependent failures and a non-discriminating probe are already attached, so this body only records the fixture-count check that was ruled out.

…of-a-test-run-share-one-realm.test.ts

A run that ended otherwise than stated failed on its summary alone,
before its lines were compared. Each test now compares the lines and
the end of a run in one object, so a failure names the expression.

Two probes of the third project could not fail. The probe of
Error.prepareStackTrace printed "text" for every string, prepared or
not: it now prints what the stack begins with. The stack limit was 3,
over the two frames that a stack has there: it is now 1.

The first two tests start a program named `true`. Where the system has
none they are skipped, and the file says why on stderr.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

The follow-up commit addresses the three points from the earlier review (lines compared together with the run's end, the prepared probe now pins prepared n, and the true-less skip). Beyond the inline note, I also checked the hardcoded summary against the fixture: 118 expressions give 118 statement files plus 117 value files (235) and 118 + 234 tests (352), with only index 057 failing, so " 351 pass / 1 fail / 235 files" is consistent; and the memoized seenOnce result holds only strings, so disposing the temp dir after the first awaiter is fine.

Extended reasoning...

Test-only change to one AOT test file and its fixtures; no runtime source, no security-sensitive surface. The second push reworked assertions to compare SEEN lines and run summary in one object, fixed two probes in the third project's fixture, and gated the first two tests on Bun.which("true"). The restructuring changes what the first test's lowest layers contain (its files now sit above the second test's project file, as the PR notes), which is a trade-off an AOT maintainer may want to weigh, so a human look remains worthwhile.

…-realm.test.ts with either line ending

A checkout that converts text files ends the lines of answers.txt with
\r\n. The lines were split at \n alone, so each stated answer kept a \r
and the third test failed there.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant