Skip to content

test(aot): run the compiled runs of errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts concurrently and state every line - #47843

Open
robobun wants to merge 2 commits into
mainfrom
robobun/4ce739d2/errors-below-callbacks-concurrent-runs
Open

robobun wants to merge 2 commits into
mainfrom
robobun/4ce739d2/errors-below-callbacks-concurrent-runs

Conversation

@robobun

@robobun robobun commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts takes 349 s on debian 13 x64-asan (build #137158).
  • Its six bun test --aot runs go one by one in one test. Four build in layers, and each compiles Bun's own library again.
  • Two of a run's 28 lines are checked, by substring. Nothing checks that both test files ran.

Fix

  • Each run is its own concurrent test. bun test --max-concurrency says how many build at once.
  • Every run must print the 28 stated lines and the summary of two passing test files, say how it was built, and exit with 0.
  • A debug build now runs the plain run and both one-program runs, and skips the four runs in layers.
  • Verified: the table. bun bd test fails on main and runs 3 tests in 23 to 30 s here.
Main This PR
CI: x64-asan, x64, aarch64 349, 150, 97 s 89, 51, 35 s
Local, release, 12 CPUs 20 s 10 s

Background

  • bun test --aot compiles a test run as one program or in layers: the package with Bun's own code, the application above it, each test file above that.
  • The builds cost the same processor time as before. The file ends sooner where processors are free.
  • The CI row is not like for like: CI runs a file that a PR changes alone. On main it runs in a batch, two tests at once.
  • AOT: hangs, wrong answers, failed builds; bun test --aot follows #44833's mocks #47605 also makes the six forms tests and runs two before a merge. Its filter fits this table.
Notes

What the file asks, and what stays as it was. The project (files) is the same text, byte for byte. The seven commands are the same: plain bun test --isolate, and bun test --isolate --aot with TestRunLayers:0, TestRunLayers:1,TestRunPartFiles:1 and TestRunLayers:2,TestRunPartFiles:1, each without and with BUN_JSC_validateAOTRuntime=1. run() of harness/run.ts still fails a run that was not built in the layers it asked for. Each run has a folder of its own now.

One variable is new for every run: BUN_JSC_showPrivateScriptsInStackTraces=0. It is the default of every build but a debug build, and test/js/bun/util/reportError.test.ts and others pass it for the same reason.

The limit of a test. The one test of main has 480 s for its seven runs, and takes 306 to 354 s of them on x64-asan (eleven recent builds, 137158 to 137216). No test has a limit of its own now. A test holds one run and has the limit of the suite: 120 s, and 270 s on a build with a sanitizer or assertions. What the slowest test took:

Where Slowest test Limit
CI, x64-asan, all runs at once (the whole file took 89 s) under 89 s 270 s
CI, x64 and aarch64, all runs at once (51 to 53 s and 34 to 35 s) under 53 s 120 s
Local, AddressSanitizer, 8 CPUs, all at once 62 to 66 s 270 s
Local, AddressSanitizer, 2 CPUs, two at once 117 s 270 s
Local, release, 4 CPUs, all at once 22 to 23 s 120 s
Local, release, 1 CPU, two at once 50 s 120 s

The rows with one and two CPUs stand for a full batch of CI, where the file has about that much of the machine. With two CPUs and AddressSanitizer this PR's file takes 231 s. Main's file takes 306 to 354 s in its batch.

Assertions, before and after.

Before After
Lines of a run 28 counted. Lines 2 and 5 checked by substring. The compiled runs compared with the plain run All 28 stated in the file and compared exactly, for the plain run and for each compiled run
Test files of the project not checked 2 pass, 0 fail, 2 expect() calls, Ran 2 tests across 2 files. for every run
How a run was built run() checks layers 1 and layers 2. Nothing checks TestRunLayers:0 A run with TestRunLayers:0 must say AOT test run | one program, and no other run may
Exit code first, alone, for the plain run last, in the same comparison as the output, for every run
A failure the test stops at the first run that differs each run that differs fails by its name

toEqual with the stated lines, not toMatchInlineSnapshot: bun:test refuses snapshot matchers in concurrent tests.

Four faults, put into both versions of the file. Release build.

Fault Main's file This PR
a.test.js prints the line "one of the package too" before it names inThePackage, in every run 1 pass 7 fail. The diff shows the three lines
b.test.js skips its test 1 pass 7 fail: 1 pass, 1 skip, 1 expect() calls
The one-program runs are asked for with TestRunLayers:O, a letter for the digit, and are built in layers 1 pass, in 31.8 s 2 fail: builtAsOneProgram: false
Only the compiled runs give each another name 1 fail, at the first compiled run 6 fail, 1 pass (the plain run)

Where the time goes. One run of each form alone, release build, 12 CPUs.

Run Wall Processor time Functions compiled Program
plain 0.05 s 0.05 s
one program 1.2 s 2.8 s 299 1.0 MB
one layer 5.4 s 29.3 s 11,041 33 MB
two layers 5.6 s 28.7 s 11,032 in the lowest layer 33 MB

The project is 5 KB. In layers the lowest program holds Bun's own code for whatever the programs above may ask: 249 units. With BUN_JSC_logAOTPhaseTimes=1 a run in two layers says that this layer takes 4.7 s of 5.3 s, 25.1 s of user time. The application layer takes 0.18 s and the two test files 0.14 and 0.25 s. So the file is four builds of Bun's library, and its processor time does not change: 100 s of user time before and after (release, 12 CPUs).

All local runs. Linux x64, 12 vCPUs, 15.6 GB, builds of 3bf1a06. taskset for fewer CPUs. Each row is one round, taken in turn.

Release (bun run build:release test <file>), 12 CPUs:

Round Main This PR
1 20.93 s 9.84 s
2 19.96 s 10.47 s
3 19.88 s 10.25 s

Release, 4 CPUs, in the environment that scripts/runner.node.ts gives a file:

Round Main This PR, --max-concurrency=4 This PR, --max-concurrency=2
1 28.6 s 23.4 s 24.2 s
2 26.3 s 23.8 s 23.8 s

Release with AddressSanitizer (bun run build:asan), 8 CPUs, in the environment of the lane with a sanitizer:

Round Main This PR, --max-concurrency=8 This PR, --max-concurrency=2
1 86.4 s 63.7 s 64.2 s
2 87.1 s 66.9 s 88.2 s
3 118.2 s 66.9 s 72.4 s
4 109.4 s 62.9 s 67.4 s
  • --max-concurrency=8 (4 on the lanes without a sanitizer) is what the runner passes when the file runs in a process of its own. --max-concurrency=2 is what a parallel batch gets.
  • The slowest test took 62 to 66 s with all at once and 31 to 41 s with two at once (AddressSanitizer), 22 to 23 s and 12 s (release, 4 CPUs).
  • Kernel time on this machine is between 46 and 132 s for the same run of main. User time is steady: 402 to 421 s for main, 408 to 457 s for this PR.
  • Most memory in use at once, beyond what was in use before the run. AddressSanitizer: 1.9 GB for main, 6.2 to 6.5 GB with all at once, 3.2 to 3.5 GB with two at once. Release, 4 CPUs: 0.3 GB, 0.9 GB and 0.55 GB.
  • With four processors that do nothing else, main's runs already keep most of them busy. There the file ends 14% sooner. With twelve it ends 50% sooner.

Debug (bun bd test <file>):

  • Main: 1 fail in 3.0 s (4.9 s with the check of the build). Received: "SEEN made, as written | made < visit < step < each < inThePackage < flatIntoArrayWithCallback < forEachOf < work < <anonymous>". A debug build sets showPrivateScriptsInStackTraces (src/jsc/bindings/ZigGlobalObject.cpp, under BUN_DEBUG), so plain Bun lists the private helper that flatMap calls. No compiled run starts.
  • This PR: 3 pass, 4 skip in 22.9, 25.6 and 29.8 s. The plain run takes 1 s and each one-program run 20 to 27 s. One SKIPPED: line on stderr names the runs in layers and the reason.
  • One run in two layers alone takes 94 s with the debug build (573 s of processor time). All seven tests at once passed in 250 s, and the four runs in layers took 218 to 243 s each: the limit of the suite is 270 s there. CI has no debug lane.

CI, before. Build #137158, the only lanes that run the file: 349.29 s on x64-asan, 150.46 s on x64, 97.04 s on aarch64. On x64-asan it ran inside a batch of 36 files, bun test --parallel=7 --max-concurrency=2, which took 413 s. This file was the longest of the batch. On x64 and aarch64 the batch has 40 files, 3 at a time. Other builds: 306 to 354 s on x64-asan (eleven builds of pull requests), 130 to 167 s on x64 and 83 to 89 s on aarch64 (four builds of main).

CI, this PR. The runner starts a file that a PR changes in a process of its own, in front of the other files that build executables: with --max-concurrency=8 beside two other files on x64-asan, with --max-concurrency=4 beside one on x64 and aarch64. So all seven tests ran at once. Build #137237 ran the first commit, which had a limit of 480 s on each compiled run. Build #137247 ran the second commit.

Lane Main, build #137158, in its batch This PR, build #137237, alone This PR, build #137247, alone
debian 13 x64-asan 349.29 s 88.79 s 88.51 s
debian 13 x64 150.46 s 53.04 s 50.67 s
debian 13 aarch64 97.04 s 33.63 s 35.48 s
  • The first column is not the same way of running the file as the others. Once this PR is merged the file is in its batch again, with two of its tests at once. The local column --max-concurrency=2 is the nearest measure of that: the same time as with all at once where processors are free.
  • What ran beside the file was slower for that time. test/aot/servers/node.test.ts took 131 and 133 s on x64-asan (89 s in build #137158). test/aot/tools/same-code-wherever-the-text-stands.test.ts took 53 and 51 s on x64 (29 s) and 34 and 35 s on aarch64 (21 s).
  • Both builds are red for the same 41 other test files, none of them this file. 40 of them are red in build #137162, which is main's own build of the base commit. The other is test/js/web/request/request-clone-leak.test.ts on x64-asan, a lane that builds of main do not run. It is also red in builds #137193 and #137212 of other pull requests.

On top of #47605. That PR leaves this file out of the list that CI runs in front of a merge, makes each form a test behind a beforeAll, and runs two of the six forms unless AOT_WHOLE=1. I put this PR's file on its test/aot/harness and test/aot/supported.ts (22621ef), with its filter on the table, and ran it with main's build of Bun: 3 pass without the variable, 7 pass with AOT_WHOLE=1, 1 skip with AOT_WHOLE=0.

Designs that were not taken.

  • One build for a run and the run that validates it. bun test --aot writes the program to an unnamed file and starts itself again on it (start_second_process in src/runtime/cli/run_compiled.rs). Nothing is left for a second run.
  • The project without new Function("f", "return f"). The package hands each callback through it, so that the compiler cannot follow the function that gets the name. That is what the file is about.
  • All runs in one test behind Promise.all. The runner gives a file of a batch two tests at once, so that no more builds run than there are processors. Tests of their own obey it.
  • A skip of the whole file on a debug build. One variable makes the plain run and the one-program runs pass there.

[auto-merge] gate passed · iteration 0 · 1 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts
bun test v1.4.4 (367d939d9)

test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts:
SKIPPED: test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts: the runs in layers: a debug build of Bun takes a minute and a half or more for each, a release build runs them
(skip) a stack through three programs names its callbacks as plain Bun does: one layer
(skip) a stack through three programs names its callbacks as plain Bun does: two layers
(skip) a stack through three programs names its callbacks as plain Bun does: one layer that validates
(skip) a stack through three programs names its callbacks as plain Bun does: two layers that validate
(pass) plain Bun names the callbacks of a stack through three programs [1196.33ms]
(pass) a stack through three programs names its callbacks as plain Bun does: one program [29519.10ms]
(pass) a stack through three programs names its callbacks as plain Bun does: one program that validates [29711.35ms]

 3 pass
 4 skip
 0 fail
 3 expect() calls
Ran 7 tests across 1 file. [38.46s]
Exit: 0
diff hotspot
...at-may-get-a-display-name-in-a-test-run.test.ts | 96 +++++++++++++++++-----
 1 file changed, 74 insertions(+), 22 deletions(-)

gate history · 2 passed · 0 rejected · iteration 0

evidence per changed file
file                                                      reads  edits  tests
…backs-that-may-get-a-display-name-in-a-test-run.test.ts      4      1     45

…rrent test and state every line

test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts
ran its six `bun test --aot` runs one after another in one test. Each
run is now a test of its own, and the tests are concurrent, so
`bun test --max-concurrency` says how many of them build at once.

Every run must print the 28 stated lines and the summary of two passing
test files, and exit with 0. A run that asks to be one program must say
that it is one.

A debug build of Bun lists a private helper of flatMap in the stack of
plain Bun, so the file failed there. It now runs the plain run and the
two runs as one program on a debug build. It skips the four runs in
layers there and says why.
@github-actions github-actions Bot added the claude label Nov 3, 2026
@robobun

robobun commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 5:55 AM PT - Nov 3rd, 2026

❌ @robobun, your commit 3a2283c has 41 failures in Build #137247 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 47843

That installs a local version of the PR into your bun-47843 executable, so you can run:

bun-47843 --bun

@robobun

robobun commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. This is a test-only change to test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts, and the file passes on its three lanes in builds 137237 and 137247.

How this was checked:

  • CI, build 137247: the file took 88.51 s on debian 13 x64-asan, 50.67 s on x64 and 35.48 s on aarch64. Build 137237, of the first commit: 88.79, 53.04 and 33.63 s. On main it takes 349.29, 150.46 and 97.04 s (build 137158).
  • Those are not like for like. CI runs a file that a PR changes in a process of its own, with all its tests at once. On main the file runs in a batch, with two of its tests at once.
  • Local, bun run build:release test <file>, 12 CPUs, three rounds in turn: 19.9 to 20.9 s on main, 9.8 to 10.5 s with this PR.
  • Local, a release build with AddressSanitizer, 8 CPUs, in the environment of the runner, four rounds in turn: 86 to 118 s on main, 63 to 88 s with this PR.
  • bun bd test <file> (debug): main fails in 3.0 s at line 104, before any compiled run starts. This PR has 3 pass and 4 skip in 23 to 30 s.
  • Four faults, put into both versions of the file: main's file passes three of them, and this PR fails all four.
  • The file also passes on the harness of AOT: hangs, wrong answers, failed builds; bun test --aot follows #44833's mocks #47605, with that PR's filter on the table.

Both builds are red for the same 41 other test files, and none is this file. 40 of them are red in build 137162, which is main's own build of the base commit. The other is test/js/web/request/request-clone-leak.test.ts on x64-asan, a lane that builds of main do not run. It is also red in builds 137193 and 137212 of other pull requests.

The description has every run and how it was taken.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The test compares complete stack output and run results between plain Bun and AOT runs. It covers one-program, one-layer, and two-layer configurations, with and without runtime validation.

Changes

AOT stack trace tests

Layer / File(s) Summary
Expected output and result collection
test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts
The test defines expected stack frames, test summaries, and exit status. Its helper collects stack output, stderr summaries, one-program status, and exit code.
Plain Bun and AOT run matrix
test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts
Concurrent tests compare plain Bun with one-program, one-layer, and two-layer AOT runs. AOT runs also cover runtime validation. Layered cases are skipped in debug builds.

Possibly related issues

Suggested reviewers: jarred-sumner

Merge Risk: Minimal · up to 3a228

No actionable merge-blocking issue is established for this test-only change. It is mergeable subject to normal checks.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title is specific and matches the changeset. The PR changes only the AOT test file so that each of its seven runs becomes a separate concurrent test, and it adds stricter checks on output, exit co…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the template's two sections, "What does this PR do?" (as "Problem", "Fix", and "Background") and "How did you verify your code works?" (as the local and CI timing tables, the d…
Full details: Title check

Explanation

The title is specific and matches the changeset. The PR changes only the AOT test file so that each of its seven runs becomes a separate concurrent test, and it adds stricter checks on output, exit code, and build mode. The title is long at 142 characters, but it names the primary change. The author describes the change in terms of concurrency and output checks, which the file changes support.

Full details: Description check

Explanation

The description follows the template's two sections, "What does this PR do?" (as "Problem", "Fix", and "Background") and "How did you verify your code works?" (as the local and CI timing tables, the debug run, the fault-injection table, and the gate evidence). It is detailed and specific: it names the file, the commands, the concurrency settings, the before/after timings, and the design choices it rejected. The main weakness is length. The notes section is very long and repeats the core claims several times, which makes the author's key points harder to find. The verification also depends on CI and local measurements that a reviewer cannot easily reproduce. Even so, the required information is present and complete, so the description passes.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 6828417 and 3a2283c.

Files selected for processing: 1
  • test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline note, I also checked two things and ruled them out: the 28 literal expected lines follow the fixture's naming sequence in a.test.js (each shownAs/asWritten step and the "others" re-run with fresh each/inThePackage objects but the still-named probe), and the builtAsOneProgram: false expectation for layered forms cannot be tripped by the non-Linux "one program: this Bun loads one program a realm" line, since scripts/runner.node.ts runs this file only on the Debian and ASAN lanes.

Extended reasoning...

The PR rewrites one test file under test/aot, splitting a serial seven-run test into one plain-Bun test plus six concurrent --aot form tests, each in its own temp dir, with the full 28-line stdout, the bun test summary counts, the build form and the exit code asserted as one object. It touches no production code and no security-sensitive surface. The expected output is a hand-written literal that I could not execute here, and the only inline finding is a pre-existing per-test timeout nit, so a human familiar with the AOT test lanes should still glance at the concurrency and memory profile on the ASAN runners.

…acks to the suite

Each test holds one run now, so it needs no limit of its own. The suite
gives a test 120 s, and 270 s on a build with a sanitizer or assertions.
In CI the whole file took 89 s on the lane with a sanitizer, 53 s on x64
and 34 s on aarch64, with all its runs at once.

With a limit of 480 s on each of six tests, runs that hang two at a time
held the file for three times as long as the one test of main did.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the follow-up commit drops the per-test 480 s ceiling, and the remaining change is a self-contained, test-only restructuring of one file.

What was reviewed:

  • The 28 hard-coded SEEN lines and the 2 pass / 0 fail / 2 expect() calls / Ran 2 tests across 2 files. summary against the unchanged fixture in files (a.test.js probes, Error.stackTraceLimit loop, b.test.js) — they match what the fixture prints.
  • builtAsOneProgram against src/runtime/cli/run_compiled.rs: AOT test run | one program is printed for TestRunLayers:0 in both spellings, and run() in test/aot/harness/run.ts still enforces expectLayers for the layered forms.
  • The ruled-out 120 s timeout concern on the ASAN lane: test/aot/harness/default-timeout.ts raises the default to 270 s when isASAN || isDebug, so each concurrent layered run has headroom beyond the measured 62–66 s.
  • test.skipIf(...).concurrent(...) chaining and { ...bunEnv, KEY: undefined } follow existing patterns in test/ (e.g. test/bundler/bundler_splitting.test.ts, test/aot/supported.ts's alwaysSkipped proxy handles the chain when --aot is unsupported).
Extended reasoning...

The PR changes only test/aot/errors-below-callbacks-that-may-get-a-display-name-in-a-test-run.test.ts (+74/-22), splitting one serial 480 s test into seven concurrent tests with exact expected output, a stderr summary check, and a one-program check, while leaving the fixture project byte-identical. It touches no production code and no security-sensitive surface. The second commit (3a2283c) removed the per-test timeout that my earlier inline comment flagged, and the bug hunt ran dry with no findings. The file has no CODEOWNER, and the timeline shows no CHANGES_REQUESTED review or unresolved human objection, so a human look is not required.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant