Skip to content

test(serve-body-leak): run the fixture without the ASAN quarantine and with eager page release - #41560

Open
robobun wants to merge 1 commit into
mainfrom
robobun/a9391ab3/serve-body-leak-asan-quarantine
Open

robobun wants to merge 1 commit into
mainfrom
robobun/a9391ab3/serve-body-leak-asan-quarantine

Conversation

@robobun

@robobun robobun commented Sep 6, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/js/bun/http/serve-body-leak.test.ts fails on the x64-asan lane: expect(report.leak).toBeLessThanOrEqual(64) received 83 (build #110910, "streaming the body incompletely"). Two earlier x64-asan hits on other PRs measured 70 and 80 ("streaming the body"). No hit on main in the 125 main builds before. No single culprit commit: the flake is allocator noise with no margin.
  • The cause is how ASAN frees memory in the fixture process. The quarantine pins up to 256 MB of freed blocks, and the allocator returns free pages to the OS at most every 5 s. report.leak is the difference of two RSS samples 2000 requests apart. With no leak, that difference moved by 60 to 90 MB (serve-body-leak.test.ts:125).

Fix

  • Spawn the fixture with quarantine_size_mb=0, thread_local_quarantine_size_kb=0 and allocator_release_to_os_interval_ms=0 in ASAN_OPTIONS. Freed memory leaves RSS at once, so RSS tracks live memory. The options are inert off ASAN.
  • This is the same idiom that archive.test.ts, fetch-leak.test.ts and harness.ts (expectRssDeltaBelow) use for RSS leak checks under ASAN. No bound changes.
  • The test still detects a leak. With these options and a fixture that retains one chunk per streaming request, leak measured 124 MB (incomplete streaming) and 1043 MB (full streaming), against the 64 MB bound.
  • Verified: bun bd test test/js/bun/http/serve-body-leak.test.ts (debug ASAN, 15 pass, max leak 4). Before the change, the same build measured leak 63 and 64 with the default options.

Background

  • The fixture is one Bun.serve process. /report runs Bun.gc(true), sleeps 50 ms and returns its RSS. The test sends batches of 512 KB bodies and samples RSS every 1000 requests.
  • ASAN's allocator does not reuse freed memory at once. Freed blocks wait in a FIFO quarantine (256 MB by default on x86_64) so a use after free can be caught. Free pages of the size-class regions are given back to the kernel only on a timer.
  • Related: test(serve-body-leak): cut the request volume and tighten the assertions #40088 rewrites this test and its fixture (fewer requests, a settle loop, a 32 MB quarantine). This PR is the minimal change to stop the failure on main.
Notes

Measurements on the local debug ASAN build, leak in MB per scenario run (each run is 3000 streaming requests, the statistic is the last sample minus the first one):

  • default ASAN options, 10 runs: 27, 41, 0, 0, 36, 64, 0, 0, 20, 0 (max 64). Full test file runs: 36, 39, 63 for the streaming scenarios.
  • quarantine_size_mb=0:thread_local_quarantine_size_kb=0, 10 runs: 0, 57, 28, 0, 0, 46, 39, 0, 0, 44 (max 57).
  • plus allocator_release_to_os_interval_ms=0, 10 runs: 3, 0, 0, 12, 0, 6, 0, 17, 3, 0 (max 17). Two full test file runs: max 1 and max 37. bun bd test after rebuild at HEAD: max 4.
  • positive control with the final options, fixture patched to retain every streaming chunk: streaming 1043, incomplete streaming 124.
  • A settle loop in the fixture (repeat Bun.gc(true) + 50 ms until RSS stops falling) did not reduce the residual noise (max 42 over 10 runs), so it is not part of this change.
  • process.memoryUsage() in the fixture at each sample shows heapUsed under 20 MB while RSS moves by 40 to 50 MB, so the residual noise is native allocator state, not JS heap.

CI scan: 400 builds between 2026-09-03 and 2026-09-06 on all branches, 3 failures of this file, all on x64-asan (leak 70, 80, 83). 125 main builds between 2026-08-20 and 2026-09-03: none.

CI on this PR (build #111033, x64-asan release): the file passes, 15 tests in 38 s. The fixture's RSS baseline dropped from 495 to 650 MB (build #110910) to 290 to 340 MB. leak per scenario: 0, 0, 3, 0, 35, 0, 0 over HTTP/1, then 0, 0, 0, 0, 2, 0, 0 over HTTP/2. The remaining 35 is in the streaming scenario (samples 300 and 343 MB, end 338 MB).


[auto-merge] gate passed · iteration 0 · 1 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/js/bun/http/serve-body-leak.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/js/bun/http/serve-body-leak.test.ts
bun test v1.4.3 (f42e98025)

test/js/bun/http/serve-body-leak.test.ts:
http://localhost:37263/
{
  leak: 2,
  start_memory: 404037632,
  peak_memory: 404037632,
  end_memory: 381464576,
  memory_examples: [ 379367424, 381464576 ],
}
(pass) request body leak (http2: false) > #10265 should not leak memory when ignoring the body [4853.32ms]
{
  leak: 0,
  start_memory: 381464576,
  peak_memory: 395354112,
  end_memory: 391725056,
  memory_examples: [ 391958528, 395354112 ],
}
(pass) request body leak (http2: false) > should not leak memory when buffering the body [9318.45ms]
{
  leak: 2,
  start_memory: 391725056,
  peak_memory: 411160576,
  end_memory: 411074560,
  memory_examples: [ 408842240, 411160576 ],
}
(pass) request body leak (http2: false) > should not leak memory when buffering a JSON body [35991.25ms]
{
  leak: 1,
  start_memory: 411074560,
  peak_memory: 411074560,
  end_memory: 400551936,
  memory_examples: [ 399282176, 400252928 ],
}
(pass) request body leak (http2: false) > should not leak memory when buffering the body and accessing req.body [12506.48ms]
{
  leak: 10,
  start_memory: 400551936,
  peak_memory: 467083264,
  end_memory: 438861824,
  memory_examples: [ 428195840, 467083264 ],
}
(pass) request body leak (http2: false) > should not leak memory when streaming the body [13434.39ms]
{
  leak: 0,
  start_memory: 438861824,
  peak_memory: 513155072,
  end_memory: 476024832,
  memory_examples: [ 493940736, 513155072 ],
}
(pass) request body leak (http2: false) > should not leak memory when streaming the body incompletely [10455.30ms]
{
  leak: 2,
  start_memory: 476024832,
  peak_memory: 476024832,
  end_memory: 392167424,
  memory_examples: [ 390070272, 391118848 ],
}
(pass) request body leak (http2: false) > should not leak memory when streaming the body and echoing it back [8365.58ms]
https://localhost:39051/
{
  l
... (truncated)
Exit: 0
diff hotspot
test/js/bun/http/serve-body-leak.test.ts | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

gate history · 1 passed · 0 rejected · iteration 0

evidence per changed file
file                                      reads  edits  tests
test/js/bun/http/serve-body-leak.test.ts      1      2     10

root cause · written by the author bot

The test's leak value is the difference between two RSS samples taken from a fixture process running under ASAN's default allocator settings, where freed blocks sit in a 256 MB quarantine and pages are only returned to the OS every 5 seconds, so this deferred freeing alone shifted the measurement by 60 to 90 MB on the x64-asan lane and tripped the 64 MB bound with no real leak present. The fix spawns the fixture with quarantine_size_mb=0, thread_local_quarantine_size_kb=0, and allocator_release_to_os_interval_ms=0 appended to ASAN_OPTIONS, the same idiom used in harness.ts and o…

…d with eager page release

Under ASAN, freed memory stays resident: the quarantine pins up to 256 MB
of freed blocks and the allocator returns free pages to the OS at most
every 5 s. The fixture's RSS then moves by 60 to 90 MB between two
samples taken 2000 requests apart with no leak, which is over the 64 MB
allowance (x64-asan, build #110910, leak 83).

Spawn the fixture with quarantine_size_mb=0,
thread_local_quarantine_size_kb=0 and
allocator_release_to_os_interval_ms=0, as archive.test.ts and
fetch-leak.test.ts already do. A retained 64 KB chunk per request still
measures about 120 MB and a retained body over 1 GB.
@github-actions github-actions Bot added the claude label Sep 6, 2026
@robobun

robobun commented Sep 6, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 5:10 AM PT - Sep 6th, 2026

❌ @robobun, your commit 1274c64 has 4 failures in Build #111033 (All Failures):

  • test/js/bun/websocket/websocket-server-rsv-frames.test.ts - code 1: error: upgrade failed: HTTP/1.1 200 OK on 🍎 any aarch64
  • test/js/node/test/parallel/test-crypto-dh-leak.js - code 1 on 🐧 13 x64-asan
  • test/js/bun/test/parallel/test-docker-build-debian-slim.ts - timeout on 🐧 25.04 x64
  • ❌ CPU instruction violation on Linux x64 — 1 check(s) failed
  • The baseline build contains instructions not available on Nehalem (SSE4.2, no AVX/AVX2/AVX512).

    • Static instruction scan

    Static scan violations

    llint_op_wide16  [INVLPGB]  (1 insns)
    

    If these are runtime-dispatched behind a CPUID gate: add each symbol to scripts/verify-baseline-static/allowlist-x64.txt with a comment pointing at the gate.

    If there's no gate: this is a real bug — a -march leaked into a subbuild.

  • ❌ CPU instruction violation on Linux x64 — 1 check(s) failed
  • The baseline build contains instructions not available on Nehalem (SSE4.2, no AVX/AVX2/AVX512).

    • Static instruction scan

    Static scan violations

    llint_op_wide16  [INVLPGB]  (1 insns)
    

    If these are runtime-dispatched behind a CPUID gate: add each symbol to scripts/verify-baseline-static/allowlist-x64.txt with a comment pointing at the gate.

    If there's no gate: this is a real bug — a -march leaked into a subbuild.


🧪   To try this PR locally:

bunx bun-pr 41560

That installs a local version of the PR into your bun-41560 executable, so you can run:

bun-41560 --bun

@robobun

robobun commented Sep 6, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: test-only change, ready for review.

Reproduced on the local debug ASAN build: leak 63 and 64 with the default ASAN options, bound 64. With the fixture's ASAN quarantine off and eager page release, bun bd test test/js/bun/http/serve-body-leak.test.ts passes (15 pass, max leak 4). A fixture patched to retain one chunk per request still measures 124 MB and 1043 MB, so the test keeps its signal.

CI (build #111033): serve-body-leak.test.ts passes on x64-asan (max leak 35, baseline RSS down from about 600 MB to about 300 MB). The red lanes are unrelated to this diff: verify-baseline on linux x64 and x64-musl fails the same way on main (build #110907, fix in #41549), test-docker-build-debian-slim.ts timed out in a docker apt-get step, websocket-server-rsv-frames.test.ts failed once on darwin aarch64, and test-crypto-dh-leak.js on x64-asan is marked pre-existing. None of them recur in the last 60 builds.

@coderabbitai

coderabbitai Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Essentials

Run ID: d2f16af2-05b3-42f8-99d9-fcac0e243865

📥 Commits

Reviewing files that changed from the base of the PR and between ae7b8f4 and 1274c64.

📒 Files selected for processing (1)
  • test/js/bun/http/serve-body-leak.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 0 remain after this review.


Walkthrough

The body-leak fixture subprocess now receives ASAN allocator settings that disable quarantines and enable immediate page release. Existing environment values remain preserved, and the settings remain inert outside ASAN.

Changes

Body-leak fixture

Layer / File(s) Summary
ASAN environment configuration
test/js/bun/http/serve-body-leak.test.ts
The fixture merges bunEnv with filtered ASAN options and joins the options with colons before spawning the subprocess.

Suggested reviewers: jarred-sumner

Merge Risk: ⚪ Minimal · up to 1274c

The fixture configures ASAN to release freed memory promptly, reducing RSS measurement noise while retaining its leak thresholds. No current merge-blocking risk is identified.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the test and the primary change: disabling the ASAN quarantine and enabling eager page release. It is concise and specific.
Description check ✅ Passed The description explains the problem, fix, background, verification steps, test results, and leak-detection coverage. Although it does not use the exact template headings, it provides the required inf…

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only ASAN de-flake following the established ASAN_OPTIONS idiom.

What was reviewed:

  • The { ...bunEnv, ASAN_OPTIONS: [bunEnv.ASAN_OPTIONS, ...].filter(Boolean).join(":") } shape matches harness.ts expectRssDeltaBelow and a dozen other RSS-leak tests exactly, so existing harness ASAN_OPTIONS (allow_user_segv_handler=1:...) are preserved.
  • The 64 MB assertion is unchanged; per the positive-control numbers a real per-request retention still measures 124 MB–1 GB, so the test still catches the regression it guards.
  • Inert on non-ASAN builds (env var is ignored); colon-separated option syntax is the standard sanitizer format on all lanes.
Extended reasoning...

Overview

This PR touches a single test file, test/js/bun/http/serve-body-leak.test.ts, changing only the env passed to the spawned fixture subprocess in beforeAll. It appends three ASAN allocator options (quarantine_size_mb=0, thread_local_quarantine_size_kb=0, allocator_release_to_os_interval_ms=0) to ASAN_OPTIONS so freed memory leaves RSS immediately, eliminating 60–90 MB of quarantine/page-release jitter that was tripping the 64 MB leak allowance on the x64-asan lane with no actual leak present.

Security risks

None. This is a test-environment tweak affecting only the ASAN runtime's allocator behavior in a spawned test fixture. No production code, no user input, no auth/crypto/permission surface.

Level of scrutiny

Low. The change is mechanical and copies an idiom already used verbatim in test/harness.ts (line 355) and ~12 other leak tests (fetch-leak.test.ts, archive.test.ts, etc.). It spreads bunEnv per REVIEW.md convention, preserves the harness-set ASAN_OPTIONS via .filter(Boolean).join(":"), and leaves the assertion threshold untouched — so it does not weaken the safety net. The PR description documents a positive control (patched fixture retaining chunks measures 124 MB / 1043 MB against the 64 MB bound), satisfying the "bound well below what the unfixed leak produces" requirement.

Other factors

The added comment is on the longer side, but it explains a genuinely non-obvious mechanism (ASAN's 256 MB quarantine + 5 s page-release timer causing RSS jitter between two samples) that a future reader would otherwise spend real effort rediscovering — it clears the CLAUDE.md rule-14 bar. Bug hunt exited on dry_streak with no findings and no ruled-out candidates. No prior reviews or outstanding objections on the timeline.

@robobun

robobun commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

The automated review raised no actionable comments, and there are no open review threads. Buildkite build #111033 is running. The test-only diff stays as pushed (1274c64).

@robobun

robobun commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks. No changes requested, so the diff stays as pushed (1274c64). Waiting on Buildkite build #111033.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant