Skip to content

test(fetch-leak): gate the streaming-abort leak fixture on off-heap growth - #36148

Closed
robobun wants to merge 2 commits into
mainfrom
farm/11017547/fetch-abort-leak-offheap-metric
Closed

robobun wants to merge 2 commits into
mainfrom
farm/11017547/fetch-abort-leak-offheap-metric

Conversation

@robobun

@robobun robobun commented Jul 27, 2026 •

Copy link
Copy Markdown
Collaborator

test/js/web/fetch/fetch-leak.test.ts "aborting in-flight streaming fetch() responses does not retain the buffered body off-heap" (added in #32662) has flaked on 86 of the last 399 Buildkite builds, almost all on the darwin lanes:

LEAK: RSS grew 57.8MB over 60 aborts (> 55MB)

86 observed values span 55.1 to 69.8 MB, median 59.1, against a 55 MB threshold. The sibling behavioural test ("discards the buffered body and errors the reader") is unaffected.

Cause

The leaked datum is the native ByteStream buffer, whose per-iteration size is bounded by one recv() (LIBUS_RECV_BUFFER_LENGTH = 512 KB) plus the kernel socket receive buffer. The fixture therefore retains a platform-dependent amount per iteration, and on the darwin CI hosts the 60-iteration RSS growth comes out as 43-58 MB fixed vs 75-81 MB unfixed. The 55 MB threshold was derived from a Windows measurement (32.8 MB noise) and sits inside darwin's noise band; there is no value between the two distributions that separates them reliably.

Both fixed and unfixed builds grow the JS heap linearly in the held Response/reader objects (~0.28 MB/iteration, identical on both). Subtracting JS-heap growth from RSS growth isolates the off-heap component, which is flat on a fixed build at any iteration count (allocator and transport overhead only) and grows per iteration on an unfixed one (each retained ByteStream buffer).

Fix

Gate on RSS growth - JS-heap growth at 120 iterations with a 64 MB threshold. Measured on the CI hosts:

platform fixed off-heap MB unfixed off-heap MB
darwin aarch64 (n=20) 24.3 - 42.3 104.0 - 112.6 (n=5)
darwin x64 (n=10) 18.1 - 29.2
linux x64 (n=5) 39.1 - 41.2

The highest fixed value (42.3 MB) is 21 MB below the threshold; the lowest unfixed (104.0 MB) is 40 MB above. Runs under 6-way concurrent load on the darwin host stayed at 25.3-39.4 MB.

Verification

bun bd test test/js/web/fetch/fetch-leak.test.ts -t "aborting in-flight streaming" passes locally. 20/20 fixture runs pass on darwin aarch64 with the latest canary; 5/5 fail with a pre-#32662 canary. Wall time goes from ~0.5 s to ~1.0 s release, ~3 s debug+ASAN.


no test proof · iteration 1 · Platform-specific test-only change; deferring to CI.

…rowth

The "aborting in-flight streaming fetch() responses does not retain the
buffered body off-heap" test has been flaking on 86 of the last 399
Buildkite builds, almost entirely on darwin (14/26 x64+aarch64):

    LEAK: RSS grew 57.8MB over 60 aborts (> 55MB)

Observed values 55.1-69.8 MB, median 59.1, against the 55 MB threshold.

The retained ByteStream buffer per iteration is bounded by one recv()
(LIBUS_RECV_BUFFER_LENGTH = 512 KB) plus the kernel socket receive
buffer, so the leak signature scales with the platform's SO_RCVBUF.
Measured on the darwin CI hosts, the 60-iteration RSS growth is
43-58 MB fixed vs 75-81 MB unfixed: the 55 MB threshold sits inside
the noise band and the separation is too small to be reliable.

Both fixed and unfixed builds grow the JS heap linearly in the held
Response/Reader objects (~0.28 MB/iter); subtracting that leaves an
off-heap component that is flat (~25-42 MB) on a fixed build at any
iteration count, and grows per iteration (~0.6 MB) on an unfixed one.
At 120 iterations:

  darwin aarch64 fixed    off-heap 24.3-42.3 MB  (n=20)
  darwin x64 fixed        off-heap 18.1-29.2 MB  (n=10)
  linux x64 fixed         off-heap 39.1-41.2 MB  (n=5)
  darwin aarch64 unfixed  off-heap 104.0-112.6 MB  (n=5)

Gate on off-heap growth at 120 iterations with a 64 MB threshold.
Runtime goes from ~0.5 s to ~1.0 s release, ~3 s debug+ASAN.
@coderabbitai

coderabbitai Bot commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 15 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: e1548d03-26c9-4e1e-b354-d4136bf9fb54

📥 Commits

Reviewing files that changed from the base of the PR and between 4eb6f99 and 398930d.

📒 Files selected for processing (2)
  • test/js/web/fetch/fetch-abort-stream-leak-fixture.ts
  • test/js/web/fetch/fetch-leak.test.ts

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — well-measured test de-flake that isolates the off-heap signal instead of raw RSS.

What was reviewed:

  • Fixture and its sole caller updated consistently (env var rename, iteration count, held= assertion); no stale MAX_GROWTH_MB references remain.
  • The new metric still asserts the original property (ByteStream buffer not retained off-heap) and the PR verifies it fails on a pre-#32662 build, so the regression guard is preserved.
  • Threshold (64 MB) sits between the measured fixed max (42.3 MB) and unfixed min (104 MB) with wide margins on both sides; wall time stays ~1s release / ~3s ASAN.
Extended reasoning...

Overview

Test-only de-flake of fetch-leak.test.ts "aborting in-flight streaming fetch() responses does not retain the buffered body off-heap" and its fixture fetch-abort-stream-leak-fixture.ts. The change switches the leak gate from raw RSS growth to RSS growth − JS-heap growth, doubles iterations (60→120), and moves the threshold from 55 MB (inside darwin's noise band) to 64 MB (between measured fixed 24-42 MB and unfixed 104-113 MB). No production code is touched.

Security risks

None. Test fixture only; no auth, crypto, or user-facing code paths.

Level of scrutiny

Low-to-medium. It's a leak-test threshold adjustment, and the primary REVIEW.md concern for de-flakes is whether the property the original assertion protected is still asserted. It is: the fixture still holds every Response/reader reachable and gates on off-heap growth, which is exactly where the #32659 leak lived. The PR description shows the new metric was validated against a pre-fix canary (5/5 fail) and 20/20 pass on the flakiest lane, satisfying the "bound well below the unfixed leak" and "still fails when the fix is reverted" rules.

Other factors

  • Verified there are no other consumers of the fixture or the renamed MAX_GROWTH_MB env var; the rename is complete.
  • heap0 is captured at the same point as rss0 (after the initial Bun.gc(true)), so the subtraction is apples-to-apples.
  • Doubling iterations keeps wall time within the ~1s-per-test budget (~1.0s release, ~3s debug+ASAN per the description).
  • The comment in the fixture documents the mechanism (why subtraction isolates the signal), which is durable non-obvious content per the repo's comment guidance.

@github-actions

Copy link
Copy Markdown
Contributor

This PR may be a duplicate of:

  1. test(fetch): widen fetch-leak RSS thresholds for macOS/Windows arm64 #33988 - Widens fetch-leak RSS thresholds for macOS/Windows arm64 to fix the same flaky leak detection tests

🤖 Generated with Claude Code

@robobun

robobun commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator Author

Not a duplicate of #33988. That PR widens thresholds on three different tests in this file (compress option, fragmented compressed, should not leak using readable stream); this one changes the metric for the aborting in-flight streaming ... off-heap test, which was added in #32662 on July 22 and did not exist when #33988 was opened. No overlapping lines; the two are complementary.

Of the 124 fetch-leak.test.ts flaky occurrences in the last 399 builds, 86 are the test this PR targets and 12 are the tests #33988 targets.

@robobun

robobun commented Jul 27, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 4:32 PM PT - Jul 27th, 2026

❌ @robobun, your commit 398930d has 1 failures in Build #83637 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 36148

That installs a local version of the PR into your bun-36148 executable, so you can run:

bun-36148 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only de-flake that tightens the leak metric rather than just widening a threshold.

What was reviewed:

  • Metric change (RSS − JS-heap growth) correctly isolates the off-heap ByteStream buffer the original test was written to catch; PR verified 5/5 failures on a pre-#32662 canary, so the assertion still protects the same property.
  • Env var rename MAX_GROWTH_MB → MAX_OFFHEAP_MB is consistent between fixture and test; no other references in the repo.
  • 64 MB threshold sits well between measured fixed (≤42.3 MB) and unfixed (≥104 MB) distributions per the PR's CI-host measurements.
Extended reasoning...

Overview

This PR de-flakes the "aborting in-flight streaming fetch() responses does not retain the buffered body off-heap" test in test/js/web/fetch/fetch-leak.test.ts and its fixture fetch-abort-stream-leak-fixture.ts. The test has flaked on 86/399 recent Buildkite builds (mostly darwin) because the 55 MB raw-RSS threshold sits inside darwin's noise band. The fix changes the gating metric from raw RSS growth to RSS growth − JS-heap growth, doubles iterations to 120, and sets a 64 MB threshold with a documented ~60 MB gap between fixed and unfixed measurements.

Security risks

None. Test-only change; no runtime code touched.

Level of scrutiny

Low-to-medium. This is a leak-test threshold adjustment, and the REVIEW.md guidance for de-flaking is explicit: "keep asserting the property the original assertion protected — branch per-platform rather than dropping precision." This PR does better than that — instead of widening or per-platform branching, it refines the metric to remove a confounding variable (JS-heap growth from held Response/reader objects, identical on fixed and unfixed builds), then re-derives a threshold with empirical measurements on the actual CI hosts. The PR description confirms the new gate still fails 5/5 on a pre-fix canary, so the regression it guards against is still caught.

Other factors

  • The env var rename is fully contained (grep confirms only the two touched files reference it).
  • heapStats().heapSize is already imported and used elsewhere in the file for the baseline, so the API is known-good.
  • Wall time increases from ~0.5s to ~1.0s release / ~3s debug+ASAN, which is acceptable for a leak test and well within the file's existing budget.
  • The comments added to both files explain the why (mechanism + measured bounds) rather than restating the code, matching the repo's comment guidance.
  • No outstanding reviewer comments; CodeRabbit was rate-limited and did not review.

@robobun

robobun commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator Author

Diff is green. The changed test ("aborting in-flight streaming fetch() responses does not retain the buffered body off-heap") passed on every lane in both build 83623 and build 83637.

Remaining CI red is unrelated to this change:

  • test/js/bun/http/serve.test.ts "releases a paused request body when the handler responds without reading it" on darwin 14 x64, both builds. This flake appears 133 times across the last 399 builds and is addressed by test(serve): fix darwin flake in the respond-without-reading-body backpressure test #36088.
  • Build 83637 also had a yellow retry on the "Request body HiveRef pool returns slot via Body.Value.deinit (does not leak) > String" test in the same file on Windows arm64 (68 MB vs 64 MB threshold). That test is untouched by this PR and has 26 prior occurrences in the same 399-build window.

Ready for review.

@robobun

robobun commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as part of a cleanup of stale pull requests. This PR has had no new commits since 2026-07-27, it conflicts with main, and its last CI run failed. This is not a judgment on the fix itself. If the problem still reproduces on a current build, reopen this PR after a rebase or open a new one against main.

@robobun robobun closed this Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant