Skip to content

test: deflake the h2 stream-release cases on debug builds - #42357

Open
robobun wants to merge 2 commits into
mainfrom
robobun/ea0f4155/h2-release-jit-plan-stragglers
Open

robobun wants to merge 2 commits into
mainfrom
robobun/ea0f4155/h2-release-jit-plan-stragglers

Conversation

@robobun

@robobun robobun commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • On a debug build, "stream release after a queued END_STREAM" in test/js/node/http2/h2-conformance.test.ts fails with Expected: <= 3, Received: 4 (or 5): 7 of 12 runs with 2 cores.
  • A heap snapshot shows each survivor as a direct root with reason JITWorkList. It is not a leak and not the stack scan.
  • liveCount() stops when the count does not move for 10 passes. The streams that a compile plan roots go at about pass 11.

Fix

  • liveCount() stops when the count is within GC_STRAGGLERS, not on a plateau. The assertion is <= GC_STRAGGLERS, so a later pass cannot change the result. The 50-pass limit stays.
  • It stops after 10 passes only if nothing at all was collected, which is the leak these cases guard against. Without the release line from node:http2: release streams whose END_STREAM was flushed from the outbound queue #38044, the five cases still fail with Received: 16.
  • Verified on the debug build: the file passes 12 of 12 runs with 2 cores, 15 of 15 with 12 cores, 8 of 8 under CPU load.

Background

  • JSC compiles hot functions on worker threads. One compile is a plan in JITWorklist.
  • The plan keeps the arguments of the call that crossed the tier-up threshold (for an Http2Stream method, this is the stream). The GC marks them as roots until the plan is installed.
  • Only a tier-up check on the main thread installs a finished plan. Few such checks run while liveCount() polls.
  • A debug build needs seconds for one compile, so several plans are pending when the requests end.
Notes

Reproduction. bun bd, then taskset -c <2 cpus> ./build/debug/bun-debug test test/js/node/http2/h2-conformance.test.ts. Unmodified file: 7 of 12 runs fail (Received: 4 or 5). With BUN_JSC_useConcurrentJIT=0 the unmodified file passes 8 of 8 in the same setup. With all 12 cores about 1 run in 5 to 10 shows a plateau above 3.

Snapshot. generateHeapSnapshotForDebugging() on a fresh turn, after the count held above 3 for three passes. Reset case (32 streams, 9 alive): 8 ServerHttp2Stream cells are direct roots with reason JITWorkList, the ninth hangs off a JITWorkList-rooted WritableState (Object -[onwrite]-> bound function -> stream). Compat case: 3 ServerHttp2Stream direct JITWorkList roots. Client case: 5 ClientHttp2Stream direct JITWorkList roots. The stalled stream shows StrongHandles, as expected. No survivor was without a reported root.

Mechanism in JSC (oven-sh/WebKit at dfd696443b):

  • operationOptimize (jit/JITOperations.cpp) fills mustHandleValues with every parameter of the triggering frame (numParameters() includes this), also for a compile at function entry, and hands them to DFG::compile.
  • DFG::Plan::checkLivenessAndVisitChildren appends them with appendUnbarriered. JITWorklist::visitWeakReferences runs that for every plan in m_plans (queued, in compilation, ready) from the "JIT Worklist" marking constraint.
  • JITWorklistThread::work moves a finished plan to m_readyPlans and notifies nobody on the main thread. completeAllReadyPlansForVM is what finalizes it. Its callers are the tier-up slow paths (operationOptimize, jitCompileAndSetHeuristics in llint/LLIntSlowPaths.cpp, the FTL triggers) and Heap::completeAllJITPlans (delete all code).

Stand-alone demonstration (debug build): call function hot(o) with 400 distinct objects, then only collect. Exactly one object survives (the argument of the call that started the compile, for example #147) while numberOfDFGCompiles(hot) is 0. It goes on the pass after the count turns 1. With BUN_JSC_useConcurrentJIT=0 nothing survives.

Why pass 11. Traces of the loop (count logged per pass, loop extended to 400 passes): a group of 4 to 6 holds from pass 0 and goes in one pass at pass 11 to 14, at 1.0 s with 12 cores, 2.4 to 2.8 s with 2 cores, 3.6 s with 1 core. So the release follows the pass count, not the time. The loop's own code makes the tier-up checks: the filter callback runs 16 times per pass and reaches the LLInt threshold near pass 6 and the DFG threshold near pass 11. In the 32-stream case the steps start at pass 4 to 6. A single straggler that misses those checks goes at pass 60 to 62 in every run (the filter loop's own DFG threshold). Code older than 5 to 15 s is discarded at a full GC, so each case starts this schedule again.

Second effect. The old loop tried to reach 0, so a case with 1 to 3 pinned stragglers spent 10 more passes for nothing (about 1 s with 12 cores, 2 s with 2). Under CPU load (24 busy loops on 12 cpus) that alone pushed the unmodified file over the default 5 s timeout in 3 of 8 runs. The new loop returns at once in that state.

Runs of the fixed file (debug build, whole file unless noted): 12 of 12 with 2 cores, 6 of 6 with 1 core, 15 of 15 with 12 cores, 8 of 8 with 24 busy loops on 12 cpus, 12 of 12 with -t "stream release after a queued END_STREAM" and 2 cores.

Leak check. Removed self.free_resources::<false>(client) from Stream::flush_queue (h2_frame_parser.rs:1651), rebuilt: the five queued cases fail with Expected: <= 3, Received: 16 in 1.0 to 1.6 s each, the reset case passes (other path). Restored, rebuilt: 70 pass.

Related. #34640 (with oven-sh/WebKit#308) proposes that JSC stops rooting must-handle values. It is open. It found the same JITWorkList roots behind the N-1/N stall in test-gc-http-client*. With it no plan would root a stream, and this loop would return after the first pass. This change does not depend on it.

Not done. A longer plateau has no bound to derive it from (the release waits for a tier-up check, not for time). getProtectedObjects() in place of collection (#39609) and a child process with the concurrent JIT off both remove the JIT from the picture, but they replace the design that #40966 settled on. This change keeps that design.


[auto-merge] gate passed · iteration 1 · 1 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/js/node/http2/h2-conformance.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "test/js/node/http2/h2-conformance.test.ts"
bun test v1.4.3 (4ff919377)

test/js/node/http2/h2-conformance.test.ts:
(pass) connection preface & SETTINGS handshake (checklist §1) > server sends a SETTINGS frame first (§1.4) [379.07ms]
(pass) connection preface & SETTINGS handshake (checklist §1) > server ACKs the client's SETTINGS frame (§3.5) [113.43ms]
(pass) connection preface & SETTINGS handshake (checklist §1) > a SETTINGS frame with a non-zero stream id is a PROTOCOL_ERROR (§3.5) [88.66ms]
(pass) connection preface & SETTINGS handshake (checklist §1) > a SETTINGS frame whose length is not a multiple of 6 is a FRAME_SIZE_ERROR (§3.5) [45.09ms]
(pass) connection preface & SETTINGS handshake (checklist §1) > a SETTINGS ACK that carries a payload is a FRAME_SIZE_ERROR (§3.5) [39.37ms]
(pass) PING (checklist §3.7) > server replies to PING with a PING ACK echoing the payload [38.28ms]
(pass) PING (checklist §3.7) > a PING with length != 8 is a FRAME_SIZE_ERROR [44.78ms]
(pass) PING (checklist §3.7) > a PING on a non-zero stream id is a PROTOCOL_ERROR [35.98ms]
(pass) WINDOW_UPDATE (checklist §6) > a connection-level WINDOW_UPDATE with a 0 increment is a PROTOCOL_ERROR [68.03ms]
(pass) WINDOW_UPDATE (checklist §6) > a WINDOW_UPDATE with length != 4 is a FRAME_SIZE_ERROR [59.27ms]
(pass) frame structure (checklist §2,§3) > an unknown frame type is ignored, not an error (§2.4) [47.65ms]
(pass) frame structure (checklist §2,§3) > RST_STREAM on an idle stream is a PROTOCOL_ERROR (§4) [42.78ms]
(pass) stream-id rules (checklist §3) > HEADERS on stream 0 is a PROTOCOL_ERROR (§6.2) [44.33ms]
(pass) stream-id rules (checklist §3) > DATA on stream 0 is a PROTOCOL_ERROR (§6.1) [32.26ms]
(pass) fixed-length frames (checklist §3) > PRIORITY with length != 5 is a FRAME_SIZE_ERROR (§6.3) [30.01ms]
(pass) fixed-length frames (checklist §3) > RST_STREAM with length !
... (truncated)
Exit: 0
diff hotspot
test/js/node/http2/h2-conformance.test.ts | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

gate history · 2 passed · 0 rejected · iteration 1

evidence per changed file
file                                       reads  edits  tests
test/js/node/http2/h2-conformance.test.ts      3      2     24

…reams

The release cases in h2-conformance.test.ts fail on a debug build with
`Expected: <= 3, Received: 4` (or 5). A heap snapshot shows the surviving
streams as direct roots with reason JITWorkList, not as leaks.

A DFG compile plan roots the arguments of the call that started it (the
stream, as `this`). A finished plan stays in the worklist until the main
thread reaches its next tier-up check. In liveCount() that check comes at
about pass 11, one pass after the 10-pass plateau exit gives up.

liveCount() now stops as soon as the count is within GC_STRAGGLERS and does
not stop on a plateau. It still stops after 10 passes that collected nothing
at all, so the leak these cases guard against fails fast with `Received: 16`.
@robobun

robobun commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 12:25 PM PT - Sep 11th, 2026

✅ @robobun, your commit a91069632163d915b4ea6ef6a40a13abf6a2f73d passed in Build #114388! 🎉


🧪   To try this PR locally:

bunx bun-pr 42357

That installs a local version of the PR into your bun-42357 executable, so you can run:

bun-42357 --bun

@robobun

robobun commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

Status: reproduced on linux x64 with the debug build from bun bd. With the process pinned to 2 cores (taskset -c <2 cpus> ./build/debug/bun-debug test test/js/node/http2/h2-conformance.test.ts), the unmodified file fails 7 of 12 runs with Expected: <= 3, Received: 4 (or 5) in the end("ok") and compat API cases. With BUN_JSC_useConcurrentJIT=0 it passes 8 of 8. With this PR the file passes 15 of 15 runs in the same setup, and bun bd test of the file passes (70 pass).

@coderabbitai

coderabbitai Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Essentials

Run ID: 45949a4b-5b3b-4e64-8c7a-883d84e7a11f

📥 Commits

Reviewing files that changed from the base of the PR and between 158ff6c and c5d83e4.

📒 Files selected for processing (1)
  • test/js/node/http2/h2-conformance.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 1 remains after this review.


Walkthrough

The HTTP/2 conformance test updates liveCount to tolerate configured garbage-collection stragglers and to stop after bounded retry passes.

Changes

HTTP/2 conformance test

Layer / File(s) Summary
Bounded live reference retries
test/js/node/http2/h2-conformance.test.ts
liveCount tolerates up to GC_STRAGGLERS surviving references. It stops after 50 passes or after 10 passes with all references still live.

Suggested reviewers: jarred-sumner

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to c5d83

The test now tolerates expected GC stragglers while retaining bounded leak detection, with no merge-blocking risk identified.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main change: reducing flakiness in HTTP/2 stream-release tests on debug builds.
Description check ✅ Passed The description is detailed and relevant. It explains the problem, the fix, the technical cause, verification results, and leak checks. It does not use the template headings exactly, but it provides t…

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

The review above has no actionable items, and no review threads are open. The format and lint checks pass. The test build (Buildkite 114382) is in progress.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — a well-justified de-flake that keeps the assertion strength intact.

What was reviewed:

  • The new loop condition stops at <= GC_STRAGGLERS (the assertion bound) instead of chasing 0, so no additional pass can flip a pass into a fail; the 50-pass cap is unchanged.
  • The pass === 10 && last === refs.length early exit still fires on the actual regression these tests guard (all streams retained → Received: 16), so leak detection is not weakened and stays fast.
  • The retention is attributed to JITWorkList roots (concurrent-compile mustHandleValues), not the conservative stack scan — a real, named retainer, so this doesn't run afoul of the "never blame the scanner" rule.
  • Checked all four liveCount call sites: each asserts <= GC_STRAGGLERS, so the changed exit condition is sound for every caller.
Extended reasoning...

Overview

This PR touches a single test-helper function, liveCount, in test/js/node/http2/h2-conformance.test.ts. The helper drives GC passes and counts surviving WeakRefs for the stream-release leak tests. The old version stopped early when the surviving count plateaued for 10 consecutive passes; the new version instead runs until the count drops to GC_STRAGGLERS (3) or fewer, keeps the 50-pass cap, and only bails early if after 10 passes literally nothing was collected (the actual leak signature). The doc comment is expanded to name the mechanism: a concurrent JIT compile plan roots the arguments (including this, the stream) of the call that triggered tier-up until the main thread installs the compiled result — on debug builds this pins 4–9 streams for a dozen passes, which the old plateau heuristic misread as a stable state.

Security risks

None. This is a test-only change to a GC-polling helper; no runtime code, no user-facing surface, no I/O or parsing of untrusted input is touched.

Level of scrutiny

Low-to-moderate, focused on REVIEW.md's de-flake rules: does the change weaken the assertion, and is the retention mechanism real (not a stack-scanner blame)? On both counts the change holds up. The assertion bound (<= GC_STRAGGLERS = 3) is unchanged and remains well below the unfixed leak (16 or 32 streams). The PR description verifies that reverting the fix under test still fails with Received: 16 in ~1–1.6s, so the early-exit path still catches the regression fast. The retainer is named as JITWorkList with heap-snapshot evidence and JSC source references (operationOptimize → mustHandleValues → Plan::checkLivenessAndVisitChildren), which is a legitimate GC root distinct from the conservative stack scan — CLAUDE.md rule 15 is not implicated. The new loop is also faster in the common case (stops at ≤3 instead of spending 10 extra passes chasing 0), which addresses the secondary timeout-under-load flake noted in the PR.

Other factors

No CODEOWNERS entry covers this file. All four liveCount call sites assert <= GC_STRAGGLERS, so terminating the loop at that threshold is sound for every consumer. The 50-pass hard cap bounds the worst case even if a partial leak (some but not all streams retained) doesn't trip the pass-10 early exit. The added comment is load-bearing — it names a non-obvious JSC mechanism the next reader would otherwise spend real effort rediscovering.

@robobun

robobun commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

The second automated review has no findings either. It confirms that each of the four liveCount call sites asserts <= GC_STRAGGLERS, so the new exit condition is sound for every caller. No review threads are open. Buildkite 114382 is still in progress. After it is green, this PR needs a maintainer.

@robobun

robobun commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

The summary above was regenerated for an empty commit. It has no new findings, and the diff is the same as at c5d83e4. No review threads are open.

Build 114382: test/js/node/http2/h2-conformance.test.ts passed on every lane. One test stayed red: test/js/bun/http/serve-pending-promise-abort-leak.test.ts on debian 13 x64-asan (Expected: 0, Received: 1 in its own 20-pass GC loop). This diff does not touch it, and it also fails in the final builds of other recently merged PRs. Every other failure passed on retry. The empty commit runs CI again (build 114388).

@robobun

robobun commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator Author

Data from #41305, in case it helps here. The <= GC_STRAGGLERS tests in this file react to what else is in the file, not only to the build type.

Command, on a linux x64 debug+ASAN build, 2 lanes in parallel: bun bd test --reporter=junit --reporter-outfile=/tmp/x.xml test/js/node/http2/h2-conformance.test.ts test/js/node/http2/node-http2-continuation.test.ts

h2-conformance.test.ts content runs that failed
as on main (4b5862f) 0 of 100
plus a 190-line describe of 17 unrelated option tests near line 450, before the GC tests 4 of 50
plus the same block at the end of the file, after every GC test 13 of 50
-t the first GC test alone 0 of 30
-t that block plus the first GC test 0 of 30
  • With the block in the middle, the failing test was always inbound stream lifecycle > releases server stream objects once the peer resets their streams (Expected: <= 3, Received: 6), which this PR does not touch.
  • With the block at the end, it was server streams answered with end("ok") behind another stream's stalled response are released (10 runs, Received: 5) and the compat API twin (3 runs).
  • The failing runs are the short ones (about 1.1 to 1.3 s for the test, against 1.4 to 2.5 s when it passes). That fits liveCount leaving through its "count did not move for 10 passes" exit.

The block is at git show 34229c9acb:test/js/node/http2/h2-conformance.test.ts. #41305 has since moved those tests to their own file, so it no longer changes this one.

robobun added a commit that referenced this pull request Sep 19, 2026
h2-conformance.test.ts is the same as on main again. That file also holds
cases that count live stream objects after a GC, and those fail now and
then on a debug build (#42357). The new file carries its own raw server,
like h2-push-refusal-staged.test.ts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant