Skip to content

test: mark the remaining h3 batch tests flaky so one handshake flake does not fail CI - #41497

Open
robobun wants to merge 1 commit into
mainfrom
robobun/40813883/h3-batch-flaky-tests
Open

robobun wants to merge 1 commit into
mainfrom
robobun/40813883/h3-batch-flaky-tests

Conversation

@robobun

@robobun robobun commented Sep 6, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/js/bun/stream/direct-readable-stream.test.tsx went red on main in build 110599 (debian 13 aarch64), failing one h3 case with HTTP3HandshakeFailed fetching "https://127.0.0.1:50025/0". A red test on main blocks every open PR.
  • The cause is not a regression in that file. HTTP3HandshakeFailed is an intermittent QUIC handshake failure that hits any h3 test when it runs in the parallel CI batch under load. It has shown on serve-protocols, fetch-http3-adversarial, serve-http3, fetch-backpressure, fetch-http3-client and others over many builds (for example 92885, 107194, 108109, 108378, 108460, 109304, 109337, 109428, 110120, 110131, 110280, 110298, 110599). The same case passes when the runner re-runs the file alone.
  • The runner re-runs a failed batch file alone only when test/flaky-tests.txt lists it (scripts/runner.node.mjs, isFlakyTest). The other h3 batch tests are all listed, so the flake self-heals. Four h3 files were omitted, so one flake fails the whole job with no retry.

Fix

  • Add the four omitted h3 batch files to test/flaky-tests.txt: serve-direct-readable-stream.test.ts, serve-protocols.test.ts, stream/direct-readable-stream.test.tsx, fetch/fetch-http3-adversarial.test.ts.
  • A batch failure on these files now triggers a run-alone, the same treatment their sibling h3 tests already get. Each entry cites the builds where it went red.
  • Correct because the failure is a known load-dependent flake that passes on the run-alone (every annotation shows "passed alone"). The change reduces no coverage: each file still runs in full and still exercises the h3 path.
  • Verified: the added paths parse through flakyTests/isFlakyTest in scripts/runner.node.mjs and match the batch-relative titles. Ran the parallel batch of 202 batch-eligible files locally 11 times with the release binary (bun test --parallel=3); the flake did not reproduce on this 16-core linux box, consistent with it being load and platform dependent.

Background

  • The CI test runner groups fast, non-flaky files into one bun test --parallel batch per shard. A file qualifies through test/parallel-allowlist.json minus test/parallel-denylist.txt.
  • A file that fails inside the batch counts as a serial failure. It re-runs alone only when test/flaky-tests.txt names it. A file not listed makes the whole job red on its first batch failure.
  • The h3 fetch client and the Bun.serve h3 listener run on separate event loops and lsquic engines and talk over loopback UDP. Under heavy, oversubscribed CI load the handshake occasionally fails fast, which surfaces as HTTP3HandshakeFailed.
Notes The deeper question of why the QUIC handshake fails under load (a dropped Initial, a stale pooled session on the shared client UDP endpoint after a server port is reused across test files, or similar) is a separate product issue. It is rare, load and platform dependent, and did not reproduce locally despite repeated parallel-batch runs, so it cannot be proven fail-before to the gate here. It is handed off for dedicated investigation. This PR restores main to green for every PR in the established way the repo already manages this exact flake.

Culprit for the visibility of the break: the parallel batch runner (#36175, #39373) plus the retry-only-if-listed rule (#41204) mean an unlisted batch-eligible h3 file fails hard on a flake. The fix follows the same pattern the sibling h3 entries already use.


no test proof · iteration 0 · no src or test change; test-proof not applicable

…does not fail CI

HTTP3HandshakeFailed is an intermittent QUIC handshake failure that
appears on any h3 test in the parallel CI batch under load. The runner
re-runs a failed batch file alone only when test/flaky-tests.txt lists
it. Every other h3 test in the batch is listed (serve-http3,
fetch-http3-client, fetch-http3-cold-post, fetch-backpressure,
serve-http2-lifecycle, serve-http2), so the flake self-heals on the
run-alone. These four were omitted, so one flake turns the whole job
red with no retry:

- test/js/bun/http/serve-direct-readable-stream.test.ts
- test/js/bun/http/serve-protocols.test.ts
- test/js/bun/stream/direct-readable-stream.test.tsx
- test/js/web/fetch/fetch-http3-adversarial.test.ts

Add them so a batch failure re-runs the file alone, where it passes.
@github-actions github-actions Bot added the claude label Sep 6, 2026
@robobun

robobun commented Sep 6, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 1:24 AM PT - Sep 6th, 2026

✅ @robobun, your commit 3d247ab2d5286648aaf9b8f1d93b6744dfe16a62 passed in Build #110792! 🎉


🧪   To try this PR locally:

bunx bun-pr 41497

That installs a local version of the PR into your bun-41497 executable, so you can run:

bun-41497 --bun

@coderabbitai

coderabbitai Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Essentials

Run ID: 3bd3f019-76ca-44f2-ad60-a09e8b3af317

📥 Commits

Reviewing files that changed from the base of the PR and between 4593030 and 3d247ab.

📒 Files selected for processing (1)
  • test/flaky-tests.txt

Included review availability: Your plan provides up to 10 included reviews per hour; 0 remain after this review.


Walkthrough

The test tracking file now lists four additional HTTP/3-related flaky tests covering readable streams, protocol tests, stream tests, and adversarial fetch tests.

Changes

HTTP/3 flaky-test tracking

Layer / File(s) Summary
Record HTTP/3 flaky tests
test/flaky-tests.txt
Added entries for HTTP/3 failures in direct readable streams, protocol tests, stream tests, and adversarial fetch tests.

Merge Risk: ⚪ Minimal · up to 3d247

The HTTP/3 tests retain coverage and will be rerun individually after intermittent parallel-run failures. No current merge-blocking risk is identified.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the main change: marking the remaining HTTP/3 batch tests as flaky so isolated retries prevent CI failures from intermittent handshake errors.
Description check ✅ Passed The description explains the problem, the four-file fix, the retry behavior, test coverage impact, verification steps, and relevant background. It provides the information required by the template, al…

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — data-only allowlist edit matching the sibling h3 entries already in the file.

What was reviewed:

  • All four added paths exist on disk and parse through flakyTests in scripts/runner.node.mjs (split on #, trim).
  • Entries land in alphabetical order and match the neighboring comment format; each names the observed failure (HTTP3HandshakeFailed) with build numbers.
  • Coverage is not reduced — the allowlist only enables the run-alone retry after a parallel-batch failure; the files still run in full.
Extended reasoning...

Overview

The PR adds four lines to test/flaky-tests.txt, listing h3-exercising test files that intermittently fail with HTTP3HandshakeFailed when run inside the parallel CI batch but pass when re-run alone. The runner (scripts/runner.node.mjs) only re-runs a batch-failed file in isolation when it appears in this allowlist; sibling h3 tests (serve-http3.test.ts, fetch-http3-client.test.ts, serve-http2-lifecycle.test.ts) are already listed for the same reason. This change extends the same treatment to four files that were omitted.

Security risks

None. This is a plaintext allowlist consumed by the CI test runner; it does not affect the shipped binary, touch no auth/crypto/network code, and introduces no executable content.

Level of scrutiny

Low. It is a mechanical data-file edit following an established repo pattern. I verified the four paths exist, that the parser (line.split("#")[0].trim()) accepts the two-space + comment format used, that alphabetical ordering is preserved at each insertion point, and that each entry documents the observed failure with specific build numbers — satisfying REVIEW.md's requirement that skips/flaky markers name the observed failure. .github/CODEOWNERS does not cover test/flaky-tests.txt.

Other factors

The allowlist does not skip or weaken tests; it only enables a run-alone retry after a parallel-batch failure, so full coverage is retained. The PR description acknowledges the underlying QUIC handshake-under-load issue as a separate investigation, which is the appropriate scoping. No prior reviewer objections appear in the timeline.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant