Skip to content

test: dial a local blackhole listener instead of TEST-NET-1 in the never-connecting tests - #39903

Open
robobun wants to merge 2 commits into
mainfrom
farm/d0f1424e/fetch-abort-slow-connect-local-blackhole
Open

robobun wants to merge 2 commits into
mainfrom
farm/d0f1424e/fetch-abort-slow-connect-local-blackhole

Conversation

@robobun

@robobun robobun commented Aug 21, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Fix

  • test/blackhole.ts exports blackholeListener(): a raw libc listener on 127.0.0.1 that nobody accepts from, with eight connections of our own in its accept queue. The kernel drops the SYN of every later connect, so a dial sits in SYN_SENT until it is closed. The three tests dial it.
  • A dropped packet produces the same socket state, so each test exercises the path it did before (register_abort_tracker after connect in src/http/lib.rs, eviction with active sockets, the valkey timeout).
  • Windows answers a full accept queue with RST, so the helper returns TEST-NET-1 there, as before.
  • Verified: the three files pass with bun bd test here, where the old slow-connect file fails. On bingus the old file fails 3 of 3 runs, the listener version passes 5 of 5. With the listener, the eviction fixture's 65 TLS fetches sit in SYN-SENT (ss).

Background

  • What a connect to TEST-NET-1 (RFC 5737) does depends on the first router that refuses it: a silent drop, an ICMP unreachable, or no route at all.
  • The accept queue (backlog) holds completed connections the process has not accepted yet. When it is full, Linux and macOS drop new SYNs. Windows sends RST.
  • Bun's own listeners accept everything and do not expose the backlog, hence libc through bun:ffi, on the test/mkfifo.ts pattern. Linux admits backlog + 1 connections. macOS treats 0 as the default, so the helper passes 1 there.
Notes
  • Build 102370 darwin shard 0 ran on agent darwin-aarch64-26.6.2-1, host darwin-arm64-bingus (172.92.21.195, LAN gateway 192.168.50.1, connected at 06:13 UTC). Build 102348 shard 0 ran on the same host and failed the same way. Shard 1 jobs on that host pass. Shard 0 jobs on hardtack, crouton and breadstick pass. The agent name darwin-aarch64-26.6.2-1 is shared by bingus and hardtack. bingus served none of the last 60 main builds (since Aug 19) while every other persistent mac did, so today's PR jobs were its first.
  • Probes from the host: 192.0.2.1:80 fails in 22 ms with No route to host (EHOSTUNREACH). 203.0.113.1 gets Network is unreachable. 198.51.100.1, 240.0.0.1 and 10.255.255.1 hang. From hardtack, 192.0.2.1 hangs for the full 5 s probe window.
  • This container has no route to 192.0.2.1 at all. The old slow-connect file fails here too: connect fails synchronously with ENETUNREACH (UnreachableError with the release build), and under the debug build both connect tests fail. The eviction test passed here before this change with every connect already failed.
  • History: Fix aborting fetch() calls while the socket is connecting. Fix a thread-safety issue involving redirects and AbortSignal. #22842 added the slow-connect file. node:http2: rewritten inbound engine, batched write path, server push, +290 node v26.3.0 tests (79% passing) #31584 added a [ DARWIN ] ... [ FLAKY ] entry to test/expectations.txt because the darwin agents of that time had no route to 192.0.2.1, and asked for a local hanging connect as the follow-up. test: prune stale entries from expectations.txt #36780 removed the entry after a probe build passed. This PR is that follow-up.
  • The first revision of this PR set the listener up inline in the slow-connect file. A self-review pointed out that dns-interleave.test.ts and valkey-gc.test.ts already carry inline copies of the same listener and that the eviction test had the same dependence, so the listener moved to test/blackhole.ts and the two other tests were converted. The inline copies in dns-interleave.test.ts (binds specific addresses inside a child script) and valkey-gc.test.ts (inside a child script) and connect-autoselectfamily-destroy-fixture.js (needs two addresses, skips itself today) are left for a follow-up. fetch: add an RFC 8305 per-attempt connect timer so a blackholed address does not stall for ~130 s #32949 adds another inline copy and could use the helper once this lands.
  • Old slow-connect file on bingus with bun 1.3.13: 3 of 3 runs fail the first test (Received: "Error", 3.3 to 4.0 ms). The listener version, byte for byte the code now in test/blackhole.ts, passes 5 of 5 runs there and 3 of 3 on a darwin x64 host (macOS 14).
  • Socket states during a run, from ss -tan and netstat -an. Linux: LISTEN with recv-q 1, one filler ESTAB, seven fillers plus the fetch in SYN-SENT. macOS 26 on bingus: one filler ESTABLISHED, seven fillers plus the fetch in SYN_SENT. With 65 TLS fetches (distinct serverNames, as in the eviction fixture) against the listener, ss shows 72 SYN-SENT sockets after 300 ms and all 65 reject with AbortError on abort.
  • The valkey describe block needs docker and is skipped in this container. The converted test body was run here as an ungated copy with the release and the debug build (passes, "Connection timeout reached after 2ms"). expect(asyncFn).toThrow... in bun:test returns synchronously, so the old form could not be followed by client.close(). The .rejects form is awaited, verified to resolve only after the rejection, and asserts the same message.
  • The second slow-connect test (AbortSignal.timeout(1)) had the same dependence. It passed on bingus only because the ICMP reply took longer than 1 ms.
  • Windows CI has passed the slow-connect file all along (test/expected-durations.json lists it at 114 ms there), so TEST-NET-1 stays as the Windows branch inside the helper. A full accept queue on Windows produces ECONNREFUSED (see the note in test/js/bun/http/request-smuggling.test.ts), not a pending connect. The eviction test now dials port 80 instead of 443 there, which makes no difference for a dropped SYN.
  • src/http/lib.rs:2884 refers to the slow-connect file by path. The path is unchanged, as are the entries in test/parallel-denylist.txt and test/expected-durations.json.
  • Also ran: connection-failures.test.ts under bun bd test (46 pass, 13 skipped for docker), the slow-connect file with USE_SYSTEM_BUN=1, and six debug-build runs of it in a row.

no test proof · iteration 0 · Platform-specific test-only change; deferring to CI.

The two abort-while-connecting tests dialed 192.0.2.1 and relied on the
network between the CI host and the internet to drop the packets. A host
whose network answers with ICMP unreachable, or that has no route at all,
fails the fetch before the signal fires.

Dial a local listener instead. It is a raw libc socket with the smallest
accept queue, and a few connections of our own fill the queue, so the
kernel drops the SYN of every later connect and the fetch stays in
SYN_SENT until the signal aborts it. Windows answers a full accept queue
with RST, so it keeps dialing TEST-NET-1.
@coderabbitai

coderabbitai Bot commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Your included review limit has been reached.

You’re in a promotional period — use the checkbox below to run this review for free:

  • Run review for free

On-demand reviews are free for the next 30 days. After that, they cost $0.25 per reviewed file.

How can I continue?

Run this review now using the option above, or comment @coderabbitai review --use-credits.

You can also wait for the limit to reset (next review available in 42 seconds), then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 60aeac33-0930-409d-9e4d-da3771adfd63

📥 Commits

Reviewing files that changed from the base of the PR and between 7c9a51e and 68bb9b1.

📒 Files selected for processing (5)
  • test/blackhole.ts
  • test/js/bun/http/fetch-abort-ssl-context-eviction.test.ts
  • test/js/bun/io/fetch/fetch-abort-slow-connect.test.ts
  • test/js/valkey/reliability/connection-failures.test.ts
  • test/tsconfig.json

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for a maintainer. Current revision: 68bb9b1.

How it was reproduced:

  • Builds 102348 and 102370 both ran darwin shard 0 on the host darwin-arm64-bingus. From that host, a connect to 192.0.2.1:80 fails in 22 ms with EHOSTUNREACH. From darwin-arm64-hardtack it hangs, as the tests assumed.
  • The old fetch-abort-slow-connect.test.ts fails 3 of 3 runs on bingus and also in a container with no route to 192.0.2.1. The listener version passes 5 of 5 runs on bingus, 3 of 3 on a darwin x64 host, and with bun bd test in that container.
  • ss -tan and netstat -an during a run show the sockets in SYN_SENT on Linux and macOS, so each test still works on a socket that is connecting.

CI for 68bb9b1 (build 102654): the three files pass on darwin aarch64, debian x64, debian x64 asan, alpine x64 and windows 2019 x64. The valkey block runs on the docker lanes (58 pass) and is skipped on darwin and windows as before. The only failure that did not pass on retry is test/js/bun/http/bun-server.test.ts on windows 2019 x64. It fails the same way on main (for example build 102370) and is reported separately. This PR does not touch it.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — well-researched test de-flake that removes the network-topology dependence.

What was reviewed:

  • The full-accept-queue technique matches the existing pattern in dns-interleave.test.ts; libcPathForDlopen() covers linux/darwin and Windows keeps the original path.
  • Checked sockaddr_in layout for both the Linux and BSD branches, and the big-endian port read after getsockname — both correct.
  • Verified beforeAll sets url before the test.concurrent bodies run, and that LIFO cleanup destroys fillers before closing the listen fd.
  • Confirmed the filler-ordering claim (all 8 connect(2) are issued in one tick before any 'connect' event can fire), so the queue is full before any test's fetch sends a SYN.
Extended reasoning...

Overview

This PR modifies a single test file, test/js/bun/io/fetch/fetch-abort-slow-connect.test.ts, to remove its dependence on how the CI host's network routes packets to TEST-NET-1 (192.0.2.1). The old test assumed a silent SYN drop; a new darwin CI host (bingus) instead returns ICMP unreachable in ~20 ms, which beats the 50 ms AbortSignal.timeout and flips the rejection to a connection error rather than a TimeoutError. The new setup creates a raw libc listener on 127.0.0.1:0 with the smallest possible backlog, fills its accept queue with 8 net.connect fillers, and points the three tests at that URL. On Linux/macOS this reliably leaves later connects stuck in SYN_SENT; Windows (where a full queue produces RST) keeps the original 192.0.2.1 path since Windows CI has always passed this file.

Security risks

None. Test-only change, no runtime code touched. The raw libc socket is bound to loopback with an ephemeral port and closed in afterAll.

Level of scrutiny

Low-to-medium. It's a targeted CI de-flake with no production surface, following an existing in-tree pattern (dns-interleave.test.ts uses the same full-accept-queue trick on Linux, and the PR description references valkey-gc.test.ts for the raw-libc-listener approach). The PR description documents empirical verification on the failing host (5/5 pass), a darwin x64 host, and multiple debug runs, plus ss/netstat state captures showing the intended SYN_SENT state.

Other factors

  • The sockaddr_in construction correctly branches on Linux vs BSD layout (16-bit sin_family vs sin_len+sin_family), and port is read big-endian from offsets 2–3 after getsockname — correct for both.
  • AF_INET=2/SOCK_STREAM=1 are correct on both Linux and macOS; libcPathForDlopen() in harness handles both platforms.
  • Cleanup is LIFO via cleanup.pop(), so fillers are destroyed before the listen fd is closed; filler sockets have an 'error' handler so late SYN-retry timeouts don't surface as unhandled.
  • The three tests remain test.concurrent; beforeAll fully awaits the first filler's 'connect' (guaranteeing all 8 SYNs are already sent) before any test starts, so there is no race on url or on queue fullness.
  • The PR notes that USE_SYSTEM_BUN=1 also passes — expected here since this is test-infrastructure hardening, not new coverage for a Bun-side fix. The test's original assertion contract (TimeoutError within 1 s) is unchanged.
  • No prior human review comments to address; only a CodeRabbit rate-limit notice on the timeline.

@robobun

robobun commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

The review above asks for no changes. One update to the body: darwin-arm64-bingus served none of the last 60 main builds (since Aug 19) while every other persistent mac did, so today's PR jobs were its first. That is why the failure started today without a change in bun or in the test.

…or the other TEST-NET-1 tests

The listener is the same code that fetch-abort-slow-connect.test.ts set up
inline. fetch-abort-ssl-context-eviction.test.ts and the valkey connection
timeout test dialed 192.0.2.1 for the same reason and now dial the
listener too. On a host that rejects TEST-NET-1, the eviction test ran with
no connecting sockets and the valkey test got a connect error instead of
the timeout.
@robobun

robobun commented Aug 21, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 6:36 AM PT - Aug 21st, 2026

❌ @robobun, your commit 68bb9b1 has 1 failures in Build #102654 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 39903

That installs a local version of the PR into your bun-39903 executable, so you can run:

bun-39903 --bun

@robobun robobun changed the title test: make fetch-abort-slow-connect independent of routing to TEST-NET-1 test: dial a local blackhole listener instead of TEST-NET-1 in the never-connecting tests Aug 21, 2026
@robobun

robobun commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed 68bb9b1 after a self-review of the first revision. It raised two points:

  • The inline listener was the third copy of this setup in the tree (dns-interleave.test.ts and valkey-gc.test.ts carry the others). The listener now lives in test/blackhole.ts (blackholeListener(), same code, test/mkfifo.ts pattern), and this file is its first consumer.
  • fetch-abort-ssl-context-eviction.test.ts and the valkey connection timeout test dialed 192.0.2.1 for the same reason. On a host that rejects TEST-NET-1 the first one runs with no connecting sockets and the second gets a connect error. Both dial the listener now. With it, the eviction fixture's 65 TLS fetches are all in SYN-SENT when the cache evicts (checked with ss).

The copies inside the child scripts of dns-interleave.test.ts and valkey-gc.test.ts, and the autoselectfamily fixture, are left for a follow-up. The PR body describes the current revision.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only de-flake that replaces network-dependent TEST-NET-1 dials with a local full-accept-queue listener.

What was reviewed:

  • test/blackhole.ts matches the existing raw-libc listener pattern in valkey-gc.test.ts and dns-interleave.test.ts; the sockaddr_in Linux/BSD branching and backlog values (0 vs 1) line up with those precedents.
  • Checked that the ssl-context-eviction fixture still works when the listener lives in the parent and children dial it — the queue is kernel state so cross-process connects also see dropped SYNs; Windows keeps the old TEST-NET-1 path unchanged.
  • The valkey connection timeout test also fixes an unawaited async toThrow (previously vacuous) into a proper await expect(...).rejects with try/finally cleanup.
  • Confirmed libcPathForDlopen covers linux/darwin and Windows returns early before reaching it; filler sockets and the fd are released via Symbol.dispose on both success and error paths.
Extended reasoning...

Overview

This PR removes a CI dependency on how the local network routes packets to TEST-NET-1 (192.0.2.1). Three test files that previously assumed a connect to that address would hang forever now dial a local listener whose accept queue is deliberately filled, so the kernel drops further SYNs and the connect stays in SYN_SENT. The pattern is extracted into a new shared helper test/blackhole.ts (with a tsconfig.json path mapping) rather than duplicated. No production code is touched.

Security risks

None. This is test infrastructure only. The FFI use is limited to socket/bind/listen/getsockname/close on 127.0.0.1, following the same pattern already present in test/js/valkey/valkey-gc.test.ts and test/js/bun/dns/dns-interleave.test.ts.

Level of scrutiny

Low-to-medium. It's a test-only change fixing environment-dependent flakiness, but it introduces a shared helper with raw libc FFI and platform-specific sockaddr layout, so I checked it against the two existing in-tree copies of the same technique — the struct layout, AF_INET/SOCK_STREAM constants, and per-kernel backlog choice all match. The PR body documents verification on the failing host (bingus, 5/5 pass), a darwin-x64 host, Linux debug, and cites ss/netstat output confirming the fetch socket sits in SYN_SENT.

Other factors

  • The valkey connection-failures change is a strict improvement independent of the flake fix: the old expect(async () => ...).toThrowErrorMatchingInlineSnapshot(...) was never awaited, so it could not fail. The new form awaits .rejects and closes the client in finally.
  • Resource cleanup in the helper is sound: the fd and filler sockets are released via [Symbol.dispose], and the helper's own error path calls dispose() before rethrowing. Callers use using or afterAll.
  • Windows behavior is unchanged everywhere (the helper returns the same 192.0.2.1:80 there), and the PR notes Windows CI has always passed this file.
  • CI on the PR passed the rewritten file across darwin-aarch64, debian x64 (+asan), alpine x64, and windows; the one unrelated failure also fails on main.

@robobun

robobun commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

One correction to the review summary above: the old valkey assertion was not vacuous. In bun:test an unawaited expect(asyncFn).toThrow... still fails the test when the rejection does not match (checked with a mismatching message). It only could not be followed by client.close(), which is why the test now awaits the .rejects form. The review asks for no code change. CI for 68bb9b1 is build 102654.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant