Skip to content

usockets(tls): key the spill slot by a connection id; park the handshake reason on the SSL - #37679

Closed
robobun wants to merge 4 commits into
mainfrom
farm/11f75065/tls-loop-slot-ids
Closed

robobun wants to merge 4 commits into
mainfrom
farm/11f75065/tls-loop-slot-ids

Conversation

@robobun

@robobun robobun commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • No user-visible bug and no behavior change intended; this is a cleanup of design debt in the TLS layer of usockets. Every teardown path on main is complete, so there is nothing to reproduce.
  • Two pieces of state that belong to one connection at a time, the ciphertext spill slot and the parked handshake failure reason, live in per-loop scratch and record their owner as the raw socket pointer.
  • Keying by address is only sound while every teardown path clears the slot before the allocator recycles the address, and while adoption re-points the owner each time it moves a live connection to a new block.
  • A missed site would drain a dead connection's spilled ciphertext into a new connection's fd, or report a dead connection's handshake reason as the new connection's own.

Fix

  • The parked reason moves onto the connection's own SSL, as the packed error code in a BoringSSL ex_data slot, and is formatted into a string at dispatch time. It dies with the SSL and can only be read back through the SSL that parked it, so the per-loop buffer, its owner field and the clears on teardown go away. The string JS receives is unchanged.
  • The spill slot stays one per loop on purpose (it is the bound on ciphertext reported as written but not yet handed to the kernel), but its owner is now a 64-bit per-loop connection id assigned when TLS is attached. An id is never handed out twice on a loop, so a connection can only drain or release a spill it made itself, whatever address it currently has.
  • The id is copied with the rest of the socket header on adoption, so the relocation hook is deleted. On epoll/kqueue the id fits in existing tail padding and the struct stays 80 bytes (pinned by a _Static_assert); on the libuv build it grows from 80 to 96, and nothing outside usockets depends on the size.
  • Verification: source-only, no new test, since one would pass with and without the change. The existing TLS suites were run under ASAN on this branch and on the same tree with usockets reset to main, with matching results; temporary local instrumentation showed existing tests exercise the park and dispatch paths, including a client and a server of one failed handshake on the same loop reporting different reasons.

Background

  • usockets is the C socket layer under Bun.serve, fetch, node:net and node:tls. struct us_socket_t is its per-socket header, and the caller's ext data is laid out directly after it, which is why the header's size matters.
  • loop_ssl_data is scratch shared by every TLS socket on one event loop (shared BIOs and read buffers), so anything kept there for a single connection has to record which connection it belongs to.
  • Ciphertext write batching: the records sealed by one TLS write are sent with one syscall. If the kernel takes only part of the batch, BoringSSL already counts those records as sent, so the rest is spilled into the loop's single slot and drained on the owner's next writable event; batching is off for every other socket until then.
  • Parked handshake reason: when a fatal SSL error closes a socket during the handshake, the OpenSSL error is saved before the close so the handshake failure dispatch can report it to JS (for example wrong version number) the way Node does.
  • Adoption (us_socket_adopt): a live socket whose ext size changes, such as a WebSocket upgrade over TLS, is copied into a new allocation, so a connection's address can change mid-life and its old address becomes free for reuse.
Original description

What does this PR do?

Tightens the ownership bookkeeping of the per-loop TLS scratch in packages/bun-usockets/src/crypto/openssl.c. No behavior change is intended; this is design debt in how two pieces of state were attributed to a connection.

struct loop_ssl_data held two things that belong to one connection at a time, both recording their owner as the raw us_socket_t *:

  1. the ciphertext spill slot (ssl_spill_owner / ssl_spill / ssl_spill_len / ssl_spill_off), filled by ssl_flush_write_batch when a batched write only partially reaches the kernel and drained from the owner's writable event and before its later writes, shutdown and close;
  2. the parked handshake failure reason (ssl_last_fatal_error / ssl_last_fatal_error_owner), written by ssl_park_fatal_reason and consumed by ssl_dispatch_parked_reason.

Keying by address is only sound while every teardown path clears the slot before the allocator can hand the same address to a new socket, and while adoption re-points the owner whenever us_poll_resize moves a live connection to a new block (us_internal_ssl_socket_relocated, called from us_socket_adopt). I traced those sites and they all hold on main, which is also why there is nothing to reproduce; but the slots' correctness rested on each of them staying complete, and a missed one would mean a new connection on a recycled address draining a dead connection's spilled records into its own fd, or reporting a dead connection's handshake reason as its own.

Change. The two pieces of state get different treatment, because only one of them is loop-level by design:

  • The parked reason is per-connection state, so it moves onto the connection's own SSL, where this file already keeps that kind of state (reneg counters, SNI state, the inline-reject verdict). ssl_park_fatal_reason stores the packed ERR_peek_error() code in a new free-func-less ex_data slot, exactly like us_ssl_inline_reject_err_ex_idx; ssl_dispatch_parked_reason takes it off the SSL and formats it with ERR_error_string_n at dispatch time. That function is a pure function of the packed code, so the string JS receives is unchanged (the longest possible output with the vendored BoringSSL tables is 106 bytes, inside ERR_ERROR_STRING_BUF_LEN). The per-loop buffer, its owner field, and the clears on detach, on the inline-reject dispatch and after the fatal-read close all go away: the state dies with the SSL and can only be read back through the SSL it was parked on. (The inline-reject dispatch needs no clear: ssl_trigger_handshake flips the state to HANDSHAKE_COMPLETED first, after which neither dispatch site runs again.)
  • The spill slot is deliberately one per loop: it is the bound on ciphertext that has been reported to SSL as written but not yet handed to the kernel (at most one spill exists at a time, and every other socket writes through per record while it is occupied; see the comments in ssl_flush_write_batch and us_internal_ssl_write, which this PR keeps). A shared slot needs an owner, and the owner is now an explicit connection id: us_internal_ssl_attach, the one place every TLS socket passes through (accept, connect, us_socket_from_fd, us_socket_adopt_tls), assigns s->ssl_id from a per-loop 64-bit counter, and every match goes through ssl_owns_spill. The id travels with the header copy in us_poll_resize, so us_internal_ssl_socket_relocated and its call in context.c are deleted. An id is never handed out twice on a loop, so a connection can only drain or release a spill it made itself, whatever address it currently has. Like the other ssl_* header fields, ssl_id is set by attach and only read behind s->ssl; us_internal_ssl_detach, the one release path that also runs for plain sockets (us_internal_socket_close_raw calls it unconditionally), now skips sockets that never attached instead of every socket constructor having to zero the field. The loop_ssl_data comment now says why the slot is per loop, so the owner key is not mistaken for an accident of the current storage.

Two alternatives I considered and did not take. Keying the slot on s->ssl instead of a new id would also survive adoption, but an SSL * is itself a recycled heap address, so it would only remove the relocation dependency and keep the address-reuse one; the id removes both, and lets a stale slot, should a future teardown path ever miss the release, degrade to "batching stays off on this loop" instead of misdirected ciphertext. Giving each connection its own spill buffer (the us_ssl_pending_session_t shape) would remove the owner key entirely, but it changes the policy above (one bounded spill per stalled connection instead of one per loop, and batching staying on for everyone else), which is a behavior change with its own memory and throughput trade-offs; it would be its own PR, and this one keeps the policy as is.

Layout. ssl_id sits where struct us_socket_t had tail padding on epoll/kqueue (72 bytes of content rounded to 80 by the 16-byte alignment of the poll header), so the struct and the ext offset behind it are unchanged there; a _Static_assert next to the existing us_socket_flags assert pins the size at 80 (checked with an offsetof/sizeof probe: ssl_id at 24, sizeof 80), and the layout comments in internal.h that the field made inaccurate now describe the current layout in one place. On the libuv build the poll header is 24 bytes and the struct goes from 80 to 96, which any added field would cause; nothing outside usockets depends on the size (the Rust side treats us_socket_t as opaque and reaches ext data through us_socket_ext).

Tests

This PR is source-only on purpose. Every current teardown path clears both slots, so there is no behavior on main a test can fail against; the change removes what the slots' correctness depended on rather than fixing an observable bug, and a test added here would pass with and without it.

To make sure the rewritten code is actually exercised by what exists, I temporarily instrumented ssl_park_fatal_reason and ssl_dispatch_parked_reason (locally, not in this PR) and counted distinct connections that park and then report their own reason: test/js/node/tls/node-tls-server.test.ts 13, node-tls-cert.test.ts 11, node-tls-ecdh-curve.test.ts 6, node-tls-connect.test.ts 5 (including the two early-write ERR_SSL_NO_SUPPORTED_VERSIONS_ENABLED cases that go through the us_internal_ssl_write park site), test/js/bun/net/socket.test.ts 3 (adopted upgradeTLS sockets), and fetch-tls-cert.test.ts "fetch applies tls.sigalgs" on the fetch client's own loop. The vendored test-tls-alert.js, test-tls-min-max-version.js, test-tls-server-failed-handshake-emits-clienterror.js, test-tls-socket-failed-handshake-emits-error.js, test-tls-junk-server.js and test-tls-close-error.js each have the client and the server of one failed handshake on the same loop reporting different reasons (for example UNSUPPORTED_PROTOCOL on the server and TLSV1_ALERT_PROTOCOL_VERSION on the client), which is the attribution property this PR is about. The spill slot is covered by the deferred spill-close and destroySoon tests in node-tls-server.test.ts, the TLS variants in node-http-backpressure.test.ts / node-http-pinned-write.test.ts, and tls-syscall-fault.test.ts.

How did you verify your code works?

Ran the suites above plus the rest of the TLS-related files (tcp-server, socket-retention, fetch.tls, serve (tls cases), node-tls-upgrade, node-tls-context, tls-connect-socket-churn, tls-syscall-fault, renegotiation, ssl-ctx-cache, hostname verification, half-open, duplex close, test/regression/issue/12117.test.ts, and test-tls-destroy-whilst-write.js / test-tls-close-event-after-write.js) with the debug (ASAN) build on this branch and on the same tree with packages/bun-usockets reset to main; the results match. test/js/web/websocket/websocket.test.js "should connect many times over https" (300 WebSocket upgrades over TLS, i.e. adoption of live TLS sockets with address churn), websocket-syscall-fault.test.ts and test/js/node/http2/node-http2.test.js pass on the branch. The failures present in both runs are this container's localhost resolving to ::1 while the tests connect to 127.0.0.1, tests that need the public internet, and two tests that sit at the 5 s default timeout under ASAN here with and without the change (the destroySoon test measured 4.6 to 6.6 s on main and 5.0 to 6.7 s on the branch, delivering every byte in both; reported separately as a test-budget issue); none of them involve the changed code.

@coderabbitai

coderabbitai Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: dc6f74e2-2c32-427b-b86b-f93cc6e8be03

📥 Commits

Reviewing files that changed from the base of the PR and between b49c971 and 8eaa718.

📒 Files selected for processing (5)
  • packages/bun-usockets/src/context.c
  • packages/bun-usockets/src/crypto/openssl.c
  • packages/bun-usockets/src/internal/internal.h
  • packages/bun-usockets/src/loop.c
  • packages/bun-usockets/src/socket.c

Walkthrough

The PR replaces pointer-based SSL spill ownership with per-socket IDs. It stores parked handshake errors in SSL ex_data, removes the relocation callback, updates socket layout checks, and initializes ssl_id for new and adopted sockets.

Changes

SSL socket lifecycle

Layer / File(s) Summary
Socket identity and layout
packages/bun-usockets/src/internal/internal.h, packages/bun-usockets/src/context.c, packages/bun-usockets/src/loop.c, packages/bun-usockets/src/socket.c
us_socket_t gains an ssl_id field and an 80-byte size assertion. New sockets initialize the field to zero. Socket adoption copies the TLS state as part of the socket header.
Ciphertext spill ownership
packages/bun-usockets/src/crypto/openssl.c, packages/bun-usockets/src/internal/internal.h
Spill state stores the owning socket ID. Ownership checks and cleanup are centralized and used by draining, teardown, retries, pending-byte reporting, batching, and shutdown.
Per-SSL handshake errors
packages/bun-usockets/src/crypto/openssl.c
Fatal handshake errors are stored and retrieved through SSL ex_data. Loop-global parked-error storage and ownership checks are removed.

Suggested reviewers: cirospaciari, jarred-sumner

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main TLS ownership changes: connection IDs for spill slots and SSL storage for handshake errors.
Description check ✅ Passed The description explains the problem, fix, background, verification steps, test coverage, and known environment-related failures.

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. Current revision is 879a612 (rebased onto main after #37675 and #31829 landed in the same files; no conflicts).

  • Source-only change in packages/bun-usockets; there is no failing behavior on main to test against (every teardown path already clears both slots), so no test is added. The Tests section of the body lists, per file, how many connections exercise the rewritten park/dispatch path (counted with a temporary local probe) and what covers the spill slot.
  • Verified by running those suites plus the rest of the TLS-related files with the debug (ASAN) build on the branch and on the same tree with packages/bun-usockets reset to main; the results match. The upgradeTLS tests, the 300-upgrade WebSocket-over-TLS churn test, the websocket fault-injection suite and node-http2.test.js pass on the branch.
  • CI: builds 92636 and 92997 (earlier revisions) were green across the matrix. Build 93254 (879a612) hit a pre-existing intermittent Windows crash in test/bake/deinitialization.test.ts (also fails on unrelated branches, builds 93237, 93192, 93056; reported separately). The re-run on the unchanged diff, build 93292, has no test failures: 179 of 181 jobs passed, including that Windows lane, and the remaining two (both macOS 26 aarch64 test lanes) expired without ever being picked up by a runner, which is what leaves the build marked failed. Nothing in either build points at the changed code.
  • sizeof(struct us_socket_t) stays 80 on epoll/kqueue (static assert); libuv builds grow by 16 bytes, nothing depends on the size.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. Because it reworks ownership tracking in the native TLS layer (per-loop spill slot + parked handshake reason) and changes struct us_socket_t layout with no new test, a human look is still worthwhile.

Checked: ssl_id_counter starts at 0 via us_calloc, so the first id handed out is 1 and 0 never matches. All four plain-socket init sites set ssl_id = 0; ssl_owns_slot refuses 0. us_poll_resize memcpy's the whole header, so the id survives adoption and dropping us_internal_ssl_socket_relocated is sound. The keyed ssl_clear_parked_reason after ssl_close in on_data reads s->group->loop post-dispatch, which us_internal_ssl_detach already does one frame down, so no new lifetime hazard. The epoll/kqueue _Static_assert (80 == 5×16) matches the layout math.

Extended reasoning...

Overview

Refactor of per-loop TLS scratch ownership in packages/bun-usockets: the ciphertext spill slot and parked handshake reason in loop_ssl_data are re-keyed from raw us_socket_t* to a monotonic uint64_t ssl_id handed out by us_internal_ssl_attach. Adds ssl_id to struct us_socket_t (fills tail padding on epoll/kqueue, +16B on libuv), removes us_internal_ssl_socket_relocated, consolidates release into ssl_spill_free / ssl_clear_parked_reason / ssl_owns_slot, and moves both releases to the top of us_internal_ssl_detach. Five files touched, all in packages/bun-usockets/src/.

Security risks

None identified. The change tightens (not loosens) the condition under which one connection can touch another's per-loop scratch: address-keying could in principle alias across allocator recycling if a teardown site were missed; id-keying cannot. No new user-controlled input reaches the changed code, and no security gates are removed.

Level of scrutiny

High. This is native C in the TLS write/close path — the exact area REVIEW.md flags as the most-blocked category (per-loop shared state, teardown ordering, adoption/relocation). A miskeyed spill would drain one connection's sealed records into another's fd. The change also touches struct layout with a platform-gated static assert (epoll/kqueue only; libuv/Windows grows and has no assert), and ships without a new test on the argument that current behavior is already correct. That justification is reasonable, but per REVIEW.md this is a maintainer call.

Other factors

I traced the specific paths the PR body describes and they check out: us_calloc zero-inits ssl_id_counter; the four plain-socket init sites (us_internal_init_listen_socket, us_internal_init_connect_socket, accept in loop.c, us_socket_from_fd) all set ssl_id = 0; us_poll_resize copies the whole old block so the id survives adoption; ssl_owns_slot guards against 0-matching-0 for plain sockets on the unconditional us_internal_ssl_detach call. The one line that changed shape — ssl_clear_parked_reason(s->group->loop, s) after ssl_close in us_internal_ssl_on_data — now dereferences s->group after JS dispatch, but that's the same pattern us_internal_ssl_detach already uses one frame down in close_raw, so it introduces no new lifetime dependency. The design decision itself (id-keying vs. address-keying, and shipping a source-only invariant refactor without a test) is what needs a maintainer sign-off.

@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

Nothing actionable in the review above; the points it traced (counter starts at 1 via us_calloc, the four ssl_id = 0 init sites, ssl_owns_slot refusing 0, the header copy in us_poll_resize, the post-close s->group->loop read matching what us_internal_ssl_detach already does) are the ones the PR body relies on.

On the libuv build there is deliberately no size assert: the poll header there is 24 bytes, so the struct goes from 80 to 96 (the same growth a 4-byte field would cause, given the 16-byte alignment), and nothing reads the size except the sizeof-based allocation and us_socket_ext. The source-only rationale is in the body and stands as written; happy to add coverage if a maintainer wants a particular path exercised.

@robobun

robobun commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 1:05 PM PT - Aug 12th, 2026

❌ @robobun, your commit f9fe907 has some failures in Build #93292 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 37679

That installs a local version of the PR into your bun-37679 executable, so you can run:

bun-37679 --bun

@robobun robobun changed the title usockets(tls): key the per-loop spill slot and parked handshake reason by a connection id usockets(tls): key the spill slot by a connection id; park the handshake reason on the SSL Aug 12, 2026
@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

Revised in 8eaa718 after a self-review pass. Two things changed:

  1. The parked handshake reason no longer lives in loop_ssl_data at all. It is per-connection state, and this file already keeps that kind of state in SSL ex_data (the inline-reject verdict is stored the same way), so ssl_park_fatal_reason now stores the packed ERR_peek_error() code on the connection's SSL and ssl_dispatch_parked_reason formats it at dispatch time. ERR_error_string_n is a pure function of the code, so the string JS sees is unchanged (and the longest possible one with the vendored tables is 106 bytes, inside ERR_ERROR_STRING_BUF_LEN). That deletes the per-loop buffer, its owner field and every clear site; the existing reason assertions (node-tls-connect, node-tls-server, fetch-tls-cert, no-cipher-match) pass unchanged.
  2. ssl_id therefore only keys the spill slot, which is the one genuinely loop-level piece of state, and the comments are reduced to saying that once. The layout comments in internal.h that the new field had made inaccurate now describe the current layout in one place next to the size assert.

The PR body also answers why the slot is keyed by an id rather than by s->ssl: an SSL * is a recycled heap address too, so that would only have removed the relocation dependency, not the address-reuse one.

@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

Automated review of 8eaa718 came back with no findings; nothing to change from it. CI for the revised push is running as build 92997 (https://buildkite.com/bun/bun/builds/92997), the previous revision was green across the matrix in build 92636.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. Because it refactors ownership bookkeeping in the usockets TLS layer — changing us_socket_t's layout, moving the parked handshake reason into SSL ex_data, and re-keying the per-loop spill slot — a human look would still be worthwhile.

What was reviewed:

  • ssl_id init at all four plain-socket sites and assignment in us_internal_ssl_attach; counter starts from us_calloc'd 0 so pre-increment never hands out 0, and ssl_owns_spill refuses 0.
  • us_poll_resize (epoll/kqueue and libuv) does memcpy the whole header, so the id survives adoption; all references to the removed us_internal_ssl_socket_relocated are gone.
  • Every prior ssl_spill_owner == s / != s site now goes through ssl_owns_spill or the == 0 free-slot check; ERR_peek_error() returns uint32_t in BoringSSL and ERR_ERROR_STRING_BUF_LEN is its own header constant.
  • The parked reason on ex_data is a packed uint32_t in a void*, cleared before the JS dispatch runs, and freed with SSL_free — no separate release needed.
Extended reasoning...

Overview

This PR refactors two pieces of per-loop TLS scratch state in packages/bun-usockets/src/crypto/openssl.c: the ciphertext spill slot and the parked handshake-failure reason. Both previously recorded their owner as a raw us_socket_t *; the spill slot now records a 64-bit per-loop connection id, and the parked reason moves off the loop entirely onto the connection's own SSL* via ex_data. A new uint64_t ssl_id field is added to struct us_socket_t (fits existing padding on epoll/kqueue, pinned at 80 bytes by a new _Static_assert; grows the libuv-build struct from 80 to 96). The now-unnecessary us_internal_ssl_socket_relocated hook and its call in us_socket_adopt are removed. Five files touched: openssl.c, internal.h, context.c, loop.c, socket.c.

Security risks

None identified. This is internal bookkeeping; no user-controlled input flows into the new id assignment or the ex_data slot, and the packed error code is only ever read back and formatted through ERR_error_string_n.

Level of scrutiny

High. This is C in a hot, memory-safety-sensitive path (TLS handshake and ciphertext batching), it changes the layout of a core struct that every socket allocation depends on, and it ships intentionally without a new test (the PR argues, reasonably, that there is no observable behavior on main a test could distinguish). The reasoning in the PR body is thorough and I traced the invariants it relies on, but layout changes to us_socket_t and the design decision to spend 8 header bytes per socket (vs. e.g. keeping the pointer key and auditing teardown sites) are the kind of thing a maintainer should sign off on.

Other factors

  • The PR description is slightly out of date relative to the second commit (it describes ssl_last_fatal_error_owner becoming an id and a ssl_clear_parked_reason helper; the final state instead moves the parked reason to SSL ex_data, which is simpler). Not a correctness concern.
  • No CODEOWNERS entry for packages/bun-usockets.
  • CI build #92997 is in flight; the author reports 22 TLS suites pass identically with and without the change under ASAN.

@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

No code changes from the review above. Its one note, that the description still described the per-loop reason owner becoming an id, was read against the body as it stood when 8eaa718 was pushed; the body was rewritten right after and now describes the final shape (reason on the SSL via ex_data, ssl_id keying only the spill slot), with the old per-loop fields mentioned only in the description of what main does today.

…nection id

loop_ssl_data holds two pieces of per-loop scratch that one TLS connection
owns at a time: the ciphertext spill slot and the parked handshake reason.
Both recorded their owner as the us_socket_t address, which only works as
long as every teardown path clears the slot before the allocator can hand
the address to a new socket, and as long as adoption re-points the owner
when it moves a connection to a new block.

Give each TLS connection a 64-bit id instead: us_internal_ssl_attach takes
it from a per-loop counter and stores it in a new us_socket_t.ssl_id, and
both slots record that id. An id is never handed out twice on a loop, so a
connection can only ever touch a slot it filled itself, whatever address it
has; plain sockets carry id 0, which never matches an occupied slot. The id
travels with the header copy in us_poll_resize, so
us_internal_ssl_socket_relocated is no longer needed and is removed.
us_internal_ssl_detach now releases both slots up front, so every teardown
path gives the scratch back in one place.

ssl_id takes the place of tail padding on epoll/kqueue; a static assert
pins sizeof(struct us_socket_t) at its previous 80 bytes there.
The reason is per-connection state, so store it where this file keeps the
rest of that state: as the packed error code in an SSL ex_data slot, the
same shape as the inline-reject verdict. ssl_dispatch_parked_reason takes
it off the SSL and formats it with ERR_error_string_n at dispatch time,
which is a pure function of the code, so the reported string is unchanged.
This removes the per-loop reason buffer, its owner field and the clears on
every path that used to have to give it back; the spill slot is now the
only loop-level state keyed by ssl_id, and the comments describe that.
ssl_id is set by us_internal_ssl_attach like the other ssl_* header
fields, so read it only behind s->ssl: us_internal_ssl_detach, the one
release path that also runs for plain sockets, now skips them instead of
every socket constructor zeroing the field. Every spill owner still has
s->ssl set when it reaches the release, so the ordering is unchanged, and
the id-0 special case in ssl_owns_spill goes away with the init lines.
Also spell out why the spill slot is per loop (it is the bound on
ciphertext reported as written but not yet handed to the kernel), which is
what makes it need an owner in the first place.
@robobun
robobun force-pushed the farm/11f75065/tls-loop-slot-ids branch from 8eaa718 to 879a612 Compare August 12, 2026 14:04
@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

Third revision, 879a612, after another self-review pass over the ex_data version (the automated summaries above have nothing to act on; the walkthrough still describes the ssl_id = 0 init lines, which this revision removes):

  • ssl_id is now treated like the other ssl_* header fields: written by us_internal_ssl_attach, read only behind s->ssl. us_internal_ssl_detach, the one release path that also runs for plain sockets, skips sockets that never attached, so the four constructor init lines and the id-0 special case in ssl_owns_spill are gone. loop.c and socket.c drop out of the diff; every spill owner still has s->ssl set when it reaches the release, so the order of operations is unchanged. This also means the new us_socket_group_listen_fd path from cluster: port Node's cluster and child_process handle-passing suites (+43 upstream tests; cluster 54 → 85) and implement what they expose — round-robin fd handoff, SCHED_NONE shared handles, UDP clustering, IPC handle passing #31829 needed nothing.
  • The loop_ssl_data comment and the body now say why the spill slot is per loop (it is the bound on ciphertext reported as written but not yet handed to the kernel), which is what makes it need an owner at all; the body also records why per-connection spill buffers were not done here (a policy change, its own PR) and why the key is an id rather than s->ssl (an SSL * is a recycled address too).
  • The Tests section was wrong in one place and vague elsewhere: node-tls-no-cipher-match-error.test.ts never opens a connection, and the SSLV3_ALERT_HANDSHAKE_FAILURE assertion in fetch-tls-cert.test.ts is inside an it.todo. I temporarily instrumented park/dispatch locally and the body now lists, per file, how many distinct connections actually park and report their own reason, plus the vendored node tests where the client and the server of one failed handshake sit on the same loop and report different reasons.
  • Rebased onto current main; the A/B run was redone against main's packages/bun-usockets on the same tree and matches.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and the automated bug hunt found no issues. Because it refactors ownership bookkeeping in the TLS write-batching and handshake-error paths (native C, crypto/openssl.c) and changes struct us_socket_t layout, a maintainer sign-off is still worthwhile.

What was reviewed:

  • ssl_owns_spill reads of s->ssl_id are all reached only when s->ssl != NULL (us_internal_ssl_detach moved the release inside the guard; us_socket_ssl_spill_pending gates on s->ssl), so uninitialized ssl_id on plain sockets is never compared.
  • loop_ssl_data is us_calloc'd, so ssl_id_counter starts at 0 and the first id handed out is 1; ssl_spill_owner == 0 correctly means "free".
  • us_poll_resize header copy carries ssl_id across adoption, so the deleted us_internal_ssl_socket_relocated has no remaining callers or purpose.
  • The parked-error ex_data slot uses no free func and stores a packed uint32_t as (void*)(uintptr_t), matching the existing us_ssl_inline_reject_err_ex_idx pattern in the same file.
Extended reasoning...

Overview

This PR refactors two pieces of per-loop TLS scratch state in packages/bun-usockets/src/crypto/openssl.c:

  1. The ciphertext spill slot's owner changes from a raw us_socket_t * to a monotonic per-loop uint64_t connection id (ssl_id), assigned in us_internal_ssl_attach and stored in a new field on struct us_socket_t.
  2. The parked handshake-failure reason moves from a per-loop 256-byte buffer + owner pointer onto the connection's own SSL * via a new ex_data slot, formatted with ERR_error_string_n at dispatch time.

The now-unnecessary us_internal_ssl_socket_relocated (which re-pointed the address-keyed owner after us_poll_resize) is deleted along with its call site in context.c. internal.h gains the ssl_id field plus a _Static_assert pinning sizeof(struct us_socket_t) == 80 on epoll/kqueue.

Security risks

The change is entirely inside the TLS layer — write batching, spill draining, and handshake error reporting. A mistake here could misdirect spilled ciphertext to the wrong fd or misattribute a handshake error to the wrong connection. I traced every read of ssl_id through ssl_owns_spill and confirmed each is gated on s->ssl != NULL, so an uninitialized ssl_id on a plain socket never participates in a comparison. The counter is zero-initialized via us_calloc and pre-incremented, so 0 is never assigned. The ex_data pattern for the parked error is copied directly from the adjacent us_ssl_inline_reject_err_ex_idx slot. I did not find a path where the refactor weakens an existing check.

Level of scrutiny

High. This is native C in crypto/openssl.c — the review guide calls out native memory-ownership changes as the most-blocked category, and this file is on the wire path for every TLS connection. The refactor is conceptually tight and the PR description traces every invariant it depends on, but it is a design change (new struct field, state relocated to SSL ex_data, deletion of a relocation hook) rather than a mechanical fix, and it ships with no new test on the grounds that main has no observable defect to test against. That reasoning is sound, but the combination of struct-layout change + TLS ownership rewrite + no new coverage is exactly the kind of thing a maintainer should confirm.

Other factors

  • The PR description claims "the four plain-socket init sites set it to 0", but the final 3-file diff does not add ssl_id = 0 to any init site — that appears to be stale wording from before commit 879a612 moved ssl_release_spill inside the if (s->ssl) guard, which made zero-init unnecessary. The code as shipped is consistent; only the description is slightly out of date.
  • No prior human review on the thread; CodeRabbit and a prior automated pass found nothing actionable.
  • Existing TLS test suites (listed in the PR body) exercise all rewritten paths and reportedly pass identically with and without the change under ASAN.

@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

Nothing to change from the review above. Its one note (the description mentioning four init sites zeroing ssl_id) was read against the body as it stood when 879a612 was pushed; the body was rewritten right after and now describes the shipped design, where ssl_id is attach-owned and us_internal_ssl_detach skips sockets that never attached.

CI for 879a612 (build 93254): every lane is green except Windows 2019 x64, where test/bake/deinitialization.test.ts crashed in the dev server fixture (plain HTTP and WebSocket, no TLS). The same crash shows up on unrelated branches in builds 93237, 93192 and 93056, so it is a pre-existing intermittent failure; it has been reported for triage separately. The previous two revisions were green across the matrix (builds 92636 and 92997), and the delta in 879a612 is the detach gating plus comments.

@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

f9fe907 is an empty commit to re-run CI once after the unrelated Windows crash in build 93254 (see the status comment); nothing in the diff changed, so there is nothing new for the automated reviews to look at. The re-run is build 93292 (https://buildkite.com/bun/bun/builds/93292).

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. Because it refactors ownership bookkeeping in the usockets TLS layer (struct layout change, spill-slot keying, SSL ex_data lifecycle) and deliberately ships without new tests, a human look from someone who owns this subsystem would still be worthwhile.

What was reviewed:

  • ssl_id counter: loop_ssl_data is us_calloc'd so the counter starts at 0, first id is 1, free slot is owner 0 — no attached socket can match a free slot.
  • ssl_release_spill moving inside if (s->ssl): only ssl_flush_write_batch fills the slot and it's only reached with s->ssl set; every constructor NULLs s->ssl, so plain sockets correctly skip.
  • us_poll_resize memcpys the whole header, so ssl_id travels on adoption; the retired old block is freed by us_internal_free_closed_sockets without calling detach, so no double-release.
  • us_internal_ssl_socket_relocated has no remaining references; ERR_error_string_n into ERR_ERROR_STRING_BUF_LEN is bounded and NUL-terminated.
Extended reasoning...

Overview

This PR refactors two pieces of per-loop TLS scratch state in packages/bun-usockets: the parked handshake failure reason moves from a per-loop buffer keyed by us_socket_t* onto the connection's own SSL via a new ex_data slot, and the ciphertext spill slot's owner changes from a raw socket pointer to a monotonically-assigned per-loop 64-bit connection id stored in a new us_socket_t.ssl_id field. us_internal_ssl_socket_relocated and its call site in us_socket_adopt are deleted, ssl_release_spill moves inside the if (s->ssl) gate in us_internal_ssl_detach, and a _Static_assert pins sizeof(struct us_socket_t) to 80 on epoll/kqueue. Files touched: crypto/openssl.c, context.c, internal/internal.h.

Security risks

The change is in TLS record delivery ordering and handshake error attribution — a wrong spill-owner match could send one connection's ciphertext to another fd, and a wrong parked-reason attribution could report a stale error. I traced the invariants (counter never hands out 0; us_calloc zeros the slot; us_poll_resize memcpys the id; retired adopted blocks are us_poll_free'd without detach; s->ssl is NULL'd by every constructor and only set by attach) and they hold. The ERR_peek_error() cast to uint32_t matches BoringSSL's return type, and the (void*)(uintptr_t) round-trip through ex_data is lossless. No new untrusted-input parsing.

Level of scrutiny

High. This is hand-written C in the networking/TLS core, where memory-safety bugs are the most-blocked review category per REVIEW.md. The change is a design refactor rather than a bug fix, alters struct layout (grows us_socket_t by 16 bytes on the libuv/Windows build), and makes an architectural choice (id vs SSL* vs per-connection buffer) that the PR body explicitly weighs. That is exactly the kind of decision a maintainer who owns this subsystem should ratify.

Other factors

The PR is source-only by design — the body argues no test can distinguish before/after because every current teardown path already clears both slots on main. REVIEW.md is explicit that every behavioral change ships an automated test in the same PR; whether this qualifies as a no-behavior-change refactor that is exempt is a maintainer call, not an automated one. The author has done thorough A/B verification and CI is green on prior revisions, but the combination of no new tests, struct layout change, and a subtle ordering change in us_internal_ssl_detach puts this outside what I'd approve without a human familiar with the usockets TLS layer signing off.

@robobun

robobun commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator Author

No findings to act on in the review above; the diff is unchanged since 879a612 (f9fe907 is empty). On its one open question, whether a refactor with no observable behavior change ships without a new test: that is deliberately left to the maintainer, and the body sets out both the reasoning and the existing coverage that exercises every rewritten path.

Re-run build 93292 has passed 177 of 181 jobs with no failures, including the Windows 2019 x64 lane that hit the unrelated test/bake/deinitialization.test.ts crash in build 93254; the remaining four are macOS test lanes waiting for runners.

@Jarred-Sumner

Copy link
Copy Markdown
Collaborator

Closing: this is a refactor with no behavior change, and the code it touches is replaced by the write-path change in #44618 (the #44529 part).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants