Skip to content

node:quic: pick a bidirectional stream's send limit by who opened it - #46322

Open
robobun wants to merge 1 commit into
robobun/26acf120/quic-default-stream-windowsfrom
robobun/26acf120/quic-send-limit-by-initiator
Open

robobun wants to merge 1 commit into
robobun/26acf120/quic-default-stream-windowsfrom
robobun/26acf120/quic-send-limit-by-initiator

Conversation

@robobun

@robobun robobun commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • lsquic picks the first send limit of a bidirectional stream by the role of the connection (apply_trans_params, lsquic_full_conn_ietf.c:3671). RFC 9000 section 18.2 picks it by who opened the stream.
  • The two agree only for a stream the client opens. With unequal windows, a stream that a node:quic server opens ends with flow control violation on stream 1, or stalls.
  • A FIN with no data before it never leaves a stream whose window is 0.

Fix

  • patches/lsquic/stream-send-limit-by-initiator.patch keeps both limits of the peer, named as in upstream's draft WebTransport and HTTP Datagram Support litespeedtech/lsquic#629. HTTP/3 connections keep their one value. A unidirectional stream opened in 0-RTT keeps the unidirectional limit.
  • stream.rs sends a raw QUIC stream's FIN at once when no data is queued.
  • Verified: quic-stream.test.ts, 21 new tests, 14 fail without this PR. Also test-quic-*.mjs, serve-http3.test.ts, fetch-http3-client.test.ts.
  • Self-reviewed: 35 concerns raised, 29 addressed. Notes lists the rest.

Background

  • The peer's initial_max_stream_data_bidi_remote limits a stream this side opens. Its bidi_local limits a stream the peer opens.
  • lsquic gives a write event only to a stream with credit. A FIN takes none. HTTP/3 keeps the write event: its FIN follows the header block.
  • Considered equal default windows only (node:quic: use one default stream window per endpoint #46321, the base): explicit windows still fail.

Downsides

  • A peer that advertises 0 for such a stream now gets no data on it, and no error. Released bun does that (1.4.2 measured). Notes has the workaround.
  • A QUIC connection grows from 10232 to 10240 bytes, Bun.serve and fetch included. mimalloc's block stays 10240 bytes.
  • Release binary: +544 bytes.
Notes

Stack. This PR is based on #46321. That one repairs the default case with no change to lsquic. This one makes the sender correct for any pair of windows. The diff here is only this PR.

How it was found. The failure is in the "Found, not in this PR" list of #46054. No user reported it.

The lsquic patch, hunk by hunk (lsquic_full_conn_ietf.c, 6 hunks, 105 lines).

  • struct ifc_cfg: max_stream_send becomes max_stream_send_bidi_local and max_stream_send_bidi_remote, the names of WebTransport and HTTP Datagram Support litespeedtech/lsquic#629. Upstream master (a9616720, v4.10.2) still has the one role-keyed limit. The vendored version is v4.9.4.
  • apply_trans_params: both limits are stored from the peer's transport parameters. The role-keyed assignment is removed.
  • create_bidi_stream_out reads bidi_remote. new_stream reads bidi_local.
  • The loop that sets the limits of existing streams again after 0-RTT: a unidirectional stream takes the unidirectional limit. Before, a resumed client sent 400000 bytes into a 12288-byte window, or failed the handshake when the server's initialMaxStreamDataBidiRemote was below the bytes already sent.
  • HTTP/3 (IFC_HTTP): the limit for server-opened streams keeps the value of the client-opened ones. This part is not in Create share image for social media #629. lsquic's transport parameter check bounds only that value from below (4096). With a limit of 0, the stream that an HTTP/3 node:quic server is not allowed to open reaches assert(size <= stream_avail) in stream_write_avail_with_frames (reproduced under gdb). A bump to an lsquic with Create share image for social media #629 has to replace this block, not only drop it.

The FIN change. drain_outbound sends the FIN, and it ran only from lsquic's write event. lsquic gives that event only when the stream and the connection have credit (stream_writeable). kick_write and bind_raw now call drain_outbound at once when only the FIN is queued. HTTP/3 sessions are excluded: createBidirectionalStream({ headers, body }) queues the end of an empty body before the header block, and a FIN before the headers makes lsquic reset the stream. Four tests pin that order. 18 HTTP/3 orderings (end before or after sendHeaders, from a microtask or a timer) and 24 create-then-abandon sequences give the same result as a build of #46321.

Measurements. Release builds of #46321 and of this PR.

Mixed versions. Two processes on loopback, a server-opened stream, 6 runs each. "1.4.2" is the released bun, which advertises a window of 0 for such a stream and accepts 16384 bytes.

Case #46321 this PR
this build on both ends, 4 MB each way complete complete
1.4.2 server sends 4 MB to this build complete complete
this build as server opens the stream with no data, a 1.4.2 client sends 4 MB complete complete
this build as server sends 1000 bytes to 1.4.2 complete 0 bytes arrive
1.4.2 server, this build as client sends 1000 bytes complete 0 bytes arrive
the same two cases with no data, FIN only complete complete
this build as server sends 262144 bytes to 1.4.2 5 of 6 connections die 0 bytes arrive

In the two "0 bytes" rows nothing reports an error. The body is never sent, the sessions end at the idle timeout, and the closed promises resolve. Workaround on the 1.4.x end: set initialMaxStreamDataBidiRemote (client) or initialMaxStreamDataBidiLocal (server). With 1048576 there, both pairings carry 262144 bytes into that window and 1000 bytes the other way in 6 of 6 runs.

Tests. Of the 21 new tests, 14 fail on a build of #46321. The other 7 guard one part each:

  • 4 HTTP/3 tests fail when the FIN change also applies to HTTP/3.
  • "ends in both directions on a stream the server opens when both windows for it are 0" fails with the lsquic patch and without the FIN change.
  • "keeps its limit when it is bidirectional" fails when the two limits are swapped in the 0-RTT loop. The equal default windows hide that from the ported test-quic-zero-rtt.mjs.
  • "carries 4096 bytes into a window of 4096 bytes" is the control for the 4097-byte row.

The tests that end a stream without data turn the idle timeout off. With it, a session whose FIN never leaves ends after 10 s and both readers see the same clean end as after a FIN.

Also run on the debug build: all 235 test-quic-*.mjs files (the same 3 fail as on main: 2 import node:stream/iter, 1 asserts an RTT below 10 ms; test-quic-callback-error-* fail in 1 to 2 of 6 runs on both builds), quic-endpoint.test.ts and quic-sni.test.ts (18 pass), serve-http3.test.ts (79 pass), fetch-http3-client.test.ts (67 pass).

Self-review, not taken.

Order with other work. The branch robobun/3fc05d46/quic-stream-credit-on-pull has the same 0-RTT hunk and a work-around for this bug. The second of the two to land drops its copy. #37491 makes Outbound.data private. only_fin_pending then reads out.is_empty().

lsquic picks the first send limit of a bidirectional stream by the role
of the connection. RFC 9000 section 18.2 picks it by who opened the
stream. The two agree only for a stream that the client opens. With
unequal windows, a stream that a node:quic server opened sent into the
wrong window in both directions.

- patches/lsquic/stream-send-limit-by-initiator.patch keeps the two
  limits of the peer, under the names of the upstream draft
  litespeedtech/lsquic#629. An HTTP/3 connection keeps its one value. A
  unidirectional stream opened in 0-RTT keeps the unidirectional limit
  when the handshake completes.
- A raw QUIC stream sends a FIN that has no data before it without a
  write event. lsquic gives a write event only to a stream that has
  flow-control credit, so such a FIN never left when the window of the
  stream or of the connection was 0.
@robobun

robobun commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 8:38 AM PT - Oct 21st, 2026

❌ @robobun, your commit 36bb05a has 1 failures in Build #130056 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 46322

That installs a local version of the PR into your bun-46322 executable, so you can run:

bun-46322 --bun

@robobun

robobun commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status

How the failures reproduce, on a build of #46321 (the base of this PR). Both ends are in one process, ALPN quic-test.

After session.opened the server opens a bidirectional stream with a 50000-byte body, and the client answers with 50000 bytes:

  • Client transportParams: { initialMaxStreamDataBidiRemote: 1000, initialMaxStreamDataBidiLocal: 100000 }: the client session reports QUIC transport error 1: flow control violation on stream 1.
  • The same two values swapped: the server stops after 1000 bytes and the stream never ends.

A stream that ends without data:

  • Server transportParams: { initialMaxStreamDataBidiRemote: 0 }. The client calls createBidirectionalStream({ body: new Uint8Array(0) }). The server never sees the stream.

With this PR all three complete. The tests are in test/js/node/quic/quic-stream.test.ts.

This PR: #46322. It is based on #46321.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the inline findings, I also checked the IFC_HTTP block in the lsquic patch (it copies one limit over the other in the same direction the old role-keyed assignment chose, so HTTP/3 send limits are unchanged for both server and client) and the 0-RTT re-limit loop (a unidirectional stream now takes ifc_max_stream_data_uni, which matches what the resumed-session test pins).

Extended reasoning...

The change adds a vendored lsquic patch splitting the bidirectional send limit into local/remote per RFC 9000 18.2, and a Rust-side fast path that sends a bare FIN synchronously on non-HTTP/3 streams; the inline findings cover the FIN fast path, so this note only records what else in the patch was examined and ruled out.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

  • 🟡 src/runtime/node/quic/stream.rs — A raw node:quic stream created with no body and destroyed in the same turn now reaches the peer as a reset stream where the base sent nothing. drain_outbound at stream.rs:526-528 now runs synchronously from kick_write for a FIN-only stream and sets wrote_to_lsquic, so abort_for_destroy at stream.rs:936 sends STOP_SENDING/RESET_STREAM at once instead of calling defer_stream_abort. The marker check in flush_deferred_aborts (session.rs:717-720), which drops the frames for Node parity when another stream wrote in between, is never reached. Fix: keep the created-and-abandoned path deferred for a stream whose only lsquic write was a bare FIN, or have the direct branch honor the same write_marker rule.

    Why this was flagged

    Trigger: on a non-HTTP/3 session, JS calls stream.setBody(null) or createUnidirectionalStream({ body: new Uint8Array(0) }) and then stream.destroy() before the turn ends, with any write on another stream of the session in between. configureOutbound (quic.ts:1377-1379) calls endWrite -> end_write (stream.rs:792) -> kick_write (stream.rs:567); only_fin_pending is true, so drain_outbound runs, calls s.shutdown(1) and sets wrote_to_lsquic (stream.rs:527-528). destroy() then calls abortForDestroy (quic.ts:2085); abort_for_destroy sees wrote_to_lsquic true (stream.rs:936) and sends stop_sending/reset directly. On the base, wrote_to_lsquic was still false (the FIN waited for a write event), so defer_stream_abort ran and flush_deferred_aborts (session.rs:717-720) called shutdown_internal when the marker had moved, putting nothing on the wire. The code path that implements the stated Node-parity rule is now unreachable for FIN-only streams. Remedy: route the FIN-only case through the deferred-abort path, or apply the marker rule in the direct branch.

    Verification: stream.rs:277-278 (bind_raw) and :568-569 (kick_write) now call drain_outbound() when only_fin_pending() holds; drain_outbound at :526-528 does s.shutdown(1); self.wrote_to_lsquic.set(true). abort_for_destroy at stream.rs:935-946: with the PR the first branch runs and frames go out at once; on the base the second branch ran and flush_deferred_aborts (session.rs:717-720) dropped them via shutdown_internal() when e.marker != marker.

Comment on lines 567 to 571
fn kick_write(&self) {
if let Some(s) = self.ls() {
if self.only_fin_pending() {
self.drain_outbound();
} else if let Some(s) = self.ls() {
s.want_write(true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 A node:quic server that opens a bidirectional stream to a released bun client (1.4.x advertises initialMaxStreamDataBidiRemote 0) now sends nothing, with no blocked event and no error. With the stream window at 0, kick_write at src/runtime/node/quic/stream.rs:571 only sets want_write(true), and lsquic never delivers the write event, so drain_outbound never runs and note_write_blocked (the only StreamBlocked producer) never fires. Fix: when data is queued and the stream has no credit, surface the blocked state without a write event, so the application sees the stall the way node's onblocked does. The PR text calls this an accepted downside; in fact it is a silent loss for every default-configured 1.4.x peer, where the base delivered up to 16384 bytes. [also at: src/runtime/node/quic/stream.rs:572 - node:quic users whose peer advertises 0 for a stream (every released Bun <=1.4.2 does so by default for server-opened bidirectional streams) now get a silent stall: queued bytes never leave, no error, no onblocked, until idle timeout.; patches/lsquic/stream-send-limit-by-initiator.patch:42 - A node:quic app that opens a server-side bidirectional stream to a Bun 1.4.x peer now delivers 0 bytes of its body, with no error, where the base delivered it.]

Why this was flagged

A server built from this PR calls createBidirectionalStream({ body }) toward a bun 1.4.2 client, which by default advertises initial_max_stream_data_bidi_remote = 0. The patch at patches/lsquic/stream-send-limit-by-initiator.patch now gives that stream max_stream_send_bidi_remote = 0 (create_bidi_stream_out hunk). set_body queues the bytes and calls kick_write (src/runtime/node/quic/stream.rs:741); only_fin_pending is false because data is non-empty, so kick_write only calls want_write(true) at stream.rs:571. lsquic's stream_writeable requires write credit, so on_stream_write (stream.rs:1184) never runs, drain_outbound never runs, and note_write_blocked (stream.rs:480), the only source of SessionEvent::StreamBlocked, never fires. The user gets a stream whose body is never sent, no 'blocked' callback, no error, and sessions that end at the idle timeout with closed resolving cleanly. On the base branch the server took the client's bidi_local window instead and delivered up to 16384 bytes.

Verification: patches/lsquic/stream-send-limit-by-initiator.patch, create_bidi_stream_out hunk: max_stream_send_bidi_remote is now the send limit of a locally-opened bidi stream; on base the server used the peer's bidi_local. src/runtime/node/quic/stream.rs:567-572 kick_write with data queued only does s.want_write(true). drain_outbound is the only caller of note_write_blocked, the sole producer of SessionEvent::StreamBlocked (stream.rs:488).

Comment on lines +560 to +565
fn only_fin_pending(&self) -> bool {
let out = self.outbound.get();
out.end == PendingEnd::Fin
&& out.data.is_empty()
&& !self.session_ref().is_some_and(QuicSession::is_http3)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Applications that set an empty or null body on a peer-opened unidirectional stream now lose the peer's data and see the stream close early. only_fin_pending() at stream.rs:560 is true for such a stream (JS setBody has no direction guard), so kick_write calls drain_outbound, which after the FIN runs s.shutdown(0) on the read side (stream.rs:531-533). Fix: skip the direct-FIN path whenever the write side is already ended (write_ended != 0, set for inbound uni streams at stream.rs:271), or make set_body/end_write ignore a FIN on a receive-only stream like the JS writer getter does. On the base want_write(true) is refused by lsquic for that stream, so nothing was shut down.

Why this was flagged

A raw (non-HTTP/3) session; the peer opens a unidirectional stream and the application's onstream handler calls stream.setBody(null) (quic.ts:1377-1380 -> handle.endWrite()). quic.ts:2373-2390 checks only outboundSet/writer, not direction. end_write (stream.rs:797-808) sets o.end = PendingEnd::Fin with empty data and calls kick_write. only_fin_pending() (stream.rs:560-565) is true because the session is not HTTP/3, so kick_write (stream.rs:568-569) calls drain_outbound directly. There s.shutdown(1) runs (stream.rs:527), and because the id has STREAM_ID_UNI_BIT it calls s.shutdown(0) (stream.rs:531-533), shutting the read side of the peer's stream. Later bytes and the peer's FIN never reach JS. On the base branch kick_write only called s.want_write(true); lsquic refuses lsquic_stream_wantwrite on a stream whose write side is done (bind_raw marks write_ended = 1 at stream.rs:271), so drain_outbound never ran. The new direct path bypasses that lsquic check and nothing in only_fin_pending looks at write_ended.

Verification: pre-existing — the base branch reaches the same read-side shutdown by the same route one lsquic tick later, so this PR does not introduce the data loss. Trigger: a raw session whose onstream handler calls setBody(null) on a peer-opened unidirectional stream. On the base, kick_write called s.want_write(true) and on_stream_write (stream.rs:1184-1186) calls the same unchanged drain_outbound (stream.rs:525-533).

@robobun

robobun commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

A second branch for this defect exists: robobun/f6399ed2/quic-stream-send-limit-by-initiator. It came from a separate report and was written without knowledge of this stack. It has no PR and gets none: #46321 and this PR cover it, and this PR also has the FIN change that the branch does not have. The lsquic patch there is the same design as the one here (one limit per initiator, the IFC_HTTP block, the unidirectional limit in the 0-RTT loop).

Three differences that can be of use here:

  1. struct ietf_full_conn does not grow. The branch moves ack_exp out of ifc_cfg into the 2 bytes of padding after ifc_max_udp_payload. Release builds of main 9634520 and of the branch both have calloc(1, 0x27f8) (10,232 bytes) in lsquic_ietf_full_conn_client_new and lsquic_ietf_full_conn_server_new. That removes the second item under Downsides here.

         struct {
             uint64_t    max_stream_send_bidi_local;
             uint64_t    max_stream_send_bidi_remote;
    -        uint8_t     ack_exp;
         }                           ifc_cfg;
    @@
         unsigned short              ifc_max_udp_payload;    /* Cached TP */
    +    uint8_t                     ifc_ack_exp;            /* Cached TP */
         lsquic_time_t               ifc_max_ack_delay;

    The two uses are in apply_trans_params and process_ack_frame.

  2. lsquic_stream_set_max_send_off counts the buffered bytes. It refuses a new limit below tosend_off, but the send credit is max_send_off - tosend_off - sm_n_buffered (stream_write_avail_no_frames). A lowered limit that falls inside the buffered bytes makes that credit wrap. The branch compares with tosend_off + sm_n_buffered. The only compiled caller is the 0-RTT loop that this patch changes. This is from a read of the code. I did not reproduce the wrap.

  3. For node:quic: use one default stream window per endpoint #46321: the default changes on raw QUIC only. The branch sets the default in us_nq_settings_init under if (!is_http), and only for the window that is 0. HTTP/3 over node:quic then keeps its transport parameters: 72 bytes (client) and 107 bytes (server, 17-byte ODCID) on main and on the branch. Raw QUIC: client 73 to 79 bytes, server +6 bytes. The branch does not raise the unidirectional window, so it does not have the 0-RTT row of node:quic: use one default stream window per endpoint #46321 for a 1.4.x client.

Other numbers from the branch, linux x64, release builds of main 9634520 and the branch:

  • create_bidi_stream_out: 98 instructions on both. new_stream: 189 on both. apply_trans_params: 1,555 to 1,596 bytes.
  • Release text: +288 bytes (size), with no FIN change.
  • A 1.4.3 server sends 1,000,000 and 20,000,000 bytes on a stream it opened to a client of the branch: both arrive whole. A client of main closes the session in 3 of 3 runs.
  • A server of the branch sends 21 bytes to a 1.4.3 client: 0 arrive, as in the table here.

@robobun

robobun commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

#46446 has the rule for a stream that ends with nothing queued, as its own change. It is meant to land before #46321 and this PR.

It differs from only_fin_pending() in two points:

  • It sends the FIN in the call only when lsquic_stream_write_avail() is 0. A stream with credit keeps the write event. Without that check, 1000 empty-body round trips at concurrency 50 send 47 client and 54 server datagrams, against 38 and 40.
  • It leaves a stream with no send side alone. drain_outbound() ends with shutdown(0) for every unidirectional stream. So setBody("") on a unidirectional stream that the peer opened closes the read side, and the peer's later bytes are lost. On main this happens only when the stream gets a write event. With no send credit main reads every byte. This point is from a read of only_fin_pending(). I did not run a build of this PR. The test "does not close a stream that the peer opened for its own data only" in node:quic: send the end of a raw QUIC stream at once when it has no flow-control credit #46446 covers the case.

On the branch of #46446, without the lsquic patch of this PR, these tests of this PR pass: the 8 tests of "a stream that ends without data" and the 4 tests of "an HTTP/3 stream whose body ends before its headers are sent".

With #46446 under this PR, these parts are not needed here:

  • only_fin_pending() and its two callers in stream.rs
  • QuicSession::is_http3() in session.rs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant