Skip to content

tls: deliver decrypted bytes before SSLWrapper answers close_notify (wss via CONNECT proxy reports 1006 on a server close) - #43198

Merged
Jarred-Sumner merged 4 commits into
mainfrom
robobun/ce0cbd45/tls-tunnel-close-frame-with-close-notify
Sep 18, 2026
Merged

Jarred-Sumner merged 4 commits into
mainfrom
robobun/ce0cbd45/tls-tunnel-close-frame-with-close-notify

Conversation

@robobun

@robobun robobun commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • A wss:// WebSocket through a CONNECT proxy reports close code 1006 Failed to write when the server closes. A direct connection reports the code of the server. A server ends TLS behind its Close frame (ws.close() in Bun.serve does), so the frame and the close_notify arrive in one read.
  • The cause is SSLWrapper::handle_reading (src/uws/lib.rs:1033). On SSL_ERROR_ZERO_RETURN it sent our close_notify (shutdown(false)) before it ran the data callback for the bytes decrypted in the same read.
  • The Close echo of the client then fails in write_data, and enqueue_encoded_bytes (src/http_jsc/websocket_client.rs:881) calls terminate(FailedToWrite).

Fix

  • handle_reading runs the data callback first, then shutdown(false), then the close callback. openssl.c uses this order for sockets with a file descriptor.
  • Scope: a server close with an empty client send queue. The Notes list what still gives 1006 and what stays different from Node.
  • Verified: five new cases in test/js/web/websocket/websocket-proxy.test.ts (the four proxy cases fail without the fix) and one for node:tls over a Duplex in test/js/node/tls/node-tls-connect.test.ts. Also test/js/bun/http/proxy.test.ts, and node-tls-namedpipes.test.ts on Windows.
  • Self-reviewed: the ten design concerns are addressed. Three test-level items did not reach me, because the review output was cut off. The tests assert the CONNECT request and one write with both TLS records.

Background

  • SSLWrapper is a TLS engine over memory BIOs. Its owners are the WebSocket proxy tunnel, the fetch proxy tunnel, node:tls over a Duplex, and Windows named pipes.
  • close_notify is the TLS alert that ends one direction of a session. SSL_write fails on the side that sent it.
  • An endpoint that receives a WebSocket Close frame echoes it. CloseEvent.code is the code of the received frame.
Notes

Trace of the tunnel on main (BUN_DEBUG_SSLWrapper=1 BUN_DEBUG_WebSocketProxyTunnel=1), frames and close_notify in one read:

[sslwrapper] just read 12
[sslwrapper] just read 0
[websocketproxytunnel] writeEncrypted: 24 bytes      <- our close_notify
[sslwrapper] triggering data callback (read 12)
[websocketproxytunnel] onData: 12 bytes              <- Close frame, the echo fails
[websocketproxytunnel] onClose
{"code":1006,"wasClean":false,"reason":"Failed to write"}

With this change:

[sslwrapper] just read 12
[sslwrapper] just read 0
[sslwrapper] triggering data callback (read 12)
[websocketproxytunnel] onData: 12 bytes
[websocketproxytunnel] writeEncrypted: 30 bytes      <- Close echo
[websocketproxytunnel] onClose
{"code":4001,"wasClean":true,"reason":""}

How often. A Bun.serve origin that calls ws.close(4001) through the plain CONNECT proxy of the test suite gives 1006 on 20 of 20 connections on the release build.

The tests. Three cases use a Bun.serve origin that calls ws.close(4001, "bye") from its message handler: direct, http proxy, https proxy. They use the plain proxy of the suite, so the network decides if both TLS records arrive in one read. On the builds without the fix they did so on every run here (release and debug, Linux and Windows). Two more cases pin it. The client arms the proxy from its open handler and sends go. A raw TLS origin answers with socket.end(frames). The proxy holds the bytes until two complete TLS records are buffered (the frames, then the alert) and forwards them in one write. The assertion also checks the CONNECT request and that exactly one write with two records reached the client. 20 reruns of all five cases pass on the debug build.

Not fixed here. These routes keep their current behavior. Each one needs its own change and test.

  • node:tls over a Duplex or a Windows named pipe: socket.write() from the data handler for the last bytes fails with ERR_SOCKET_CLOSED, and the peer never gets the bytes. Node v26.3.0 and Bun's file descriptor path deliver them. The write is refused in socket_body.rs, because SSLWrapper::is_shutdown() includes received_ssl_shutdown. us_internal_ssl_is_shut_down in openssl.c checks only the sent side. The follow-up is a write-side predicate for the wrapper. It needs the order from this PR first, because write_data failed on sent_ssl_shutdown before.
  • WebSocket tunnel with queued outbound data: the close dispatch waits in close_dispatch_pending, and WebSocketProxyTunnel::on_close calls fail(Ended) without a look at it. The client reports 1006 Connection ended. The direct path honors it in handle_close.
  • WebSocketProxyTunnel::write maps WantRead and WantWrite to ConnectionClosed, so an echo during a TLS 1.2 renegotiation still gives 1006 Failed to write (see also tls: fix renegotiation leaving the handshake state latched (lost drain, bogus ECONNRESET on close) #37094).
  • The arm of handle_reading that refuses a renegotiation closes without a flush of the bytes already decrypted in that read.
  • The WebSocket tunnel ends TLS without close_notify after a clean close. This is not new: on main the tunnel sent close_notify only in the one-read flow, where the WebSocket close failed. After a close that worked (Close frame and close_notify in separate reads, or a client ws.close()), clear_data() tears the tunnel down with a fast shutdown, which queues close_notify but does not flush it. The one-read flow is now the same. The follow-up is a graceful shutdown(false) in the clean close of the tunnel, after the Close frame is flushed. The fast path itself cannot flush, because it is also destroy() for node:tls over a Duplex, where Node sends no close_notify (tls: close a destroyed TLS socket with a bare FIN, no close_notify #40412).

Other owners of SSLWrapper.

  • node:tls over a Duplex: the data event for the last bytes now fires before the close_notify reply is written to the Duplex. Before, it fired after. Node v26.3.0 emits the data first too. The new case in node-tls-connect.test.ts pins this order with an in-memory Duplex pair.
  • fetch proxy tunnel: a response and close_notify in one read still completes the response. received_ssl_shutdown is still set before the data callback, so is_shutdown() is true there as before, and tunnel_poolable still refuses to pool such a tunnel.
  • An owner that tears the wrapper down from the last data callback (shutdown(true)) now takes the normal fast shutdown path, not the sent_ssl_shutdown early return. Both run the close callback, so the guard after the data callback still returns. A fast shutdown queues close_notify but does not flush it, as for every other fast shutdown. So in this case the tunnel no longer writes a close_notify reply. The WebSocket tunnel already behaves this way for a client ws.close().
  • The early return in shutdown(true) stays reachable from a fatal read. response + corrupt TLS record in one packet in proxy.test.ts covers it.

Suites run with the change. Linux debug build: websocket-proxy.test.ts, websocket-proxy-close-reentrancy, websocket-proxy-tunnel-client-leak, websocket-proxy-tunnel-upgrade-leak, test-ws-bidir-proxy, first_party/ws/ws-proxy.test.ts, bun/http/proxy.test.ts (92 pass, includes the ASAN test from #31959), node-tls-connect, node-tls-upgrade, node-tls-duplex-close-throw-uaf, node-tls-duplex-write-throw-error-value, node-tls-socket-allow-half-open-option, renegotiation, and the Node parallel tests test-tls-js-stream, test-tls-inception, test-tls-destroy-stream, test-tls-socket-allow-half-open-option, test-tls-streamwrap-buffersize, test-http2-generic-streams. Windows x64 debug build: websocket-proxy.test.ts (four proxy cases fail before, all pass after) and node-tls-namedpipes.test.ts (6 pass, same run time with and without the change).


no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/http/proxy.test.ts

A wss:// WebSocket through a CONNECT proxy reported close code 1006
"Failed to write" when the server closed. A server that closes sends its
Close frame and ends the TLS session behind it (ws.close() in Bun.serve
does), so the frame and the close_notify usually arrive in one read. A
direct wss:// connection reports the server's code.

SSLWrapper::handle_reading called shutdown(false) as soon as SSL_read
returned SSL_ERROR_ZERO_RETURN. That sent our close_notify before the
data callback ran for the bytes decrypted in the same read. write_data
refuses to write after that, so the WebSocket client could not echo the
Close frame and failed the connection.

Run the data callback first, then send the close_notify, then run the
close callback. The uSockets path in openssl.c uses the same order.
@coderabbitai

coderabbitai Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Essentials

Run ID: 86aa232a-e26b-412e-ab98-ee79f271096d

📥 Commits

Reviewing files that changed from the base of the PR and between 422179d and 583f404.

📒 Files selected for processing (4)
  • src/uws/lib.rs
  • test/js/bun/http/proxy.test.ts
  • test/js/node/tls/node-tls-connect.test.ts
  • test/js/web/websocket/websocket-proxy.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 0 remain after this review.


Walkthrough

The change reorders SSL remote-close handling so buffered data and pending events are processed before shutdown. New TLS and WebSocket proxy tests verify event ordering and clean close handling when shutdown records arrive with application data.

Changes

TLS shutdown handling

Layer / File(s) Summary
Shutdown ordering and safety documentation
src/uws/lib.rs, test/js/bun/http/proxy.test.ts
SSLWrapper now flushes buffered data and pending events before completing the two-step shutdown. Comments document the close-callback lifetime sequence.
TLS event-order regression
test/js/node/tls/node-tls-connect.test.ts
A TLS 1.2 duplex test verifies the order of data, client close_notify, transport end, and end.
WebSocket proxy shutdown coverage
test/js/web/websocket/websocket-proxy.test.ts
Tests cover direct, HTTP-proxy, and HTTPS-proxy routes, including coalesced WebSocket Close and TLS close_notify records.

Suggested reviewers: cirospaciari

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 583f4

The TLS shutdown ordering change preserves safe teardown and has targeted regression coverage for the affected event ordering. No actionable merge risk remains.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the main SSLWrapper behavior change and the affected WSS CONNECT proxy failure. It is long but remains specific and relevant.
Description check ✅ Passed The description explains the problem, fix, scope, remaining limitations, and verification results. It does not use the exact template headings, but it provides the required information, including how …

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator Author

Status

Reproduction, on 1.4.3-canary.1 (release) and on a debug build of main, Linux x64 and Windows x64:

USE_SYSTEM_BUN=1 bun test test/js/web/websocket/websocket-proxy.test.ts -t "server close with the TLS close_notify"
  • Without the change, the four proxy cases report { code: 1006, reason: "Failed to write", wasClean: false }. The direct case reports the server's code.
  • With the change (bun bd test, same arguments), all five cases report { code: 4001, reason: "bye", wasClean: true }.

PR: #43198

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

  • 🟡 src/uws/lib.rs — Every wss client that closes from inside the tunnel's last data callback now ends TLS without ever sending close_notify, where the base branch always sent it. After the fix the Close echo runs before shutdown(false), so the WebSocket client's clear_data() calls shutdown(true) at src/uws/lib.rs:1049 with sent_ssl_shutdown still false and takes the fast path (631-665), which queues close_notify into the wbio but never calls handle_writing; the guard at 1049 then returns before the deferred shutdown(false) at 1062 runs. Fix: the fast path must drain the write BIO (handle_writing) before trigger_close_callback, for every owner (WebSocket tunnel, fetch ProxyTunnel::close_raw, UpgradedDuplex::close).

    Extended reasoning...

    Server-initiated close through a CONNECT proxy is the population this PR targets; it happens on every such close (20/20 per the PR). Trace: SSL_read returns 12 bytes then ZERO_RETURN; received_ssl_shutdown set (1035); data callback at 1047 delivers the Close frame. websocket_client.rs:1146 enqueue_encoded_bytes -> WebSocketProxyTunnel::write -> write_data (759) succeeds, SSL_write queues the echo, handle_traffic re-enters (1148) and flushes it. websocket_client.rs:1148-1150: send_buffer empty -> clear_data() -> WebSocketProxyTunnel::shutdown -> w.shutdown(true). At 596 sent_ssl_shutdown is false (the whole point of the fix), so the early-return branch is skipped and the fast path runs: SSL_shutdown twice (631,651) queues close_notify in the memory BIO, set_received_ssl_shutdown, trigger_close_callback (663), return. No handle_writing call anywhere on that path; the only drain is at 683 in the non-fast branch. Back in handle_reading, closed_notified is true so 1049 returns false; 1056-1064 never run. The queued close_notify is freed with the SSL. On the base, shutdown(false) ran before…

    Verification: nit; acknowledged in diff (PR description only, not in code): "A fast shutdown queues close_notify but does not flush it ... So in this case the tunnel no longer writes a close_notify reply" — the bound stated there is accurate, but the description does not weigh what an origin that waits for the peer's close_notify does. Trigger: a wss origin behind a CONNECT proxy sends its Close frame and…

Comment thread src/uws/lib.rs
…ly over a Duplex

node:tls over a Duplex uses the same SSLWrapper as the WebSocket proxy
tunnel. The peer's last application data and its close_notify reach the
engine in one chunk. The 'data' event must fire before the engine writes
its close_notify reply to the transport, as in Node.
Comment thread src/uws/lib.rs Outdated
Comment thread src/uws/lib.rs Outdated
@robobun

robobun commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 6:19 PM PT - Sep 17th, 2026

@robobun, your commit 583f404 is building: #117450

@robobun

robobun commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator Author

Reply to the two review findings.

1. Order test for node:tls over a Duplex. Added in 4c492f6 (test/js/node/tls/node-tls-connect.test.ts). Without the change the log is write close_notify then data last. With the change it is data last then write close_notify. Node v26.3.0 also emits the data first. The test fails on the release build and passes on the debug build, on Linux and on Windows.

2. No close_notify when the owner closes from the last data callback. I did not change the fast path of shutdown(true) in this PR. These are the reasons:

  • The one-read flow is now the same on the wire as the flows where the close already worked. I recorded the TLS records that the tunneled client sends after the server's Close frame (TLS 1.3, the Close echo is 30 bytes, a close_notify is 24 bytes):

    build Close frame and close_notify client sends close event
    main separate reads echo 4001, clean
    this PR separate reads echo 4001, clean
    main one read close_notify, no echo 1006 Failed to write
    this PR one read echo 4001, clean

    On main the tunnel sent close_notify only in the flow where the WebSocket close failed. After a close that worked, and after a client ws.close(), it never sent one, because clear_data() tears the tunnel down with a fast shutdown.

  • The fast path is also destroy() for node:tls over a Duplex and a named pipe. Node sends no close_notify from destroy(), and tls: close a destroyed TLS socket with a bare FIN, no close_notify #40412 made the file descriptor path match that. A drain of the write BIO in the fast path sends an alert on every such destroy().

  • The write callback must not run after the close callback, because the owner can free handlers.ctx (http: fix use-after-free when proxy tunnel closes after response completes mid-read #31959). So the drain cannot move behind the guard in handle_reading.

The correct place for this is the clean close of the WebSocket tunnel: a graceful shutdown(false) after the Close frame is flushed, before clear_data(). That matches the direct path, where ssl_close sends close_notify after the echo. It changes every clean close through the tunnel, also the client ws.close(), and it touches the teardown that #37487 reworks. I added it to the list of follow-ups in the PR notes.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

@Jarred-Sumner
Jarred-Sumner merged commit 91d89f4 into main Sep 18, 2026
8 of 10 checks passed
@Jarred-Sumner
Jarred-Sumner deleted the robobun/ce0cbd45/tls-tunnel-close-frame-with-close-notify branch September 18, 2026 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants