Repository navigation
perf(cmux-tui): the inline key write checks room with one kernel queue per connection and sends without a copy (#35413 s17ga) - #35988
Conversation
…room check (#35413 s17ga; red on macOS) The s17ga typing A/B for #35413 (STATUS 13245, 13254) kept the S1 input lane off: daemon part p50 was +0.016 ms with ON, against an OFF round-to-round spread of 0.009 ms, with both ON rounds above both OFF rounds. The whole difference was the key's write call (daemon.send.write -> written): 0.019 and 0.020 ms at p50 with ON, 0.005 ms with OFF. The inline key write (#35872) checks for room with poll(POLLOUT, 0) before its send. On XNU each poll() opens a kernel event queue, registers a knote on the socket under the socket's lock, and frees both again (sys_generic.c poll_nocancel; the #35872 reviewer's note 3). The test sends 100 keys to one idle connection and counts the kernel event queues that their room checks opened, with a test-only per-thread counter at the room check. It fails on this code: 100 keys open 100 queues, one per key. macOS only (Linux's poll opens no queue).
… per connection and sends without a copy (#35413 s17ga) The test is the commit before this one (red on macOS: 100 keys to one connection opened 100 kernel event queues, one per key's room check). s17ga (STATUS 13245, 13254, 13302) kept the S1 input lane off. On one boundary for both arms, ON was worse at p50 beyond the OFF spread. The whole difference was the write call on the key's own request thread: 0.019-0.020 ms with the inline write, against 0.005 ms for the old path's direct write. The inline write did three things the old path did not: 1. poll(POLLOUT, 0) before the send. On XNU each poll() opens a kernel event queue, registers a knote under the socket lock and frees both. 2. Two heap copies of the key (Frame::new(payload.to_vec()), encode_frame). 3. One send of a frame it had just built. Now (terminal_host_runtime/shared/attachment/room_check.rs): - macOS: each host connection keeps one kernel event queue, with the socket's EVFILT_WRITE registered once at its first key. Each key reads it with kevent64(KEVENT_FLAG_IMMEDIATE): one call, no allocation, no sleep. The filter's data is the socket's free room; the key goes inline only when it holds the whole frame. EV_EOF or EV_ERROR is no room, so the key goes to the port and its reconnect path. A failed kqueue() falls back to poll(). The connection's attachment owns the queue and shares it with every route clone; it closes with the attachment, and a reconnect makes a new one. - Linux: no check. MSG_DONTWAIT is honoured, and a Unix stream socket takes a frame this small whole or not at all. - Every Unix target: the frame header is built on the stack (terminal_host_protocol::input_frame_header, pinned to encode_frame's output by the_inline_frame_header_is_the_encoders_header). The header and the key go out in one sendmsg of two iovecs, with no heap copy of the key. The order rules, the size limit and the fallbacks to the port are the same. The room check still runs under the connection's writer guard, so nothing takes the room before the send.
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
web and size checks: green Shipped in:
|
|
Found 298 test failures on Blacksmith runners: Failures
...and 288 more test failures. View all on Blacksmith
|
…line again; the header test covers Paste (#35988 review) Review of #35988 (SHIP at 0fffc79), should-fix 1: no test sent a key after the socket was full and the host then read everything. On macOS that is the one state the per-connection room-check queue keeps between keys: if its write filter did not report room again, every later key on the connection would go to the port and nothing would say so. keys_to_a_host_that_stopped_reading_never_block_their_requests now ends by waiting for an idle port, holding the port's writer and sending one more key: it must reach the host inline. The order test uses the same pattern. Note 3: the_inline_frame_header_is_the_encoders_header also covers MessageKind::Paste, which reaches the inline write too.
…ote shape, warm and cold (#35988) The live app daemon under the L6 typing load (cmux21, build e8291edc, lskq and proc_pidinfo) holds 264-266 knotes: 13-14 on fds, 250-253 hashed (234-237 EVFILT_PROC, one per host child), and 1052 open fds. This bench builds that shape in the test process and times poll(POLLOUT, 0) against kevent64 (IMMEDIATE) on a queue registered once, warm and with the caches swept before each call. Fails on purpose so the step log shows the report.
…aemon's own call and switch rates (#35988) The live app daemon under the L6 load ran 390-560 threads, about 375k Unix calls and 145k context switches a second (proc_pidinfo, cmux21, build e8291edc). At its knote count alone, cold, poll() cost 4.75 us more than kevent64 (step 4a530efa4a2276db2caa564e), a third of the A/B difference. This bench adds socket ping-pong and EVFILT_USER wake threads paced to about those rates and times both calls warm and cold, idle and under that load.
…ter #35988 at the daemon's shape and load Times the old write (poll room check, two copies, send) against the new one (kevent64 on the connection's queue, stack header, one sendmsg) end to end, and each half alone, interleaved, warm and cold, idle and under paced socket and EVFILT_USER load, at the live daemon's knote shape, with a host thread that reads the socket. The knote-only benches explain about a third of the 12-13 us the A/B removed; this measures the rest of the path the same way. Fails on purpose so the step log shows the report.
|
Review (claude-reviewer-rds-cost): SHIP at 058c3a2 Independent review subagent of the delta 0fffc79..058c3a2 (test only:
|
…inline-write cost) into fix/p0-typing-s1-8-delete-old-path
…ted_typing_surface without the lane argument S1.8 removed #35988 (179bb0f) added a_connections_keys_share_one_kernel_queue_for_their_room_check with hosted_typing_surface(name, true). S1.8 removed the lane argument, so the merged tree does not compile on Apple targets (the test is cfg(target_vendor = "apple"), so the Linux steps cannot see it). The call drops the argument; the test is unchanged otherwise.
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
Summary
The s17ga typing A/B for #35413 kept the S1 input lane off (STATUS 13245, 13254, 13302). The daemon part was worse with the lane on at p50: +0.016 ms against an OFF round-to-round spread of 0.009 ms, with both ON rounds above both OFF rounds. On one boundary for both arms (from the key's arrival at the daemon to its bytes on the host socket), the difference stays: +0.018 ms at p50. It is all in the key's write call on its own request thread: 0.019-0.020 ms at p50 with the inline write (#35872), against 0.005 ms for the old path's direct write. Both arms write the host socket on the request thread, so this is the inline write's own cost.
The inline write did three things the old direct write did not:
poll(POLLOUT, 0)before the send. On macOS it is needed: there asendwithMSG_DONTWAITon a blocking stream socket still waits for room (perf(cmux-tui): a key to an idle input port is written on its own turn, with no thread hand-off (#35413 A/B) #35872). But XNU'spoll()opens a kernel event queue, registers a knote on the socket under the socket's lock, and frees both again on every call. Freeing the queue walks every knote of the process: the fd knote lists under the process's fd lock, then every hashed knote under the hash lock. The daemon holds one hashedEVFILT_PROCknote per host child, so this cost grows with the session's terminals.Frame::new(payload.to_vec()), thenencode_frame).sendof the frame it had just built.Now (
terminal_host_runtime/shared/attachment/room_check.rs):EVFILT_WRITEregistered once, at its first key. A key reads it withkevent64(KEVENT_FLAG_IMMEDIATE): one call, no allocation, no sleep (a zero-timeoutpoll()also slept about 10 us when there was no room). The filter'sdatais the socket's free room; the key goes inline only when that holds the whole frame.EV_EOForEV_ERRORis no room, so the key goes to the port and its reconnect path. A failedkqueue()falls back topoll(). The connection's attachment owns the queue and shares it with every route clone; it closes with the attachment (a reconnect makes a new attachment and a new queue). Closing the socket removes the registration, so a stale queue never reports room.MSG_DONTWAITis honoured there, and a Unix stream socket takes a frame this small whole or not at all.terminal_host_protocol::input_frame_header, pinned byte for byte toencode_frame's output by a test). The header and the key go out in onesendmsgof two iovecs, with no heap copy of the key.The order rules, the 1 KB size limit and every fallback to the port are unchanged. The room check still runs under the connection's writer guard, so no other daemon writer can take the room before the send.
Testing
Focused command:
cmux-ci run --class isolated --script scripts/ci/cmux-tui-rust-check.sh --os macos|linux --arg test --arg input_port_inline --ref SHA.Pasteas well asInput. Head 058c3a2: macOSinput_port_inline6 of 6 (step f25adbd3739062d9b540f9e6), Linux 5 of 5 (step e1c696e6dc6436d3597240cb; the first Linux run died on a zig TLS fetch while building ghostty-vt-sys, so the same SHA ran again), Linux clippy -D warnings (step 076708c6cfd10fc2eca60557). Review of the delta: SHIP at 058c3a2 (receipt in the PR comments).a_connections_keys_share_one_kernel_queue_for_their_room_checkat 737b57c (the test only, on feat 539facd), macOS step a56f9142fa5dac3cc0a4e0a6: "100 keys to one connection opened 100 kernel event queues for their room checks: one per key".input_port_inline6 of 6 on macOS (step 4abaf6fbccc02d57fa8c01bd). These are the new queue test, the new header test (the_inline_frame_header_is_the_encoders_header) and perf(cmux-tui): a key to an idle input port is written on its own turn, with no thread hand-off (#35413 A/B) #35872's four tests, which include the stalled-host test (1000 keys into a host that reads nothing). Linux: 5 of 5 (step 2a4b1da9b1a6502c24ab1d5b; the queue test is macOS only).surface::tests::hosted40 of 40 on macOS (step a08306a8069afa5a572d8403). Linux: 39 of 39 (step d9a428e849b4ddc077dd6a9c; the writer QoS test is macOS only).Observed / Not run
poll(POLLOUT, 0)375,kevent64(IMMEDIATE)on a registered queue 167,Frame::new+encode_frame42, a 33-bytesend1625. On an idle Mac this saves about 0.25 us per key. The same benchmark with 14 spinning threads and the measuring thread at user-interactive QoS (before the fix: step b9f9e486b09b70b51035d3b2; after: step fde1d207ce50ee8d1d22a4fc; different minis, so compare within a run): poll 416 ns, kevent64 208 ns at p50 in both runs. These benchmarks did not reproduce the A/B's 0.014 ms write-call difference: in the test process the lane-onwrite_bytes_replyingis not slower than the lane-off one at p50 in either run. A cost that the test process does not have (never-merge macOS bench, step 9e2ae6e0d493356babb1a379): a zero-timeoutpoll()costs more the more kernel event registrations the process holds, because XNU's poll builds and tears down a kernel queue against the process's file table on every call. p50 375 ns with no other registrations, 1083 ns with 1024, 3584 ns with 4096.kevent64(IMMEDIATE)on a queue registered once stays at 208 ns at every count. The daemon's own count, read from the live app daemon under the L6 load (see below), is 264-266, not thousands; the benches below are at that count. In the after run, the tight 20,000-key loop sent most keys through the port (lane-on p50 83 ns: a queue push, no syscall), because the writer outran the reader and the port then stays busy; the paced test above shows that keys with room go inline.lskqandproc_pidinfo; rounds s17hb-1-off at 02:44Z and s17hp-head-1 at 03:20Z, the same in both): 264-266 knotes. 13-14 are on fds (highest fd 67-79). 250-253 are hashed, and 234-237 of them areEVFILT_PROC, one per host child, so the count grows with the session's terminals. 1052 open fds (768 sockets), highest fd 1095. The chief daemon holds 29. The daemon under that load runs 390-560 threads, 365k-388k Unix calls and 142k-149k context switches a second, at about 180% CPU (s17hp-head-2 at 03:41Z).poll(POLLOUT, 0): 791 warm, 6500 cold (the caches swept before each call).kevent64(IMMEDIATE): 208 warm, 1750 cold. So at the daemon's count the old check cost +0.58 us warm and +4.75 us cold per key. A knote list grown to the highest fd changes nothing (792 ns): the hashed knotes set the cost. With 1000 hashed knotes: +1.96 us warm, +23.9 us cold, so the old cost grows with the session size. The same shape with paced socket andEVFILT_USERthreads at about the daemon's rates (398k Unix calls and 262k switches a second; step e7934de9e7d37a98dcf3461b): +1.58 us warm, +5.38 us cold; poll's p95 rose to 11.2 us against 1.5 us for kevent64.sendagainst a stack header andsendmsg) +2.75 / +2.6 us. Warm (200 us apart): +2.0 / +8.2 us. Idle and cold: +5.9 / +6.3 us.Performance
Counted on the live daemon: 264-266 knotes (250-253 hashed). At that count the bench gives +0.58 us warm and +4.75 us cold per key for the old check (step 4a530efa4a2276db2caa564e), about a third of the 12-13 us the write call lost. The whole old write against the new one, at that count and at about the daemon's call rates, with the caches cold: +8.0 us at p50 and +14.4 us at p95 (step 582752f33f1c775cf20b9cc3), about two thirds of what the A/B removed (12-13 / 20-22 us). The old room check alone is +5.3 / +12.5 us of that, and it grows with the hashed knotes (one per host child): +23.9 us cold at 1000. The rest has no measured cause.
Changelog
none (the input lane is on by default since #35413, which has the What's New highlight; this removes about 8-13 us per key on macOS, which no one can see)
Proof
No capture: daemon-only change with no UI. The typing A/B is reported on #35413.