[teamleaderleo]

A missing Mac reply can leave the workspace list, pairing, chat initialization, or terminal input waiting indefinitely. Terminal input also paused new writes once one reply was 500 ms old, even on a healthy high-latency path. This change bounds requests, keeps ordered terminal writes pipelined, and bounds terminal event and chat replay buffers.

Design: design/ios.md, items B21 and B23.

Leo's request:

well i just think we can improve it a f***ton more than it is now... it's slow it's bad it belabours the point in too many tooltips it's ugly in places it doesn't even work in places there's lots of functionality that needs to get in

Daemon requests, batches, and submitted replies have 15-second deadlines; ordinary chat RPCs have 30 seconds. Prompts can include human approval and keep a finite 24-hour turn budget. Pairing has a 15-second handshake budget and a separate 180-second human-approval budget. The eager workspace setup and version response include that authorization budget because the Mac deliberately holds them until approval. Expiry reaches the existing error/retry paths. Late replies are discarded and mutations are not automatically retried.

Terminal input keeps its ordered 64-request window without an RTT-dependent pause. Uncertain writes are never resent. A byte-bounded daemon event queue stops receiving when the terminal falls behind, allowing the existing link flow control to reach the sender. Chat replay retains at most 2,000 events; after overflow, new readers fetch a complete history snapshot while existing readers continue live.

Validation

Draft pending completed fleet tests, the paired benchmark, and independent review at 1c8696ca9805fee0244e4ab49ae4375cece2ad24. No local build or test suite ran. git diff --check passes. This changes transport behavior and error propagation; there are no layout or styling changes.

Full kit run: https://github.com/manaflow-ai/cmux/actions/runs/41168331197 (pending). Requested: full CmuxMobileKit tests on macOS and iOS simulator, plus repeated focused runs of AcpmuxReplayResyncTests|LinkRequestDeadlineTests|FeedRefreshTests. Compare any failure by test name and error against a real base run. Earlier validation exposed a compiler closure-lifetime hang, a duplicate test helper, and fixture scheduling/lifetime failures; those have code changes but still require passing receipts at this head.

The replay fixture retains both asynchronous owners through live delivery and waits for both readers. The approval fixture replies 1.1 seconds after entering its approval phase, beyond the 1-second handshake budget and within the separate 10-second approval budget. The base integration preserves closed-channel handling for every replay generation and the existing acpmux timeout error code.

Benchmark: run TerminalInputThroughputTests/keystrokeThroughputAt500msRTT at this head and at base 5587de5f8379ceae40f34edc9a160a438e3dc13d. Copy only the unchanged head benchmark file into the base test target; its FakeTerminalDaemon fixture and tailProbeDelay already exist on base. Record both TERMINAL_INPUT_PERF lines: clean path and first reply dropped. The performance table will contain the measured values and run links before landing.

The change is about 1,500 lines including regression fixtures and the benchmark. Request expiry, input recovery, and event pressure share the same daemon actor and lifecycle, so their implementation and tests stay together; the small replay-cache change completes the related bounded-stream fix.

Performance

Metric	Before	After	Delta	Harness or run link
Terminal keystroke throughput at 500 ms RTT, clean	Pending fleet measurement	Pending fleet measurement	Pending	TerminalInputThroughputTests/keystrokeThroughputAt500msRTT, same harness on base and head
Terminal keystroke throughput at 500 ms RTT, first reply lost	Pending fleet measurement	Pending fleet measurement	Pending	Same harness and RTT; one deliberately omitted reply

Risk and cleanup

A slow request now fails instead of waiting indefinitely. Under sustained terminal overload, the existing link stall budget may close that channel and trigger its normal recovery. A new chat reader may make another history request after its replay budget is exhausted. No heartbeat or foreground reconnect policy changes.

Deadline tasks stop on response, failure, cancellation or channel closure; the losing operation is cancelled. Abandoned pairing/version channels and timed-out writes are closed, and late-opened channels are disposed. The bounded queue wakes blocked producers and consumers when closed; cancelled chat readers unregister. Benchmark ACK tasks end after one delayed reply or are cancelled when the synthetic channel closes. No animation loop or observer is added.

Related work: #26500 implements server-event gap refill and pagination; this change bounds the local replay cache and preserves that separate fix. Overlap awareness has been requested. pr-overlap was run but its GitHub read was refused by the broker; open and merged PR searches found no duplicate of these request-deadline changes.

Agent session: agent session
🤖 Generated with Claude Code

[blacksmith-sh]

Found 3 test failures on Blacksmith runners:

Failures

Test	View Logs
nightly cmux-tui resolve/nightly cmux-tui resolve	View Logs
nightly.yml must not read vars.NIGHTLY_SIGN while nightly-sign.yml is in shadow mode/ nightly.yml must not read vars.NIGHTLY_SIGN while nightly-sign.yml is in shadow mode	View Logs
the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac k/ the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac k	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Bead cx-yvpcnj created for this PR by the cmux coordinator.

[teamleaderleo]

Receipts at a50755a1cd56faaaa099ec94969c878139f87cb9 (confirmed with git ls-remote origin refs/pull/27323/head): the macOS kit build at this head doesn't finish, so there are no test results and no bench numbers.

What hangs: swift build --package-path cmux-next-ios/CmuxMobileKit --build-tests (Xcode 26.6, Swift 6, debug), with the FFI xcframework built first (exit 0). Every target compiled up to [684/690]. Then one swift-frontend job, the CmuxMobileTransport primary file Sources/CmuxMobileTransport/LinkDeadline.swift, ran for 90 minutes at 100% CPU and never finished. That's 46 minutes of CPU time, against about 4 minutes for the rest of the package. A sample of that process shows it inside the mandatory SIL diagnostic pipeline (runSILDiagnosticPasses → insertDeallocOfCapturedArguments / DeadEndBlocksAnalysis / DominanceAnalysis), so this is a compiler hang in LinkDeadline.run, not host load. The likely trigger is the nested escaping closures: Task {} inside withCheckedThrowingContinuation inside withTaskCancellationHandler, with Task { await onDiscard(value) } capturing a generic Value.

Independent corroboration: this PR's own cmux-next-ios kit tests / CmuxMobileKit tests job at this head (run 40842151392) stalls the same way. Its last line is [677/691] Emitting module CmuxMobileOnboardingTests at 08:44:18Z, there's no output after that, and the job is cancelled at 08:55:58Z in swift test (macOS). That run shows as "cancelled", not as a base red. The kit workflow has no run at the base d6623d8c, because of path filters.

Fleet: step 7339da19dcacc5961e432a75 (simulator compile plus the full kit suite) was submitted at 08:30Z and queued until 10:30Z. It's still running, with no result yet. I'll add a follow-up only if it reports.

Filtered suites (DaemonClientTests, DaemonBackpressureTests, LinkDeadlineTests, LinkRequestDeadlineTests, DaemonTerminalSessionTests, AcpmuxReplayTests, AcpmuxReplayResyncTests, PairInviteTests, MacPinTests, PushPairingTests, RemoveThisPhoneUnpairTests): not run, because the test bundle never linked. That's 0 passed and 0 failed, not a pass.

Performance

metric	before	after	delta	harness/run
TERMINAL_INPUT_PERF 500 ms RTT, clean	no number (build hangs)	no number (build hangs)	n/a	TerminalInputThroughputTests/keystrokeThroughputAt500msRTT, 3 runs per variant planned, 0 completed
TERMINAL_INPUT_PERF 500 ms RTT, dropped first reply	no number (build hangs)	no number (build hangs)	n/a	same

"Before" means this head with only DaemonTerminalSession.swift and DaemonTerminalSessionTests.swift restored from d6623d8c. That variant keeps this head's LinkDeadline.swift, so it hits the same hang. No numbers were recorded, and none are shown.

To unblock: split LinkDeadline.run so that no Task {} closure capturing the generic Value is nested inside both the continuation closure and the cancellation-handler closure. For example, move the worker and timer creation into a LinkDeadlineRace method, and spawn the discard task from a non-generic helper. Then confirm that swift build --build-tests for the kit finishes before re-running these receipts.

Agent session: agent session

[teamleaderleo]

The typing benchmark now has a fleet route: scripts/measure/swift-test-bench.sh (#27352, #27395) runs one swift test filter by itself as a measurement step. It has a source= argument for this head and restore_from=/restore= arguments for the two-file base run.

There are no TERMINAL_INPUT_PERF numbers yet. At 9b7735f196e47cf4c10f55b6006857a8a4be52dd, swift build --build-tests of cmux-next-ios/CmuxMobileKit hangs in the compiler right after Emitting module CmuxMobileOnboardingTests (about [686/690]). It never reaches the test run:

• fleet step 6a56dc28fd5a9e335526e512: Xcode 26.6, silent for 45 min after [686/690] Emitting module CmuxMobileOnboardingTests, then stopped
• fleet step 03766723fae500ea84eb8dce: the same, on another host
• this PR's own CmuxMobileKit tests job: swift test (macOS) stops after [681/691] Emitting module CmuxMobileOnboardingTests at 09:26Z and is cancelled at 10:21Z

On base, feat-cmux-next builds and runs the macOS kit tests in about a minute (run 40866990807: 1289 tests passed). So the hang comes from this head. The remaining targets are most likely the CmuxMobileLive tests this PR changes. Once the head compiles, both runs are:

cmux-ci run --class exclusive --script scripts/measure/swift-test-bench.sh --ref <feat-cmux-next tip> \
  --arg cmux-next-ios/CmuxMobileKit --arg TerminalInputThroughputTests/keystrokeThroughputAt500msRTT --arg source=<head> \
  [--arg restore_from=d6623d8cb536956dd96c2d7dbdf84c9d5ac7ac89 \
   --arg restore=Sources/CmuxMobileLive/DaemonTerminalSession.swift,Tests/CmuxMobileLiveTests/DaemonTerminalSessionTests.swift] \
  --env CMUX_CI_XCODE_APP=/Applications/Xcode_26.6.app --artifact artifacts/swift-test-bench/output.txt

[teamleaderleo]

Fleet diagnosis: head 9b7735f196e47cf4c10f55b6006857a8a4be52dd. The iOS simulator build hangs in the Swift compiler while it compiles the module CmuxMobileTransport. The build system isn't deadlocked, and no test is waiting: the build never reaches the tests.

Command: cmux-ci run --class simulator --target-worker <a third worker> --script scripts/ci/cmux-next-ios-sim.sh --ref 9b7735f196e47cf4c10f55b6006857a8a4be52dd --arg build. This was the build-only mode, on a different Mac from the two earlier runs that hung. I read the step's live build.log and the process list on the worker. The step is fleet step 9dede7d3273df5429f20a3e6, and I cancelled it once the hang was clear.

• Step: the FFI xcframework and xcodegen finish. xcodebuild … -configuration Demo -destination 'generic/platform=iOS Simulator' build then stops making progress about 4 minutes in.
• Last compile unit: build.log stops at CompileSwift normal arm64 (in target 'CmuxMobileTransport' from project 'CmuxMobileKit') and stops growing. One swift-frontend -frontend -c … -module-name CmuxMobileTransport -O (whole module) sat at ~98% CPU for 16 min 49 s, with no other compile running for this step.
• Where the compiler is stuck: a stack sample of that swift-frontend shows the main thread inside the mandatory SIL pipeline. runSILDiagnosticPasses → ClosureLifetimeFixup::run → fixupClosureLifetimes → tryExtendLifetimeToLastUse → insertDeallocOfCapturedArguments → computeDominatedBoundaryBlocks. The pass that fixes the lifetimes of closures converted from escaping to non-escaping is looping.
• Same failure on the earlier runs: the two earlier steps hung at the same point. On 9144759e the build went past 70 min; 0caecec8 hit the 90 min timeout. Other PRs build in 5–20 min on the same hosts. The PR's own "CmuxMobileKit tests (iOS simulator)" job was cancelled too.

Suspects (not narrowed to one function yet). This PR's changes to CmuxMobileTransport against its merge base d6623d8c are BoundedEventStream.swift (+142, new), LinkDeadline.swift (+88, new) and DaemonClient.swift (+112/−61). They add withTaskCancellationHandler wrapping withCheckedThrowingContinuation: twice in BoundedEventStream, once in LinkDeadline, once in DaemonClient. A Task { [weak self] in … } is also stored into deadlines[id]. Those nested cancellation-handler and continuation closures are what this pass rewrites. Splitting such a closure out into a separate function usually gets past the pass. Removing one of the three files from the target and building again would name the culprit.

Agent session: agent session

[teamleaderleo]

Published compiler-repair head cff151071b98c6191b3f8d7809bcd8d82c2f8d45. GitHub now reports the PR mergeable; it remains draft.

The previous compiler hang is addressed by splitting nested task/cancellation/continuation closures in LinkDeadline, BoundedEventStream and DaemonClient into named methods. Base conflict resolution preserves ordered workspace replies/events and chat gap refill. Added a regression combining ordered wire positions with an expired batch reply. git diff --check passes; compile success is not yet established.

The new full kit run is 40941478331. Fleet validation and the controlled baseline typing benchmark still need completed receipts; exact commands and baseline revision are in the body. The validation dispatch message failed because the relay could not read its session directory; a direct broker retry is pending. Keep this draft until the compile/tests, benchmark and independent review at the new head complete.

[teamleaderleo]

Fleet iOS evidence at 3f924ee (base c00c321): blocked on a red of its own

Compile: PASS, no compiler hang. The simulator build (FFI, xcodegen, Demo app) finished in 334 s (step 32ea1280) and 255 s (step b5386bd4). The ClosureLifetimeFixup loop in CmuxMobileTransport is gone at this head.

Full CmuxMobileKit run (step 32ea1280, iPhone 17 Pro, iOS 26.5, Xcode 26.6): 1533 passed, 2 failed, 1 skipped. Nothing crashed.

• CmuxMobileLiveTests/AcpmuxReplayResyncTests/newReaderResyncsWithoutInterruptingTheExistingReader(), AcpmuxReplayTests.swift:82: Expectation failed: (ChatOpenSnapshotLiveTests.userMessages(first.events) → []).contains("live")
• CmuxMobileLiveTests/LinkRequestDeadlineTests/approvalCanFinishAfterTheHandshakeDeadline(), LinkRequestDeadlineTests.swift:70: Expectation failed: await task.value == .paired(macDevice: "test-mac")

Focused rerun of those two suites (step b5386bd4): 8 passed, 1 failed.

• AcpmuxReplayResyncTests.newReaderResyncsWithoutInterruptingTheExistingReader() failed again, with the same assertion at :82.
• LinkRequestDeadlineTests passed this time.

Head and base: this PR adds both suites. At base c00c321, AcpmuxReplayTests.swift has no AcpmuxReplayResyncTests, and LinkRequestDeadlineTests.swift doesn't exist, so neither can be a base red.

Blocker: AcpmuxReplayResyncTests.newReaderResyncsWithoutInterruptingTheExistingReader() fails 2 of 2 at head, and it is this PR's own test. The existing reader's events never contain the "live" user message. approvalCanFinishAfterTheHandshakeDeadline() failed once and passed once; it is a new test too, so its timing needs to hold up under load.

Not run: the exact-head review and the TerminalInputThroughputTests/keystrokeThroughputAt500msRTT benchmark (scripts/measure/swift-test-bench.sh). Both wait for a head that fixes the red, since numbers at this head would be superseded. Push a new head and the same focused run, the benchmark at head and base, and a review follow.

[teamleaderleo]

Fleet run at head b8079db (simulator class, scripts/ci/cmux-next-ios-sim.sh test).

Result: BLOCK, the iOS build fails to compile, so no test ran.

Step	Scope	Outcome
9564437fbfca842689612948	targeted: AcpmuxReplayTests, AcpmuxReplayResyncTests, DaemonTerminalSessionTests, ReattachSurfacesTests, TerminalRenumberTests, LinkRequestDeadlineTests, TerminalInputThroughputTests, DaemonBackpressureTests, DaemonClientTests, LinkDeadlineTests	BUILD FAILED, 0 tests run (0 passed / 0 failed)
0d82aad56468e646468de224	full CmuxMobileKit	cancelled: same module would fail to compile identically

Compile errors (own, in a file this PR changes: it turned sources into [String: [WeakSource]] but two readers still use the old single-value shape):

• Sources/CmuxMobileLive/AcpmuxHub.swift:120:54: value of type '[AcpmuxHub.WeakSource]' has no member 'source'
• Sources/CmuxMobileLive/AcpmuxHub.swift:153:49: value of type '[AcpmuxHub.WeakSource]' has no member 'source'

These are not base failures: the old dictionary held a single WeakSource, so the line only exists broken after this diff. No crashes. No step queued for a display worker (gui-session-v1) over 30 minutes. Fix those two sites (e.g. compactMap(\.source) as in the other changed call sites), push, and ask for a rerun.

[teamleaderleo]

The two compile errors reported on b8079db are fixed in the published head 1c8696ca9805fee0244e4ab49ae4375cece2ad24. Both connection-closure paths now iterate liveSources, so they notify every active source in the session's array.

Requested a fresh fleet iOS compile and full CmuxMobileKit run at this head, including AcpmuxReplayResyncTests|LinkRequestDeadlineTests|FeedRefreshTests. The previous failed/cancelled steps did not run tests and do not validate this head. Automatic kit run 41168331197 is queued. The PR remains draft pending completed validation, paired throughput measurements, and independent review at the final head.