Skip to content

test: hold the daemon writer before the PTY attach deadline handler runs - #15921

Merged
lawrencecchen merged 2 commits into
mainfrom
fix-pty-attach-timeout
Sep 30, 2026
Merged

lawrencecchen merged 2 commits into
mainfrom
fix-pty-attach-timeout

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

RemoteDaemonRPCClient timeout isolation / a timed-out PTY attach does not wait for a blocked cancellation write has failed in about half of main's macos / swift-package-tests runs since 2026-09-29 07:43 PDT, always at line 182 (unexpectedTermination.wait(timeout: .now() + 10) returned .timedOut).

The test held the transport writer only after the fake daemon echoed a pty.data event, so the writer had to be held within the 1 s pty.attach deadline. That round trip (a shell sed fork, the stdout reader, the state queue, a hop to a global queue, and one more to the write queue) has no bound on a loaded runner. When it took longer than 1 s, the pty.attach.cancel write ran first on an idle writer, succeeded, and cancelled the write deadline, so the transport stop the test waits for never happened. Raising the final wait (2 s to 10 s in #15116) can't help, because nothing is left to stop the transport. Delaying the fake daemon's event by 1.2 s reproduces the CI failure locally with the same line and duration (11.36 s).

The test now holds the writer before running the handler a timed-out pty.attach invokes (sendPTYAttachCancellation), and keeps it held until the test ends. With that order fixed, only the product's 1 s cancellation-write deadline can stop the transport, and the wait budgets are failure bounds only. The test also no longer needs the 5 s safety timer that released the writer. The companion test in the same suite still drives a real pty.attach timeout end to end and checks that the cancellation reaches the daemon with the right request id and tokens.

Validation: swift test in Packages/macOS/CmuxRemoteDaemon passed 3 times in a row locally. The target test passed in 40 of 40 concurrent full-package runs with 60 CPU hogs on an 18-core Mac. The full ci.yml run on this branch is linked below.

Changelog

none

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Fixes the flaky timedOutPTYAttachBoundsCancellationWrite test by holding the transport writer before the attach deadline handler runs. The test previously held the writer only after a fake daemon echo, so on a loaded runner the echo could arrive after the 1 s attach deadline, letting the cancellation write complete and the expected transport stop never happen.

  • Holds the writer first, then invokes sendPTYAttachCancellation directly, keeping the writer held until the test ends.
  • Replaces the fake daemon with an idle transport that answers hello and then only drains input.
  • Removes the 5 s safety timer that no longer serves a purpose.
  • The companion test still covers timeout-to-cancellation wiring end to end.

Written for commit d7b51d6. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Tests
    • Expanded coverage for remote terminal connection cancellation, including cases where background communication is delayed.
    • Verified that cancellation can complete without waiting for a blocked write operation.
    • No user-facing feature or behavior changes are included in this update.

The blocked-cancellation test held the transport writer only after a fake
daemon event arrived, so the writer had to be held within the 1 s attach
RPC deadline. When that round trip took longer on a loaded runner, the
cancellation write ran first, succeeded, cancelled the write deadline, and
the expected transport stop never came (line 182 timed out after 10 s).

The test now holds the writer first and then runs the handler a timed-out
pty.attach invokes, keeping the writer held until the test ends. The
companion test still covers the timeout-to-cancellation wiring.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 68f6617e-fc75-4da8-aa7d-9f59d972b0f8

📥 Commits

Reviewing files that changed from the base of the PR and between 6d2b5d1 and d4b47a7.

📒 Files selected for processing (1)
  • Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.


📝 Walkthrough

Walkthrough

The timeout-isolation test now uses an idle transport and blocks the client write queue before invoking PTY-attach cancellation on a separate queue. It checks that cancellation returns while the write remains blocked. The test no longer checks the timed-out pty.attach call or its process-termination outcome.

Changes

PTY Attach Cancellation

Layer / File(s) Summary
Idle transport and cancellation check
Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift
The test uses an idle transport that replies to the initial handshake and drains later input. It blocks the client write queue and checks that sendPTYAttachCancellation returns on a separate queue.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Suggested reviewers: austinywang

Merge Risk: ⚪ Minimal · up to d4b47

The change isolates cancellation under a blocked writer while retaining transport-termination and attach-timeout coverage. No merge-blocking issue remains.

🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (24 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input ✅ Passed PASS: The PR changes only a timeout-isolation test. It adds an idle fake transport and blocks the existing daemon writer before sendPTYAttachCancellation; it does not change Cloud terminal creation,…
Cmux Swift Actor Isolation ✅ Passed PASS: The pull request changes only Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. The diff updates test setup and adds a fake idle tra…
Cmux Swift Blocking Runtime ✅ Passed PASS: The PR changes only Packages/macOS/CmuxRemoteDaemon/Tests/.../RemoteDaemonRPCClientTimeoutIsolationTests.swift. The added DispatchSemaphore waits and writer hold are deterministic test scaff…
Cmux Browser Automation Off-Main ✅ Passed The PR changes only Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. The diff changes PTY timeout test setup and adds an idle fake transp…
Cmux Expensive Synchronous Load ✅ Passed The pull request changes only Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. The diff updates test synchronization and adds an idle fak…
Cmux Cache Substitution Correctness ✅ Passed PASS: The pull request changes only Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. The diff updates test synchronization and adds a fak…
Cmux No Hacky Sleeps ✅ Passed PASS: The PR changes only Packages/macOS/CmuxRemoteDaemon/Tests/.../RemoteDaemonRPCClientTimeoutIsolationTests.swift, which is Swift test code. The custom check applies to non-Swift TypeScript, Java…
Cmux Algorithmic Complexity ✅ Passed PASS: The PR changes only Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. The added transport helper and queue coordination are test-onl…
Cmux Swift Concurrency ✅ Passed PASS: The PR changes only RemoteDaemonRPCClientTimeoutIsolationTests.swift. The added DispatchQueue and semaphore usage deliberately controls test interleaving around a blocked writer. The moderni…
Cmux Swift @Concurrent ✅ Passed The diff changes only a Swift test. The added makeIdleTransport and modified test methods are synchronous, with no nonisolated async, @concurrent, or actor-isolated declarations. The `DispatchQu…
Cmux Swift Package Boundaries ✅ Passed PASS: The pull request changes only Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. The diff updates a test and adds a fake transport te…
Cmux Swiftpm Lockfiles ✅ Passed PASS: The pull request changes only Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. It does not change a Package.swift, `Package.resol…
Cmux Swift Logging ✅ Passed PASS. The pull request changes only RemoteDaemonRPCClientTimeoutIsolationTests.swift, which is test code. The diff adds no Swift print, debugPrint, dump, NSLog, Logger declaration, diagnosti…
Cmux User-Facing Error Privacy ✅ Passed PASS: The review-scoped diff changes only Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. It modifies test synchronization and a fake tr…
Cmux Full Internationalization ✅ Passed PASS. The authoritative diff changes only Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. The changes are test setup, assertions, commen…
Cmux Swiftui State Layout ✅ Passed PASS: The pull request changes only RemoteDaemonRPCClientTimeoutIsolationTests.swift, a daemon timeout test. The diff adds no SwiftUI code and contains no ObservableObject, @Published, `@Observa…
Cmux Architecture Rethink ✅ Passed PASS: The diff changes only RemoteDaemonRPCClientTimeoutIsolationTests.swift. It adds a test-only DispatchSemaphore/serial-queue synchronization that blocks client.writeQueue before invoking the…
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed PASS. The PR changes only RemoteDaemonRPCClientTimeoutIsolationTests.swift, a test-only fixture. The diff adds no NSWindow, NSPanel, NSWindowController, SwiftUI Window/WindowGroup, or auxi…
Cmux Source Artifacts ✅ Passed PASS. The PR changes only the tracked Swift test source Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. The diff adds test logic and a r…
Cmux No Test Or Debug Seam In Production Source ✅ Passed The pull request changes exactly one file: Packages/macOS/CmuxRemoteDaemon/Tests/CmuxRemoteDaemonTests/RemoteDaemonRPCClientTimeoutIsolationTests.swift. The diff contains no Swift file under a produ…
Title check ✅ Passed The title clearly identifies the main change: holding the daemon writer before the PTY attach deadline handler runs. It is specific and concise.
Description check ✅ Passed The description explains the flaky test, the cause, the fix, and the validation results. It includes a changelog entry of none. The explicit Testing heading and checklist are omitted, and the linked C…
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

CI failure attribution

CI failed on d4b47a77b1 (run 36699125313 attempt 1): 1 code.

Job Verdict Why
macos / macOS compile admission code a compile error
Matched log lines
macos / macOS compile admission: /tmp/cmux-ci/src/Sources/TerminalSharingDisplay.swift:102:27: error: cannot find type 'TabPresence' in scope

Not re-run automatically: macos / macOS compile admission is not a machine failure.

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's.

@github-actions

Copy link
Copy Markdown
Contributor

Dogfood tours of d4b47a77

sidebar-and-chrome-tour at d4b47a77: not run

skipped: CI left no app build for this head (its compile failed or was cancelled)

Tours are picked by the paths globs in dogfood/scenarios/*.json; a Dogfood-tours: a, b line in the description picks them instead (none turns this off). Look at every frame before merging: a green tour only means no step failed.

@teamleaderleo

Copy link
Copy Markdown
Collaborator

Your package-test fix looks right and macos / swift-package-tests is passing here. The red that is holding this PR is not yours.

macos / macOS compile admission fails with cannot find type 'TabPresence' in scope and cannot find type 'BonsplitContrastPalette' in scope. Both types exist only in bonsplit 83857fa043b. This branch pins that commit, but pull request CI compiles the merge with main, and main's pointer went back to b32f48b9200 in #15747, so the gitlink merge takes main's side and the compile loses the types.

#15930 restores main's pointer. Once it lands, a rerun of macos / macOS compile admission here should clear this.

Separately, thank you for this one. The same test was failing on #14876, #14855 and #15229 at the same time and your diagnosis of the ordering is better than the one I had.

@austinywang

Copy link
Copy Markdown
Contributor

Same diagnosis as mine for the #15488 main full-suite repair: the product is right, and the test races its own setup. The writer block is queued from a .global() pty.data handler. It sometimes runs after the 1 s attach deadline, because .global() work stalls while RemoteDaemonRPCClientSocketForwardTests (added in #15116) hold every Swift Testing thread. The cancel write then succeeds, and onUnexpectedTermination never fires. It failed in 7 of 16 runs after #15116 and 0 of 14 before; raising the wait (2 → 5 → 10 s) can't pass that ordering.

Two gaps in this version:

  1. It calls sendPTYAttachCancellation directly, so it skips waitForCall. A regression that made the timeout path wait for the blocked cancel write would still pass both tests, since the sibling's writer is idle.
  2. The sibling test still takes its events on .global(). Its 5 s wait only hides the same starvation.

An alternative keeps the real path. Call through callIfIdle, whose admission hook runs inside the writeQueue section that writes the attach request. The blocker queued there runs after that request and ahead of the deadline's cancellation, however late any thread runs. Both tests also move to a private serial event queue:

_ = try client.callIfIdle(method: "pty.attach", params: [...], timeout: 1) {
    client.writeQueue.async {
        writeBlockEntered.signal()
        releaseWrite.wait()  // held until every expectation is checked; a 60 s safety timer is failure-only
    }
}

The file goes from 256 to 261 lines. I have the full diff and can push it here or open it separately, whichever you prefer.

@austinywang

Copy link
Copy Markdown
Contributor

This branch's red checks come from a 10:00 UTC run, when main didn't compile: the vendor/bonsplit pin had moved backwards, and #15930 fixed that. main has since removed automated PR catch-up (#15959), so the branch needs a manual main merge (scripts/merge-main.sh) to get a current run. The #15488 full-suite validation ran this change merged with the other fixes, and the RemoteDaemon suites passed there.

Catch-up merge by scripts/ci/catch_up_pr.py (RFC #14631).
Merged by scripts/merge-main.sh: origin/main at 086c8cb, the newest commit with green CI fast guards (1 newer skipped).
@austinywang

Copy link
Copy Markdown
Contributor

I merged main into this branch with scripts/merge-main.sh (d7b51d6, main at 086c8cb, the newest commit with green guards) so it gets a current CI run. The merge brings in only main's changes; your commit is untouched.

main still hits this failure: swift-package-tests in run 36741080703 failed at RemoteDaemonRPCClientTimeoutIsolationTests.swift:182 with .timedOut after 11.2 s. I checked the new order against the client: sendPTYAttachCancellation only enqueues the write and arms the 1 s timer, and the keepalive probe arms its own deadline only after it reserves an idle writer. So with the writer held, only the cancellation deadline can stop the transport. I plan to merge this once the run is green, as part of #15488.

@austinywang

Copy link
Copy Markdown
Contributor

Pre-merge review (correctness first, subagent): approve.

  • With the writer held from the start, only the 1 s cancellation-write deadline can stop the transport. The keepalive's callIfIdle needs writeQueue.sync (RemoteDaemonRPCClient+RPC.swift:318), so it can neither write nor arm its watchdog, and the start-up hello finishes before the hold.
  • A synchronous unbounded cancellation write fails the 5 s cancellationReturned check, and a cancellation write with no deadline fails the 10 s termination wait.
  • The companion test still checks that a timed-out attach sends pty.attach.cancel with the matching request id, session, attachment and token.
  • Non-blocking gap: neither test now times out a real pty.attach while the writer is blocked. Replacing the call at +RPC.swift:354 with a synchronous notify would pass both. The old test caught that through cleanupFired. call() writes its request with writeQueue.sync, so there is no reliable way to block the writer between the request and the deadline. Splitting the check across two tests is a reasonable trade, and I'm not asking for a change.
  • Fixed in the description: the ## Changelog line now reads none, the form CONTRIBUTING.md asks for.

This run's one red check is swift-package-tests, in a different package: the CmuxRemoteWorkspace test process died on SIGPIPE (signal 13), unrelated to this change. I'm tracking that separately under #15488.

@lawrencecchen
lawrencecchen merged commit a8ac4a7 into main Sep 30, 2026
53 of 56 checks passed
@lawrencecchen
lawrencecchen deleted the fix-pty-attach-timeout branch September 30, 2026 19:23
@github-actions

Copy link
Copy Markdown
Contributor

Merge receipt for d7b51d6ea9, merged 2026-09-30 19:23:38 UTC

  • Not verified at merge: ci-status (not reported), macOS compile admission (in progress), swift-package-tests (failure)
  • Verified: CI fast guards, detect-ios-changes, Fast static checks, GhosttyKit release check, guards (7), ios-tests, linux-preflight, macOS admission gate, package-conventions-lint, runner, Testbox broker trust boundary, Web complexity, and 1 more
  • Skipped by policy: admission-placement, browser, Claude wrapper regressions, Dogfood build #​${{ github.event.pull_request.number }}, ios-simulator, ios-simulator-build, mobile-core-package, remote-daemon, suite-coverage, ui-tests, web, web-build, and 2 more
  • Full suite: runs on main after merge.

Labeled merged-unverified: if main breaks near this merge, look here first.

@github-actions github-actions Bot added the merged-unverified A judging check was not green at merge; see the merge receipt comment label Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merged-unverified A judging check was not green at merge; see the merge receipt comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants