Skip to content

remote-daemon: tear a PTY session down once - #8643

Merged
lawrencecchen merged 1 commit into
manaflow-ai:mainfrom
ejc3:fix/pty-teardown-idempotent
Jul 23, 2026
Merged

lawrencecchen merged 1 commit into
manaflow-ai:mainfrom
ejc3:fix/pty-teardown-idempotent

Conversation

@ejc3

@ejc3 ejc3 commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

TestTerminateProcessesRunsOnlyOnce has failed on main since it was added in #8438:

--- FAIL: TestTerminateProcessesRunsOnlyOnce
    ws_pty_test.go:809: process teardown ran 2 times, want exactly once
FAIL github.com/manaflow-ai/cmux/daemon/remote/cmd/cmuxd-remote

The test asserts that a PTY session is torn down once, but the guard it was written against never landed, so terminateProcessesWithForegroundGroupLookup redoes the foreground-group ioctl and re-sends both SIGKILLs on every call.

Two paths reach teardown for the same session. waitSessionProcess runs it after the session leader exits. The hub runs it when a client sends a close frame, when a non-persistent attachment's connection ends, on closeAll, and on an idle reap. Those four hub paths cannot collide with each other, because each one removes the session from the hub map under h.mu before it signals anything, but waitSessionProcess is outside that bookkeeping. A session that outlives its leader and is then closed runs teardown twice.

This wraps the body in a sync.Once, matching closeTTYOnce and closePTYOnce already on the struct. A wsPTYSession owns one exec.Cmd for its whole life and nothing reassigns it, and reconnecting to a live session only adds an attachment, so the guard covers exactly the one process group it is meant to. Once a session is closed the attach path builds a fresh wsPTYSession with its own sync.Once.

What a duplicate pass actually costs is a second pair of SIGKILLs and a second walk of the process table. terminatePTYSessionMembers reads every /proc/<pid>/stat on Linux and forks ps on macOS, and it rescans until the discovered member set stops growing. A second pass can also only run after cmd.Wait has returned, so the leader pid it signals has already been reaped and no longer belongs to the daemon.

The guard does change one thing. When a hub path tears the session down first, the sweep waitSessionProcess does after cmd.Wait becomes a no-op, and that sweep's own comment says it is there to catch a SIGHUP-shielded background job still holding the PTY slave. I do not believe anything is lost in practice. The hub pass runs the same rescanning member sweep, the leader is a member of its own session so it gets killed inside that sweep, and a process cannot fork once SIGKILL has been delivered to it, which is what makes the scan converge. TestWebSocketPTYCleanupTerminatesEverySessionProcessGroup already drives that exact case from all three hub paths with set -m; trap '' HUP; sleep 300 &, and with this guard in place it is the hub pass alone that has to clean up. I could not construct a member that the hub pass misses and the skipped pass would have caught, but it is one fewer sweep than before, so please check that reasoning rather than take it from me.

I left the stale-pid signalling alone. Signalling a pid the daemon has already reaped is wrong on the first pass too: waitSessionProcess calls terminateProcesses after cmd.Wait, so the leader is gone by then, and a sync.Once cannot suppress a first call. The window is narrow, because Linux keeps a pid allocated while any process still has it as a session id, so recycling requires the session to be empty and then Kill(-pid, SIGKILL) just returns ESRCH. Fixing it properly means capturing the member set before cmd.Wait instead of rederiving it from the leader pid afterwards, which is a bigger change than this test fix and belongs in its own PR.

I could not compile or run any of this locally, since the machine I wrote it on has no Go toolchain. The linux remote-daemon-tests job is the gate. An earlier workflow_dispatch run of this branch on my fork reported ok github.com/manaflow-ai/cmux/daemon/remote/cmd/cmuxd-remote 1.783s.

This failure also blocked CI that has nothing to do with the daemon. linux-preflight fails when a routed linux job with a true route fails, the macOS jobs gate on linux-preflight succeeding, and any non-pull_request event marks every route true. So while this test was red, every manually dispatched ci.yml run skipped the entire macOS matrix.

Summary by CodeRabbit

  • Bug Fixes
    • Improved PTY session shutdown reliability by ensuring termination actions run only once.
    • Prevented repeated shutdown attempts from causing duplicate process termination or inconsistent session cleanup.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 84163578-286d-47dc-b7e0-7b4163dc7fbf

📥 Commits

Reviewing files that changed from the base of the PR and between 184d61e and a79dfd6.

📒 Files selected for processing (1)
  • daemon/remote/cmd/cmuxd-remote/ws_pty.go

📝 Walkthrough

Walkthrough

wsPTYSession now uses sync.Once to ensure PTY process termination and foreground-group cleanup execute only once across repeated or concurrent teardown paths.

Changes

PTY termination lifecycle

Layer / File(s) Summary
Guard PTY process teardown
daemon/remote/cmd/cmuxd-remote/ws_pty.go
Adds a terminateOnce guard to wsPTYSession and wraps foreground-group lookup, session-member termination, and process-group killing in a one-time teardown closure.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • manaflow-ai/cmux#8438: Both changes modify PTY teardown and foreground-process-group termination in ws_pty.go.

Suggested reviewers: austinywang, lauzierj


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Cmux Cache Substitution Correctness ❓ Inconclusive placeholder Need to inspect the diff and cache-substitution rules to see if the change affects a cache vs fresh-read path.
✅ Passed checks (24 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: making PTY session teardown run only once.
Description check ✅ Passed Description covers the change, rationale, and testing, though it omits template sections like Demo Video, Review Trigger, and Checklist.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Swift Actor Isolation ✅ Passed PASS: The diff only changes a Go file (ws_pty.go); no Swift files or actor-isolation-sensitive Swift code are introduced.
Cmux Swift Blocking Runtime ✅ Passed Diff only changes daemon/remote/cmd/cmuxd-remote/ws_pty.go; no Swift files or Swift sync/runtime code are touched.
Cmux Browser Automation Off-Main ✅ Passed PR only changes daemon/remote/cmd/cmuxd-remote/ws_pty.go; no browser.* routing or WebKit/AppKit automation code is touched, so the rule doesn’t apply.
Cmux Expensive Synchronous Load ✅ Passed Not applicable: the PR only changes Go PTY teardown in daemon/remote/cmd/cmuxd-remote/ws_pty.go; no Swift/main-actor load paths were added.
Cmux No Hacky Sleeps ✅ Passed No sleeps, timers, polling loops, or wall-clock waits were added; the change only wraps teardown in sync.Once.
Cmux Algorithmic Complexity ✅ Passed Diff adds sync.Once around existing PTY teardown; it removes duplicate process-table rescans rather than introducing new scalable scans.
Cmux Swift Concurrency ✅ Passed The commit only changes a Go file; no Swift files or concurrency patterns are introduced, so the Swift-concurrency rule is not applicable.
Cmux Swift @Concurrent ✅ Passed PR diff touches only ws_pty.go; no Swift files or @concurrent annotations are involved.
Cmux Swift Package Boundaries ✅ Passed No Swift files were changed; the diff only touches ws_pty.go, so the Swift package-boundary rule is not applicable.
Cmux Swiftpm Lockfiles ✅ Passed HEAD only changes daemon/remote/cmd/cmuxd-remote/ws_pty.go; no .gitignore, Package.resolved, Xcode, or workflow files were touched.
Cmux Swift Logging ✅ Passed Diff changes only daemon/remote/cmd/cmuxd-remote/ws_pty.go; no Swift code or new Swift logging is involved.
Cmux User-Facing Error Privacy ✅ Passed Diff only adds sync.Once and internal comments in ws_pty.go; no user-facing errors, alerts, or sensitive payloads changed.
Cmux Full Internationalization ✅ Passed The commit only changes daemon/remote/cmd/cmuxd-remote/ws_pty.go, adding sync.Once and internal comments; no user-facing Swift/web/xcstrings/Info.plist text was touched.
Cmux Swiftui State Layout ✅ Passed Only daemon/remote/cmd/cmuxd-remote/ws_pty.go changed; no SwiftUI files or SwiftUI state/layout patterns are present.
Cmux Architecture Rethink ✅ Passed This PR only changes Go PTY teardown in ws_pty.go; no Swift files or referenced Swift architecture rule are involved.
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed Diff only changes daemon/remote/cmd/cmuxd-remote/ws_pty.go; no Swift window/controller code or cmuxAuxiliaryWindowIdentifiers changes are present.
Cmux Source Artifacts ✅ Passed Only ws_pty.go changed, and the diff is intentional Go source adding sync.Once guard logic, not an artifact or generated output.
Cmux No Test Or Debug Seam In Production Source ✅ Passed The diff only changes daemon/remote/cmd/cmuxd-remote/ws_pty.go; no Swift production Sources/ files were touched, so the rule is not applicable.
Cmux No Ambient Global State ✅ Passed Only an instance-owned sync.Once field was added on wsPTYSession; no new top-level func, var, or singleton appears in the diff.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

TestTerminateProcessesRunsOnlyOnce has failed since it landed in manaflow-ai#8438. The
test asserts teardown is idempotent, but the guard it was written against was
never added, so the foreground-group lookup and both SIGKILLs run again on
every call.

Two paths reach teardown for the same session. waitSessionProcess runs it once
the session leader exits, and the hub runs it on a client close frame, when a
non-persistent attachment's connection ends, on closeAll, and on an idle reap.
The hub paths exclude each other by removing the session from the hub map under
h.mu, but waitSessionProcess sits outside that bookkeeping, so a session that
outlives its leader and is then closed tears down twice.

A second pass can only run after cmd.Wait has returned, so the leader pid it
signals has already been reaped. It also repeats the member scan, which reads
every /proc/<pid>/stat on Linux and forks ps on macOS. Guard the body with a
sync.Once, matching closeTTYOnce and closePTYOnce on the same struct.
@ejc3
ejc3 force-pushed the fix/pty-teardown-idempotent branch from abe046e to a79dfd6 Compare July 22, 2026 09:10
@lawrencecchen
lawrencecchen marked this pull request as ready for review July 22, 2026 19:43
@lawrencecchen

Copy link
Copy Markdown
Contributor

@codex review

@coderabbitai review

Please review exact head a79dfd6cc6e634bf0ec7ab4e1f69abcb25015f6e, especially the sync.Once ownership argument and PTY process-group teardown behavior. The head passes full Go and Go race tests on GCP and has zero unresolved review threads.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

requested='a79dfd6cc6e634bf0ec7ab4e1f69abcb25015f6e'

echo '== checked-out commit =='
git rev-parse HEAD
echo '== requested commit exists =='
git rev-parse "${requested}^{commit}"

echo '== PR metadata =='
gh pr view 8643 --repo manaflow-ai/cmux --json number,headRefOid,baseRefOid,files --jq '{number, headRefOid, baseRefOid, files: [.files[].path]}'

echo '== diff against first parent =='
git diff --find-renames --unified=80 "${requested}^" "${requested}" -- \
  'daemon/remote/cmd/cmuxd-remote/*'

echo '== candidate source and test files =='
fd -t f -e go . daemon/remote/cmd/cmuxd-remote | sort

echo '== structural outline =='
ast-grep outline daemon/remote/cmd/cmuxd-remote --items all --type struct,function,method \
  --match 'wsPTYSession|terminateProcessesWithForegroundGroupLookup|waitSessionProcess|closeTTYOnce|closePTYOnce|terminatePTYSessionMembers'

echo '== teardown and lifecycle references =='
rg -n -C 5 \
  '\b(terminateProcessesWithForegroundGroupLookup|terminateProcesses|terminatePTYSessionMembers|waitSessionProcess|closeTTYOnce|closePTYOnce|wsPTYSession)\b' \
  daemon/remote/cmd/cmuxd-remote --glob '*.go'

Length of output: 52730


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '== session start / attach / reconnect lifecycle =='
sed -n '720,970p' daemon/remote/cmd/cmuxd-remote/ws_pty.go

echo '== hub teardown paths =='
sed -n '1080,1270p' daemon/remote/cmd/cmuxd-remote/ws_pty.go
sed -n '1489,1645p' daemon/remote/cmd/cmuxd-remote/ws_pty.go
sed -n '1848,1885p' daemon/remote/cmd/cmuxd-remote/ws_pty.go

echo '== POSIX process-session member enumeration =='
cat -n daemon/remote/cmd/cmuxd-remote/ws_pty_session_processes_linux.go
cat -n daemon/remote/cmd/cmuxd-remote/ws_pty_session_processes_darwin.go

echo '== relevant termination tests =='
sed -n '700,830p' daemon/remote/cmd/cmuxd-remote/ws_pty_test.go
rg -n -C 12 \
  'TestWebSocketPTYCleanupTerminatesEverySessionProcessGroup|shielded|trap.*HUP|terminateProcesses|waitSessionProcess|closeAll|reapIdleSession' \
  daemon/remote/cmd/cmuxd-remote --glob '*_test.go'

echo '== all wsPTYSession constructions and cmd assignments =='
rg -n -C 3 \
  'wsPTYSession\s*\{|\.cmd\s*=|cmd:\s*' \
  daemon/remote/cmd/cmuxd-remote --glob '*.go'

Length of output: 78863


@lawrencecchen Focused review of exact head a79dfd6cc6e634bf0ec7ab4e1f69abcb25015f6e found no issues.

The sync.Once ownership is correct: wsPTYSession receives its exec.Cmd at construction and production code does not replace it. Reattaching uses that same live session; after removal from the hub map, a later attach creates a new wsPTYSession and therefore a new guard.

The teardown reasoning also holds. All hub teardown paths and waitSessionProcess converge on terminateProcesses. Whichever path wins executes the full session-member sweep; the other becomes a no-op. The sweep is based on POSIX session ID on both Linux and macOS, so it covers the foreground and SIGHUP-shielded background process groups belonging to that PTY session. The Linux cleanup test covers hub-first explicit close, closeAll, and idle-reap cleanup with a shielded background job.

No findings from this focused review.

@greptile-apps

greptile-apps Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes PTY process teardown run once per session. The main changes are:

  • Adds a session-owned sync.Once guard.
  • Wraps foreground lookup, member scanning, and process signaling in the guard.
  • Prevents duplicate teardown from hub and process-wait paths.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
daemon/remote/cmd/cmuxd-remote/ws_pty.go Adds a per-session guard that prevents competing cleanup paths from repeating the complete process teardown.

Reviews (1): Last reviewed commit: "remote-daemon: tear a PTY session down o..." | Re-trigger Greptile

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: a79dfd6cc6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@austinywang austinywang mentioned this pull request Jul 23, 2026
6 of 8 tasks
@lawrencecchen
lawrencecchen merged commit 9f0613e into manaflow-ai:main Jul 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants