Skip to content

acpmux: wait for live daemon peers - #35174

Merged
teamleaderleo merged 18 commits into
feat-cmux-nextfrom
lane/cli-deadlines
Nov 4, 2026
Merged

teamleaderleo merged 18 commits into
feat-cmux-nextfrom
lane/cli-deadlines

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Local acpmux CLI calls now wait while the daemon peer is alive and fail promptly when the peer is gone. Default acpmux send/new prompt collection waits for the live peer; explicit --stall and --timeout remain caller choices. Daemon handover, lock, socket and startup paths use peer liveness, with only cleanup and a documented hung-child readiness fallback bounded.

Leo's words (2026-11-04): “shouldn't be this short. no reason for this. i imagine there's other nonsensical gates like this to remove in the broader app/acpmux”.

Gates audited

Location Old gate What it guarded Verdict
cmux-tui/crates/acpmux/src/cli/command.rs:249 --stall default 30s default prompt collection Removed default wall-clock failure; live peer waits; explicit flag preserved
cmux-tui/crates/acpmux/src/cli/command_new.rs:71 --stall default 30s new-session prompt collection Removed default wall-clock failure; explicit flag preserved
cmux-tui/crates/acpmux/src/daemon.rs:1255 lock handover 60s daemon takeover Removed; kernel lock release is liveness
cmux-tui/crates/acpmux/src/daemon.rs:1270,1275,1294 connect/status/shutdown 5s/5s/10s takeover peer calls Removed; closed sockets fail through transport, live peers may answer later
cmux-tui/crates/acpmux/src/daemon.rs:1393,1621 handover 20s start/stop handover Removed; lock/socket liveness controls retry
cmux-tui/crates/acpmux/src/daemon/running.rs:42 HANDOVER_BUDGET acpmux ls Removed; live lock holder is retried and a released lock fails promptly
cmux-tui/crates/acpmux/src/daemon/handover.rs:19 HANDOVER_BUDGET ready-fd start race Removed; retries until lock ownership or live winner answers
cmux-tui/crates/acpmux/src/daemon.rs:1432,1452 systemd probe 1s service probing Probe timeout falls through to the real launch path
cmux-tui/crates/acpmux/src/daemon.rs:1321 detached readiness 8s child that never reaches readiness Kept as a real hung-child hazard bound, documented beside the constant
cmux-tui/crates/acpmux/src/daemon.rs:1797, daemon/windows.rs:276 no winner bound after losing start hung lock winner Restored START_BUDGET; process/lock liveness still controls ordinary handover
cmux-tui/crates/acpmux/src/daemon/snapshot_clear.rs:12 connect could hang forever accepted socket that never initializes Kept 2s initialization grace so lock-protected offline cleanup can decide safely
cmux-tui/crates/acpmux/src/daemon.rs:1325 shutdown exit grace cleanup after stop/death Kept; cleanup bound, not a live-peer answer deadline

Regression coverage

  • daemon::running::tests::a_slow_but_alive_daemon_is_not_cut_off_by_the_old_budget waits through 120 failed probes before success.
  • daemon::running::tests::a_closed_socket_fails_fast_even_when_the_connect_future_is_slow covers closed/refused peer failure.
  • daemon::running::tests::no_daemon_is_an_error_at_once_and_nothing_is_started covers no live peer.
  • Fleet green at the final exact head: fast-checks run 41829007548, tested 6c626322732c64e03f9e5502b4ef577efaec387d; 5 passed.
  • Earlier fleet red runs exposed and were fixed: 41815398549 (duplicate lock helper/readiness type), 41828393439 (readiness error conversion).

Changelog

Fixed: local acpmux CLI and daemon handover calls wait for live peers instead of failing on arbitrary default wall-clock limits.

🤖 Generated with Codex

Agent session: agent session

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 96d1a32: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr35174 -f ref=lane/cli-deadlines -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 4, 2026
@blacksmith-sh

blacksmith-sh Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Found 6 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmpa26dtv9z/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpyliowc0n/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
@cursor

cursor Bot commented Nov 4, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review): BLOCK at 7a7c65d

BLOCK at 7a7c65d

Visual verdict: no images; Leo tries the build

  • [P1] [blocking] [category:reliability] [finding:F1] [outcome:pending] cmux-tui/crates/acpmux/src/daemon.rs:1759-1772 (and Windows daemon/windows.rs:255): detached readiness no longer has the documented 8-second hung-child bound. A child that remains alive but stalls before writing readiness leaves every caller blocked forever, contrary to the PR table’s claim that this hazard remains bounded.

  • [P1] [follow-up] [category:reliability] [finding:F2] [outcome:pending] cmux-tui/crates/acpmux/src/daemon/running.rs:40: connect_running_with now awaits Client::connect() without a timeout or concurrent lock check. A peer that accepts the socket but never answers initialization can hang acpmux ls forever; the previous handover budget returned daemon_not_serving.

  • [P1] [follow-up] [category:reliability] [finding:F3] [outcome:pending] cmux-tui/crates/acpmux/src/daemon/snapshot_clear.rs:12: removing the 2-second connection bound means a socket that accepts but does not answer prevents the function from reaching its lock-protected offline path, so snapshot cleanup can hang indefinitely despite the comment promising failure for an unresponsive writer.

Follow-ups:

  • F2: restore a liveness-aware bound or race connection/initialization against lock release.
  • F3: retain a finite connection/initialization grace before attempting the lock-protected fallback.

@teamleaderleo teamleaderleo added the leo-named-fix An obvious fix to something Leo named: lands on push, review runs in parallel label Nov 4, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review): BLOCK at bcfbb2d

BLOCK at bcfbb2d

Visual verdict: no images; Leo tries the build

    • [P1] [blocking] [category:correctness] [finding:F1] [outcome:pending] cmux-tui/crates/acpmux/src/daemon.rs:1727-1782 declares path only under #[cfg(target_os = "macos")], but the Unix startup function references path.display() unconditionally in the timeout error. Linux and other Unix builds therefore fail with an unresolved path identifier.

Follow-ups: none

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review): BLOCK at 116c0e7

BLOCK at 116c0e7

Visual verdict: unchanged by this diff.

  • [P1] [blocking] [category:reliability] [finding:F1] [outcome:pending] [daemon/snapshot_clear.rs:12] Removing the 2-second connect bound makes snapshot cleanup hang forever when a live daemon accepts the socket but never completes initialization, preventing sign-out/clear-peer-snapshots from returning.
  • [P1] [blocking] [category:reliability] [finding:F2] [outcome:pending] [daemon.rs:1797-1818, daemon/windows.rs:266-295] The macOS launchd and Windows lock-winner paths now wait indefinitely after a losing start exits. A live but hung winner that never binds its socket keeps the caller blocked forever, despite the retained START_BUDGET intended to bound hung startup.

Follow-ups: none

cmux-bot Bot and others added 2 commits November 4, 2026 12:04
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 4, 2026 12:04 Inactive
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at d7e6af0

BLOCK at d7e6af0

Visual verdict: no images; Leo tries the build.

  1. [P1] [blocking] [category:correctness] [finding:F1] [outcome:pending] daemon.rs:1793 returns read directly, but it is Result<String, std::io::Error> while the function returns anyhow::Result<String>. Rust does not implicitly convert the nested error type, so this change fails to compile; map the error (for example, Ok(read) => Ok(read?)).

Follow-ups: none

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review): BLOCK at 6c62632

BLOCK at 6c62632

Visual verdict: no images; Leo tries the build

  • [P1] [blocking] [category:security] [finding:F1] [outcome:pending] [daemon.rs:1991] Key handoff now runs on an unbounded blocking thread. If a live child never sends KEY_READY, hand_key waits forever and keeps the Zeroizing<String> person key resident indefinitely; repeated keyed starts can also accumulate permanently blocked threads. Retain a bounded handoff or otherwise abort and zeroize the key when readiness fails.

  • [P2] [follow-up] [category:reliability] [finding:F2] [outcome:pending] [daemon.rs:1479] A timed-out systemctl start is discarded and retried every 250 ms forever. Since cancelling Command::output() does not necessarily terminate the child process, a slow/unresponsive systemd can accumulate start processes while this loop never falls through to detached launch. Track/kill the timed-out child or bound retries before fallback.

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at 13a93b9

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 13a93b9

Visual verdict: no images; Leo tries the build

  • [P2] [follow-up] [category:reliability] [finding:F1] [outcome:pending] daemon/running.rs:41 and daemon/handover.rs:26 await Client::connect (including initialization) before rechecking lock liveness. A stale listener that accepts but never responds can therefore hang forever even after its lock is released; race the connection with periodic lock checks.

Follow-ups: [F1] Poll lock liveness while waiting for initialization so stale accepted sockets fail promptly.

cmux-bot[bot]
cmux-bot Bot previously approved these changes Nov 4, 2026

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 13a93b9: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@cursor

cursor Bot commented Nov 4, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 4, 2026 15:08 Inactive
@cmux-bot
cmux-bot Bot dismissed their stale review November 4, 2026 15:14

A newer verdict blocks 8580711 (no independent SHIP at the head).

teamleaderleo and others added 2 commits November 4, 2026 15:24
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-58861fa38207c3a8ed1a
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at 96d1a32

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 96d1a32

Visual verdict: no images; Leo tries the build

  • [P2] [follow-up] [category:reliability] [finding:F1] [outcome:pending] daemon.rs:1817 compares the lock holder PID with the launcher PID, but Linux’s systemd-run --scope --no-block path at daemon.rs:1926 makes those different processes. A hung daemon that acquired the lock before readiness therefore resets the 8-second timeout forever instead of returning the documented startup error.

  • [P2] [follow-up] [category:reliability] [finding:F2] [outcome:pending] daemon/handover.rs:19 now waits indefinitely on the requested socket. If a live daemon owns the home lock but serves a different ACPMUX_SOCKET, the losing starter never reaches start_and_connect’s holder_elsewhere check and hangs permanently.

Follow-ups:

  • [finding:F1] Fix readiness ownership detection for wrapped/systemd launches.
  • [finding:F2] Detect a live holder serving another socket during handover.

@teamleaderleo
teamleaderleo merged commit 29ea10f into feat-cmux-next Nov 4, 2026
160 of 161 checks passed
@teamleaderleo
teamleaderleo deleted the lane/cli-deadlines branch November 4, 2026 15:51
teamleaderleo added a commit that referenced this pull request Nov 4, 2026
teamleaderleo added a commit that referenced this pull request Nov 4, 2026
…wlist (#35601)

* ci: run the short-timeout ratchet on the tree, shrink its list, drop a new 2 s app gate

The ci-guards step ran only the checker's unit tests, never the checker, so two
new short hard timeouts landed after #35209 and nine fixed hits stayed listed.

- ci-guards runs scripts/ci/check_short_timeouts.py after its tests.
- AcpmuxSessionPage.summary waits AcpmuxEnvironment.answerDeadline, not 2 s,
  like its neighbours since #35075.
- agent.rs HOST_KIND_BUDGET carries its `// timeout:` reason: a late host
  falls back to its profile and is asked again, nothing fails.
- The allowlist drops the eight entries whose hits are gone and lowers
  host_load.rs to 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fb75b7af9902231028aa

* ci: lower daemon.rs in the short-timeout allowlist to 6 after #35174

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fb75b7af9902231028aa

* ci: select the short-timeout ratchet test for every path it scans

The ci-guards step sits in the `ci` matrix group, which workflow-guard-tests
excludes, so a step there never runs on these PRs. The linux-guard lane runs
tests/test_short_timeout_guard.py (whose first case scans the whole tree) only
when the focused map selects it, and nothing mapped the scanned source paths
to it. focused-tests-extra.json now maps the six scanned prefixes to the test;
the dead ci-guards edit is reverted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Agent-Session: https://cmux.com/deeplink/agent?id=aid-fb75b7af9902231028aa

This branch was previously deployed

3 inactive deployments
artifacts — 96d1a32c Deployed Nov 4, 2026 by teamleaderleo via publish tree-addressed cmux-tui #112242
sentry-dsyms — 96d1a32c Deployed Nov 4, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #112242
ci-routing-agent — 96d1a32c Deployed Nov 4, 2026 by teamleaderleo via cmux-tui tree preflight #112242
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

leo-named-fix An obvious fix to something Leo named: lands on push, review runs in parallel risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant