Conversation
cmux's `workspace list` is not read-your-writes against `new-workspace`: a list issued right after a successful create can be served a snapshot taken before the new workspace was published, so the adapter's duplicate-check-then-create-then-resolve sequence aborted every spawn with "could not resolve a cmux workspace id ... after creation". Give both post-creation reads a bounded retry window through a new fm_backend_cmux_settle helper. The bound is explicit, and an exhausted bound keeps the existing loud refusal rather than becoming a silent success or an unbounded wait. The duplicate check stays, because cmux enforces no title uniqueness of its own. Covered by a portable regression over the fake CLI and a live guard that drives the real app and names cmux and its version on failure.
…dupe settle bound
… workspace resolve
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Launching a worker through the cmux backend fails every time, so nothing can be dispatched on
cmux in this home. Fix it, and get the fix reviewed and merged before anything else runs.
The failure, reproduced four times from
bin/fm-spawn.sh:error: could not resolve a cmux workspace id for
'fm-firstmate-e2cba94a-wf-wfsmoke-orient' after creation
It was traced to
bin/backends/cmux.sh,fm_backend_cmux_create_task. That function makesthree cmux calls in order: a duplicate-check
workspace list, thennew-workspace, then asecond
workspace listto resolve the new workspace's id. The third call is served a stalesnapshot that does not yet contain the workspace just created, so the function aborts.
Reproduced outside firstmate:
query -> EMPTY (the duplicate check)
create -> OK
query -> EMPTY (stale; this is the read that aborts the spawn)
sleep 1; query -> 7E517926-4039-48D3-BAA5-7EFF5B2ECE73 (correct)
Dropping the leading duplicate-check query makes the immediate post-create resolve return the
right id, which is why the same two calls pass in a standalone test and fail inside the adapter.
Two knock-on effects were observed and should not be made worse. Each failed attempt left an
orphan cmux workspace carrying the crewmate's title, and the next attempt's duplicate check then
refused with
already exists. An earlier interrupted spawn had left such a workspace behind anda later agent session ended up running inside it, so the duplicate check named a workspace that
was in fact a live session.
Boundary stated: fix the cmux launch fault. Nothing else about cmux is being asked for.
What Changed
fm_backend_cmux_create_tasknow resolves a freshly created workspace's id through a newfm_backend_cmux_workspace_id_settledhelper, which re-readsworkspace listover a bounded retry window (FM_BACKEND_CMUX_SETTLE_TRIES, default 40;FM_BACKEND_CMUX_SETTLE_DELAY, default 0.25s, clamped byfm_backend_cmux_settle_tries) instead of trusting a single post-create read that cmux can serve from a snapshot predating the create. An exhausted bound keeps the original loud refusal, now naming the attempt count it spent; the pre-create duplicate check and the un-retriedlist-panessurface read are unchanged.tests/fm-backend-cmux.test.shdriving the fake CLI through a stale post-create list (asserting the retry resolves and that no second workspace is created) and through a never-resolving list (asserting the refusal and the retry bound), plus a live guardtests/fm-cmux-workspace-settle-live-e2e.test.shthat runs realcreate_taskover the duplicate-check-then-create sequence, gated viafm_live_gate/FM_CMUX_WORKSPACE_SETTLE_LIVEand closing only thefm-test-workspaces it creates; the new live test is registered inbin/fm-test-run.sh's live-e2e family.docs/cmux-backend.mdas adapter finding feat: add scout task lifecycle #7 and indocs/verification/runtime-backends.mdwith the dated cmux 0.64.25 / macOS aarch64 evidence, the observed query/create/query trace, and the command that refreshes the entry.Risk Assessment
✅ Low: The bounded lookup retry addresses the reported stale-read sequence while preserving duplicate refusal, single creation, and existing identity checks; no new material issues were substantiated.
Testing
Drove the cmux launch path against the real cmux 0.64.25 app on this host. The reported fault reproduced 4/4 times on the base-commit adapter with the intent's exact error message and left the described orphan workspace each time; the fixed adapter passed the identical sequence 4/4, and the repo's new default-on live guard passed all five rounds through bin/fm-test-run.sh. The end-user entrypoint bin/fm-spawn.sh --backend cmux launched a worker in an isolated firstmate home, its workspace was live in the cmux app under its home-scoped title, its launch output was capturable from the surface, and bin/fm-teardown.sh removed it cleanly. Adversarially, reducing the settle window to one attempt reproduced the original failure at the real entrypoint with the bounded, loud refusal preserved, and the duplicate check still refused a repeat launch of a live label. The portable regression added in this change was exercised only against the suite's fake CLI — it fails on the pre-fix adapter and passes on the fixed one there, but that is not a drive of the real cmux app, so it is reported as untested rather than a live pass. No visual artifact is attached because the change alters a shell adapter's CLI call sequence rather than any rendered surface; the reviewer-visible surfaces here are the fm-spawn transcript and cmux's own workspace listing, both captured as text, and a full-screen macOS capture would have published unrelated operator screen content. Every workspace created during testing was closed through tests/cmux-test-safety.sh's guarded close and the app ended in exactly its starting state.
spawned ... window=<ws>:<surface>, the workspace appears in the live cmux list under…create_task('fm-test-cmuxspawn-7515') -> rc=1 error: cmux workspace '...' already existsEvidence: Pre-fix adapter reproduces the spawn refusal 4/4 against real cmux
Source: Pre-fix adapter reproduces the spawn refusal 4/4 against real cmux
round 1: FAIL error: could not resolve a cmux workspace id for 'fm-firstmate-85fb1818-test-prefix-3432-1' after creation round 2: FAIL ... round 3: FAIL ... round 4: FAIL summary: 4/4 rounds failedEvidence: Fixed adapter: same sequence, 4/4 resolve the new workspace/surface
Source: Fixed adapter: same sequence, 4/4 resolve the new workspace/surface
round 1: PASS create_task -> 5DAC0CE0-... 03C6B3F5-... round 2..4: PASS summary: 0/4 rounds failedEvidence: End-user launch: bin/fm-spawn.sh --backend cmux, live workspace, duplicate refusal, clean teardown
Source: End-user launch: bin/fm-spawn.sh --backend cmux, live workspace, duplicate refusal, clean teardown
spawned test-cmuxspawn-7515 harness=sh kind=ship mode=no-mistakes yolo=off window=8C5A52E5-...:F8F4473E-... worktree=~/.treehouse/scratch-project-c3223c/1/scratch-project workspace 8C5A52E5-... title fm-firstmate-f234c421-test-cmuxspawn-7515 <capture of the worker surface> cmux-spawn-live-ok create_task('fm-test-cmuxspawn-7515') -> rc=1 error: cmux workspace 'fm-firstmate-f234c421-test-cmuxspawn-7515' already exists teardown test-cmuxspawn-7515 complete ... --- workspaces carrying this task's title after teardown --- <none>Evidence: Adversarial: settle window collapsed to one attempt reproduces the fault at the real entrypoint
Source: Adversarial: settle window collapsed to one attempt reproduces the fault at the real entrypoint
error: could not resolve a cmux workspace id for 'fm-firstmate-f234c421-test-cmuxspawn-tries1-12371' after creation (gave up after 1 attempts) --- exit: 1 --- workspace 0BBC1E75-... title fm-firstmate-f234c421-test-cmuxspawn-tries1-12371 (the orphan the intent describes)Evidence: Orphan from the failed launch closed; cmux app back to its starting state
Source: Orphan from the failed launch closed; cmux app back to its starting state
orphan left by the failed spawn: 0BBC1E75-... (fm-firstmate-f234c421-test-cmuxspawn-tries1-12371) guarded close: done --- workspaces now --- 638CC0CB-44C2-4A89-ABC6-FA7583837ADF wayfinder · wayfinder-smokeEvidence: Live settle guard through the real runner (5 rounds, real cmux)
Source: Live settle guard through the real runner (5 rounds, real cmux)
ok - real cmux (cmux 0.64.25 (106) [b685a275c]): create_task resolved its new workspace and surface on all 5 rounds FM_TEST_END ... exit=0 duration_ms=76447 gate_skip=falseEvidence: New portable regression fails against the pre-fix adapter (fake-CLI suite, red before / green after)
Source: New portable regression fails against the pre-fix adapter (fake-CLI suite, red before / green after)
not ok - create_task should retry a stale post-create workspace list, got '' (pre-fix adapter; passes on the fixed adapter)Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
spawned ... window=<ws>:<surface>, the workspace appears in the live cmux list under…create_task('fm-test-cmuxspawn-7515') -> rc=1 error: cmux workspace '...' already existsbash tests/fm-backend-cmux.test.sh(portable cmux suite incl. the two new settle regressions) — 63 ok, 0 not okSame suite run against a temp root whose bin/backends/cmux.sh is the base-commit (pre-fix) adapter —not ok - create_task should retry a stale post-create workspace list(red-before/green-after)bin/fm-test-run.sh tests/fm-cmux-workspace-settle-live-e2e.test.sh— default-on live guard, 5 real create_task rounds against cmux 0.64.25, exit=0Manual live repro: pre-fixfm_backend_cmux_create_taskx4 rounds against the real cmux app (duplicate-check-then-create shape)Manual live control: fixedfm_backend_cmux_create_taskx4 rounds, same lab FM_HOME, same appbin/fm-spawn.sh test-cmuxspawn-<pid> <scratch-project> "sh -c 'echo cmux-spawn-live-ok; exec sleep 600'" --mode no-mistakes --yolo off --backend cmuxin an isolated FM_*_OVERRIDE home, thenfm_backend_cmux_captureof the worker surface andbin/fm-teardown.sh <id>Same live spawn withFM_BACKEND_CMUX_SETTLE_TRIES=1(adversarial: settle window removed)Live duplicate-check probe:fm_backend_cmux_create_taskon a label already live in the appcmux workspace list --jsonbefore and after the whole session to confirm no orphan fm-titled workspacesbin/fm-test-run.sh --list --lane portable-serialand--check-coverageto confirm the new guard is scheduled, not orphaned✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.