Skip to content

ask-6ef2624788: fence exact workflow prompt cancellation durably - #33812

Merged
teamleaderleo merged 18 commits into
feat-cmux-nextfrom
lane/native-workflow-cancel
Nov 3, 2026
Merged

teamleaderleo merged 18 commits into
feat-cmux-nextfrom
lane/native-workflow-cancel

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Leo-ask: ask-6ef2624788

Leo asked: “go do all those/help out where you can + examine how you can improve yourself and your available options and capabilities and speed”.

Pausing a workflow fences its accepted prompt and automatic retry descendants, including a prompt dequeued before its turn is installed. A failed journal write or flush returns an error; the stop is cached and acknowledged only after its tombstone reaches stable storage and is readable. Malformed history and a torn log tail preserve ordinary next-person prompts and durable stop restoration.

The additive event needs no journal migration. Lock order remains retry state → append → turn. Unrelated retries and background work are preserved.

Validation

  • Red at 9995846344b2c17280e67237077425f87f235cee: fleet run 41636505772, cargo test -p acpmux exact_prompt_stop --lib: three passed; persistence regression failed as expected with “stop acknowledged despite failing persistence operation 1”.
  • Head 2cdab546c4dc789c8624ba9e0ee8017d50d8a41b repairs the malformed-history blocker identified in independent review. A real LocalStore regression verifies Stop, the next ordinary prompt and tombstone restoration after malformed and unterminated records.
  • Fleet run 41639052241 passed exact_prompt_stop (5), prompt_accept_tests (3), turns_copy_stop_tests (10), and auto_resume::tests (8) at the final head; all selections passed.
  • Opus 5.5 medium review: SHIP at 2cdab546c4dc789c8624ba9e0ee8017d50d8a41b.
  • The cache-reset test helper uses poison-safe locking; the crash-ratchet's counted unwrap is removed.
  • Diff check and scoped prepush gate passed. No local Cargo build or test ran.

Prior art: #31709, #31593, #28859, #30526, #30439, #30280, #30157, #29879, #27924, #29502, #29175, #28111, #28503, #28099, #28477 were read at their source diffs. The parent lane acknowledged remaining mandatory prior art and open #33791.

Review follow-ups: #33840 (indexed scans), #33841 (origin-scoped subagent retries), and #33842 (idempotent repeated Stop); the native workflows lane owns them.

Risk: exact-stop admission reads a lazy per-session tombstone set. Storage read errors fail admission closed; malformed records are skipped, and acknowledgements verify a readable durable boundary. Existing cancellation escalation remains in place.

Changelog

none

Agent session: agent session

teamleaderleo and others added 14 commits November 3, 2026 10:38
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d8021dc40c61c7f7b852
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d8021dc40c61c7f7b852
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d8021dc40c61c7f7b852
@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 2cdab54: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33812 -f ref=lane/native-workflow-cancel -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): BLOCK at 867f561

Visual verdict: unchanged by this diff (no UI code).

Checked: the descendant fence in stop_prompt_retry (root matching via split_attempt, Waiting/Ending go to Stopped, Firing goes to Stopping, a late schedule_auto_resume is refused), the durable tombstone (append_stored and flush_session run before the cache insert and the ack, and a failure surfaces and retries), the person's next turn and mark left alone, and lock order (mark, then append, then turn, as in finish_turn). There are no new production unwraps or panics, the only new test helper is cfg(test), and nothing is dead in non-test builds. Hunks in #33791's cancel.rs don't overlap these and I found no semantic conflict. If the fix below adds a kind to kind_index.rs, it textually meets #33791's edits there.

  • [blocking] prompt_stop_set uses scan_kinds_strict, and start_turn now returns that error to every turn, so one unreadable line anywhere in a chat's log refuses every prompt in that chat forever. The cache stays None, so it fails again on each try. Today that chat works. Unreadable lines do happen: open_writer never repairs a torn tail, so a power loss or panic mid-append leaves a merged, unparseable line mid-log, and kind_index.rs already models it (parsed = false). transfer_authority uses the same strict scan but only warns. The same applies to the second strict scan in stop_prompt_retry: an error there returns before cancel_matching, so the workflow's running turn is never interrupted. Fix: read the tombstones with the non-strict scan_kinds, or warn and continue on the error, so an unreadable log can't fence the whole chat.

Follow-ups:

  • [follow-up] Performance: neither prompt_stop_requested nor auto_resume_scheduled is in INDEXED_KINDS, so both scans parse the whole log. The first start_turn per restored session does this, and so does every cancel_prompt. Both run while holding the auto_resume std mutex, which retry_row snapshots also wait on, and an fsync runs under it too. Large logs (the 100 MB case in cold_open_perf) will feel it. Add the kinds to the index, or record the tombstone in session meta.
  • [follow-up] An exact-prompt cancel no longer calls stop_subagent_retries. A subagent retry that the fenced workflow turn left waiting (its prompt id is subagent-retry-*, not root~*) still fires later in a paused workflow's child chat. Fence the retries whose origin is the stopped root.
  • [follow-up] Pressing Stop again on a root whose mark is already Stopped appends auto_resume_stopped and auto_resume_settled again. This is harmless, but the journal records the stop twice.

teamleaderleo and others added 2 commits November 3, 2026 12:44
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d8021dc40c61c7f7b852
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d8021dc40c61c7f7b852
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d8021dc40c61c7f7b852
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at 2cdab54

Visual verdict: unchanged by this diff

I re-checked the earlier blocking finding at 867f561 using git diff 867f561..2cdab54 (three commits, no base merge). It is fixed. prompt_stop_set, the new readback, and the descendant fence scan in stop_prompt_retry all use the tolerant scan_kinds now. The only strict scan left in hub/ is transfer_authority, which is unchanged and also on base. A malformed line no longer fences the chat, and it no longer stops Stop from reaching cancel_matching. The retry runs at most twice. A second prompt_stop_requested is written only when the first can't be read back (it was swallowed by a torn tail), and duplicate tombstones fold into a set, so nothing loops or double-applies. The cache is filled only after the readback succeeds. prompt_stop_requested is not an indexed kind, so the readback parses the flushed file itself and is not fooled by a stale copy. The new LocalStore test covers a malformed line, a torn tail, the person's next prompt, and a restore after the cache is cleared. The new unwraps are test-only, clear_prompt_stop_cache_for_test is poison-tolerant and cfg(test), and there is no dead code.

Follow-ups:

  • [follow-up] cancel_prompt still returns early with ? when the tombstone can't be persisted, flushed or read back (disk full, an I/O error), so in that case the running workflow turn is not interrupted. On base, Stop always interrupts. Consider calling cancel_matching anyway and then returning the persist error.
  • [follow-up] Carried from the earlier review, still open: tombstone and fence scans parse the whole log under the auto_resume mutex, and a Stop now scans twice (add the kinds to INDEXED_KINDS or keep them in meta). Exact-prompt cancel doesn't fence subagent-retry-* retries from the stopped root. A repeated Stop on a Stopped mark re-journals auto_resume_stopped/settled.

@blacksmith-sh

blacksmith-sh Bot commented Nov 3, 2026

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)/
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)
View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 2cdab546 Deployed Nov 3, 2026 by teamleaderleo via cmux-tui tree preflight #106142
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant