Skip to content

workflow: execute durable local agent and machine activities - #33779

Merged
teamleaderleo merged 44 commits into
feat-cmux-nextfrom
lane/native-workflow-executor
Nov 3, 2026
Merged

teamleaderleo merged 44 commits into
feat-cmux-nextfrom
lane/native-workflow-executor

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Workflow agent() and machine() calls reserve durable child identities before effects, reattach interrupted work, and preserve the exact final assistant reply or complete machine output. Parallel activities enqueue independently and receive distinct session names. A fallback after a stalled daemon retry reserves its own prompt identity before admission.

Agent replies get one schema correction; inactivity nudges once, then restarts on a configured permitted harness or route, then fails. Account, capacity and permission waits keep the existing recovery path. Exact child ownership prevents cancelling logical siblings, and waiting for a machine task cannot delay daemon shutdown. Uncertain machine launches fail closed.

This executor replaces one seam of #33755 and builds on the merged journal, reply policy, lifecycle and cancellation foundations. QuickJS, remote agent placement and public CLI/tool wiring remain separate PRs. CI placement stays refused here and is tracked in #33778. At 2,357 added lines this is one coupled executor state machine, its authenticated activity RPC, complete-output adapter and regression tests. The independent seams—QuickJS engine, CLI/tool wiring, remote placement and CI adapter—stay in their own PRs; splitting these handlers from their recovery tests would leave neither independently useful.

Validation: exact-head fleet run https://github.com/manaflow-ai/cmux/actions/runs/41662804193 passed at 46beb0d5c95d851ce9b590f42c3fae4fcbf8b4d6: the route-nudge crash regression fails with the persisted cancellation target omitted, the source is restored, then formatting, all 83 workflow tests, 15 background-task tests and strict acpmux all-targets clippy pass. Static prepush checks pass. The separate protocol IR check reports inherited drift for two MCP connect methods; its registry, source and committed spec are unchanged from the merged base, and all other protocol tests pass. The separate clippy job fails only the inherited indexing panic count: all emitted diagnostics were compared against this diff, with zero hits on added lines; strict all-targets clippy passes in the focused run.

Independent review: Opus 5.5 medium SHIP at 46beb0d5c95d851ce9b590f42c3fae4fcbf8b4d6: #33779 (comment).

Follow-up: complete machine output is retained in its spool but still collected and serialized whole in memory; #33851 tracks reading it in pages. Additional review follow-ups are #33976 (coalescing detached lock waits), #33977 (explicit-null placement semantics), and #33978 (validating an alternate before recording Restart). #33841’s route-nudge crash boundary is covered here. Signing, notarizing and publishing credentials remain outside the workflow runtime.

Leo-ask: ask-6ef2624788
Leo asked: “go do all those/help out where you can + examine how you can improve yourself and your available options and capabilities and speed”. Design: teamleaderleo/leo-workspace#624 §3.2–3.3.

🤖 Generated with Codex

Agent session: agent session

teamleaderleo and others added 18 commits November 3, 2026 10:38
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
…-workflow-executor

# Conflicts:
#	cmux-tui/crates/acpmux/src/lib.rs
… outcomes

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
…errupted prompts

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
… lifecycles

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 46beb0d: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33779 -f ref=lane/native-workflow-executor -f platform=macos -f backend_mode=local

Shipped in:

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
@blacksmith-sh

This comment has been minimized.

teamleaderleo and others added 3 commits November 3, 2026 12:19
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d8021dc40c61c7f7b852
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-911d6e3b0a43b52512dd
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-911d6e3b0a43b52512dd
@cursor

cursor Bot commented Nov 3, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at 644ca8e

Delta since the SHIP at 44b2ead: one hunk in workflow_activity.rs. When the stall deadline makes RecoveryAction::Restart, it clears waiting_for and route_nudge before the policy write.

Checked: current_id is state.waiting_for.clone().unwrap_or_else(..) (workflow_activity.rs:289), and automatic is waiting_for.is_some() (:326). Before this fix, the fallback attempt after continue 'attempt reused the daemon-retry id on the -restart session with a different route or harness. It also took the automatic admission path. That is the identity and input reuse fleet run 41658129526 hit. Now the fallback gets a fresh -g{epoch}-{gen}-{c} id, and it is persisted before the cancel, so recovery after the write also starts clean. This matches what the error-path Restart branch already did (:526-527). No legitimate wait is lost: the wait belonged to the original session's account, and that prompt is cancelled right after (:463-464). The Nudge and Wait paths are unchanged.

Findings:

  • [follow-up] workflow_activity.rs:441-447: the delta adds no regression test. A hub test should cover an automatic daemon retry (waiting_for set) that stalls, gets nudged, stalls again and restarts onto fallbackHarness, and assert the fallback's prompt id is fresh and not waiting_for. The focused fleet re-run covers it only indirectly.
  • [follow-up] workflow_activity.rs:441-460: the cleared state is persisted before the alternate check. With no fallback configured, the activity errors with waiting_for already gone. That is harmless today, because the terminal cancel still reaches the registered agent_prompt child. Checking alternate before the write would keep the record exact.
  • Note: the PR currently shows CONFLICTING against feat-cmux-next, so a base merge is needed before landing. Checks at this head: 15/15 green.

Follow-ups: regression test for stalled automatic retry -> fallback prompt identity; check alternate before persisting the cleared wait.

Persist the original retry prompt cancellation target alongside Restart before clearing its admission identity. Replay cancels that target before admitting the fallback; a regression simulates the crash gap with an active original retry. Also flatten the activity-cleanup guard for strict clippy.

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c3deabb1b73f18aa4204
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at bb14a21

Delta 644ca8e..bb14a21 is one commit (no base merge): Policy gains restart_prompt_id (#[serde(default)]), both Restart transitions set it in the same update_activity_policy write that clears waiting_for, and restart recovery cancels that exact id on the original session, falling back to the old computed id. It adds a regression test and fixes a clippy collapsible_if in server/workflow_activity.rs.

What I checked:

  • Durability: in both the stall path (hub/workflow_activity.rs:449-453) and the route-failure path (:529-545), restart_prompt_id and waiting_for = None are set in the same serialized state, and that one store write happens before remove_queued/cancel_prompt. A crash after the write leaves the exact retry id on disk.
  • Recovery (:212-220): with the field set, it cancels only that id on the original session. With an older row (None through serde default), it falls back to the previous computed id, so behaviour matches before. A stale id does nothing harmful: cancel_matching returns Ok without cancelling when the running turn's prompt_id differs (cancel.rs:343-348).
  • Test (:1248-1342): it starts a real ~resume-1 retry on the original session, then persists a Restart-state policy with restart_prompt_id and runs workflow_execute with no in-process cancel. The old computed id (-g1-0-0) would not match the ~resume-1 retry, so the worker would never finish and session.turn() would stay set. The test therefore covers the crash window.
  • Panics: the delta adds no unwrap or expect outside mod tests.

CI at bb14a21: 8 passed, 0 failed, 3 pending. GitHub reports the PR as CONFLICTING with feat-cmux-next, so it needs a base update before it can land.

Follow-ups: [follow-up] restart_prompt_id is never cleared. This is harmless today because only one restart is allowed and the cancel is exact-match, but clearing it once the fallback is admitted would keep the field from outliving its purpose.

…-workflow-executor

# Conflicts:
#	cmux-tui/crates/acpmux/src/bg_run.rs
@cursor

cursor Bot commented Nov 3, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

…unch

Preserve the newly merged background description and completion-notification arguments in the stable-ID adapter, including the Arc receiver its watcher needs. Ordinary tasks retain the base behavior; workflow tasks use their own durable completion path.

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c3deabb1b73f18aa4204
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium, carried): SHIP at 0b9a897

This carries the SHIP at bb14a21 (#33779 (comment)). The new head merges base a34003d. I compared the PR's own diff against its base parent at both heads. Apart from the conflict resolution in adjacent bg_run.rs tests, which keeps both sides, the only difference is two lines in the regression test. The test now drives the real gap: a RouteError event and a -route-nudge~resume-1 retry id, where it used to use Stalled and ~resume-1. That makes the test stronger. Production code is unchanged.

Follow-ups: cmux#33976, cmux#33977, cmux#33978 (filed by the lane).

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (claude-opus-5-5 medium, coordinator): SHIP at 46beb0d

This is a delta review from the carried SHIP at 0b9a897. The only new commit is 46beb0d, in bg_tasks.rs and workflow_activity.rs. It brings the stable-ID launch in line with the background description/notify arguments added on base by #33317.

There is no behaviour change beyond that. The exact-head run 41662804193 passed fmt, 83 workflow tests, 15 bg_ tests, strict clippy and the mutation proof.

Follow-ups: none new (cmux#33976, cmux#33977 and cmux#33978 stand).

@teamleaderleo
teamleaderleo merged commit 74c166b into feat-cmux-next Nov 3, 2026
152 of 155 checks passed
@teamleaderleo
teamleaderleo deleted the lane/native-workflow-executor branch November 3, 2026 15:34
teamleaderleo added a commit that referenced this pull request Nov 3, 2026
#33317 added result_line to bg_start and the Spec, and #33779 moved the spawn into
bg_start_with_id without it, so feat-cmux-next fails with E0425 at
hub/bg_tasks.rs:160. Workflow activities start with result_line false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-92fa78a91e29868c5eaa
(cherry picked from commit 124e4c1)
teamleaderleo added a commit that referenced this pull request Nov 3, 2026

This branch was previously deployed

3 inactive deployments
artifacts — 46beb0d5 Deployed Nov 3, 2026 by teamleaderleo via publish tree-addressed cmux-tui #106617
sentry-dsyms — 46beb0d5 Deployed Nov 3, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #106617
ci-routing-agent — 46beb0d5 Deployed Nov 3, 2026 by teamleaderleo via cmux-tui tree preflight #106617
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant