Skip to content

feat(acpmux): durable native workflow CI machines [ask-6ef2624788] - #33982

Merged
teamleaderleo merged 80 commits into
feat-cmux-nextfrom
lane/native-workflow-ci-machine
Nov 3, 2026
Merged

teamleaderleo merged 80 commits into
feat-cmux-nextfrom
lane/native-workflow-ci-machine

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Native workflow machine(command, {on: "ci"}) now submits through the installed authenticated cmux-ci client. Each activity persists its exact repository, source SHA, relative working directory, requested worker, command, and stable request key before submission; retries attach to the same controller binding and cancellation uses its exact receipt and digest.

The adapter reads verified complete output and the command's actual exit status, refuses missing or conflicting bindings, and never falls back to local execution. Arbitrary peer command execution remains refused.

Leo-ask: ask-6ef2624788

go do all those/help out where you can + examine how you can improve yourself and your available options and capabilities and speed

Validation: static prepush checks pass. The initial adapter compiled and passed its focused remote test run 41662771589 at 100e4bba495adb5a1dbc453d1e09b5ec0383fdbe. Integrated verification 41670356514 passed 91 workflow library tests and strict acpmux library clippy at ef86e85e13feead6c1a5441f772c56cf3346d2d6. Final checked-range successor cf8224938b9f51e366dacbe91512ab7432b3b67a passed the same 91 workflow tests and strict clippy in 41672647323, exit 0. Independent Opus 5.5 medium review is SHIP at that exact head. Coverage exercises lost acknowledgement, immutable resume inputs, conflicting binding, exact conditional cancellation, complete output and nonzero exit status, and unavailable client.

Follow-ups from independent review: #34029 retains retryable cleanup after cancellation failure; #34030 handles pause before first submission; #34031 preserves late background-descendant output. Additional Opus follow-ups: #34037 hardens exit-marker framing, #34038 avoids duplicated durable logs, #34039 retries transient wait errors, #34040 verifies unpinned assigned-worker echoes, and #34041 improves unpushed-HEAD diagnostics. The CI adapter lane owns these follow-ups.

The panic ratchet has the same base count (acpmux: clippy::indexing_slicing 714 > baseline 686), with no added indexing sites. The protocol IR check fails the same protocol::tests::protocol_ir_is_current test on base 63381baf63019d678cffac2288ad00b3cfc2e871, run 41669629482, confirmed by the repository base-red comparator.

Risk: replay/cancellation behavior depends on the existing request-key controller contract; older controllers fail closed. No controller, capabilities, or security-policy changes.

Waiting uses bounded controller long polls and rechecks the run generation between waits. Client processes have a 300-second deadline, stop when their future is dropped, and keep the same saved request key. Temporary receipts and complete logs are removed with their activity-owned temporary directory.

Built on merged workflow foundations #33779 and #33811; the integrated diff contains only this CI adapter.

🤖 Generated with Codex

Agent session: agent session

teamleaderleo and others added 30 commits November 3, 2026 10:38
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
…-workflow-executor

# Conflicts:
#	cmux-tui/crates/acpmux/src/lib.rs
… outcomes

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
…errupted prompts

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
… lifecycles

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d39ec4d1a1c69bb721ec
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-62212972239ec6a32fed
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-d8021dc40c61c7f7b852
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-911d6e3b0a43b52512dd
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f13eb5f2f92091514f98
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f13eb5f2f92091514f98
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-911d6e3b0a43b52512dd
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f13eb5f2f92091514f98
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f13eb5f2f92091514f98
Adapted from 2379048f121b; exact prompt cancellation is a separate dependency.

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-911d6e3b0a43b52512dd
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f13eb5f2f92091514f98
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-b1f8de9bd6990c23fb3c
Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-b1f8de9bd6990c23fb3c
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review cf82249: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33982 -f ref=lane/native-workflow-ci-machine -f platform=macos -f backend_mode=local

Shipped in:

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at ef86e85

Visual verdict: unchanged by this diff (no UI code).

I read the whole diff from 63381ba to ef86e85. Here is what I checked:

  • Trust. The repo comes only from the run cwd's origin. The host must be github.com, userinfo, query and fragment are dropped, and the path must be a single owner/repo. The test confirms no credential reaches the intent. The ref is a full 40-hex HEAD. targetWorker is passed to cmux-ci as a request, so the controller's own policy still applies and acpmux adds no way around it. The principals are the same as for local machine(), which already runs bg_start without approval, so this adds no escalation. Git stderr, which can contain a credentialed URL, is never shown.
  • Idempotency. The request key is derived from the run id and activity. prepare() runs before register_activity, so the intent is persisted before the spawned execute does any POST. On resume the saved child is reused and nothing is resolved again. After a lost ack the code runs a read-only status, checks the binding, then repeats the identical keyed POST, so it cannot submit twice.
  • Cancel. Before cancelling, the code compares the full key, command, repo, ref and kind, and checks the digest's format. The cancel itself sends the id, key and digest to the controller as a conditional request, so it cannot hit someone else's job. Terminal receipts are skipped, and stall_stopping is still cancelled.
  • Safety and CI. Non-test code has no panics: slice bounds are checked and there are no unwraps. The only cfg(test) seam is Client.program, and release builds always run cmux-ci. The one red, acpmux protocol IR is current, is not this PR's: the regenerated IR adds only _acpmux/mcp_connect_request and _acpmux/mcp_connect_answer, which already exist on base. This diff touches no protocol types.

Findings:

  • [follow-up] hub/workflow_ci.rs:325-342 Marker forgery. The exit marker can be forged. The request key is visible to the command, for example in its parent shell's argv. A background child that prints \ncmux-workflow-exit-<key>:N\n after the wrapper exits, or a marker printed just before the wrapper is killed, becomes the rposition match, so the reported exitCode and the text slice come from the command and not the wrapper. Success still requires controller state == done and a zero exit from the wait, so this can only misreport the workflow's own result; it cannot turn a failure into a success. Prefer the controller's exit status when it is present, or check that the marker is the log's final line.
  • [follow-up] hub/workflow_ci.rs:317,345-347 Log size. The full log is read into memory and returned as both text and base64 data. That view then goes into the activity outcome in the durable run record, so a large CI log is stored about twice over in the run JSON. Consider keeping a reference or tail in the record and serving the bytes on request.
  • [follow-up] hub/workflow_ci.rs:385-405 Wait errors. Any cmux-ci wait exit other than 2 ends the activity. A transient client or network failure leaves no receipt, or an earlier receipt that is not terminal, so the activity errors and the post-execute workflow_cancel_activity then cancels a fleet job that is still healthy. Retry wait on errors that are not terminal, and cancel only on run cancellation or pause.
  • [follow-up] hub/workflow_ci.rs:288-291 Unpinned target. When targetWorker is unset and the controller echoes the assigned target_worker without pin_requested, the binding fails with "requested target differs" even though nothing was pinned. Confirm against the controller's job schema, or compare only pin_requested (absent counts as empty).
  • [follow-up] hub/workflow_ci.rs:118 Unpushed HEAD. A HEAD that was never pushed passes prepare and only fails at the controller. A clearer early error ("HEAD is not on origin") would help.

Follow-ups: marker forgery affects only the reported exitCode; full log stored twice in the run record; a transient wait error cancels a live job; unpinned target_worker echo; early error for an unpushed HEAD.

Co-authored-by: Codex <199175422+chatgpt-codex-connector[bot]@users.noreply.github.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-b1f8de9bd6990c23fb3c
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (claude-opus-5-5 medium, coordinator delta): SHIP at cf82249

This is a delta from SHIP at ef86e85 (#33982 (comment)). The new head adds one commit, in hub/workflow_ci.rs only. Two byte-slice indexes become checked .get ranges: the first-line split now uses and_then(|i| v.get(..i)), and the command-output range returns incomplete("CI output has an invalid command range") instead of panicking. That removes two panic sites, which clears the panic ratchet (716 back to 714), and an invalid range now reports incomplete output rather than crashing. Nothing else changed.

Carried findings:

  • The older automatic review's blocking finding (cache_remote in server/requests/workflow_dispatch.rs does not check the peer row's briefId/durable fields, so put_durable can overwrite another brief's key) is in code that came from merged workflow: dispatch durable agents to paired machines #33811 and is unchanged on base 63381ba. It is not in this PR's diff, so it does not block this PR. It is a base issue whose focused fix-forward is in flight by the remote owner.
  • The protocol IR red is the base red from b6e681e.

Follow-ups: cmux#34029, cmux#34030, cmux#34031, plus the five from the ef86e85 review.

@teamleaderleo
teamleaderleo merged commit 4836ed0 into feat-cmux-next Nov 3, 2026
164 of 166 checks passed
@teamleaderleo
teamleaderleo deleted the lane/native-workflow-ci-machine branch November 3, 2026 16:27

This branch was previously deployed

3 inactive deployments
artifacts — cf822493 Deployed Nov 3, 2026 by teamleaderleo via publish tree-addressed cmux-tui #106778
sentry-dsyms — cf822493 Deployed Nov 3, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #106778
ci-routing-agent — cf822493 Deployed Nov 3, 2026 by teamleaderleo via cmux-tui tree preflight #106778
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant