Skip to content

feat(acpmux): issue enrollment-backed turn attestations - #28099

Merged
teamleaderleo merged 25 commits into
feat-cmux-nextfrom
lane/turn-attestation-acpmux
Oct 31, 2026
Merged

teamleaderleo merged 25 commits into
feat-cmux-nextfrom
lane/turn-attestation-acpmux

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

What it does

Adds daemon-owned turn attestations for live acpmux turns. The RPC accepts only {sessionId, ttlMs?} from the authenticated local caller; it does not accept install, team, or user identity arguments. acpmux reads the existing journal turn_started sequence, refuses closed, ended, missing, purged, or unloaded turns, and signs {sessionId, turnSeq, teamInstall, team, user, issuedAt, expiry, nonce} with the validated ES256 key from a completed server enrollment. No second turn counter or session manager is introduced.

The narrow EnrolledIdentity signing accessor retains the validated private key in memory. Missing, pending, mismatched, or incomplete enrollments fail closed. Enrollment-derived paired metadata supplies install, team, and user.

Trust model

The enrolled acpmux daemon signs. cmux-platform verifies the signature against the current serverKey for the authenticated caller's signed team/user/install grant, and checks current TeamDO membership before key lookup. The current grant, daemon install binding, team binding, and revocation are therefore rechecked on every verification; key rotation replaces the enrolled public key and revocation makes lookup fail. Replay protection is a durable atomic nonce claim port. Receipt routes remain inert/404 in deployed Workers.

Unix enrollment storage is protected by a same-UID 0600 boundary; unsandboxed code running as the same OS UID can read it. That is a known limitation. Follow-up work is an in-memory delegated daemon key certified through app enrollment, with the public key registered through the existing enrollment path.

Shared signing vector

Both the Rust signer and TypeScript verifier assert this fixed synthetic vector:

  • Canonical tuple: ["cmux-acpmux-turn-attestation-v1","vector-session",7,"inst_vector","team_vector","user_vector",1700000000000,1700000060000,"vector-nonce"]
  • Signing bytes (base64): WyJjbXV4LWFjcG11eC10dXJuLWF0dGVzdGF0aW9uLXYxIiwidmVjdG9yLXNlc3Npb24iLDcsImluc3RfdmVjdG9yIiwidGVhbV92ZWN0b3IiLCJ1c2VyX3ZlY3RvciIsMTcwMDAwMDAwMDAwMCwxNzAwMDAwMDYwMDAwLCJ2ZWN0b3Itbm9uY2UiXQ==
  • Expected ES256 signature (base64url): fmZ2JgEiGrBe1KTUFBQEGA-5j0xfdBwkkk6zCt3rp1NFr8_f79i4f4iGI_YCb-HcbSADTdRQOK4MeU-uh-47eg

Transfer replacement fence

Both admission checks reject session.replaced while holding the existing journal/admission locks. transfer_finish shares its unchanged seal, append-lock drain, and kill sequence through retire_transfer_instance. The regression holds kill at the harness-use lock after the old instance is fenced and append_lock is released, requests an attestation for that still-resolvable live turn, then lets kill finish. A second test fences the instance after journal validation to cover the final recheck.

At a1406f91350be2768bcf44c6a2c9a32f4c0be4c7, remote run 41115640512 confirms both regressions fail with the exact attestation source from 0ebd491e0e35a549b6e917a54f02dd7fb8b73bda, then pass with the fix. All 10 attestation tests pass, covering reconstructed Closed sessions, real detach, real kill, transfer replacement, changed turn identity/sequence, and late closing/purged flags. The Stop/completion lock-order regression, 49 retry tests, 12 carried-turn tests, Clippy with warnings denied, and production panic ratchet also pass. Receipt: exit 0, source tree a7049f7970a25e250a4902be8ece23dc630e1ecb, matching the final head. No lock order or baseline changed.

Live-turn admission race

After journal validation, issuance reacquires the existing live-turn mutex and rechecks both the turn ID and journal sequence, plus Closed/closing/purged/log-loaded state. The guard remains held through signature generation and JSON encoding, so kill, detach, or turn startup cannot clear or replace the admitted turn while it is signed. Lock order stays auto_resume -> append_lock -> turn for completion and append_lock -> turn for attestation; attestation never acquires auto_resume.

At 0ebd491e0e35a549b6e917a54f02dd7fb8b73bda, remote validation 41109311867 proves red with the exact attestation source from e28f7745811b5d3d548e1b974ccd20dabb71153c plus the test-only pause: a real detach finishes after the journal scan, yet the old issuer signs a stale proof. On the fixed head, the detach and replacement regressions pass along with all 6 attestation tests, the Stop/completion lock-order regression, 12 carried-turn tests, Clippy with warnings denied, and the production panic ratchet. Receipt: exit 0, source tree 5d08eb3b1c227c93182796d1c32ad906933a0a2b, identical to the pushed head.

Validation

Stop and turn completion acquire auto_resume before append_lock; attestation still shares the append lock around the live-turn clear. Completion no longer takes the retry mutex while already holding the append mutex, eliminating the Stop/completion deadlock.

Remote red/green run 41104293710 tests the exact production implementations from reviewed head e54f0ead72bc5748d2fde6898cef314f6b342e13 with only test synchronization hooks added. The new test runs real Stop and finish_turn concurrently, forces the conflicting interleaving, and fails with Stop/completion lock inversion left Stop blocked. At fixed head 15d3dda5d35500aef14caeaf5ff3fee3023a2915, it passes and preserves Stop's retry fence and journal ordering. The same run passes 49 retry tests, 12 carried-turn tests, 4 attestation tests, Clippy with warnings denied, and the production panic ratchet. The receipt has exit 0 and source tree cbfba3559447b8d4f58c90e4df9a69415989eb98, matching that head. The follow-up e28f7745811b5d3d548e1b974ccd20dabb71153c uses poison recovery in the two test-only signal hooks so the static crash ratchet adds no unwraps; exact-head validation run 41105337847 repeats the red/green proof and passes the same tests, Clippy and panic ratchet. Its receipt reports exit 0 and source tree 23d13e339e755bd09769a75883e701e405129ad4, matching the final head. The final head also passes PR fast gates, the static crash ratchet, changed-package Clippy and the capture check.

Earlier closed-session validation run 41099269918 confirms the reconstructed-Closed-session regression fails with 5f30651897400a688bf1a540ab9eb1b71a76204c's attestation source and passes after the persistent Closed guard. Its 4 attestation unit tests, authenticated-caller integration test, protocol check, Clippy, and panic ratchet pass.

All keys and enrollment records are synthetic. Removed the three indexing hits introduced by attestation; the five inherited hits were fixed upstream by #28429. No baseline was raised.

Leo said: “hey you need to look at the issues in hq and cmux-platform... lotta cloud related stuff goes in latter now” and “make sure it's not nonsense”.

Part of manaflow-ai/cmux-platform#42.

Risk: the attestation is short-lived and redacted, but deployment must still bind the trusted-source and durable nonce ports before receipt routes can be enabled.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Adds daemon-owned turn attestations for live acpmux turns, letting an authenticated local session prove the current turn's origin to cmux-platform.

  • New _acpmux/turn_attestation RPC accepts only {sessionId, ttlMs?} from the authenticated local caller; forged or legacy identity arguments are refused.
  • Attestations are signed with the ES256 key from a completed server enrollment via new EnrolledIdentity::sign and load_current accessors, and are refused when the enrollment is missing, pending, mismatched, or incomplete.
  • Liveness is checked against the live session state and the existing journal turn_started sequence; the turn is revalidated and held through issuance so detach or replacement mid-request cannot yield a stale attestation. Closed, purged, unloaded, or ended turns are refused.
  • Replacement fencing and finish_turn share the append lock with attestation; finish_turn acquires auto-resume before the journal lock to prevent a Stop/completion deadlock, and the replacement fence holds the same lock order so it cannot race a terminal retry append. The race tests were hardened so the seal thread is joined rather than panicking on a failed ordering.
  • Replay protection relies on a durable atomic nonce claim port in cmux-platform; receipt routes remain inert until that port is deployed.

Migration

  • Deployment must bind the trusted-source and durable nonce ports before receipt routes can be enabled.
  • Unix enrollment storage is protected by same-UID 0600 permissions only; unsandboxed same-UID code can read it.

Written for commit 3c7ca44. Summary will update on new commits.

View guided diff Turn on auto-fix

Replacement fence and lifecycle audit

seal_retry_replacement now writes session.replaced while holding auto_resume -> append_lock, the same order used by completion. Attestation uses append_lock -> turn and never takes auto_resume; its final check and signing remain serialized against the replacement fence. A race test starts the real replacement seal after attestation's final check; it fails on b3db459 because the old seal can flip the flag before signing, and passes after the seal shares auto_resume -> append_lock.

The production lifecycle writers were audited:

  • auto_resume::start_turn installs a turn under the turn mutex while holding auto_resume; auto_resume::take_turn clears or carries it under auto_resume -> append_lock -> turn.
  • turns.rs unstarted-turn cleanup, fallback restoration, kill, and detach_child write session.turn under the turn mutex; kill sets closing before that write while holding harness_use, and both terminal paths call set_status, whose meta mutex guards Closed.
  • carried_turn.rs and hosts.rs recovery write or update session.turn under the turn mutex while their carried or host turn_lock is held; shutdown.rs clears it under the turn mutex.
  • lifecycle.rs writes closing as an atomic close fence while holding harness_use and spawn_lock; lane_reports.rs uses the atomic close fence before its set_status call. set_status/reopen serialize Closed and all other statuses under the meta mutex.
  • auto_resume::seal_retry_replacement is the sole production writer of replaced and now holds auto_resume -> append_lock; its readers (start_turn, retry scheduling, retry admission, and finish_auto_resume) hold auto_resume while checking and acting, while the attestation reader holds append_lock -> turn. Test fixtures are excluded from the runtime path.
  • Detach has no separate state bit: detach_child clears turn under turn and changes status through set_status under meta; closing, purged, replaced, and Closed are all checked again before signing.

At remote validation 41131698851, the finish/replacement race is red with the append-only fence from 1fabc97, then green at 3c7ca44eba8598d6abe4f69ff3d5f22dd41ccb31. The run also passed all attestation tests, Stop/completion, auto-resume, carried-turn, protocol freshness, acpmux Clippy with warnings denied, and the panic ratchet. Receipt exit code 0, source tree 5dcf9053f7cf7b03d3264114888b978591c5162e (matching the pushed tree).

No capture: the only path mapped by the checker is the generated acpmux protocol methods.ts; this changes no native app UI or visuals.

Merge validation at b3db459079cc255bdbc56925986a63191ca8e944: merged feat-cmux-next at 214b514c8c3f3dabe9fb3055681a09b460f9d7be without rebasing, preserving the base's continuation-release notifications and the attestation lock order. Regenerated protocol files from the merged specification. Remote validation passed: 10 attestation tests, the Stop/completion regression, 49 auto-resume tests, 12 carried-turn tests, protocol IR freshness, 26 inbox tests, acpmux all-target Clippy with warnings denied, and the panic ratchet. Receipt exit code 0, source tree 7885b309388fb933078e7f6eb0e9f98250426afe (matches this head).

Agent session: agent session

Leo: "hey you need to look at the issues in hq and cmux-platform... lotta cloud related stuff goes in latter now"

Leo on PRs: "make sure it\x27s not bullshit"

Agent-Session: https://cmux.com/deeplink/agent?id=aid-38c4fa0ab4b2d8730974
@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 3c7ca44: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28099 -f ref=lane/turn-attestation-acpmux -f platform=macos -f backend_mode=local

Gallery before/after for 3c7ca44: 16 states rendered, 0 images differ, 2510 more touched states not rendered. Images are in the run's artifact, not on this public PR.

gh run download 41131603485 -R manaflow-ai/cmux -n gallery-advisory-28099-3c7ca44eba8598d6abe4f69ff3d5f22dd41ccb31 (then read gallery-review.json)

States
State Theme Status Diff Picked by
agent-pane.live-chat-watch/choice-alone Apple System Colors Light unchanged 0% imports methods.ts (1 hops)
agent-pane.live-chat-watch/choice-alone Apple System Colors unchanged 0% imports methods.ts (1 hops)
agent-pane.live-chat-watch/view-only Apple System Colors Light unchanged 0% imports methods.ts (1 hops)
agent-pane.live-chat-watch/view-only Apple System Colors unchanged 0% imports methods.ts (1 hops)
agent-pane.auth-health/signed-in Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/signed-in Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/signed-out Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/signed-out Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/expired Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/expired Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/unknown Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.auth-health/unknown Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/standard Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/standard Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/wide Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/wide Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/wide-blocks Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.chat-width/wide-blocks Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-edge/short-edge Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-edge/short-edge Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-edge/wide-edge Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-edge/wide-edge Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-edge/no-edge Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-edge/no-edge Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-floats/composer-grown Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.composer-floats/composer-grown Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.continuation-spacing/shipped Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.continuation-spacing/shipped Apple System Colors unchanged 0% imports methods.ts (2 hops)
agent-pane.continuation-spacing/two-thirds Apple System Colors Light unchanged 0% imports methods.ts (2 hops)
agent-pane.continuation-spacing/two-thirds Apple System Colors unchanged 0% imports methods.ts (2 hops)

Pick other states with a Gallery: entry.id, entry.id/variant line in the PR body.

web and size checks: green

Shipped in:

@blacksmith-sh

blacksmith-sh Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)
View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex-review-box/gpt-6-astra high, lane-place): BLOCK at 0ebd491

BLOCK
Visual verdict: N/A — non-UI diff; no hosted screenshots were supplied.

  1. Stale attestation during replacement: [turn_attestation.rs:29-34](<>) and [:68-74](<>) never check session.replaced. transfer_in marks the old instance replaced before releasing append_lock and awaiting kill; during that window the old live turn remains resolvable and can receive a signed attestation. Reject replaced sessions in both admission checks.

@teamleaderleo teamleaderleo removed the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex-review-box/gpt-6-astra high, lane-place): BLOCK at b3db459

BLOCK
Visual verdict: BLOCK — webviews/ changed, but no after screenshot was supplied; the PR body explicitly says “No capture” (review/pr.body:79, review/images/README.txt:2).

  1. Stale attestation race: turn_attestation.rs:67 rechecks replaced before signing, but seal_retry_replacement can set that flag concurrently under a different lock. A transfer can therefore fence the old instance between the check and signature, returning an attestation after replacement has begun.

  2. Visual review cannot be completed: The required after-image review and landing-lane statement are missing.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex-review-box/gpt-6-astra high, lane-place): SHIP at 3c7ca44

SHIP

Visual verdict: No UI changes; no after screenshots were supplied.

  1. No actionable findings.

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 3c7ca44: an independent review passed this exact head (review box review-28099-3c7ca44 receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

This branch was previously deployed

3 inactive deployments
artifacts — 3c7ca44e Deployed Oct 30, 2026 by teamleaderleo via add the Windows daemon to the tree #85158
sentry-dsyms — 3c7ca44e Deployed Oct 30, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #85158
ci-routing-agent — 3c7ca44e Deployed Oct 30, 2026 by teamleaderleo via cmux-tui tree preflight #85158
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant