Repository navigation
feat(acpmux): issue enrollment-backed turn attestations - #28099
Conversation
Leo: "hey you need to look at the issues in hq and cmux-platform... lotta cloud related stuff goes in latter now" Leo on PRs: "make sure it\x27s not bullshit" Agent-Session: https://cmux.com/deeplink/agent?id=aid-38c4fa0ab4b2d8730974
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
Gallery before/after for
States
Pick other states with a web and size checks: green Shipped in:
|
|
Found 2 test failures on Blacksmith runners: Failures
|
|
Review (codex-review-box/gpt-6-astra high, lane-place): BLOCK at 0ebd491 BLOCK
|
|
Review (codex-review-box/gpt-6-astra high, lane-place): BLOCK at b3db459 BLOCK
|
|
Review (codex-review-box/gpt-6-astra high, lane-place): SHIP at 3c7ca44 SHIP Visual verdict: No UI changes; no after screenshots were supplied.
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
What it does
Adds daemon-owned turn attestations for live acpmux turns. The RPC accepts only
{sessionId, ttlMs?}from the authenticated local caller; it does not accept install, team, or user identity arguments. acpmux reads the existing journalturn_startedsequence, refuses closed, ended, missing, purged, or unloaded turns, and signs{sessionId, turnSeq, teamInstall, team, user, issuedAt, expiry, nonce}with the validated ES256 key from a completed server enrollment. No second turn counter or session manager is introduced.The narrow
EnrolledIdentitysigning accessor retains the validated private key in memory. Missing, pending, mismatched, or incomplete enrollments fail closed. Enrollment-derived paired metadata supplies install, team, and user.Trust model
The enrolled acpmux daemon signs. cmux-platform verifies the signature against the current
serverKeyfor the authenticated caller's signed team/user/install grant, and checks current TeamDO membership before key lookup. The current grant, daemon install binding, team binding, and revocation are therefore rechecked on every verification; key rotation replaces the enrolled public key and revocation makes lookup fail. Replay protection is a durable atomic nonce claim port. Receipt routes remain inert/404 in deployed Workers.Unix enrollment storage is protected by a same-UID 0600 boundary; unsandboxed code running as the same OS UID can read it. That is a known limitation. Follow-up work is an in-memory delegated daemon key certified through app enrollment, with the public key registered through the existing enrollment path.
Shared signing vector
Both the Rust signer and TypeScript verifier assert this fixed synthetic vector:
["cmux-acpmux-turn-attestation-v1","vector-session",7,"inst_vector","team_vector","user_vector",1700000000000,1700000060000,"vector-nonce"]WyJjbXV4LWFjcG11eC10dXJuLWF0dGVzdGF0aW9uLXYxIiwidmVjdG9yLXNlc3Npb24iLDcsImluc3RfdmVjdG9yIiwidGVhbV92ZWN0b3IiLCJ1c2VyX3ZlY3RvciIsMTcwMDAwMDAwMDAwMCwxNzAwMDAwMDYwMDAwLCJ2ZWN0b3Itbm9uY2UiXQ==fmZ2JgEiGrBe1KTUFBQEGA-5j0xfdBwkkk6zCt3rp1NFr8_f79i4f4iGI_YCb-HcbSADTdRQOK4MeU-uh-47egTransfer replacement fence
Both admission checks reject
session.replacedwhile holding the existing journal/admission locks.transfer_finishshares its unchanged seal, append-lock drain, and kill sequence throughretire_transfer_instance. The regression holds kill at the harness-use lock after the old instance is fenced and append_lock is released, requests an attestation for that still-resolvable live turn, then lets kill finish. A second test fences the instance after journal validation to cover the final recheck.At
a1406f91350be2768bcf44c6a2c9a32f4c0be4c7, remote run 41115640512 confirms both regressions fail with the exact attestation source from0ebd491e0e35a549b6e917a54f02dd7fb8b73bda, then pass with the fix. All 10 attestation tests pass, covering reconstructed Closed sessions, real detach, real kill, transfer replacement, changed turn identity/sequence, and late closing/purged flags. The Stop/completion lock-order regression, 49 retry tests, 12 carried-turn tests, Clippy with warnings denied, and production panic ratchet also pass. Receipt: exit 0, source treea7049f7970a25e250a4902be8ece23dc630e1ecb, matching the final head. No lock order or baseline changed.Live-turn admission race
After journal validation, issuance reacquires the existing live-turn mutex and rechecks both the turn ID and journal sequence, plus Closed/closing/purged/log-loaded state. The guard remains held through signature generation and JSON encoding, so kill, detach, or turn startup cannot clear or replace the admitted turn while it is signed. Lock order stays
auto_resume -> append_lock -> turnfor completion andappend_lock -> turnfor attestation; attestation never acquiresauto_resume.At
0ebd491e0e35a549b6e917a54f02dd7fb8b73bda, remote validation 41109311867 proves red with the exact attestation source frome28f7745811b5d3d548e1b974ccd20dabb71153cplus the test-only pause: a real detach finishes after the journal scan, yet the old issuer signs a stale proof. On the fixed head, the detach and replacement regressions pass along with all 6 attestation tests, the Stop/completion lock-order regression, 12 carried-turn tests, Clippy with warnings denied, and the production panic ratchet. Receipt: exit 0, source tree5d08eb3b1c227c93182796d1c32ad906933a0a2b, identical to the pushed head.Validation
Stop and turn completion acquire
auto_resumebeforeappend_lock; attestation still shares the append lock around the live-turn clear. Completion no longer takes the retry mutex while already holding the append mutex, eliminating the Stop/completion deadlock.Remote red/green run 41104293710 tests the exact production implementations from reviewed head
e54f0ead72bc5748d2fde6898cef314f6b342e13with only test synchronization hooks added. The new test runs real Stop andfinish_turnconcurrently, forces the conflicting interleaving, and fails withStop/completion lock inversion left Stop blocked. At fixed head15d3dda5d35500aef14caeaf5ff3fee3023a2915, it passes and preserves Stop's retry fence and journal ordering. The same run passes 49 retry tests, 12 carried-turn tests, 4 attestation tests, Clippy with warnings denied, and the production panic ratchet. The receipt has exit 0 and source treecbfba3559447b8d4f58c90e4df9a69415989eb98, matching that head. The follow-upe28f7745811b5d3d548e1b974ccd20dabb71153cuses poison recovery in the two test-only signal hooks so the static crash ratchet adds no unwraps; exact-head validation run 41105337847 repeats the red/green proof and passes the same tests, Clippy and panic ratchet. Its receipt reports exit 0 and source tree23d13e339e755bd09769a75883e701e405129ad4, matching the final head. The final head also passes PR fast gates, the static crash ratchet, changed-package Clippy and the capture check.Earlier closed-session validation run 41099269918 confirms the reconstructed-Closed-session regression fails with
5f30651897400a688bf1a540ab9eb1b71a76204c's attestation source and passes after the persistent Closed guard. Its 4 attestation unit tests, authenticated-caller integration test, protocol check, Clippy, and panic ratchet pass.All keys and enrollment records are synthetic. Removed the three indexing hits introduced by attestation; the five inherited hits were fixed upstream by #28429. No baseline was raised.
Leo said: “hey you need to look at the issues in hq and cmux-platform... lotta cloud related stuff goes in latter now” and “make sure it's not nonsense”.
Part of manaflow-ai/cmux-platform#42.
Risk: the attestation is short-lived and redacted, but deployment must still bind the trusted-source and durable nonce ports before receipt routes can be enabled.
Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Adds daemon-owned turn attestations for live acpmux turns, letting an authenticated local session prove the current turn's origin to cmux-platform.
_acpmux/turn_attestationRPC accepts only{sessionId, ttlMs?}from the authenticated local caller; forged or legacy identity arguments are refused.EnrolledIdentity::signandload_currentaccessors, and are refused when the enrollment is missing, pending, mismatched, or incomplete.turn_startedsequence; the turn is revalidated and held through issuance so detach or replacement mid-request cannot yield a stale attestation. Closed, purged, unloaded, or ended turns are refused.finish_turnshare the append lock with attestation;finish_turnacquires auto-resume before the journal lock to prevent a Stop/completion deadlock, and the replacement fence holds the same lock order so it cannot race a terminal retry append. The race tests were hardened so the seal thread is joined rather than panicking on a failed ordering.Migration
Written for commit 3c7ca44. Summary will update on new commits.
Replacement fence and lifecycle audit
seal_retry_replacementnow writessession.replacedwhile holdingauto_resume -> append_lock, the same order used by completion. Attestation usesappend_lock -> turnand never takesauto_resume; its final check and signing remain serialized against the replacement fence. A race test starts the real replacement seal after attestation's final check; it fails onb3db459because the old seal can flip the flag before signing, and passes after the seal sharesauto_resume -> append_lock.The production lifecycle writers were audited:
auto_resume::start_turninstalls a turn under theturnmutex while holdingauto_resume;auto_resume::take_turnclears or carries it underauto_resume -> append_lock -> turn.turns.rsunstarted-turn cleanup, fallback restoration,kill, anddetach_childwritesession.turnunder theturnmutex;killsetsclosingbefore that write while holdingharness_use, and both terminal paths callset_status, whosemetamutex guardsClosed.carried_turn.rsandhosts.rsrecovery write or updatesession.turnunder theturnmutex while their carried or hostturn_lockis held;shutdown.rsclears it under theturnmutex.lifecycle.rswritesclosingas an atomic close fence while holdingharness_useandspawn_lock;lane_reports.rsuses the atomic close fence before itsset_statuscall.set_status/reopenserializeClosedand all other statuses under themetamutex.auto_resume::seal_retry_replacementis the sole production writer ofreplacedand now holdsauto_resume -> append_lock; its readers (start_turn, retry scheduling, retry admission, andfinish_auto_resume) holdauto_resumewhile checking and acting, while the attestation reader holdsappend_lock -> turn. Test fixtures are excluded from the runtime path.detach_childclearsturnunderturnand changes status throughset_statusundermeta;closing,purged,replaced, andClosedare all checked again before signing.At remote validation 41131698851, the finish/replacement race is red with the append-only fence from
1fabc97, then green at3c7ca44eba8598d6abe4f69ff3d5f22dd41ccb31. The run also passed all attestation tests, Stop/completion, auto-resume, carried-turn, protocol freshness, acpmux Clippy with warnings denied, and the panic ratchet. Receipt exit code 0, source tree5dcf9053f7cf7b03d3264114888b978591c5162e(matching the pushed tree).No capture: the only path mapped by the checker is the generated acpmux protocol
methods.ts; this changes no native app UI or visuals.Merge validation at
b3db459079cc255bdbc56925986a63191ca8e944: mergedfeat-cmux-nextat214b514c8c3f3dabe9fb3055681a09b460f9d7bewithout rebasing, preserving the base's continuation-release notifications and the attestation lock order. Regenerated protocol files from the merged specification. Remote validation passed: 10 attestation tests, the Stop/completion regression, 49 auto-resume tests, 12 carried-turn tests, protocol IR freshness, 26 inbox tests, acpmux all-target Clippy with warnings denied, and the panic ratchet. Receipt exit code 0, source tree7885b309388fb933078e7f6eb0e9f98250426afe(matches this head).Agent session: agent session