[teamleaderleo]

Adds machine-join-e2e.yml, a workflow you can only start by hand. It runs one bounded end-to-end check of the one-line machine join against the development platform: mint a join token, pipe it to acpmux host join on stdin, wait for a heartbeat, check status, the platform list, consent and the credential file mode, revoke, and scan for secrets. All of it lives in one bun script, scripts/cmux-next/machine-join-e2e.ts, with its tests.

Leo: "just have this sort of thing for, I guess, general onboarding of adding remote machines and whatnot and whatever, which includes, I guess, screen sharing potentially."

The ask from the acpmux join owner: "Please provide the CI-safe harness invocation for mint+join (where the signed-in CLI bearer is injected transiently by the CI job, never argv/env/log/disk) and the script's expected redacted evidence fields. I will run one bounded cmux-ci job only; it must mint in-process, pipe cmj_ directly to acpmux host join stdin, then emit only machineId/teamId/name/hubOrigin/credentialPresent/heartbeat/revoked and join list correlation by joinTokenId, with no secrets. Also confirm which CI origin/development route and whether the job may use an ephemeral temporary OS home that is destroyed by the job. Do not send any token/credential to this lane."

The secret rules, as given: "DEVELOPMENT platform only (https://cmux-platform-development.debussy.workers.dev), minted with the DEV stack UI-test account (CMUX_UITEST_STACK_* in CI) inside a CI job that already holds those secrets. Never put a CMUX_UITEST_STACK_* value, cmj_ or cmm_ on a lane disk, in argv, in an env file or in a log. The production mint is an [outward] item for Leo, not this."

Scope

• Trigger: on: workflow_dispatch and nothing else (no pull_request, pull_request_target or workflow_run). The job's if: and its first step both refuse any ref but refs/heads/feat-cmux-next. Inputs: name (default ci-join-<run id>) and acpmux_ref, which accepts only the run's own commit (the feat-cmux-next tip). acpmux is always built from the run's commit, because the build runs as the runner user that later holds the secrets, so no other code may build there.
• Token: top-level permissions: contents: read and nothing else; no job-level grants.
• Environment: the job runs in the existing cmux-next-vm-image-dev environment. GitHub's branch policy on that environment admits feat-cmux-next only, so the branch limit is enforced by GitHub as well as by the job's own checks. tests/test_supplychain_security.py requires every dispatchable job that reads secrets to run in an environment. That environment's own secret (FREESTYLE_API_KEY) is never referenced; tests/test_machine_join_e2e_workflow.py pins the job to exactly the two dogfood secret references.
• Secrets: it uses only the repo-level CMUX_DOGFOOD_STACK_EMAIL / CMUX_DOGFOOD_STACK_PASSWORD that already exist. No new secret, environment or variable. They reach one step, the mint step, through env only, mapped to CMUX_UITEST_STACK_*. ios-e2e.yml (lines 353-367) uses the same pair the same way.
• Development only: the origin is hard-coded (HUB_ORIGIN), and the script refuses any other origin before it makes a request. The Stack project is the development one that cmux-api-development trusts (454ecd03-…, backend/apps/api/wrangler.jsonc env.development). The access token comes from https://cmux-api-development.debussy.workers.dev (issuer https://cmux-api/development, the issuer cmux-platform development verifies). A token from that issuer cannot sign in to staging or production.
• Runner: an ephemeral Blacksmith VM (16 vCPU, or 32 while the overflow switch's record exists). It deliberately does not use LINUX_RUNNER, which may name a persistent host, because this job adds an OS user and holds the secrets. 45-minute job bound and 15 minutes for the mint step. No concurrency group, so it cancels nothing.
• It is never dispatched until this PR is reviewed and merged.

How the secrets are held

• In memory only. The script reads the email and password from its own env and deletes them from process.env before anything starts. It then holds these values in memory and never anywhere else: the Stack session, a throwaway cmux cli install key (non-extractable), the cmux access token, the cmj_ and the cmm_.
• Masked as they appear. Each value gets ::add-mask:: the moment it exists: the Stack tokens on sign-in, the access token on mint, the cmj_ on mint, and the cmm_ once it has been read from the credential file.
• Children see none of them. Every child process (sudo, acpmux, cat, stat, find) runs with exactly {PATH} as its environment, and argv carries no secret. acpmux runs as the disposable user through sudo -n -u USER -- env -i HOME=… PATH=… acpmux ….
• Nothing runs as root, and untrusted values stay plain. Every child runs as the disposable user, including the credential read, its stat and the home scan. So a path that user controls, such as a symlink, reaches nothing beyond what that user could read anyway. The credential path must be exactly $HOME/.acpmux/machine-credential.json, and stat without -L must show, before the read and again after the heartbeat, that ~/.acpmux is the user's own directory (not a link) and the credential is the user's own regular file with exactly one link (so no hard link to it exists elsewhere) and mode 600, checked right after the join as well as after the heartbeat. Ids from the platform and from acpmux (machine, join token, team, label) must be plain [A-Za-z0-9._-], not only dots (so . and .. can never step up a revoke URL), and never token-shaped (cmj_, cmm_, eyJ, bearer), or they are dropped before they reach a URL, a log line or the evidence. Every line the script prints, apart from its own ::add-mask:: and ::warning:: commands, is made plain: CR and LF become a space, :: is split, and ##[ is broken. So an id, a server code or acpmux's output can never start a workflow command (::stop-commands:: would turn later masks into plain text). The evidence check prints values from the evidence only as plain characters, and the smoke step prints acpmux's output only through sed that splits :: and breaks ##[, never raw.
• The join token goes in on stdin only: host join --hub <dev> --name <name> --json gets cmj_…\n on stdin and then stdin is closed.
• Error text names a step, an HTTP status and a server code only. It never includes a response body. On failure, the tail of acpmux's output is printed with every held value, cmj_/cmm_ token and JWT redacted.
• Two evidence checks. The script runs the evidence through evidenceLeaks (prefixes, JWT shape, bearer/authorization, any held value both as written and in its JSON-escaped form), and the workflow's "Check the evidence" step runs a second grep (cmj_|cmm_|eyJ…\.…\.|bearer|authorization). Either check fails the job and blocks the upload.

Cleanup (every path)

• One cleanup pass. It runs in the script's finally; on SIGINT or SIGTERM (a cancel or a step timeout; the step execs bun); and at the script's own 12-minute deadline (the step allows 15). The order:

  1. abort every request in flight; from here, the main flow starts no new join, daemon or acpmux process;
  2. stop the join's own launcher (SIGTERM, which sudo relays), then every process of the disposable user (kill -TERM -1, then -KILL -1, run as that user). Each step is confirmed before the next. A process that outlives its SIGKILL by 500 ms has its pipes closed and counts as ended, so a child holding them open cannot stall cleanup. A cancel ends a long wait that is already running but never skips the confirmation (after a cancel, at most 3 s in all). A join still running after that is reported, so no join can still redeem once the search for its machine begins;
  3. revoke the machine. If the join did not report it, the machine is found by its joinTokenId (6 list reads). After a cancel, everything works to one deadline, 7.5 s after the cancel: the search for the machine stops in time to leave the revoke request 2.5 s, and the process exits at 9 s, inside the runner's grace (SIGINT, then SIGTERM 7.5 s later, then SIGKILL). If a cancel lands during cleanup and aborts the revoke, the revoke is tried again;
  4. revoke the run's cmux install;
  5. sign out of Stack.

  A mint without a joinTokenId never joins.

• Sweep. The mint label is always ci-join-<run id>, whatever name is. Each run starts by revoking this account's own CI machines (that label, minted by the same user) that are still joined 30 minutes after they were made. This covers a run killed before its revoke (a lost runner or SIGKILL); a machine credential otherwise lives 365 days.

• Secret scan of the home. One process with a fixed argv, tar -C HOME --one-file-system -cf - ., runs as the disposable user. The harness reads the archive in memory and counts cmj_/cmm_ in every file name, link target and body, and looks for every value it holds. No file name reaches an argv. tar must read everything: any warning on its stderr fails the scan, because tar skips a socket with a warning and still exits 0. Child text output is read as UTF-8 and the archive as bytes, and every held value is matched both as text and as its UTF-8 bytes, so a non-ASCII value is caught in a file name, a file body or child output.

• Sign-in is undone even mid-way. The sign-in, install and token stage stops between steps once cleanup has begun. A cancel never aborts the install registration (it keeps its own bounded timeout), and cleanup waits for that stage, with what is left of the cancel budget less 1.5 s, before it revokes the install and signs out. So an install the server committed just as the cancel came is still revoked.

• Heartbeats are checked against the daemon's run. Status's heartbeat (ms) must fall within the daemon's first 90 s, with no future times. Each status read gets only what is left of that budget. The list's (RFC 3339, or epoch seconds or ms) must fall within a minute of it.

• "Check the evidence" (always()) fails the job when no evidence was written, or when a joined machine is not shown revoked.

• The disposable user's base is root's. The base directory is root:root 0755 before useradd. The home is made empty (no /etc/skel copy, which is hundreds of MB on some runner images), so it holds only what acpmux writes. Every ancestor of the home and of the installed binary is then checked: it must be owned by root or the user, with no group or other write unless sticky, and no extended ACL. acpmux refuses a state directory with an ancestor that another user owns (config.rs validate_private_state_ancestors), so a runner-owned base would fail the join before enrollment. A root-owned base also keeps the disposable user from replacing the binary.

• A real-binary smoke check before the secrets. "Smoke-test acpmux as the disposable user" runs the built acpmux as that user with no secret in its env: host join --hub <dev> --name ci-join-smoke --json with not-a-join-token on stdin. The join takes its lock first, so it creates and checks its state directory and every ancestor before it reads stdin. The step requires exit 2 (join.malformed, before any request), fails on exit 5 (join.store, the directory refusal), and checks that ~/.acpmux is the user's, mode 700. Rehearsed on a Linux runner with a real build: root-owned base, exit 2 and a 700 state directory; the old runner-owned base, exit 5.

• "Remove the disposable user" (always()) starts from step outputs written before useradd, so it runs even if user creation fails partway. It runs kill -KILL -1 as that user, which signals exactly that user's processes and nothing else. Then it runs userdel --remove and deletes the user's base and socket directories. The VM is then discarded.

Evidence (the run summary and the only artifact, kept 3 days)

run{id,sha}, hubOrigin, name, machineId, teamId, joinTokenId, status{credentialPresent,joinIncomplete,revoked,hubReachable,heartbeatAgeSeconds}, consent{screen,control} (only none or granted; anything else is dropped and fails the check), credentialFileMode, list{machineId,label,revoked,lastHeartbeatPresent}, secretScan{home{cmj,cmm,credentialFileCmm},log{cmj,cmm},evidence{cmj,cmm}}, revoke{revokedListed,statusRevoked}, result, reason.

Tests

• scripts/cmux-next/machine-join-e2e.test.ts (bun) runs against a fake fetch, a fake sudo and a fake acpmux, with no network and no secret, on virtual time: the clock moves only when nothing else can run. It covers:
  • the passing run's exact evidence keys;
  • for every case: no secret value, cmj_, cmm_, JWT, bearer or authorization in the evidence; every child env exactly {PATH}; no secret in argv, URLs or printed lines (masks aside);
  • the token arriving on stdin only, as the disposable user, and every value masked before use;
  • origin refusal (production, staging, a trailing slash, http, other hosts), which makes no request and spawns nothing;
  • failure paths:
    • mint refused, or a mint with no joinTokenId (never joins);
    • join exit 3;
    • a join that redeemed and then failed, with the machine listed late (found and revoked);
    • a cancel during the join (waits for it, then revokes);
    • no heartbeat (still revokes);
    • a revoke that fails (retried in cleanup; the run fails);
    • a credential file without a credential;
    • acpmux printing a token (the run fails and the output is redacted);
    • a credential copy outside the credential file (the run fails).
    • a cancel while the join is still in flight: its processes are killed first, the machine it redeemed is found and revoked, and no process of the disposable user starts afterwards;
    • after a cancel, a machine listed late (with 1.5 s list reads that end at their own timeout) is still revoked before the deadline, and a search that finds nothing stops in time to leave the revoke its reserve;
    • after a cancel: slow kills and a join that ends only later (the search waits for it), a launcher the user's kills cannot reach (stopped by its own signal), and a join that never stops (reported; the search still leaves the revoke its reserve, with reads at least 0.1 s apart);
    • an id or acpmux output shaped like a workflow command (printed as plain text; every test's log is checked the way the runner parses commands);
    • a cancel during cleanup's wait for a hung daemon to exit (the wait ends, the user's processes are killed, the machine is revoked);
    • a cancel during cleanup's revoke (the request is aborted and the revoke tried again);
    • a custom name still mints with the run's label;
    • an incomplete home scan (the run fails);
    • a register whose answer arrives 1 s after a cancel (still revoked), a password echoed as consent (dropped, never in the evidence), a JSON-escaped held value in the evidence (caught), dot-only ids (refused), a credential file not mode 600 right after the join (the run fails), and a child that keeps its pipes open after SIGKILL (cleanup still ends);
    • a token-shaped machineId from the join (never printed, used in a URL or written to the evidence), a credential path outside the credential file, and a credential file that is a link (never read).
  • The sweep revokes only this account's stale ci-join-* machines.
  • Masks are escaped (%, CR, LF) and set line by line, and the stack email and password are masked before the sign-in.
• tests/test_machine_join_e2e_workflow.py pins the scope above: dispatch-only, the feat-cmux-next refusal, the token permissions, the secrets reaching one step through env, no xtrace, the dev origin, the ephemeral runner, the always() cleanup and evidence check, the root-owned base before useradd with its ancestor check, the smoke step's place before the mint, its exit handling and its plain-text output, and a single artifact with no cache.
• Wiring: both run in ci-guards.yml, the bun test in the preflight-agent-chat group (which sets up bun) and the Python test in the ci group. The workflow runs the bun test itself too, before it reaches the secrets.

Depends on

The acpmux host join stdin path and --status --json (already on feat-cmux-next), and the platform's join-token routes (on development). Nothing here waits on machine-join.sh; a second pass through that script comes later.

🤖 Generated with Claude Code

Agent session: agent session

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review fa07b29: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27453 -f ref=p0m/machine-join-e2e -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on facadd8 to compare; do not merge until green; see lane-rules

[teamleaderleo]

Owned by an agent session · session

[blacksmith-sh]

Found 7 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmp_hlo8nb2/summary-missing.json (live):/summary-missing.json (live):	View Logs
/tmp/tmpyf7e7pyd/summary-fail.json (live):/summary-fail.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/ test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Bead cx-f1uas7 created for this PR by the cmux coordinator.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex gpt-6-astra high, next-review): SHIP at fa07b29

No blocking findings.

The 12 workflow guard tests passed. Live acpmux/API compatibility remains unverified: repository MCP reads required approval, which this session disallows.

[github-actions]

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40959656345 at b844975 failed: cmux-next checks (god files, concurrency, crash safety, l10n).
Those jobs last passed at 7f9242d. This pull request is one of 3 merged since: #26942, #27453, #27534.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

[github-actions]

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4096337330601 (4400b7aaef)
• cmux NEXT nightly 1.0.0-nightly.4096585469301 (3bc5c64c46)
• cmux NEXT nightly 1.0.0-nightly.4096721596601 (e4fc50922c)