[teamleaderleo]

Leo asked (2026-11-04):

let's also get our iteration loop good for the whole "working on cloud" type thing

Getting a NEXT Cloud chat working end to end took days, because every check waited on an app build or a nightly. This PR adds one command that checks a backend end to end with no app build, and runs it on every PR preview and after every staging and production deploy.

What changed

• scripts/cmux-next/cloud-journey.ts: bun scripts/cmux-next/cloud-journey.ts --backend local|dev|preview-N|staging|prod (or --api URL --stack dev|prod). It signs in as the agent test account and walks the routes the app and CLI use on the cmux API Worker:

  • health, sign in, user.ensure, install token
  • create machine, running, listed, first exec
  • start agent (cmux-tui acp ensure on the machine), send chat, reply
  • connect (connect_info + link_token), handoff record, a folder round trip
  • pause, resume, exec after resume, delete

  Each step prints PASS/FAIL/SKIP and its latency. --out DIR writes journey.json and summary.md. The machine it made is deleted at the end, pass or fail. --team runs as another team, and --cmux PATH sends every exec through cmux machine access exec, so the journey also checks the CLI. A failed machine fails its step at once, with the row's error.

• .github/workflows/backend.yml:

  • journey-preview runs after the preview job, against cmux-api-preview-<PR>. Its step table is the job summary, and a failed step makes that PR's check red.
  • journey-staging and journey-production run after each deploy (including promote). They run as the test team and, when CMUX_JOURNEY_PRODUCT_TEAM is set, as a product team on the never-pause path.
  • Secrets: staging reads the pair cloud-chat-proof.yml already uses (cmux-next-vm-image-dev). Preview and production read cmux-next-cloud-journey (CMUX_JOURNEY_DEV_STACK_*, CMUX_JOURNEY_PROD_STACK_*). Without them, the job posts a notice and passes.
  • The journey script's own paths trigger the workflow.

• scripts/cmux-next/cloud-dev-loop.sh --tag NAME [--platform-checkout DIR]: the inner loop. It deploys this worktree's API Worker as your own cmux-api-preview-NAME and, if given, a cmux-platform worktree as cmux-platform-preview-NAME. Then it runs the journey there and prints the seconds for each part.

• backend/scripts/deploy-worker.sh: CMUX_PREVIEW_PLATFORM_WORKER=cmux-platform-preview-<n> points a preview's PLATFORM service and CLOUD_DO at a platform preview instead of development.

• --app-cli PATH --socket SOCK runs the commands a person runs, through a running NEXT build's own CLI: cloud list-machines, vm token create, cloud new-machine, machine-status, exec, start-session, start-chat --wait, and kill-machine (always, at the end). It is a mode of the same script rather than the scripts/cloud/cli-journey.sh the issue sketched, so each step gets the same timing and table. The backend named per step comes from the answer when the answer gives one.

Closes #35086.

First result

Production, from a team Mac, as the production test account: health, sign in, user.ensure, install token and create passed (55 ms to 1.8 s each). Then the machine went to failed after 34 s, both times, with cloud.provider.unavailable: VM API: snapshot capture is still being confirmed. That is a live production bug that no existing smoke catches, and it is handed to its own lane in cmux-platform.

Notes

• The repo note says not to add CI jobs. Leo asked for exactly these: the journey on previews and as the deploy smoke.
• The preview journey sends the dev Stack test account's short-lived session to the PR's own preview Worker. That happens only on same-repo PRs, which the preview job already limits to, and with a test account that holds no data.

Observed / Not run

• Ran: two production journeys (steps above); actionlint and a YAML parse of backend.yml; prepush-gate.
• Not run here: cloud-journey.test.ts (sent to the fleet), and the staging, preview and local backends. This host has no dev-project pair, so those first run in CI.

Changelog

none

Agent session: agent session

[teamleaderleo]

Owned by an agent session · session

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[blacksmith-sh]

Found 8 test failures on Blacksmith runners:

Failures

Test	View Logs
/tmp/tmp48g8dymi/summary-fail.json (live):/summary-fail.json (live):	View Logs
/tmp/tmpxid26ckz/summary-missing.json (live):/summary-missing.json (live):	View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/ [new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows	View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames	View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/ [tabs] debug.tab_switch timed 0 of 40 switches	View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames	View Logs
nightly cmux-tui resolve/nightly cmux-tui resolve	View Logs
the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac k/ the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac k	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (subagent opus-5.5 medium, correctness): SHIP at 3a56cd8
Visual verdict: no UI in this diff.
Checked secret scrubbing on every output, delete-in-finally, the three backend.yml jobs (fork exposure, environments, needs/if, timeouts, runs-on), the deploy-worker.sh rebind and the no-secrets notice path. No blocking finding.
Follow-ups (filed as one issue, owned by this PR's lane): F1 no-credentials run reports PASS; F2 cancel/lost-reply can leak a machine (SIGTERM handler or jr-* sweeper); F3 delete wait never sees "gone" (api.read throws); F4 cleanup delete reuses the in-flow idempotency key; F5 deploy-worker.sh name check should be an exact charset; F6 confirm prod Stack accepts empty client key (local prod runs sign in fine); F7 journey-staging on dispatch from other refs if the env gains a branch policy. outcome:pending

[teamleaderleo]

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at 3a56cd8

BLOCK at 3a56cd8
Visual verdict: no images; Leo tries the build

1. [P1] [blocking] [category:tests] [finding:F1] [outcome:pending] cloud-journey.ts:448: The handoff step cannot pass against the checked-in backends. Development/staging require sessionId, which this request omits, and production intentionally lacks the handoff bindings and returns 404. Both responses become recorded failures, making the configured CI journeys red even on healthy backends. Send the required payload and handle production’s unsupported capability explicitly.

Follow-ups:

2. [P2] [follow-up] [category:correctness] [finding:F2] [outcome:pending] cloud-journey.ts:273: The generated app CLI machine name contains a space. CloudMachineRequest.isPlainName rejects spaces, so this mode always fails at creation. Use a valid name such as journey-${J.meta.tag}.

3. [P1] [follow-up] [category:reliability] [finding:F3] [outcome:pending] cloud-journey.ts:275: The app returns pending: true without a machine ID when creation exceeds its 30-second answer deadline; creation continues afterward. This branch throws while machine remains null, so cleanup never runs and the eventual machine remains allocated. Track the creation by its unique name or creation ID until its machine can be cleaned up.

4. [P2] [follow-up] [category:tests] [finding:F4] [outcome:pending] cloud-journey.ts:356: Missing credentials throw outside J.step, and the outer catch discards the error. With a healthy health endpoint, the command prints PASS, writes a passing summary, and exits 0 without running the journey. Record this failure and return nonzero.

5. [P2] [follow-up] [category:tests] [finding:F5] [outcome:pending] cloud-journey.ts:218: Freestyle uses null or missing statusCode for a command timeout; the existing CLI maps that to exit 124. This implementation maps it to 0, allowing partial expected output to pass an exec step after timeout. Preserve the CLI’s timeout handling.

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review e3941d9: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr35103 -f ref=lane/cloud-journey -f platform=macos -f backend_mode=local

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4182499138501 (e0aa712b3b)
• cmux NEXT nightly 1.0.0-nightly.4184087390301 (08052d331e)
• cmux NEXT nightly 1.0.0-nightly.4184224384601 (a3abfc4160)
• cmux NEXT nightly 1.0.0-nightly.4184409870901 (78ee0e108e)

[teamleaderleo]

Review (codex gpt-6-astra high, review box): SHIP at e3941d9
Visual verdict: no images; Leo tries the build
Checked the fixes for the BLOCK at 3a56cd8: the handoff payload with a SKIP on 404, the exec timeout mapped to 124, the app CLI machine name, and pending-create cleanup. Also checked secrets, machine cleanup and the workflow jobs. Nothing blocking.
Follow-ups (added to cmux#35132):

• F1 [follow-up] cloud-journey.ts:270: the pending-create lookup matches name, but the CLI list returns title. outcome:pending
• F2 [follow-up] cloud-journey.ts:495: run the handoff fetch inside the step, so a transport failure gets recorded instead of skipping the remaining checks. outcome:pending