Skip to content

cmux-next: screenshot and video tours for UI pull requests - #18004

Closed
azooz2003-bit wants to merge 11 commits into
feat-cmux-nextfrom
feat-cmux-next-ci-media
Closed

azooz2003-bit wants to merge 11 commits into
feat-cmux-nextfrom
feat-cmux-next-ci-media

Conversation

@azooz2003-bit

@azooz2003-bit azooz2003-bit commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Part of the cmux-next parity work listed in #16174 (comment).

Problem

UI pull requests into feat-cmux-next have no visual evidence from CI. The old app's media pipeline (pr-media.yml, test-e2e.yml, DogfoodScenarioUITests) depends on pieces this branch deleted, and cmux-next CI only runs swift test.

What this adds

cmux-next-media.yml runs on pull requests into feat-cmux-next that touch Packages/macOS/CmuxNext/Sources, webviews/src (agent pane) or the tour files:

  1. route picks an owned Mac's gui runner (scripts/ci/cmux_next_media_runner.py, built on pr_runner_pool). It never falls back to Blacksmith, whose macOS sessions sit at a locked screen and cannot capture. Fork heads and an empty gui pool skip the tour, and the comment says why.

  2. tour builds the head with scripts/reload.sh --tag ci-media. The fixed tag keeps the mini's DerivedData warm. The job then launches the app on a private socket (the smoke-signed-app-cli.sh knobs) and runs scripts/cmux-next/media/tour.py. The tour drives the app only through the bundled cmux CLI from cmux-next: the Rust cmux CLI (cmux-tui) with acpmux and cmux acp; delete the Swift CLI #16174 and captures:

    • one screenshot per step, covering the window's screen rect so browser child windows are included;
    • a frame about every 0.5 s for the video.

    Capture runs in the console user's session, as main's E2E recorder does.

  3. publish makes an mp4 and a GIF from the frames. It uploads them and the screenshots to pr-media/<pr>/<sha8>/next-<tour>/ (the layout prune_pr_media.py already bounds) through scripts/pr-media.py's writer, and keeps one sticky comment with:

    • the GIF;
    • a step table;
    • failed steps' screenshots inline;
    • the other screenshots folded.

Tours are data: scripts/cmux-next/media/tours/*.json. A step is a title plus a cmux argv (with daemon: true for daemon verbs), wait, shot, and optional: "<why>" for features not ported yet. Other lanes add steps or a tour file. tour.py --check validates them, and the tour job runs that check too.

The starter tour covers:

  • a new workspace;
  • typing in the terminal;
  • split right and split down;
  • a new terminal tab;
  • a browser tab;
  • hiding and showing the sidebar;
  • a notification;
  • the notifications panel and agent chat. These two are marked optional because they are bindUnavailable today; they show as "unavailable" until cmux-next: agent pane hosts the React acpmux pane (New Agent Chat) #16231 and the sidebar/notifications lane land.

cmux-next has no XCUITest target, so the tour is the UI check: a failed step fails the tour job and its screenshot shows inline in the comment.

Choices to review

  • Base feat-cmux-next-acpmux, stacked. The tour drives cmux-next: the Rust cmux CLI (cmux-tui) with acpmux and cmux acp; delete the Swift CLI #16174's CLI (--app-socket, action run); on feat-cmux-next the app still bundles the Swift CLI. This stays open until cmux-next: the Rust cmux CLI (cmux-tui) with acpmux and cmux acp; delete the Swift CLI #16174 merges and GitHub retargets it, then gets feat-cmux-next merged in and re-checked. It carries 822db55 (the Swift 6.2 ResourceControl build fix already on feat-cmux-next) so the base compiles for the tour; that commit drops out of the diff after the retarget. Red checks inherited from the base (cmux-next swift test, the 239-commit gap; the fork-routing guard on cmux-next.yml:60) are not from this diff.
  • Write token: approved by Leo. The publish job holds contents: write and pull-requests: write, on same-repository heads only. It never checks out or runs the pull request's code: publish.py and pr-media.py come from the base SHA, and the tour artifact is read as data (tour folders must be slugs, shot paths must be shots/NN-slug.png regular files inside the tour folder, PNG by content, and manifest text is HTML-escaped). The route and tour jobs stay contents: read. Until the base has publish.py (this PR's own runs), the publish job notes that and leaves the media in the run artifact.

What the CI runs found

  • The gui runner works for build and launch: reload.sh --tag ci-media builds on the mini, the app answers on its private socket, and workspace new, terminal typing, tab new-browser, the sidebar toggle and notify all pass.
  • action run splitRight / splitDown / newSurface fail on this base with terminal_id must be a 32-character lowercase UUIDv4 hex value: an app-side bug the tour caught (reported on cmux-next: the Rust cmux CLI (cmux-tui) with acpmux and cmux acp; delete the Swift CLI #16174). They stay required steps, so the tour job is red until it is fixed.
  • Screen capture: the gui runner's job process has no Screen Recording permission and no passwordless sudo, so screencapture and the TCC grant main's E2E lane uses do not work there. The tour now also tries an approved CuaSshScreenCapture.app (main's release-media helper), and the comment says which route failed when none works.

Testing

  • tests/test_cmux_next_media.py (wired into the preflight guard group) covers:
    • the runner pick: a gui label for a same-repo head; nothing for a fork, owned pools off, no gui slots, another Xcode pin, or malformed slots;
    • tour validation;
    • the comment rendering.
  • scripts/ci/guards-local.sh --all passes except for failures that already exist on feat-cmux-next-acpmux (fork routing in cmux-next.yml:60, the determinism gate, the iOS conventions, the bash shell checks). None of them touch these files.
  • Live CI runs on this PR follow; I will post the first tour comment's state here.

Example output

From CI run 4 on be0c62d, published by hand with publish.py while the base has no publisher yet. Split and new tab fail on an app bug in the acpmux base (reported on #16174).

Core UI: 7 ok, 3 failed, 2 unavailable

tour

Video (mp4)

# Step Result Command
1 Launch ok
2 New workspace ok cmux workspace new --name Tour
3 Type in the terminal ok cmux terminal current write --text "echo hello from the cmux-next tour\r"
4 Split right failed: exited 1 cmux action run splitRight
5 Split down failed: exited 1 cmux action run splitDown
6 New terminal tab failed: exited 1 cmux action run newSurface
7 Browser tab ok cmux tab new-browser --url https://example.com
8 Hide the sidebar ok cmux action run toggleSidebar
9 Show the sidebar ok cmux action run toggleSidebar
10 Notification ok cmux notify --title Tour --body "Hello from CI"
11 Notifications panel unavailable: exited 1 (the notifications panel is not ported yet) cmux action run showNotifications
12 Agent chat unavailable: exited 1 (agent chat views are not ported yet) cmux agent new-chat

Failed steps:

4. Split right
Split right

5. Split down
Split down

6. New terminal tab
New terminal tab

Screenshots (8)

1. Launch
Launch

2. New workspace
New workspace

3. Type in the terminal
Type in the terminal

7. Browser tab
Browser tab

8. Hide the sidebar
Hide the sidebar

10. Notification
Notification

11. Notifications panel
Notifications panel

12. Agent chat
Agent chat

🤖 Generated with Claude Code

— Zoetrope g1 🗝️ · run run_cc_next_ci_media_20261001_e712de82


Migrated from #16427 after correcting the PR author identity. The original head commit aba3a6d is preserved.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

teamleaderleo and others added 11 commits October 1, 2026 10:44
A new cmux-next-media workflow builds the head as a tagged dev app on an
owned Mac's gui runner, launches it on a private control socket, drives a
tour through the bundled cmux CLI and captures a screenshot per step plus
frames for a video. The publish job uploads them to pr-media/<pr>/<sha8>/
and keeps one sticky comment on the pull request.

Tours are JSON in scripts/cmux-next/media/tours so other lanes can add
steps. The runner is picked by scripts/ci/cmux_next_media_runner.py and
never falls back to Blacksmith, whose sessions cannot capture.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rget function)

ResourceControl.run declares `let target = try await MainActor.run { try
target(...) }`. Swift 6.3 (Xcode 27) resolves the call to the static
function; Swift 6.2 (Xcode 26.3 and 26.6, which the fleet workers use)
resolves it to the local being declared and fails with "generic parameter
'T' could not be inferred", so every fleet build of feat-cmux-next since
3d9c39f failed (fleet job 20ae3ebca821c9f69f0b857f). Call Self.target.
Reproduced and checked with a minimal file under Xcode 26.3 swiftc.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 822db55)
…ne and tab steps by action id

Run 36903298606: every capture failed with "could not create image from
rect", and pane split-right / tab new-terminal were taken by the daemon
grammar as terminal ids, so those steps run their app actions by id.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… data, capture through the cua-ssh helper

The publish job holds the pr-media write token, so it checks out its
scripts from the base SHA and validates tour names and shot paths taken
from the artifact. Run 36904395488 showed the gui runner has no Screen
Recording and no passwordless sudo, so the tour also tries an approved
CuaSshScreenCapture.app (main's release-media helper backend), and the
comment says why when nothing could be captured.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…re, inert comment text)

- tour.py: SIGTERM unwinds through cleanup; the app stops before the
  recorder; a leftover app or daemon from a cancelled run is stopped
  before launch; recording starts once the window is on screen; shots
  never fall back to the whole shared display; the probe image is
  deleted; the helper runs in the console session; the crop rect is
  clamped to the display; the recorder's frame list is locked.
- publish.py: manifest steps are normalized to plain strings and ints,
  HTML, Markdown and @mentions are escaped, a bad tour is reported
  instead of failing the job, the video has a fixed size, and the head
  is checked again right before the comment is written.
- The typing step runs before the new workspace so it lands on screen.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…failing the publish

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e app stops, signal-safe cleanup)

- A newline or CR in manifest text or a command can no longer end the
  code span or the table row.
- The recorder stops before the app is killed, so no frame shows what
  was behind the window; a window off the display clears the rect.
- Cleanup ignores further signals, and the step execs python so the
  runner's cancel signal reaches it.
- The helper's raw desktop image is written outside shots/ and frames/.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e window rect

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ci-media tag keeps its daemon state in Application Support across
runs, so run 5 launched into run 4's Tour workspace and browser tab.
--fresh-state deletes that tag's state before and after a tour; it is
opt-in so a local run against a developer's tag keeps their workspaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…und ffmpeg

Run 6 still launched into two earlier Tour workspaces: the daemon
outlives the app and keeps its workspaces, so deleting the state
directory under it changed nothing. The tour now ends the tagged
daemon through its socket (server stop --end-terminals) before the
pkill fallback and the state removal. An untagged build's daemon is
the user's and is never stopped.

Run 6's publish job spent its 15 minutes in apt installing ffmpeg.
The install is now capped at 4 minutes and may fail (the comment then
has screenshots and no video), and nothing installs when the base
revision has no publisher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stuck tour recorded a frame every half second for as long as it ran.
The recorder now stops at five minutes of frames and the publisher's
ffmpeg reads no more than that, so neither the runner nor the publish
job grows with a slow tour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 6, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@coderabbitai

coderabbitai Bot commented Oct 6, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 53576044-1b70-46d0-a4b5-c8e766de2b7c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants