Skip to content

ci(cmux-next media): let tours run side by side on one Mac - #22046

Merged
teamleaderleo merged 18 commits into
feat-cmux-nextfrom
ci/tour-concurrent
Oct 20, 2026
Merged

teamleaderleo merged 18 commits into
feat-cmux-nextfrom
ci/tour-concurrent

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Media tours can now run several at once on one Mac, each capturing only its own app's window. That lets the runner hook drop its one-tour-per-Mac limit.

What changes:

  • The tour no longer activates its app, and launches it with CMUX_NEXT_NO_ACTIVATE=1. It drives the app through the app's own control socket and screenshots it with debug.window_snapshot, so being in front never mattered. Concurrent tours would otherwise trade focus and the key window back and forth.

  • The fallback captures the window, not the screen. A build without debug.window_snapshot used to be captured with screencapture -R, a rectangle of the display, which would include whatever covers the window, such as another tour's app. It now captures that window alone: screencapture -l <id>, or the screen-capture helper's window <id>. The window server id comes from the window list by the app's pid. With no id there is no shot, never the display. The manifest records capture_window. This route leaves out child windows (sheets, popovers) that debug.window_snapshot includes; it is only for old bases without that verb.

  • A tour holds its tag (scripts/ci/media_tag_lock.py, new step "Hold this tour's tag"). One tag's warm DerivedData, sockets and daemon session take one tour at a time; other tags never wait. The concurrency group already keeps two runs of a pull request apart, but not these cases:

    • a cancelled run that is still stopping;
    • a dispatch;
    • a merge tour beside a dispatch, which share the feat-cmux-next tag.

    The lock lets go when the job's Runner.Worker exits, or after 3 h. A second tour of a tag waits up to 60 min: longer than a warm merge tour holds the shared tag (54 min seen), while leaving the waiter 45+ min of its 120-min job for a cold build and the tour. A longer wait would only end in a job-timeout cancel instead of the "still held" error. When the pull request head predates the lock, the step takes the lock script and the lock-aware prune from the run's commit, so unrebased PRs still run and their prune never deletes a held tag.

  • The DerivedData prune skips a held tag and holds a tag's lock itself until its dir is gone, so a tour never builds into DerivedData that is being deleted, or has its DerivedData deleted by another pull request's tour.

What still needs the console, and stays on the console runner, is unchanged here: app-host XCTest (testmanagerd), E2E video recording and frame-pacing benches.

Validation

  • tests/test_cmux_next_tour.py and the new tests/test_media_tag_lock.py (linux-guard lane), red then green:
    • At 3b6b740 (tests only) against the old tour.py and with no lock script, test_the_tour_never_activates_the_app and test_without_a_window_snapshot_the_tour_uses_a_screen_route fail, and so do 3 of the 4 lock tests.
    • At ff2f66f all 4 lock tests pass, and so do the tour's activation, screen-route and app-window tests.
    • At 1286a4a (tests only) the prune test fails: the old prune has no --lock-dir. At 7468a34 it passes, and so do the lock and tour tests.
  • actionlint .github/workflows/cmux-next-media.yml is clean.
  • At 071bcb0 (test only) the wait test fails against the 7200 s wait (7200 not less than or equal to 4500); at bd085a7 it passes. tests/test_media_tag_lock.py was in the linux-guard lane but no workflow ran it; ci-guards.yml now runs it in the release-notary group. All four tour/lock/prune tests pass on Huge Blue at bd085a7.
  • Screenshots now show the app's window as not key (CMUX_NEXT_NO_ACTIVATE), so focus styling looks inactive in tour media.
  • Missing ExplicitPrecompiledModules/Darwin-*.pcm tour failures (14 of 22 recent failed tours) are not from the prune: every one I checked (jobs 118285853270, 118197207461, 118076888558, 118006980751) followed media_module_cache_stamp.py clear removing ExplicitPrecompiledModules after a binary pin change, while the build's dependency records still point at those .pcm files. The prune only removes whole idle cmux-ci-media-* dirs, and now only under the tag lock. fix(cmux-next media): a pin change also drops Xcode's build description #22181 fixes that clear.
  • Base reds on this head, not from this PR: release-notary fails NEXT post-publication dependency contract the same way on feat-cmux-next (job 118292665776), and cmux-next checks fails the same six checks (state roots, concurrency, strings, type scale, script tests, web bundle key) as merged agent pane: permission UI keeps its last state across a chat switch (no blink) #22054 (job 118309312276), being fixed in cmux-next: clear every base red in cmux-next checks (type scale, strings, tree key, state root, concurrency, bundle key) #22117.
  • Old PR heads run their own tour.py, which still activates the app and captures a rectangle, until they merge main.
  • Not run yet: a tour on a Mac with two tours in flight. The runner-side change that lets them share a Mac follows this PR.

Runtime test: tests/test_media_tag_lock.py, tests/test_ci_prune_media_derived_data.py and tests/test_cmux_next_tour.py in the release-notary/preflight guard jobs at bd085a7

Changelog

none

Review: SHIP at bd085a7 (cmux-review subagent; correctness first)

🤖 Generated with Claude Code

teamleaderleo and others added 2 commits October 19, 2026 00:46
The tour never activates its app, a build without a window snapshot is
captured by its window server id rather than a rectangle of the display,
and a tour holds its tag for the rest of its job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tour.py no longer activates the app it drives through its socket, and its
fallback for a build without debug.window_snapshot captures that window
alone (screencapture -l), never the display. A new step holds the tour's
tag (media_tag_lock.py) so one tag's DerivedData, sockets and daemon take
one tour at a time while other tags run beside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 040b1e8d-ece4-4e33-a1c2-ce3790349775

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 5bfa69b: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr22046 -f ref=ci/tour-concurrent -f platform=macos -f backend_mode=local

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-jf8t2 created for this PR by the cmux coordinator.

The app refuses activation itself, so tours side by side never take focus
or the key window from each other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blacksmith-sh

This comment has been minimized.

teamleaderleo and others added 2 commits October 19, 2026 01:04
Red: prune_media_derived_data.py has no --lock-dir and removes a held
tag's base and head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ld it

- prune skips a tag whose lock a tour holds, so a tour of another pull
  request never deletes DerivedData a tour is about to build into
- the hold step fetches media_tag_lock.py from the run's commit when the
  pull request head predates it, and waits at most 30 minutes
- the last-resort window capture picks the on-screen window that matches
  the app's frame

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent, correctness first) on f0fe23e, and what 7468a34 does about each finding:

  1. Unrebased PRs would fail the new hold step, since their head has no media_tag_lock.py. Fixed: the step fetches the script from the run's commit when it is missing.
  2. screencapture -l leaves out child windows (sheets, popovers). Kept, and documented: it is only the last-resort route for bases without debug.window_snapshot, which includes them.
  3. The window lookup could pick an off-screen or helper window. Fixed: on-screen windows only, layer 0, matching the app's frame within 1 pt, else the largest.
  4. The DerivedData prune could delete a tag another tour is about to build into. Fixed: prune skips any tag whose lock is held (new scenario_held_tag, red at 1286a4a). The hold waits at most 30 min.
  5. Test holders outlived their tests. Fixed: they now let go after 4 s.

defaults-snapshot is red from the base, not this PR.

@blacksmith-sh

blacksmith-sh Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
NEXT post-publication dependency contract/NEXT post-publication dependency contract View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@greptile-apps

greptile-apps Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes affect shared Mac resources used by the cmux-next media-tour workflow.

Fix the display-shot caller before merging; it currently stops tours that request a display shot.

Findings

  1. P1 Display shots stop tours ▶

Summary

Allows media tours to share one Mac while keeping each tour's captures and build files separate.

  • Tours leave focus alone and capture only their app’s window.
  • Tours of one tag take turns while different tags run together.

Both earlier unnumbered findings are fixed: the prune keeps its lock through deletion, and older heads receive both lock-aware scripts.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Tour starts] --> B[Acquire tag lock]
  B --> C[Prune idle DerivedData]
  C --> D[Build base and head]
  D --> E[Launch without activation]
  E --> F[Capture app window]
  F --> G[Job ends and releases lock]
  C --> H[Skip tags held by other tours]
Loading

Reviews (6) · Last reviewed commit: "Merge branch 'feat-cmux-next' into ci/to..." · Reviewed by Greptile

Comment thread scripts/ci/prune_media_derived_data.py Resolved
Comment thread .github/workflows/cmux-next-media.yml Resolved
teamleaderleo and others added 7 commits October 19, 2026 13:17
The prune probes a dir's tag lock and lets it go before its rmtree, so a tour can take the tag and
start building into a dir that is being removed. This drives the real prune main with the real
media_tag_lock.py hold asking for the tag inside the delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claim_tag takes the tag's lock (creating it, as media_tag_lock.py does) without waiting and keeps it
across allocated_gib and rmtree. A tour that asks for the tag meanwhile waits on its lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…b can run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dispatch of feat-cmux-next waits behind a merge tour that holds the tag
for up to an hour warm (54 min seen) and longer cold; 30 min failed it.
Wait the tour job's 120 min timeout instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A head branched before the tag lock has a prune that ignores held tags, and
the prune step ran it from the head. The hold step now takes both scripts
from the run's commit for such a head and hands their dir to the prune step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 19, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Pushed 82b9d03 (merges feat-cmux-next 86efbba):

  • d013c33 / 41f9b19 (red first): a tour waits for a held tag up to the job's 120-min timeout, not 30 min. Merge tours ran up to 54 min in the last week.
  • c40e202 / a77a158 (red first): an unrebased head takes the lock-aware prune from the run's commit too (greptile's second thread).
  • 1b8a89f / e7d0698: the prune holds the tag lock until rmtree returns. Its test was red before the fix.

Reds on 1b8a89f that aren't this PR's:

Local runs on Huge Blue: test_media_tag_lock.py, test_ci_prune_media_derived_data.py and test_cmux_next_tour.py all pass. actionlint is clean.

teamleaderleo and others added 2 commits October 19, 2026 14:31
tests/test_media_tag_lock.py sits in the linux-guard lane but no workflow ran
it, so its tour-wait and prune-lock checks never executed in CI. The
release-notary group now runs it next to the prune test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The job timeout counts from job start, so a 120-min wait for a 120-min job
is never reached: a starved tour is cancelled instead of failing with
"still held", and one that gets the tag late times out mid-build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The wait ends while the waiter still has time to build and tour. The
old-head fallback fetches the run's commit without --depth so the reused
self-hosted checkout does not turn shallow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

For whoever lands this: #22174 is built on this branch's head (82b9d03) and replaces the per-PR tag with a per-Mac slot claimed through this PR's lock (media_tag_lock.py claim, tags ci-media-slot<N>). It doesn't change hold or the prune's lock handling, so please land this one first and I'll merge feat-cmux-next into #22174 afterwards. Also #22170 fixes the 'module file …Darwin-*.pcm not found' tour failures (14 of 22 today). Those come from #21971's partial clear, not from this PR.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Re-review (subagent, correctness first) of the commits since the last review: no defects. SHIP at .

@teamleaderleo teamleaderleo added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 20, 2026

def use_display(self) -> None:
"""Capture the window's rect on the display from now on; follow() again after."""
def use_display(self, pid: Any) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Display shots stop tours

Screen.use_display() now requires pid, but display_capture() still calls it without arguments. When a screen capture route is available, a step with display: true raises an unhandled TypeError and stops the tour. The standard core.json tour hits this through its Profile menu step, even though that step is optional.

Update display_capture() to pass the app's PID.

@teamleaderleo
teamleaderleo merged commit 9d0cbeb into feat-cmux-next Oct 20, 2026
@teamleaderleo
teamleaderleo deleted the ci/tour-concurrent branch October 20, 2026 01:02
teamleaderleo added a commit that referenced this pull request Oct 20, 2026
…ur-slots

Conflicts came from #22046 (squash-merged into base) and #22170. Each
conflicting file is base's version plus this PR's own slot-claim diff:
- cmux-next-media.yml: the claim step replaces base's one-hour hold; the
  old-head fallback keeps base's fetch without --depth.
- test_ci_media_module_cache_stamp.py: #22170's docstring and warm rebuild
  check stay; the slot tours run in the slot-numbered dirs.
- test_media_tag_lock.py: the claim tests replace base's hold --wait test,
  since a tour no longer waits for a tag.
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

teamleaderleo added a commit that referenced this pull request Oct 21, 2026
… tour (#22717)

All merge (push) runs shared one concurrency group. Each merge waited behind
the running tour, and the next merge replaced the waiting run: 95 of the 100
cancelled cmux-next-media runs on 2026-10-20 were zero-job push runs replaced
this way, and each lost its tour and its PR comment. Every merge now gets its
own group; tours already run side by side in a Mac's slots (#22046), and a
merge with nothing for a Mac to tour still ends in the route job.

Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=e8f63fd9-94e8-4878-bcd4-704659aba1f2&host=cosmic-gray
teamleaderleo added a commit that referenced this pull request Oct 21, 2026
…id, reload shim test, fleet-test-compile runners) (#23553)

* fix(cmux-next media): a display step's shot passes the app's pid to the screen route

Screen.use_display takes the app's pid since #22046 (it finds the window
server id by owner pid), and run_tour passes it; display_capture still
called use_display() with no argument, so every `display` tour step died
with a TypeError. tests/test_cmux_next_tour.py's display-step cases were red
on feat-cmux-next (preflight guard group).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15c31-89b3-7d62-8e8c-b116a23ae693&host=cosmic-gray

* fix(ci): the reload DerivedData test loads the shim writer from scripts/lib/cmux-dev-shim-install

reload.sh now sources write_dev_cli_shim and reload_socket_is_live from
scripts/lib/cmux-dev-shim-install (the one source of the managed shim), so
the test's awk extraction from reload.sh found nothing and failed with
"write_dev_cli_shim not found in reload.sh" on feat-cmux-next (preflight
guard group). The test sources the library as reload.sh does, and checks
reload.sh still sources it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15c31-89b3-7d62-8e8c-b116a23ae693&host=cosmic-gray

* fix(ci): fleet-test-compile.yml runs on Blacksmith, not a GitHub-hosted label

#23423's plan job and its empty-plan compile leg selected ubuntu-24.04, which
tests/test_ci_self_hosted_guard.sh refuses (a GitHub billing block must not
stop CI): "Validate macOS runner guards" was red on feat-cmux-next (preflight
guard group). Both take the standard selector: the fork branch, then
vars.LINUX_RUNNER with the Blacksmith fallback; a needed plan still takes
vars.CI_COMPILE_CHECK_GATEWAY.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15c31-89b3-7d62-8e8c-b116a23ae693&host=cosmic-gray

* fix(ci): fleet-test-compile.yml sends fork pull requests to Blacksmith before any runner variable

The previous commit put vars.LINUX_RUNNER (and the compile gateway) right after
the owner branch, so tests/test_ci_fork_runner_routing.py flagged both lines: a
fork pull request could reach an owned runner label. Both selectors now use the
standard form (agent-brands.yml, backend.yml): the fork pull-request branch
first, then the gateway and LINUX_RUNNER.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15c31-89b3-7d62-8e8c-b116a23ae693&host=cosmic-gray

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant