Skip to content

cmux NEXT: agents can screenshot windows, panes and tabs; browser pages come back masked - #23972

Merged
teamleaderleo merged 26 commits into
feat-cmux-nextfrom
live-qa-masked-screenshots
Oct 30, 2026
Merged

teamleaderleo merged 26 commits into
feat-cmux-nextfrom
live-qa-masked-screenshots

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Agents and scripts can capture windows, panes and tabs with cmux window|pane|tab screenshot. Automation receives an image with browser chrome, tab strips, chats, saved chat frames and unchecked app pages filled opaque gray. Raw page capture remains restricted to the browser host's guarded path.

The screenshot uses the live view tree and samples mask regions across capture. It omits child/floating windows and refuses an image when masking fails or geometry does not settle. Every control-socket invocation is marked as automation, including one claiming origin: user; only an unmarked in-app user invocation without caller metadata receives raw pixels.

The existing screenshot mask from #23983 is consolidated into ScreenshotMask. Geometry/timing live in CmuxNextScreenshots; app-specific view rules remain in AppWindowImage.swift.

Review fix

Requested: “mask the strip, or mask by page-derived content instead of by one view” and “add a test where a tab title is masked in a window and a pane capture”.

TabStripView is masked independently of BrowserChromeView, covering the sibling strip in the default pane layout. The separate saved-launch-image rule remains reachable. Regression coverage verifies raw title/launch sentinels are visible, then masked in full-window and offset pane captures; a separate geometry test requires both mask regions. Crop assertions account for the pane's flipped coordinate system.

Validation

  • Clean base merge to 7d7ec329, then 5d6d95a5 fixed two fail-open review findings: unchecked native page tabs are now masked whole (shownNativePages), and an area that moves between readings now refuses the capture.
  • Display-worker fleet run (CMUX_TEST_REQUIRE_GUI=1) at 5d6d95a5: MaskedScreenshotTests 12/12 and ActionOriginTests 11/11 pass. On base, ActionOriginTests 11/11 pass.
  • Independent Astra review (high, fast): SHIP at 5d6d95a5.

Risk: high, because this changes automation screenshot privacy. Each capture owns a sampler stopped with defer; settling uses a one-shot timer, and capture retries stop after three attempts. Frames are compared within one sampler only.

Proof

No capture: this changes the returned automation image rather than app UI. The display regression captures a real window and checks raw versus masked pixels.

Changelog

Changed: agents and scripts can screenshot windows, panes and tabs with sensitive page and chat regions masked.

Agent session: agent session

…es come back masked

Socket runs of window.screenshot, pane.screenshot and tab.screenshot are no
longer person-only. Any run the control socket carries (whatever origin it
claims) gets the window without child or floating windows, with every
browser page area, agent chat and conversation filled with an opaque mask
read from the view tree on every run loop turn around the capture. A window
that keeps animating or resizing, or a mask that cannot be painted, refuses
the capture. A browser tab no window shows stays refused to automation.
Design mode and the raw page captures stay person-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16047-2854-7333-9aff-632422c609d7&host=cmux16
@teamleaderleo teamleaderleo added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 22, 2026
@teamleaderleo

teamleaderleo commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review a752545: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23972 -f ref=live-qa-masked-screenshots -f platform=macos -f backend_mode=local

Not verified: macOS jobs did not run: compile and app tests were skipped on a752545cb5; ci-status is green without them.

CI passes on a752545cb5 (run 40979183585 attempt 1).
macOS jobs did not run: compile and app tests were skipped: Fast static checks skipped (the macos job needs both to succeed). Re-run CI once they pass, before merging.

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

red: cmux-next checks (god files, concurrency, crash safety, l10n); NEW: green on the base 79cdae2; do not merge until green; see lane-rules

@github-actions github-actions Bot added risk:normal Risk tier (docs/ci/risk-tiers.md) and removed risk:high Risk tier (docs/ci/risk-tiers.md) labels Oct 22, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Found 8 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp4a_lyxye/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpy858kw7w/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
test_list_supported_matches_schema_settings_paths (main.SupportedPathsTests.test_li
st_supported_matches_schema_settings_paths)/
test_list_supported_matches_schema_settings_paths (main.SupportedPathsTests.test_li
st_supported_matches_schema_settings_paths)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next crash-class rules / cmux-next crash-class rules is red on 11 of 11 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

teamleaderleo and others added 2 commits October 21, 2026 18:32
…ug.toast runs

Review findings on the masked socket capture:
- App pages are masked unless listed in ScreenshotMask.shownAppPages (keys,
  changelog, icon picker, apps, diff, markdown, editor, files, debug
  settings), so Settings, Passwords, History, Home Channels, Agent Activity,
  the Chief pages and any new page are masked by default.
- The Home list, dock, draft and transcript views are masked too (the list
  shows each conversation's latest message).
- debug.toast's runs are socket runs: a screenshot it starts stays masked.
- The handler's raw-or-masked choice is tested through
  ScreenshotHandlers.shown with real invocations, pane crop included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16047-2854-7333-9aff-632422c609d7&host=cmux16
…fence

withExtendedLifetime with an async body does not compile on CI's SDK. The
header now names everything a socket capture masks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16047-2854-7333-9aff-632422c609d7&host=cmux16
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-klr1d created for this PR by the cmux coordinator.

teamleaderleo added a commit that referenced this pull request Oct 22, 2026
Two open PRs (#23972, #23983) rework cmux window screenshot in
AppWindowImage and WindowImageLayer. Leave both files as on base: the
overlay cut is an NSWindow helper painted as an ordinary window layer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Orphaned: no activity for 16 h, and no lane holds a lease on it. The orphan sweep could not merge feat-cmux-next in: conflicts in Packages/macOS/CmuxNext/Sources/CmuxNextActions/Catalog/PaneActionCatalog.swift, Packages/macOS/CmuxNext/Sources/CmuxNextActions/Catalog/TabActionCatalog.swift, Packages/macOS/CmuxNext/Sources/CmuxNextActions/Catalog/WindowActionCatalog.swift, Packages/macOS/CmuxNext/Sources/CmuxNextApp/Screenshots/ScreenshotHandlers.swift, Packages/macOS/CmuxNext/Sources/CmuxNextApp/Screenshots/ScreenshotStrings.swift, Packages/macOS/CmuxNext/Sources/CmuxNextApp/Screenshots/WindowImageLayer.swift, Packages/macOS/CmuxNext/Tests/CmuxNextControlTests/ActionOriginTests.swift. Listed for a lane to resolve.

…nshotPrivacyMask

#23983 landed the same feature (agents screenshot the app, browser pages
blacked out) five hours after this PR opened, so base now has a second, weaker
mask. Resolved by keeping this PR's ScreenshotMask, which does everything
ScreenshotPrivacyMask does and more, and deleting the duplicate:

- the mask comes from the window's view tree, so a page a pane holds while its
  next tab paints, a sleeping page's picture and a page under no chrome view are
  covered; chats, Home views and every app page not in `shownAppPages` too;
- it samples each run loop turn across the capture, retries an unsettled window
  and refuses rather than return pixels it could not mask;
- `viaControlSocket` marks every socket run, so a caller claiming `origin: "user"`
  no longer gets the raw image (base's `applies(to:)` let it through).

Kept from #23983, which this PR's `isPersonsRun` did not cover: a run that names
a caller (`callerApp`, `callerAgentSession`, `callerTerminal`) is masked too,
even carrying `user` from the person's gesture inside it.

Takes back #23983's refusal of pane/tab screenshots over the socket when the
pane or tab shows a browser: masking the page area whole keeps that guarantee
without the refusal, which is this PR's point. `handlers.screenshot.browserNeedsREPL`
went with it.

Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a174ff-00c4-7cf2-b0d3-58b1149a6809&host=huge-blue
…pplies the view rules

The app-size guardrail refuses new logic in CmuxNextApp, and refuses appending
it to an existing App file too. ScreenshotMask could not move whole: it names
BrowserChromeView, AgentPaneView, the Home views, PageWebView and
PaneContentView, which the App owns until the Workbench rows of the refactor
plan move.

So the geometry and the timing (the regions, the settle wait, the attempts, the
run-loop Sampler, whether a view's pixels are where its frame says) go to
CmuxNextScreenshots beside WindowImageLayer, and the view types stay in the App
as ScreenshotMask.Rules.app in AppWindowImage.swift: covers, area,
isAppPage, coversContents. CmuxNextScreenshots gains one dependency,
CmuxNextWakeups, for the DemandTimer the settle wait uses.

Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a174ff-00c4-7cf2-b0d3-58b1149a6809&host=huge-blue
@teamleaderleo teamleaderleo added risk:high Risk tier (docs/ci/risk-tiers.md) and removed risk:normal Risk tier (docs/ci/risk-tiers.md) labels Oct 25, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

cmux-next guards / nightly-guards / workflow-guard-tests / release-notary is red on 17 of 20 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

The fleet compile at 018aac1 found the two files the move left without it:
ScreenshotHandlers' `catch let failure as ScreenshotMask.Failure` and
ScreenshotStrings' `reason(_ failure: ScreenshotMask.Failure)`.

Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a174ff-00c4-7cf2-b0d3-58b1149a6809&host=huge-blue
@cursor

cursor Bot commented Oct 25, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@github-actions github-actions Bot removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 25, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (Codex gpt-6-astra high fast, review-router): SHIP at 331c0be

Independent final-head review: no blocking findings. Reviewed full diff and supporting code, including privacy masking, socket-origin handling, tests, and base-merge correctness. Static checks passed; macOS display tests were not run in this environment.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): BLOCK at 331c0be

  • Packages/macOS/CmuxNext/Sources/CmuxNextApp/Screenshots/AppWindowImage.swift:102 — The mask covers only BrowserChromeView. In the default layout, PaneContentView.stripView is a sibling outside that view, so browser tab titles/badges remain unmasked in window and pane screenshots. Those values come from the page and can contain secrets, violating the PR’s automation privacy guarantee now that browser pane screenshots are allowed.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (Codex gpt-6-astra high fast, review-router): SHIP at 5aba021

Independent final-head verdict: the exact head masks TabStripView independently (including sibling strips), masks PaneContentView.launchImageView saved frames, applies masks before both window and pane crops, fails closed for unsettled windows or unpaintable masks, has valid CmuxNextWakeups and CmuxNextTabs dependencies, and covers full-window and pane-crop masking. No blocking findings; static review only.

…hose areas move

An independent review at 7d7ec32 blocked on two fail-open paths:
- a native page tab (InternalPageView without a PageWebView: Phone and Mac
  Screen frames, Passwords, Stashed, a new page) was not masked, against
  the rule that an app page is masked until someone checks it. It is now
  covered whole unless its page id is in shownNativePages.
- the sampler kept every reading's areas but never noticed an area that
  moved between readings, whose pixels can sit between two frames. Such a
  move now makes the mask unsettled, so the capture retries or refuses.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (Codex gpt-6-astra high fast, review-router): SHIP at 5d6d95a

Round 1 at 7d7ec329 (clean base merge of 5aba0214) was BLOCK with two fail-open findings, both fixed in 5d6d95a5:

  1. Native page tabs (InternalPageView without a PageWebView: Phone and Mac Screen frames, Passwords, Stashed, a new page) were not masked. They are now covered whole unless their page id is in shownNativePages, the native counterpart of the checked shownAppPages list.
  2. The sampler never noticed an area that moved between readings, whose pixels can sit between two frames. A frame change between readings now marks the mask unsettled, so the capture retries and then refuses.

Round 2 at 5d6d95a5: SHIP. One non-blocking note: the movement test drives sample() explicitly rather than through the run-loop observer.

Focused fleet run on a display worker with CMUX_TEST_REQUIRE_GUI=1 (no skips allowed): cmux-ci run --class light --script scripts/ci/package-test-lane.sh --arg=suite --arg=Packages/macOS/CmuxNext --arg=MaskedScreenshotTests,ActionOriginTests

  • head 5d6d95a5, step 3d4d51bf2243e02288136945: MaskedScreenshotTests 12/12 passed (including the two new tests), ActionOriginTests 11/11 passed.
  • head 7d7ec329, step 876b3c0aee104f666ce2c59a: both suites passed.
  • base 3b7d2c99, step 75e65cabc3206931175fc958: ActionOriginTests 11/11 passed (MaskedScreenshotTests does not exist on base).

The generated-files check found schemas/defaults/cmux-next.json stale at
5d6d95a. Only this PR's 24 keys (the screenshot mask, handlers and
strings, DebugToast.run) are updated, from the shared build's
regenerated file; keys that are equally stale on the base are left for
the base's own fix.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (Codex gpt-6-astra high fast, review-router): SHIP at e6b4f18

e6b4f18e changes only schemas/defaults/cmux-next.json. The generated-files check found it stale at 5d6d95a5, so this commit applies this PR's own 24 keys, verbatim from the shared build's regenerated copy. Keys equally stale on base (setting.labs.machineJoin, ActionCatalog groups and others) are left for base's fix. Reviewer: the delta is exactly this PR's 24 tunable-source keys, the JSON is valid and sorted, and only source hashes change, no default values. The Swift tree is identical to 5d6d95a5, which got the full-PR SHIP above.

Red matching by test name at 5d6d95a5 against base 3b7d2c99, using focused display-worker runs with identical failure messages: everyActionIsInThePaletteUnlessExempt, theNewestStagedBuildIsKeptAndProbedOncePerInterval, aScheduleChangeReplacesThePendingProbe, emptyRootKeepsCommonIntentsAndEightFamilies and paletteTitlesUseTitleCase fail on both (head step 4acf7c1e07950d4ae3844c7a, base step e5d597049f5e90e84b635d0d). darkTextOnALightGlassThemeStaysDark and leavingTakesTheGroupOutOfMyList pass at head; they were red only in the broad job, which was cut short (exit 143). The broad job's other 26 reds match base's tip Swift suite runs by name.

Only schemas/defaults/cmux-next.json conflicted: it takes the base's
snapshot plus this PR's own 24 screenshot keys.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (Codex gpt-6-astra high fast, review-router): SHIP at a752545

a752545c merges feat-cmux-next 79cdae21 into e6b4f18e. Only schemas/defaults/cmux-next.json conflicted. It takes base's snapshot plus this PR's own 24 source-hash keys; base changed none of the PR's other files. Reviewer: the diff against base is exactly the PR's 23 files, the snapshot delta is exactly the 24 PR keys (16 added, 6 updated, 2 removed), no base change is dropped, and no default value changes. The Swift tree is the one reviewed (SHIP at 5d6d95a5) and tested (focused display-worker steps 3d4d51bf2243e02288136945 at head and 75e65cabc3206931175fc958 at base).

@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

This branch had an error being deployed

1 failed (outdated) and 3 inactive deployments
artifacts — a752545c Deployed Oct 30, 2026 by teamleaderleo via add the Windows daemon to the tree #78643
sentry-dsyms — a752545c Deployed Oct 29, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #78643
ci-routing-agent — a752545c Deployed Oct 29, 2026 by teamleaderleo via cmux-tui tree preflight #78643
cmux-tui-merge-tree — 4012b4cd Deployed Oct 26, 2026 by teamleaderleo via publish the merge tree #6548
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant