Skip to content

fix(cua-driver): isolate screenshot transforms between clients - #3628

Closed
injaneity wants to merge 3 commits into
mainfrom
fix/session-owned-screenshot-transforms
Closed

fix(cua-driver): isolate screenshot transforms between clients#3628
injaneity wants to merge 3 commits into
mainfrom
fix/session-owned-screenshot-transforms

Conversation

@injaneity

@injaneity injaneity commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Superseded by issue #3630

At maintainer request, this work has moved to #3630 as a defect/evidence record for architectural review alongside RFC #3473 and active PR #3616. This prototype is closed without merging. Its branch and commits are retained for regression-test reuse and provenance; do not treat the session-registry design or its refusal changes as approved. The validation below is the historical prototype record, not a current readiness claim.

Scope

Refs #1882. This draft fixes cross-client screenshot transform interference, rather than claiming that the metadata/documentation issue is fully resolved. It does not introduce image-token machinery or supersede RFC #3473 / PR #3616.

Current head: 21fad867081f6b69d91418c39572aededdbace22.

Confirmed defect

Two persistent MCP clients captured the same unchanged 1470×932-point disposable AppKit window using installed macOS cua-driver 0.23.2. Client A requested a 200×127 image; client B requested native resolution. A clicked the same image coordinate (99.5, 62.5) throughout:

Sequence Native receiver landing, window-top coordinates Outcome
A capture → A click (731.325, 459.375) Target
A capture → B native capture → A click (99.5, 62.5) Wrong region
A refresh → A click (731.325, 459.375) Target

The receiver's mouse events, not the driver's unverifiable action response, are the oracle. Window geometry and backing scale remained unchanged. The baseline binary SHA-256 was 67ccfc99e69ebb5881fdfc3787d85abcd8cc2beb7423f255549557623cab6907; its reported source SHA was null. This is installed-version diagnostic evidence, not exact-main or candidate certification. No existing user browser tabs were targeted.

Implementation

  • Shared core resize registry keyed by trusted private lifecycle identity, PID, and window ID.
  • macOS, Windows, and Linux captures and resize consumers use that registry, including delegated pixel-focus paths, Linux held-pointer/recording conversion, and Windows zoom crop conversion.
  • Native-size captures publish identity transforms instead of clearing another client's scale.
  • Session cleanup removes only the owner's transforms; late capture completion cannot resurrect an ended session.
  • Common dispatch refuses missing or ambiguous window-pixel context with screenshot_context_missing, rather than borrowing another client's transform or guessing native pixels.
  • Documented persistent MCP / explicit CLI session usage. Unnamed one-shot CLI calls cannot retain screenshot context across processes.
  • Shared unit and trusted-dispatch regressions, plus same-daemon two-client interference in existing AppKit, WPF, and supported GTK3 pixel rows. Their fixture-state and desktop oracles remain required; the observer connection does not create an unrelated recording.

Validation so far

  • cargo test -p cua-driver-core --locked: 620 passed (615 library, 5 integration); one documentation test ignored. Rerun against the final local source before the latest commit.
  • cargo test -p platform-macos --lib --locked: 365 passed, 2 ignored, at dfd15de01f00b85d3685aba9039e5529ffcd5acc. The subsequent product diff is a Windows-only zoom call-site correction; remaining changes are tests/testkit/docs.
  • cargo test -p cua-driver-testkit --lib --locked: passed after adding the peer-connection helper.
  • AppKit native regression target compiles after the peer-connection update; this is not a native execution result.
  • Source driver build, formatting, and diff checks passed.
  • Previous-head Linux native unit/compile, Linux/macOS portable contract parity, generated contract/SDK checks, and release metadata passed.
  • Previous-head Windows CI caught one missed zoom resize lookup; this head supplies session/window identity there. Windows compilation and dependent checks are rerunning, not yet claimed green.

Readiness blockers / boundaries

  • Native candidate replay and the canonical stable-SHA desktop matrix. The source-built private daemon's read-only permission check reported Accessibility=false and Screen Recording=false; the separately installed signed application is authorized. No TCC reset, installed-app replacement, or permission prompt was performed. The private probe daemon has been stopped.
  • Reconcile and verify Wayland refusal precedence: when a compositor cannot supply a window screenshot, the new common missing-context precondition can precede the adapter's background-unavailable refusal. Existing refusal rows must explicitly cover that ordering before readiness; X11 success is not Wayland certification.
  • Latest-head Windows/Linux CI, native harness compilation, and documentation/contract checks.

A later capture in the same session still replaces that session's previous transform. This draft does not claim image-ID freshness, window/display-change invalidation, zoom-context ownership, or every recording-rendering alignment case. screenshot_scale retains its existing native-backing-scale meaning.

The PR remains draft and must not be treated as merge-ready or exact-candidate desktop-certified.

Refs #1882. Establish shared session-owned resize storage before wiring platform adapters.
Refs #1882. Wire capture, pointer consumers, lifecycle cleanup, and common missing-context refusal. Extend the canonical AppKit pixel row with independent-client capture interference.
Add a second unrecorded connection to the same test daemon for AppKit, WPF, and supported GTK3 pixel rows. Fix the Windows zoom resize consumer found by native CI. Document remaining compositor refusal-precedence validation.
@injaneity

Copy link
Copy Markdown
Collaborator Author

superseded by #3630 at maintainer request. the issue preserves the reproduction, prototype provenance, validation limits, and architectural follow-up with #3473 / #3616. closing without merging or deleting the branch; no expansion of the selected rfc scope is implied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant