feat(cua-driver): let get_window_state skip the a11y tree and return capture metadata

[ddupont808]

What

Folds the capture-only / live-window-preview capability into the existing get_window_state tool, instead of adding a standalone tool.

A separate capture-only tool would duplicate get_window_state's existing per-window screenshot path and create a tool-selection hazard — an agent that wants a window's pixels might reach for one when it meant the other. One entry point is clearer and cheaper to maintain.

Why this is a small change

get_window_state already:

• returns a per-window screenshot scoped to window_id,
• supports include_screenshot:false (tree-only, the cheap re-index path),
• caps the walk with max_elements / max_depth.

So this is the mirror image plus the missing metadata.

Changes

1. include_accessibility_tree (bool, default true). When false, SKIP the AX / AT-SPI / UIA walk entirely (the expensive part — up to 20 s on macOS) and return just the screenshot + window metadata. This is the capture-only / picture-in-picture path. Mirrors include_screenshot; default true keeps existing behavior unchanged.
2. Both-false guard. include_accessibility_tree:false and include_screenshot:false returns a clear "nothing to return" error rather than an empty payload (screenshot_out_file still forces a capture).
3. max_dimension (optional) downscales the returned screenshot's long edge (aspect preserved), folded with the session/global max_image_dimension ceiling — the tighter wins — reusing the existing resize_png_if_needed path.
4. Metadata enrichment (additive), adding only what each platform didn't already emit:
   • macOS already returned window_bounds, screenshot_scale, screenshot_width/height, screenshot_mime_type; this adds app_name + window_title.
   • Linux / Windows already returned screenshot_width/height + screenshot_mime_type; this adds window_bounds + app_name + window_title.

pid + window_id stay required — the schema is not loosened.

Preserved invariants

• Capture path unchanged, so the Wayland per-window limitation still stands: on Wayland, get_window_state returns the tree with a typed surface_identity_unproven screenshot error rather than faked pixels — never a fabricated capture.
• No new risk surface. get_window_state already carries screen-capture scope (it always screenshots), so no new policy rule, capability token, or risk-class entry is introduced.
• Roster unchanged (56 tools). The MCP-compat manifest and its guard test need no edit; outputSchema counts are unchanged.

Platform note on screenshot_scale

screenshot_scale is emitted on macOS, where the capture frame is validated against WindowServer bounds and the backing scale (1x/2x Retina) is proven. Linux (X11) and Windows get_window_state do not compute a proven backing scale today, so this change does not fabricate one there (it would be exactly the kind of misleading cross-platform behavior the repo's contract forbids). Those platforms gain window_bounds + app_name + window_title. Emitting a proven per-platform scale on Linux/Windows is a separate follow-up.

Tests

• macOS get_window_state unit tests: schema advertises include_accessibility_tree + max_dimension, keeps pid/window_id required, documents the both-false error; max_dimension folding (tighter cap wins, unlimited ceiling defers to per-call, absent passes through). The existing screenshot_only_path_does_not_degrade unit test already covers the tree-skipped degradation rung.
• Behavioral capture-only coverage (screenshot+metadata with no tree; both-false error) runs through the macOS Lume desktop E2E harness, which is the canonical desktop gate.

Local validation on macOS: cargo build (full workspace incl. Linux/Windows stub paths) + release build, cargo test (core 587 + macOS 360 lib, get_window_state 28), cargo fmt --check, docs regenerated. Linux/Windows native get_window_state compiles only on their OS and is covered by CI's Rust Linux/Windows lanes.

Refs CUA-1106

[Copilot]

🟡 Changes recommended

The new capture-only path can produce successful responses with an empty content stream (and on Wayland can yield neither tree nor screenshot on surface_identity_unproven), which risks breaking consumers and contradicts the documented behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends the existing get_window_state tool to support a capture-only path by allowing callers to skip the expensive accessibility-tree walk, while also enriching the returned window metadata and adding an optional per-call screenshot downscale cap.

Changes:

• Add include_accessibility_tree (default true) and a both-false guard to prevent empty tool outputs.
• Add max_dimension and fold it with the configured max_image_dimension ceiling to downscale screenshots.
• Enrich structured output with window_bounds, app_name, and window_title across platforms, and update MCP tool docs accordingly.

File summaries

File	Description
libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs	Adds capture-only controls, max-dimension folding, and window metadata to Windows get_window_state.
libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs	Adds capture-only controls, max-dimension folding, metadata enrichment, and unit tests for schema + folding behavior on macOS.
libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs	Adds capture-only controls, max-dimension folding, and window metadata to Linux get_window_state (AT-SPI/X11/Wayland).
docs/content/docs/reference/cua-driver/mcp-tools.mdx	Documents the new include_accessibility_tree and max_dimension arguments and the capture-only behavior.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.