From 95b49c15c41d5fbd5b03eb8958c80631b5c03433 Mon Sep 17 00:00:00 2001 From: Dillon DuPont Date: Tue, 1 Sep 2026 16:05:03 -0700 Subject: [PATCH 1/2] feat(cua-driver): let get_window_state skip the a11y tree and return capture metadata MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the capture-only / window-preview capability into the existing get_window_state tool instead of adding a separate tool. A standalone capture-only tool would duplicate get_window_state's existing per-window screenshot path and create a tool-selection hazard — an agent that wants a window's pixels might call one when it meant the other. One entry point is clearer and cheaper to maintain. get_window_state already returns a per-window screenshot scoped to window_id, already supports include_screenshot:false (tree-only), and already caps the walk with max_elements/max_depth. This adds the mirror image plus metadata: - include_accessibility_tree (bool, default true). When false, SKIP the AX / AT-SPI / UIA walk entirely (the expensive part) and return just the screenshot + window metadata — the capture-only / picture-in-picture path. Mirrors include_screenshot; default true keeps existing behavior unchanged. - Guard the degenerate case: include_accessibility_tree:false AND include_screenshot:false returns a clear "nothing to return" error rather than an empty payload. - max_dimension (optional) downscales the returned screenshot's long edge, folded with the session/global max_image_dimension ceiling (the tighter wins), reusing the existing resize_png_if_needed path. - Enrich the structured screenshot metadata with the fields the preview path needs that weren't already present: app_name + window_title on every platform, and window_bounds on Linux/Windows (macOS already emitted window_bounds + screenshot_scale + width/height + mime). All additive. pid + window_id stay required; the schema is not loosened. The capture path is unchanged, so the Wayland per-window limitation still stands: get_window_state on Wayland returns the tree with a typed surface_identity_unproven screenshot error rather than faked pixels. get_window_state already carries screen-capture scope (it always screenshots), so no new policy, capability token, or risk surface is introduced. Docs (mcp-tools.mdx) regenerated. The tool roster is unchanged (56), so the compat manifest and its guard need no edit. Refs CUA-1106 Co-Authored-By: Claude Opus 4.8 --- .../docs/reference/cua-driver/mcp-tools.mdx | 4 + .../crates/platform-linux/src/tools/impl_.rs | 87 ++++++++++-- .../src/tools/get_window_state.rs | 131 +++++++++++++++++- .../platform-windows/src/tools/impl_.rs | 79 ++++++++++- 4 files changed, 284 insertions(+), 17 deletions(-) diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx index 638759ee0c..ad7e90627d 100644 --- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx +++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx @@ -65,6 +65,8 @@ PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row Always returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with `h:1` frames). You choose the modality at ACTION time, not here: an element ax action (pass `element_index`/`element_token` → the accessibility rung) or an element px action (pass `x`,`y` → the pixel rung, read straight off this screenshot). `capture_mode` is deprecated and ignored. Pass `include_screenshot:false` to skip the grab and get the tree only — the cheap path when you're just re-indexing before an element ax action. +The mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata — `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` — the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_dimension` caps the returned screenshot's long edge in pixels (aspect preserved) for a cheap thumbnail. + The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window — act by pixel there. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available. Optional `query` projects both tree_markdown and structured `elements` to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and `element_count` continues to report its total size; `filtered_element_count` reports the projected response size. @@ -74,8 +76,10 @@ Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-wind **Arguments:** - `capture_mode` (string, optional): DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_index/element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. +- `include_accessibility_tree` (boolean, optional): Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). - `include_screenshot` (boolean, optional): Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk. - `max_depth` (integer, optional): Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees. range: 1–unbounded +- `max_dimension` (integer, optional): Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. range: 1–unbounded - `max_elements` (integer, optional): Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows. range: 1–unbounded - `pid` (integer, required): Target process ID. - `query` (string, optional): Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values. diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs index a148640ac3..93a95b2294 100644 --- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs @@ -562,6 +562,18 @@ mod list_windows_tests { // ── get_window_state ───────────────────────────────────────────────────────── +/// Fold a per-call `max_dimension` cap with the configured +/// `max_image_dimension` ceiling. `resize_png_if_needed` treats `0` as "no +/// limit", so an unlimited ceiling defers to the per-call cap; otherwise the +/// tighter (smaller, non-zero) of the two wins. +fn fold_max_dimension(ceiling: u32, per_call: Option) -> u32 { + match per_call { + Some(md) if ceiling == 0 => md, + Some(md) => ceiling.min(md), + None => ceiling, + } +} + pub struct GetWindowStateTool { state: Arc, } @@ -598,6 +610,14 @@ impl Tool for GetWindowStateTool { the requested surface's identity, the truthful tree is returned without \ a screenshot and `screenshot_error.code` is \ `surface_identity_unproven`.\n\n\ + The mirror image: pass `include_accessibility_tree:false` to SKIP \ + the AT-SPI walk entirely and return just the screenshot plus \ + window metadata (window_bounds, app_name, window_title) — the \ + capture-only path for a live window preview / picture-in-picture. \ + Setting BOTH `include_accessibility_tree:false` and \ + `include_screenshot:false` is an error. Optional `max_dimension` \ + caps the returned screenshot's long edge in pixels for a cheap \ + thumbnail.\n\n\ Optional `max_elements` / `max_depth` bound the AT-SPI walk to \ mitigate context-window blow-up on Electron / large web apps \ that produce 10k+ element trees. When applied, BOTH \ @@ -608,13 +628,16 @@ impl Tool for GetWindowStateTool { "pid":{"type":"integer"}, "window_id":{"type":"integer","description":"Native window identifier from list_windows."}, "capture_mode": cua_driver_core::capture_mode::capture_mode_schema(), + "include_accessibility_tree":{"type":"boolean", + "description":"Default true — walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."}, "include_screenshot":{"type":"boolean", "description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."}, "screenshot_out_file":{"type":"string", "description":"When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead."}, "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."}, "max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees."}, - "max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."} + "max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."}, + "max_dimension":{"type":"integer","minimum":1,"description":"Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview. Applied on top of the configured max_image_dimension ceiling; the tighter wins. Omit for the configured default."} },"additionalProperties":false}), read_only: true, destructive: false, idempotent: true, open_world: false, }) @@ -630,9 +653,15 @@ impl Tool for GetWindowStateTool { Ok(v) => v, Err(e) => return e, }; + // Optional per-call cap on the returned screenshot's long edge, folded + // with the configured ceiling below (the tighter wins). + let max_dimension = args + .get("max_dimension") + .and_then(|v| v.as_u64()) + .map(|v| v.max(1) as u32); let max_dim = { let cfg = self.state.config.read().unwrap(); - cfg.max_image_dimension + fold_max_dimension(cfg.max_image_dimension, max_dimension) }; // `capture_mode` is DEPRECATED and ignored — get_window_state always // returns BOTH the AT-SPI tree and a screenshot now, so the agent grounds @@ -647,6 +676,13 @@ impl Tool for GetWindowStateTool { // tree only (the cheap re-index path before an element ax action). A // screenshot_out_file still forces a capture (to disk), regardless. let include_screenshot = args.get("include_screenshot").and_then(|v| v.as_bool()); + // `include_accessibility_tree` (default true) mirrors include_screenshot: + // set false to SKIP the AT-SPI walk and return just the screenshot + + // window metadata (the capture-only / preview path). + let want_tree = args + .get("include_accessibility_tree") + .and_then(|v| v.as_bool()) + != Some(false); // screenshot_out_file: when set, write the PNG to disk and surface the // path instead of embedding base64 in the response. `~` expands. let screenshot_out_file = args.opt_str("screenshot_out_file").map(|s| { @@ -682,6 +718,12 @@ impl Tool for GetWindowStateTool { "Window target pid {pid}, window_id {xid} is stale or no longer running; refresh list_windows." )); } + // Window identity metadata (additive), sourced from the same enumerator + // list_windows uses. Names the surface and its on-screen rectangle even + // on the capture-only path, where no AT-SPI tree identifies it. + let window_meta = crate::wayland::list_windows_dispatch(Some(pid)) + .into_iter() + .find(|w| w.xid == xid); // Always walk the AT-SPI tree; capture the screenshot by default. The // tree+screenshot pair is the default so the agent grounds on both and @@ -689,6 +731,13 @@ impl Tool for GetWindowStateTool { // `include_screenshot:false` skips the grab; an unproven Wayland surface // returns the tree with a typed screenshot error instead of unrelated pixels. let should_capture = include_screenshot != Some(false) || screenshot_out_file.is_some(); + if !want_tree && !should_capture { + return ToolResult::error( + "Nothing to return: both include_accessibility_tree:false and \ + include_screenshot:false. Set at least one to true, or pass \ + screenshot_out_file to force a capture.", + ); + } let observation_only = args .get("_observation_only") .and_then(|value| value.as_bool()) @@ -697,13 +746,19 @@ impl Tool for GetWindowStateTool { let query_for_walk = query.clone(); let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> { - let tree_result = Some(crate::atspi::walk_tree_bounded( - pid, - xid, - query_for_walk.as_deref(), - max_elements, - max_depth, - )); + // Skip the AT-SPI walk on the capture-only path + // (include_accessibility_tree:false). + let tree_result = if want_tree { + Some(crate::atspi::walk_tree_bounded( + pid, + xid, + query_for_walk.as_deref(), + max_elements, + max_depth, + )) + } else { + None + }; // Bounds and element indices come from the same captured AT-SPI // traversal. Joining two live walks by ordinal mis-associated // Chromium controls when its lazy subtree changed between walks. @@ -940,6 +995,20 @@ impl Tool for GetWindowStateTool { structured["screenshot_frame_valid"] = json!(false); structured["screenshot_error"] = surface_identity_unproven_error(xid, reason); } + // Window identity metadata (additive): app + title + on-screen + // rectangle for the requested window_id, useful on the + // capture-only path where no AT-SPI tree names the surface. + if let Some(meta) = &window_meta { + if !meta.app_name.is_empty() { + structured["app_name"] = json!(meta.app_name); + } + if !meta.title.is_empty() { + structured["window_title"] = json!(meta.title); + } + structured["window_bounds"] = json!({ + "x": meta.x, "y": meta.y, "width": meta.width, "height": meta.height + }); + } ToolResult { content, diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs index 378023c169..f49892b8f1 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs @@ -47,6 +47,16 @@ fn def() -> &'static ToolDef { and ignored. Pass `include_screenshot:false` to skip the grab and get \ the tree only — the cheap path when you're just re-indexing before an \ element ax action.\n\n\ + The mirror image: pass `include_accessibility_tree:false` to SKIP the \ + AX walk entirely (the expensive part, up to 20 s) and return just the \ + screenshot plus window metadata — `window_bounds`, `screenshot_scale`, \ + `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` \ + — the capture-only path for rendering a live window preview / \ + picture-in-picture without paying for perception. Setting BOTH \ + `include_accessibility_tree:false` and `include_screenshot:false` is an \ + error (nothing to return). Optional `max_dimension` caps the returned \ + screenshot's long edge in pixels (aspect preserved) for a cheap \ + thumbnail.\n\n\ The snapshot is SCOPED to `window_id`: a window_id that no longer exists is \ refused with `window_id_not_found`, and one owned by another process is \ refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry \ @@ -79,6 +89,10 @@ fn def() -> &'static ToolDef { "window_id": { "type": "integer", "description": "Target window ID from list_windows." }, "query": { "type": "string", "description": "Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values." }, "capture_mode": cua_driver_core::capture_mode::capture_mode_schema(), + "include_accessibility_tree": { + "type": "boolean", + "description": "Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)." + }, "include_screenshot": { "type": "boolean", "description": "Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk." @@ -96,6 +110,11 @@ fn def() -> &'static ToolDef { "type": "integer", "minimum": 1, "description": "Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees." + }, + "max_dimension": { + "type": "integer", + "minimum": 1, + "description": "Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default." } }, "additionalProperties": false @@ -107,6 +126,19 @@ fn def() -> &'static ToolDef { }) } +/// Fold a per-call `max_dimension` cap with the session/global +/// `max_image_dimension` ceiling. `resize_png_if_needed` treats `0` as "no +/// limit", so when the ceiling is unlimited the per-call cap stands alone; +/// otherwise the tighter (smaller, non-zero) of the two wins. Returns `0` only +/// when neither imposes a limit. +fn fold_max_dimension(ceiling: u32, per_call: Option) -> u32 { + match per_call { + Some(md) if ceiling == 0 => md, + Some(md) => ceiling.min(md), + None => ceiling, + } +} + fn chromium_browser_window(pid: i32) -> bool { let identity = format!( "{} {}", @@ -211,6 +243,29 @@ impl Tool for GetWindowStateTool { // still forces a capture (an explicit "write the frame to disk"). let include_screenshot = args.get("include_screenshot").and_then(|v| v.as_bool()); let should_capture = include_screenshot != Some(false) || screenshot_out_file.is_some(); + // `include_accessibility_tree` (default true) is the mirror image of + // `include_screenshot`: set false to SKIP the AX walk (the expensive + // part) and return just the screenshot + window metadata — the + // capture-only / preview path. With BOTH the tree and the screenshot + // opted out there is nothing to return, so refuse rather than emit an + // empty payload. + let want_tree = args + .get("include_accessibility_tree") + .and_then(|v| v.as_bool()) + != Some(false); + if !want_tree && !should_capture { + return ToolResult::error( + "Nothing to return: both include_accessibility_tree:false and \ + include_screenshot:false. Set at least one to true, or pass \ + screenshot_out_file to force a capture.", + ); + } + // Optional per-call cap on the returned screenshot's long edge, folded + // with the session/global ceiling below (the tighter wins). + let max_dimension = args + .get("max_dimension") + .and_then(|v| v.as_u64()) + .map(|v| v.max(1) as u32); // Internal direct-tool mode used by verify_state. Registry ingress // strips underscore-prefixed arguments before public dispatch; only // a trusted direct in-process invocation can enable this mode. @@ -233,8 +288,10 @@ impl Tool for GetWindowStateTool { .map(|v| v.max(1) as usize) .unwrap_or(crate::ax::tree::DEFAULT_MAX_DEPTH); - // Always walk the AX tree (perception returns both tree + screenshot). - let tree_result = { + // Walk the AX tree unless the caller opted out via + // `include_accessibility_tree:false` (the capture-only / preview path, + // which skips the expensive walk and returns screenshot + metadata). + let tree_result = if want_tree { let q = query.clone(); // Keep the product deadline below the public client's 25-second // deadline so callers receive a structured driver error. The AX @@ -263,6 +320,8 @@ impl Tool for GetWindowStateTool { )); } } + } else { + None }; // The window can close, or its CGWindow can be re-parented onto another @@ -299,7 +358,9 @@ impl Tool for GetWindowStateTool { // against. Skipped only when `include_screenshot:false` (and no // screenshot_out_file). With `screenshot_out_file` set, write to disk and // surface the path instead of embedding base64; otherwise embed base64. - let max_dim = effective_max_dim; + // Fold the per-call `max_dimension` with the session/global ceiling + // (the tighter of the two wins). + let max_dim = fold_max_dimension(effective_max_dim, max_dimension); // Returns the encoded/file capture, delivered dimensions, optional // downscale source width, the WindowServer bounds it was validated // against, and the raw capture's backing scale. @@ -487,7 +548,7 @@ impl Tool for GetWindowStateTool { .as_ref() .map(|r| r.nodes.iter().filter(|n| n.element_index.is_some()).count()) .unwrap_or(0); - let snapshot_id = if scope_matched && !observation_only { + let snapshot_id = if scope_matched && !observation_only && tree_result.is_some() { Some(cua_driver_core::element_token::global().register_snapshot( pid, window_id, @@ -641,6 +702,19 @@ impl Tool for GetWindowStateTool { if let Some(ref fp) = screenshot_file_path { structured["screenshot_file_path"] = serde_json::json!(fp); } + // Window identity metadata (additive): the owning app and the window's + // title for the requested window_id. A cheap WindowServer lookup that + // names the surface even on the capture-only path, where no AX tree is + // present to identify it. Omitted per-field when WindowServer reports an + // empty string. + if let Some(info) = crate::windows::window_info_by_id(window_id) { + if !info.app_name.is_empty() { + structured["app_name"] = serde_json::json!(info.app_name); + } + if !info.title.is_empty() { + structured["window_title"] = serde_json::json!(info.title); + } + } cua_driver_core::window_inspection::mark_browser_chrome_capture_coverage( &mut structured, chromium_browser_window(pid).then_some( @@ -1006,6 +1080,55 @@ mod window_scope_contract_tests { ); } } + + /// The capture-only fold-in: get_window_state advertises the new + /// `include_accessibility_tree` / `max_dimension` controls, keeps pid + + /// window_id required (schema not loosened), and documents the degenerate + /// both-false case in its description. + #[test] + fn schema_advertises_capture_only_controls() { + let d = def(); + let props = &d.input_schema["properties"]; + assert!( + props.get("include_accessibility_tree").is_some(), + "schema must advertise include_accessibility_tree" + ); + assert!( + props.get("max_dimension").is_some(), + "schema must advertise max_dimension" + ); + let required: Vec<&str> = d.input_schema["required"] + .as_array() + .expect("required array") + .iter() + .map(|v| v.as_str().unwrap()) + .collect(); + assert!( + required.contains(&"pid") && required.contains(&"window_id"), + "pid and window_id must stay required: {required:?}" + ); + assert!( + d.description.contains("include_accessibility_tree:false") + && d.description.contains("include_screenshot:false"), + "description must document the both-false error" + ); + } + + /// The per-call `max_dimension` folds with the session/global ceiling: the + /// tighter non-zero cap wins, an unlimited (0) ceiling defers to the + /// per-call cap, and absent inputs pass the ceiling through unchanged. + #[test] + fn max_dimension_folds_tighter_cap() { + // Ceiling wins when it is tighter than the per-call cap. + assert_eq!(fold_max_dimension(1024, Some(2048)), 1024); + // Per-call wins when it is tighter than the ceiling. + assert_eq!(fold_max_dimension(4096, Some(512)), 512); + // Unlimited ceiling (0) defers entirely to the per-call cap. + assert_eq!(fold_max_dimension(0, Some(768)), 768); + // No per-call cap → the ceiling passes through (0 stays unlimited). + assert_eq!(fold_max_dimension(1600, None), 1600); + assert_eq!(fold_max_dimension(0, None), 0); + } } #[cfg(test)] diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs index ef1477e1d6..e52e6253c1 100644 --- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs @@ -1064,6 +1064,18 @@ mod list_windows_z_index_tests { // ── get_window_state ───────────────────────────────────────────────────────── +/// Fold a per-call `max_dimension` cap with the configured +/// `max_image_dimension` ceiling. `resize_png_if_needed` treats `0` as "no +/// limit", so an unlimited ceiling defers to the per-call cap; otherwise the +/// tighter (smaller, non-zero) of the two wins. +fn fold_max_dimension(ceiling: u32, per_call: Option) -> u32 { + match per_call { + Some(md) if ceiling == 0 => md, + Some(md) => ceiling.min(md), + None => ceiling, + } +} + pub struct GetWindowStateTool { state: Arc, } @@ -1108,6 +1120,13 @@ impl Tool for GetWindowStateTool { ACTION time: an element ax action (element_index/element_token → \ accessibility rung) or an element px action (x,y → pixel rung off this \ screenshot). capture_mode is deprecated and ignored.\n\n\ + The mirror image: pass `include_accessibility_tree:false` to SKIP the \ + UIA walk entirely and return just the screenshot plus window metadata \ + (window_bounds, app_name, window_title) — the capture-only path for a \ + live window preview / picture-in-picture. Setting BOTH \ + `include_accessibility_tree:false` and `include_screenshot:false` is an \ + error. Optional `max_dimension` caps the returned screenshot's long edge \ + in pixels for a cheap thumbnail.\n\n\ Uses `IUIAutomationCacheRequest` to batch-fetch all element properties in a \ single COM call (Chrome's ~5000-element tree returns in ~2-3s instead of \ timing out at 4s with per-property RPCs).\n\n\ @@ -1130,11 +1149,13 @@ impl Tool for GetWindowStateTool { "pid":{"type":"integer","description":"Process ID from `list_apps`."}, "window_id":{"type":"integer","description":"HWND of the target window. Must belong to `pid`. Enumerate via `list_windows` or read from `launch_app`'s `windows` array."}, "capture_mode": cua_driver_core::capture_mode::capture_mode_schema(), + "include_accessibility_tree":{"type":"boolean","description":"Default true — walk the UIA tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the UIA walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."}, "include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."}, "screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead."}, "query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."}, "max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees."}, - "max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."} + "max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."}, + "max_dimension":{"type":"integer","minimum":1,"description":"Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview. Applied on top of the configured max_image_dimension ceiling; the tighter wins. Omit for the configured default."} },"additionalProperties":false}), // Swift annotation: idempotent: false (each call is a fresh snapshot). read_only: true, destructive: false, idempotent: false, open_world: false, @@ -1177,11 +1198,34 @@ impl Tool for GetWindowStateTool { {pid}}})` for candidates." )); } + // Window identity metadata (additive): title + on-screen rectangle from + // the enumeration we already did, plus the owning process's executable + // name. Names the surface on the capture-only path, where no UIA tree + // identifies it. + let win_geom = windows_for_pid + .iter() + .find(|w| w.hwnd == hwnd) + .map(|w| (w.title.clone(), w.x, w.y, w.width, w.height)); + let app_name = tokio::task::spawn_blocking(move || { + crate::win32::list_processes() + .into_iter() + .find(|p| p.pid == pid) + .map(|p| p.name) + }) + .await + .ok() + .flatten(); + use cua_driver_core::tool_args::ArgsExt; + // Optional per-call cap on the returned screenshot's long edge, folded + // with the configured ceiling (the tighter wins). + let max_dimension = args + .get("max_dimension") + .and_then(|v| v.as_u64()) + .map(|v| v.max(1) as u32); let max_dim = { let cfg = self.state.config.read().unwrap(); - cfg.max_image_dimension + fold_max_dimension(cfg.max_image_dimension, max_dimension) }; - use cua_driver_core::tool_args::ArgsExt; // `capture_mode` is DEPRECATED and ignored — get_window_state always // returns BOTH the UIA tree and a screenshot now, so the agent grounds on // both and cross-checks (the UIA tree lies often enough that a grounding @@ -1217,8 +1261,21 @@ impl Tool for GetWindowStateTool { .get("_observation_only") .and_then(|value| value.as_bool()) == Some(true); - let do_tree = true; + // `include_accessibility_tree` (default true) mirrors include_screenshot: + // set false to SKIP the UIA walk and return just the screenshot + window + // metadata (the capture-only / preview path). + let do_tree = args + .get("include_accessibility_tree") + .and_then(|v| v.as_bool()) + != Some(false); let do_shot = include_screenshot != Some(false) || screenshot_out_file.is_some(); + if !do_tree && !do_shot { + return ToolResult::error( + "Nothing to return: both include_accessibility_tree:false and \ + include_screenshot:false. Set at least one to true, or pass \ + screenshot_out_file to force a capture.", + ); + } let state = self.state.clone(); let q = query.clone(); @@ -1497,6 +1554,20 @@ impl Tool for GetWindowStateTool { structured["screenshot_error"] = json!(err); } + // Window identity metadata (additive): title + on-screen + // rectangle + owning process name for the requested window_id, + // useful on the capture-only path where no UIA tree names it. + if let Some((title, x, y, w, h)) = &win_geom { + if !title.is_empty() { + structured["window_title"] = json!(title); + } + structured["window_bounds"] = + json!({ "x": x, "y": y, "width": w, "height": h }); + } + if let Some(name) = app_name.as_deref().filter(|n| !n.is_empty()) { + structured["app_name"] = json!(name); + } + cua_driver_core::window_inspection::mark_browser_chrome_capture_coverage( &mut structured, is_standalone_chromium_browser_process(pid).then_some( From 278a267c5480aef5402d6356e941813c65e26d93 Mon Sep 17 00:00:00 2001 From: Dillon DuPont Date: Wed, 2 Sep 2026 10:04:05 -0700 Subject: [PATCH 2/2] fix(cua-driver): never return empty get_window_state content on the capture-only path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address AI code-review feedback on the include_accessibility_tree capture-only path. - Linux: enumerate the pid's windows once (list_windows_dispatch) and reuse the result for both the Wayland window-ownership check and the additive window metadata, instead of calling the compositor/X11 enumerator twice per get_window_state call. - Linux + Windows: on the capture-only path (include_accessibility_tree:false) the response could be a "success" with no content parts — when the screenshot was written to disk via screenshot_out_file (image not embedded, and no tree markdown), or on Wayland when per-window capture cannot prove surface identity (no tree, no image). Push a text content part when the screenshot goes to disk, and return a structured error (carrying the screenshot_error reason) when nothing at all was produced, matching the macOS path which already guards this. Refs CUA-1106 Co-Authored-By: Claude Opus 4.8 --- .../crates/platform-linux/src/tools/impl_.rs | 47 ++++++++++++++----- .../platform-windows/src/tools/impl_.rs | 21 ++++++++- 2 files changed, 56 insertions(+), 12 deletions(-) diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs index 93a95b2294..0c7a8a44d6 100644 --- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs @@ -705,10 +705,16 @@ impl Tool for GetWindowStateTool { .map(|v| v.max(1) as usize); let process_is_live = crate::proc_fs::is_process_live(pid); + // Enumerate the pid's windows ONCE and reuse the result for both the + // window-ownership check (Wayland) and the additive window metadata + // below, instead of paying for the compositor/X11 enumeration twice. + // `window_meta` also names the surface + its on-screen rectangle on the + // capture-only path, where no AT-SPI tree identifies it. + let window_meta = crate::wayland::list_windows_dispatch(Some(pid)) + .into_iter() + .find(|w| w.xid == xid); let window_matches = if crate::wayland::is_wayland() { - crate::wayland::list_windows_dispatch(Some(pid)) - .iter() - .any(|window| window.xid == xid && window.pid == Some(pid)) + window_meta.as_ref().is_some_and(|w| w.pid == Some(pid)) || crate::wayland::window_was_listed_for_pid(pid, xid) } else { crate::x11::window_belongs_to_pid(xid, pid) @@ -718,12 +724,6 @@ impl Tool for GetWindowStateTool { "Window target pid {pid}, window_id {xid} is stale or no longer running; refresh list_windows." )); } - // Window identity metadata (additive), sourced from the same enumerator - // list_windows uses. Names the surface and its on-screen rectangle even - // on the capture-only path, where no AT-SPI tree identifies it. - let window_meta = crate::wayland::list_windows_dispatch(Some(pid)) - .into_iter() - .find(|w| w.xid == xid); // Always walk the AT-SPI tree; capture the screenshot by default. The // tree+screenshot pair is the default so the agent grounds on both and @@ -978,8 +978,15 @@ impl Tool for GetWindowStateTool { } // ax mode + screenshot_out_file writes the PNG to disk and // returns b64=None — never embed the image bytes in that case. + // Keep a text content part when the image went to disk so the + // response is never empty on the capture-only path (which has + // no tree markdown either). if let Some(b64) = b64_opt { content.push(cua_driver_core::protocol::Content::image_png(b64)); + } else if let Some(fp) = &file_path { + content.push(cua_driver_core::protocol::Content::text(format!( + "window_id={xid} pid={pid} size={w}x{h} screenshot written to {fp}" + ))); } structured["screenshot_width"] = json!(w); structured["screenshot_height"] = json!(h); @@ -991,9 +998,10 @@ impl Tool for GetWindowStateTool { structured["screenshot_file_path"] = json!(fp); } } - if let Some(reason) = screenshot_error { + if let Some(reason) = &screenshot_error { structured["screenshot_frame_valid"] = json!(false); - structured["screenshot_error"] = surface_identity_unproven_error(xid, reason); + structured["screenshot_error"] = + surface_identity_unproven_error(xid, reason.clone()); } // Window identity metadata (additive): app + title + on-screen // rectangle for the requested window_id, useful on the @@ -1010,6 +1018,23 @@ impl Tool for GetWindowStateTool { }); } + // The capture-only path (include_accessibility_tree:false) leaves + // `content` empty when the screenshot was also unavailable — most + // often on Wayland, where per-window capture cannot prove surface + // identity. Return a structured error rather than a "successful" + // response with no content parts. + if content.is_empty() { + let reason_note = match &screenshot_error { + Some(reason) => format!(" and no screenshot could be captured ({reason})"), + None => " and no screenshot was returned".to_string(), + }; + return ToolResult::error(format!( + "No content produced for window_id {xid}: the accessibility tree was \ + skipped (include_accessibility_tree:false){reason_note}." + )) + .with_structured(structured); + } + ToolResult { content, is_error: None, diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs index e52e6253c1..f15bf10ae6 100644 --- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs @@ -1527,9 +1527,15 @@ impl Tool for GetWindowStateTool { } // base64 is embedded only when no out_file was given (vision // path). With `screenshot_out_file` the bytes went to disk and - // we surface the path instead — never both. + // we surface the path instead — never both. Keep a text content + // part when the image went to disk so the response is never + // empty on the capture-only path (which has no tree markdown). if let Some(b64) = b64_opt { content.push(cua_driver_core::protocol::Content::image_png(b64)); + } else if let Some(fp) = &file_path { + content.push(cua_driver_core::protocol::Content::text(format!( + "window_id={hwnd} pid={pid} size={w}x{h} screenshot written to {fp}" + ))); } structured["screenshot_width"] = json!(w); structured["screenshot_height"] = json!(h); @@ -1575,6 +1581,19 @@ impl Tool for GetWindowStateTool { ), ); + // The capture-only path (include_accessibility_tree:false) leaves + // `content` empty if the screenshot was also unavailable. Return a + // structured error rather than a "successful" response with no + // content parts (consistent with the tree+screenshot path, which + // always carries at least the tree markdown). + if content.is_empty() { + return ToolResult::error(format!( + "No content produced for window_id {hwnd}: the accessibility tree was \ + skipped (include_accessibility_tree:false) and no screenshot was returned." + )) + .with_structured(structured); + } + ToolResult { content, is_error: None,