Skip to content

feat(cua-driver): let get_window_state skip the a11y tree and return capture metadata #3516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ddupont808 merged 2 commits into from
Sep 2, 2026
Merged

feat(cua-driver): let get_window_state skip the a11y tree and return capture metadata #3516

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
95b49c1
feat(cua-driver): let get_window_state skip the a11y tree and return …
ddupont808 Sep 1, 2026
278a267
fix(cua-driver): never return empty get_window_state content on the c…
ddupont808 Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/content/docs/reference/cua-driver/mcp-tools.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row

Always returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with `h:1` frames). You choose the modality at ACTION time, not here: an element ax action (pass `element_index`/`element_token` → the accessibility rung) or an element px action (pass `x`,`y` → the pixel rung, read straight off this screenshot). `capture_mode` is deprecated and ignored. Pass `include_screenshot:false` to skip the grab and get the tree only — the cheap path when you're just re-indexing before an element ax action.

The mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata — `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` — the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_dimension` caps the returned screenshot's long edge in pixels (aspect preserved) for a cheap thumbnail.

The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window — act by pixel there. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.

Optional `query` projects both tree_markdown and structured `elements` to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and `element_count` continues to report its total size; `filtered_element_count` reports the projected response size.
Expand All @@ -74,8 +76,10 @@ Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-wind
**Arguments:**

- `capture_mode` (string, optional): DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_index/element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect.
- `include_accessibility_tree` (boolean, optional): Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).
- `include_screenshot` (boolean, optional): Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk.
- `max_depth` (integer, optional): Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees. range: 1–unbounded
- `max_dimension` (integer, optional): Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. range: 1–unbounded
- `max_elements` (integer, optional): Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows. range: 1–unbounded
- `pid` (integer, required): Target process ID.
- `query` (string, optional): Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values.
Expand Down
122 changes: 108 additions & 14 deletions libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -562,6 +562,18 @@ mod list_windows_tests {

// ── get_window_state ─────────────────────────────────────────────────────────

/// Fold a per-call `max_dimension` cap with the configured
/// `max_image_dimension` ceiling. `resize_png_if_needed` treats `0` as "no
/// limit", so an unlimited ceiling defers to the per-call cap; otherwise the
/// tighter (smaller, non-zero) of the two wins.
fn fold_max_dimension(ceiling: u32, per_call: Option<u32>) -> u32 {
match per_call {
Some(md) if ceiling == 0 => md,
Some(md) => ceiling.min(md),
None => ceiling,
}
}

pub struct GetWindowStateTool {
state: Arc<ToolState>,
}
Expand Down Expand Up @@ -598,6 +610,14 @@ impl Tool for GetWindowStateTool {
the requested surface's identity, the truthful tree is returned without \
a screenshot and `screenshot_error.code` is \
`surface_identity_unproven`.\n\n\
The mirror image: pass `include_accessibility_tree:false` to SKIP \
the AT-SPI walk entirely and return just the screenshot plus \
window metadata (window_bounds, app_name, window_title) — the \
capture-only path for a live window preview / picture-in-picture. \
Setting BOTH `include_accessibility_tree:false` and \
`include_screenshot:false` is an error. Optional `max_dimension` \
caps the returned screenshot's long edge in pixels for a cheap \
thumbnail.\n\n\
Optional `max_elements` / `max_depth` bound the AT-SPI walk to \
mitigate context-window blow-up on Electron / large web apps \
that produce 10k+ element trees. When applied, BOTH \
Expand All @@ -608,13 +628,16 @@ impl Tool for GetWindowStateTool {
"pid":{"type":"integer"},
"window_id":{"type":"integer","description":"Native window identifier from list_windows."},
"capture_mode": cua_driver_core::capture_mode::capture_mode_schema(),
"include_accessibility_tree":{"type":"boolean",
"description":"Default true — walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return)."},
"include_screenshot":{"type":"boolean",
"description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."},
"screenshot_out_file":{"type":"string",
"description":"When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead."},
"query":{"type":"string","description":"Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count."},
"max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees."},
"max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."}
"max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."},
"max_dimension":{"type":"integer","minimum":1,"description":"Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview. Applied on top of the configured max_image_dimension ceiling; the tighter wins. Omit for the configured default."}
},"additionalProperties":false}),
read_only: true, destructive: false, idempotent: true, open_world: false,
})
Expand All @@ -630,9 +653,15 @@ impl Tool for GetWindowStateTool {
Ok(v) => v,
Err(e) => return e,
};
// Optional per-call cap on the returned screenshot's long edge, folded
// with the configured ceiling below (the tighter wins).
let max_dimension = args
.get("max_dimension")
.and_then(|v| v.as_u64())
.map(|v| v.max(1) as u32);
let max_dim = {
let cfg = self.state.config.read().unwrap();
cfg.max_image_dimension
fold_max_dimension(cfg.max_image_dimension, max_dimension)
};
// `capture_mode` is DEPRECATED and ignored — get_window_state always
// returns BOTH the AT-SPI tree and a screenshot now, so the agent grounds
Expand All @@ -647,6 +676,13 @@ impl Tool for GetWindowStateTool {
// tree only (the cheap re-index path before an element ax action). A
// screenshot_out_file still forces a capture (to disk), regardless.
let include_screenshot = args.get("include_screenshot").and_then(|v| v.as_bool());
// `include_accessibility_tree` (default true) mirrors include_screenshot:
// set false to SKIP the AT-SPI walk and return just the screenshot +
// window metadata (the capture-only / preview path).
let want_tree = args
.get("include_accessibility_tree")
.and_then(|v| v.as_bool())
!= Some(false);
// screenshot_out_file: when set, write the PNG to disk and surface the
// path instead of embedding base64 in the response. `~` expands.
let screenshot_out_file = args.opt_str("screenshot_out_file").map(|s| {
Expand All @@ -669,10 +705,16 @@ impl Tool for GetWindowStateTool {
.map(|v| v.max(1) as usize);

let process_is_live = crate::proc_fs::is_process_live(pid);
// Enumerate the pid's windows ONCE and reuse the result for both the
// window-ownership check (Wayland) and the additive window metadata
// below, instead of paying for the compositor/X11 enumeration twice.
// `window_meta` also names the surface + its on-screen rectangle on the
// capture-only path, where no AT-SPI tree identifies it.
let window_meta = crate::wayland::list_windows_dispatch(Some(pid))
.into_iter()
.find(|w| w.xid == xid);
let window_matches = if crate::wayland::is_wayland() {
crate::wayland::list_windows_dispatch(Some(pid))
.iter()
.any(|window| window.xid == xid && window.pid == Some(pid))
window_meta.as_ref().is_some_and(|w| w.pid == Some(pid))
|| crate::wayland::window_was_listed_for_pid(pid, xid)
} else {
crate::x11::window_belongs_to_pid(xid, pid)
Expand All @@ -689,6 +731,13 @@ impl Tool for GetWindowStateTool {
// `include_screenshot:false` skips the grab; an unproven Wayland surface
// returns the tree with a typed screenshot error instead of unrelated pixels.
let should_capture = include_screenshot != Some(false) || screenshot_out_file.is_some();
if !want_tree && !should_capture {
return ToolResult::error(
"Nothing to return: both include_accessibility_tree:false and \
include_screenshot:false. Set at least one to true, or pass \
screenshot_out_file to force a capture.",
);
}
let observation_only = args
.get("_observation_only")
.and_then(|value| value.as_bool())
Expand All @@ -697,13 +746,19 @@ impl Tool for GetWindowStateTool {
let query_for_walk = query.clone();

let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
let tree_result = Some(crate::atspi::walk_tree_bounded(
pid,
xid,
query_for_walk.as_deref(),
max_elements,
max_depth,
));
// Skip the AT-SPI walk on the capture-only path
// (include_accessibility_tree:false).
let tree_result = if want_tree {
Some(crate::atspi::walk_tree_bounded(
pid,
xid,
query_for_walk.as_deref(),
max_elements,
max_depth,
))
} else {
None
};
// Bounds and element indices come from the same captured AT-SPI
// traversal. Joining two live walks by ordinal mis-associated
// Chromium controls when its lazy subtree changed between walks.
Expand Down Expand Up @@ -923,8 +978,15 @@ impl Tool for GetWindowStateTool {
}
// ax mode + screenshot_out_file writes the PNG to disk and
// returns b64=None — never embed the image bytes in that case.
// Keep a text content part when the image went to disk so the
// response is never empty on the capture-only path (which has
// no tree markdown either).
if let Some(b64) = b64_opt {
content.push(cua_driver_core::protocol::Content::image_png(b64));
} else if let Some(fp) = &file_path {
content.push(cua_driver_core::protocol::Content::text(format!(
"window_id={xid} pid={pid} size={w}x{h} screenshot written to {fp}"
)));
}
structured["screenshot_width"] = json!(w);
structured["screenshot_height"] = json!(h);
Expand All @@ -936,9 +998,41 @@ impl Tool for GetWindowStateTool {
structured["screenshot_file_path"] = json!(fp);
}
}
if let Some(reason) = screenshot_error {
if let Some(reason) = &screenshot_error {
structured["screenshot_frame_valid"] = json!(false);
structured["screenshot_error"] = surface_identity_unproven_error(xid, reason);
structured["screenshot_error"] =
surface_identity_unproven_error(xid, reason.clone());
}
// Window identity metadata (additive): app + title + on-screen
// rectangle for the requested window_id, useful on the
// capture-only path where no AT-SPI tree names the surface.
if let Some(meta) = &window_meta {
if !meta.app_name.is_empty() {
structured["app_name"] = json!(meta.app_name);
}
if !meta.title.is_empty() {
structured["window_title"] = json!(meta.title);
}
structured["window_bounds"] = json!({
"x": meta.x, "y": meta.y, "width": meta.width, "height": meta.height
});
}

// The capture-only path (include_accessibility_tree:false) leaves
// `content` empty when the screenshot was also unavailable — most
// often on Wayland, where per-window capture cannot prove surface
// identity. Return a structured error rather than a "successful"
// response with no content parts.
if content.is_empty() {
let reason_note = match &screenshot_error {
Some(reason) => format!(" and no screenshot could be captured ({reason})"),
None => " and no screenshot was returned".to_string(),
};
return ToolResult::error(format!(
"No content produced for window_id {xid}: the accessibility tree was \
skipped (include_accessibility_tree:false){reason_note}."
))
.with_structured(structured);
}

ToolResult {
Expand Down
Loading
Loading