From fc2160b6223740647189dd6563ee27addae7250d Mon Sep 17 00:00:00 2001 From: Wangxiaoxiaoa <1059908179@qq.com> Date: Thu, 3 Sep 2026 14:08:18 +0800 Subject: [PATCH] feat(cua-driver): expose action names in get_window_state elements Expose AT-SPI/AX/UIA action names in the structured `elements` array of `get_window_state` on Linux, macOS, and Windows. The action list is omitted when empty, and Linux filters blank/whitespace-only names. macOS observation-only snapshots (`_observation_only`) continue to emit action names but do not emit `element_token`, since no snapshot is registered in that path. Refs #3376. Salvaged from #2622 (macOS). Co-authored-by: Haoqing Wang --- .../docs/reference/cua-driver/mcp-tools.mdx | 407 +++++++++--------- .../crates/platform-linux/src/tools/impl_.rs | 168 +++++--- .../src/tools/get_window_state.rs | 223 ++++++---- .../platform-windows/src/tools/impl_.rs | 160 ++++--- 4 files changed, 574 insertions(+), 384 deletions(-) diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx index ad7e90627d..e6a875f472 100644 --- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx +++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx @@ -12,7 +12,7 @@ description: Reference for every MCP tool Cua Driver exposes import { Callout } from 'fumadocs-ui/components/callout'; -`cua-driver` exposes 56 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver ''`. +`cua-driver` exposes 60 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver ''`. Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a text content block prefixed with a `✅` summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them. See the [CLI reference](/reference/cua-driver/cli-reference) for CLI-specific options like `--socket` and `--screenshot-out-file`. @@ -30,62 +30,55 @@ For the cross-cutting parameter contract (shared parameters, required-parameter ### `list_apps` -List macOS apps — both currently running and installed-but-not-running — with per-app state flags: +List Linux apps — both currently running and installed-but-not-running — with per-app state flags: - running: is a process for this app live? (pid is 0 when false) -- active: is it the system-frontmost app? (implies running) -- launch_path: filesystem path to the `.app` bundle, when known. Pass this to `launch_app` to start the app cold. -- kind: `"desktop"` for `.app` bundles on macOS. -- last_used: RFC3339 timestamp from the bundle's filesystem mtime, when readable; otherwise null. +- active: reserved (Linux X11/Wayland focus model differs from frontmost-app); always false. +- kind: `"desktop"` for XDG `.desktop` launcher entries. +- launch_path: the launcher command from `Exec=` (field codes stripped). Pass to `launch_app(launch_path=...)`. +- bundle_id: the XDG "desktop file id" — the `.desktop` file's path relative to its XDG `applications/` root with the `.desktop` suffix stripped and path separators replaced with `-` (e.g. `kde4/konqbrowser.desktop` → `kde4-konqbrowser`). +- last_used: RFC3339 mtime of the `.desktop` file, when readable. -Only apps with NSApplicationActivationPolicyRegular are included — background helpers and system UI agents are filtered out. Installed apps come from scanning /Applications, /Applications/Utilities, ~/Applications, /System/Applications, and /System/Applications/Utilities. +Running apps come from `/proc`. Installed apps come from XDG Desktop Entry files in $XDG_DATA_HOME/applications and each $XDG_DATA_DIRS entry's applications/ subdir. Entries with `NoDisplay=true` or `Hidden=true` are filtered. A `.desktop` file whose launcher matches a running process (by basename) is merged into a single entry with `running: true`. -Use this for "is X installed?" as well as "is X running?". For per-window state — on-screen, on-current-Space, minimized, window titles — call list_windows instead. For just opening an app — running or not — call launch_app({bundle_id: ...}) directly; list_apps is not a prerequisite. +Use this for "is X installed?" as well as "is X running?". For per-window state — visibility, geometry, titles — call list_windows instead. **Arguments:** none. ### `list_windows` -List all layer-0 top-level windows currently known to WindowServer. Includes off-screen windows (minimized, on another Space, hidden-launched). Use this to find a window_id before calling get_window_state. - -Per-record fields: window_id, pid, app_name, title, bounds (x/y/width/height, top-left origin), z_index (integer or null; higher values are closer to the front; null means stacking order is unavailable and callers must not infer one), is_on_screen, space_ids, current_space_id (the active Space on that window's display), and on_current_space. The top-level current_space_id is WindowServer's main/global active Space and can differ from a record's current_space_id when displays use independent Spaces. To select a frontmost candidate, take the maximum integer z_index; if every value is null, use an explicit fallback instead of relying on array order. +List top-level windows. Each record includes z_index (integer or null; higher values are closer to the front; null means stacking order is unavailable and callers must not infer one). To select a frontmost candidate, take the maximum integer z_index; if every value is null, use an explicit fallback instead of relying on array order. **Arguments:** -- `on_screen_only` (boolean, optional): When true, drop windows not on the current Space. Default false. -- `pid` (integer, optional): Optional pid filter. When set, only this pid's windows are returned. +- `on_screen_only` (boolean, optional): When true, filter to visible windows only. Default false. +- `pid` (integer, optional) ### `get_window_state` -Walk a running app's AX tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array — pass those indices to click, type_text, press_key, etc. - -INVARIANT: call get_window_state once per turn per (pid, window_id) before any element-indexed action. The index map is replaced by the next snapshot. - -PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value` (the element's text/AXValue when present — use it to verify what a field holds), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side. +Walk a running app's AT-SPI tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array. -Always returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with `h:1` frames). You choose the modality at ACTION time, not here: an element ax action (pass `element_index`/`element_token` → the accessibility rung) or an element px action (pass `x`,`y` → the pixel rung, read straight off this screenshot). `capture_mode` is deprecated and ignored. Pass `include_screenshot:false` to skip the grab and get the tree only — the cheap path when you're just re-indexing before an element ax action. +PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `actions` (names of AT-SPI actions exposed by the element, omitted when empty), `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side. Set `query` to project BOTH representations to matching rows plus their ancestor chain while preserving original indices. `total_element_count` reports the complete snapshot and `returned_element_count` reports the projection. -The mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata — `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` — the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_dimension` caps the returned screenshot's long edge in pixels (aspect preserved) for a cheap thumbnail. +Always returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_index/element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`. -The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window — act by pixel there. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available. +The mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_dimension` caps the returned screenshot's long edge in pixels for a cheap thumbnail. -Optional `query` projects both tree_markdown and structured `elements` to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and `element_count` continues to report its total size; `filtered_element_count` reports the projected response size. - -Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25). +Optional `max_elements` / `max_depth` bound the AT-SPI walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour. **Arguments:** - `capture_mode` (string, optional): DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_index/element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. -- `include_accessibility_tree` (boolean, optional): Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). -- `include_screenshot` (boolean, optional): Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk. -- `max_depth` (integer, optional): Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees. range: 1–unbounded -- `max_dimension` (integer, optional): Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. range: 1–unbounded -- `max_elements` (integer, optional): Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows. range: 1–unbounded -- `pid` (integer, required): Target process ID. -- `query` (string, optional): Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values. -- `screenshot_out_file` (string, optional): When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output will contain screenshot_file_path instead. +- `include_accessibility_tree` (boolean, optional): Default true — walk the AT-SPI tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). +- `include_screenshot` (boolean, optional): Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action). +- `max_depth` (integer, optional): Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps. range: 1–unbounded +- `max_dimension` (integer, optional): Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview. Applied on top of the configured max_image_dimension ceiling; the tighter wins. Omit for the configured default. range: 1–unbounded +- `max_elements` (integer, optional): Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees. range: 1–unbounded +- `pid` (integer, required) +- `query` (string, optional): Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count. +- `screenshot_out_file` (string, optional): When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. -- `window_id` (integer, required): Target window ID from list_windows. +- `window_id` (integer, required): Native window identifier from list_windows. ```json {"pid":844,"window_id":10725} @@ -93,15 +86,15 @@ Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-wind ### `get_accessibility_tree` -Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. +Return a lightweight snapshot of the desktop: running processes and on-screen visible X11 windows with their bounds and owner pid. -For the full AX subtree of a single window (with interactive element indices you can click by), use `get_window_state` instead — that's the heavy per-window tool. This one is a fast discovery read that needs no TCC grants. +For the full AT-SPI subtree of a single window (with interactive element indices you can click by), use get_window_state instead — this is a fast discovery read. **Arguments:** none. ### `get_desktop_state` -Capture the full display in true screen pixels with no downscale. Use its native-size PNG as the coordinate source for actions whose target is {kind:"desktop",display_id:"primary"}. Returns the true screen size and backing scale factor. Vision-only: no AX tree walk. +Capture the full display in the desktop action coordinate frame. Use the returned PNG directly as the coordinate source for actions whose target is {kind:"desktop",display_id:"primary"}. No AT-SPI walk. **Arguments:** @@ -126,7 +119,7 @@ Return the current mouse cursor position in screen points (origin top-left). ### `get_config` -Return the current cua-driver-rs configuration. +Return current cua-driver-rs configuration. **Arguments:** none. @@ -154,34 +147,19 @@ Return the session cursor's theme, semantic playback, position, visibility, and ### `launch_app` -Launch a macOS app in the background — the target does NOT come to the foreground. - -Provide either `bundle_id` (preferred — unambiguous, e.g. `com.apple.calculator`) or `name` (e.g. "Calculator"). If both are given, bundle_id wins. - -Optional `urls` are handed to the app as open targets — for Finder, pass a folder path to open a backgrounded Finder window there. - -Browser DevTools setup belongs to `browser_prepare`, which can prove that a separate isolated profile is driver-owned before enabling CDP. - -Optional `webkit_inspector_port`: opens a WebKit inspector server on the specified port (sets WEBKIT_INSPECTOR_SERVER=127.0.0.1:N + TAURI_WEBVIEW_AUTOMATION=1). Use this for Tauri/WebKit-based apps. - -Optional `creates_new_application_instance`: when true, forces a new app instance even if one is already running (passes -n to open). Reach for this when another agent or session may drive the SAME app concurrently — it returns a fresh pid + window so each session acts on its own isolated window instead of clobbering one shared instance. Without it, single-instance apps (Calculator, many utilities) hand every caller the same window, so two sessions fight over it. - -Optional `additional_arguments`: extra argv strings appended after --args. - -Returns the launched app's pid, bundle_id, name, and a `windows` array (same shape as `list_windows`) so callers can skip an extra round-trip before `get_window_state(pid, window_id)`. `launch_state` distinguishes whether the request was sent, the process is running, and a window is ready. When the focus-steal belt-and-braces demotion check ran (target pid ≠ prior frontmost), the response also includes `self_activation_suppressed: bool` — true if focus stayed with the prior frontmost, false if the launched app held focus despite the re-demote attempt. +Launch a Linux app in the background. Provide launch_path (preferred — round-trip the value from list_apps), name (tried as a direct command, then matched against installed .desktop applications, then handed to xdg-open if it is a URL or existing file path), bundle_id (ignored on Linux), or urls (list of URLs to open). Resolution precedence: launch_path > name > bundle_id. Errors when the name resolves to nothing launchable. **Arguments:** -- `additional_arguments` (array of string, optional): Extra arguments appended after --args when launching. -- `bundle_id` (string, optional): App bundle identifier, e.g. com.apple.calculator. Preferred over name. -- `creates_new_application_instance` (boolean, optional): When true, force a new app instance even if already running (open -n). Use for concurrent multi-agent/multi-session work so each session gets an isolated instance + window instead of sharing one — on single-instance apps (e.g. Calculator) every caller otherwise gets the same window and the sessions clobber each other. -- `name` (string, optional): App display name. Used only when bundle_id is absent. -- `urls` (array of string, optional): Optional file paths or URLs to open with the app (e.g. a folder path for Finder). -- `webkit_inspector_port` (integer, optional): Open a WebKit inspector server on this port (sets WEBKIT_INSPECTOR_SERVER env var). +- `additional_arguments` (array of string, optional): Extra command-line arguments passed to the launched process. +- `bundle_id` (string, optional): Ignored on Linux (macOS/Windows concept). +- `launch_path` (string, optional): Round-trip the `launch_path` returned by `list_apps` — the Exec= command from the .desktop file with XDG field codes already stripped. Highest precedence on Linux; spawned directly via the system shell. +- `name` (string, optional): App name or command to launch. Tried as a direct command first, then matched against installed .desktop applications (exact display name, desktop-file id, or Exec basename; else an unambiguous display-name substring). +- `urls` (array of string, optional): URLs to open via xdg-open. ### `kill_app` -Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill /F equivalent on Windows). Use as escalation when the cooperative close path (hotkey cmd+q on macOS, click-the-X on Windows) failed to make the process exit. Unsaved state is lost — prefer the cooperative path first. +Force-terminate a process by pid (kill -9 equivalent on Linux). Use as escalation when the cooperative close path failed to make the process exit. Unsaved state is lost — prefer the cooperative path first. **Arguments:** @@ -193,12 +171,12 @@ Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill ### `bring_to_front` -Persistently activate an app and leave it in the foreground. Most input does not need this; use it only for a focus-proxy surface that must remain foreground across interactions. With window_id, success means the exact ordinary macOS window was independently verified as the focused window and first in WindowServer layer-0 order. Request acceptance alone is reported as a partial result, never as activation. This DOES steal foreground. +Persistently activate a window so subsequent input lands on it. This deliberately breaks the no-foreground contract and is not part of the normal input ladder. For an ordinary `background_unavailable` response, retry only the refused action with `delivery_mode:"foreground"`; the input tool performs its own activate, act, and restore sequence. Use `bring_to_front` only for a focus-proxy surface that must remain foreground across multiple calls, such as a remote desktop session, or when repeated action-scoped activation prevents the remote surface from accepting input. X11: EWMH _NET_ACTIVE_WINDOW activation (the `wmctrl -a` equivalent, with proper timestamp handling to beat focus-stealing prevention). Wayland: activates through a target-addressable compositor adapter (wlroots foreign-toplevel or the GNOME Shell helper) and refuses when the compositor offers no safe adapter. Matches the macOS / Windows bring_to_front rung. **Arguments:** - `pid` (integer, required) -- `window_id` (integer, optional) +- `window_id` (integer, optional): X11 window id (xid) to activate. If omitted, the first window of `pid` is used. ```json {"pid":844} @@ -225,31 +203,26 @@ accepts it. Omit it to use the authenticated transport's implicit lifecycle sess ### `click` -Click against a target pid. **Prefer `element_token` over pixel coordinates** — the token works on backgrounded / minimized / hidden / off-Space windows, identifies one exact snapshot element, and tells you what you're clicking via the cached element's role + label. Reach for `x, y` only when the target is a canvas / video / WebGL / custom-drawn surface that doesn't appear in the AX tree. - -Two addressing modes: +Click against a target pid. **Prefer `element_index` over pixel coordinates** — element_index works on backgrounded / hidden windows, surfaces a stable handle, and tells you what you're clicking via the cached AT-SPI element's role + label. Reach for `x, y` only when the target is a canvas / custom-drawn surface that doesn't appear in the AT-SPI tree. -- element_token, or element_index + snapshot_id (from get_window_state): AX action path. Works on backgrounded/hidden windows. No cursor move, no focus steal. The snapshot cache is scoped per (pid, window_id) and is replaced by the next snapshot of the same window — re-snapshot every turn before clicking. +Provide either (window_id + x/y) or (pid + element_index). Routes via XSendEvent (no focus steal). element_index cache is scoped per (pid, window_id) and is replaced by the next get_window_state of the same window — re-snapshot every turn before clicking. -- x, y (window-local screenshot pixels, top-left origin of the PNG returned by get_window_state): CGEvent path. Synthesizes mouse events and posts to pid. Use modifier for cmd/shift/option/ctrl. Needs a visible on-screen window to anchor the conversion. +After a zoom call, pass from_zoom=true to auto-translate zoom-image coords back to full-window space. -button: "left" (default), "right", or "middle". Defaults to left so the field is fully back-compat — omit it and you get the legacy left-click behaviour. Pixel path: routes through the CGEvent left/right/middle mouse-button primitives. AX path: "right" maps to AXShowMenu (same surface as the dedicated `right_click` tool); "middle" has no AX equivalent and falls back to a pixel middle-click at the element's center. -action: press (default), show_menu, pick, confirm, cancel, open. -from_zoom: set true after a zoom call to auto-translate zoom-image pixel coordinates to full-window space. +button: "left" (default), "right", or "middle". Defaults to left so the field is fully back-compat. X11: routes through XSendEvent ButtonPress/Release with the matching button code. Native Wayland: only left-button is supported via the virtual-pointer protocol — right/middle return an error rather than silently degrading to left. `modifier` holds ctrl/shift/alt/super for the click on X11. Native Wayland refuses modified pointer clicks until its input protocol can carry keyboard modifier state. **Arguments:** -- `action` (string, optional): AX action: press, show_menu, pick, confirm, cancel, open. -- `button` (string, optional): Mouse button. Default: "left" — omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps "right" to AXShowMenu and falls back to a pixel middle-click at the element's center for "middle". -- `count` (integer, optional): Click count (pixel path only). Default 1. -- `debug_image_out` (string, optional): Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom. -- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require "foreground" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground". +- `button` (string, optional): Mouse button. Default: "left" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error. +- `count` (integer, optional) +- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'. +- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"` - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. -- `from_zoom` (boolean, optional): When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates. -- `modifier` (array of string, optional): Modifier keys: cmd, shift, option/alt, ctrl. -- `pid` (integer, optional): Target process ID. -- `scope` (string, optional): Coordinate frame for a windowless screen-absolute click (default "window"). Pass "desktop" when sending x,y with NO pid/window_id — the coordinates are then true screen pixels (read from get_desktop_state with scope="desktop"). Per-call; not a setting. +- `from_zoom` (boolean, optional): Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space. +- `modifier` (array of string, optional): Modifier keys held during the action: cmd, shift, option/alt, ctrl. +- `pid` (integer, optional) +- `scope` (string, optional): default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. @@ -257,29 +230,27 @@ from_zoom: set true after a zoom call to auto-translate zoom-image pixel coordin `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. -- `window_id` (integer, optional): Target window ID. Required for element_index. Optional when element_token is supplied (the token carries it). -- `x` (number, optional): X in screenshot pixels. A window target uses the get_window_state PNG; a desktop target uses the native get_desktop_state PNG. The driver reverses Retina backing scale and any window-image downscale. -- `y` (number, optional): Y in screenshot pixels from the image selected by target. +- `window_id` (integer, optional) +- `x` (number, optional) +- `y` (number, optional) ### `double_click` -Double-click at (x, y) or on an AX element identified by element_index + window_id. - -AX path (element_index provided): performs `AXOpen` when the element advertises it (Finder items, openable list rows/cells); otherwise resolves the element's on-screen center and falls back to a pixel double-click there. - -Pixel path (x, y provided): two down/up pairs ~80 ms apart at the given coordinates. +Double-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No focus steal. Provide either (window_id + x/y) or (pid + element_index). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords. **Arguments:** -- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. +- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'. +- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"` - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. +- `from_zoom` (boolean, optional): Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space. - `pid` (integer, required) - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. -- `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it). -- `x` (number, optional): Screen X coordinate (pixel path). -- `y` (number, optional): Screen Y coordinate (pixel path). +- `window_id` (integer, optional) +- `x` (number, optional) +- `y` (number, optional) ```json {"pid":844} @@ -287,26 +258,22 @@ Pixel path (x, y provided): two down/up pairs ~80 ms apart at the given coordina ### `right_click` -Right-click against a target pid. Two addressing modes: - -- `element_index` + `window_id` (from the last `get_window_state` snapshot) — performs `AXShowMenu` on the cached element. Pure AX RPC, works on backgrounded / hidden windows, no cursor move or focus steal. Requires a prior `get_window_state(pid, window_id)` in this turn. - -- `x`, `y` — synthesizes `rightMouseDown` / `rightMouseUp` CGEvent pair posted to the pid. Driver converts image-pixel → screen-point internally. `modifier` forces the CGEvent path (AX actions don't propagate modifier keys). - -Exactly one of `element_index` or (`x` AND `y`) must be provided. `pid` always required. `window_id` required when `element_index` is used. +Right-click at (x,y) or an element_index (AT-SPI bounds) via XSendEvent. No focus steal. Provide either (window_id + x/y) or (pid + element_index). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords. **Arguments:** -- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. +- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'. +- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"` - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. -- `modifier` (array of string, optional): Modifier keys held during the right-click: cmd/shift/option/ctrl. Pixel path only. -- `pid` (integer, required): Target process ID. +- `from_zoom` (boolean, optional): Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space. +- `modifier` (array of string, optional): Modifier keys held during the action: cmd, shift, option/alt, ctrl. +- `pid` (integer, required) - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. -- `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it). -- `x` (number, optional): X in window-local screenshot pixels. Must be provided together with y. -- `y` (number, optional): Y in window-local screenshot pixels. Must be provided together with x. +- `window_id` (integer, optional) +- `x` (number, optional) +- `y` (number, optional) ```json {"pid":844} @@ -314,37 +281,30 @@ Exactly one of `element_index` or (`x` AND `y`) must be provided. `pid` always r ### `drag` -Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window. - -Use for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel. - -`duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures. - -`modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture. - -When `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching. +Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels via XSendEvent (ButtonPress + MotionNotify × steps + ButtonRelease). duration_ms (default 500), steps (default 20). No focus steal. **Arguments:** -- `button` (string, optional): Mouse button used for the drag. Default: left. -- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. -- `duration_ms` (integer, optional): Wall-clock duration of the drag path between mouseDown and mouseUp. Default: 500. range: 0–10000 -- `from_x` (number, required): Drag-start X in window-local screenshot pixels. Top-left origin. -- `from_y` (number, required): Drag-start Y in window-local screenshot pixels. Top-left origin. -- `from_zoom` (boolean, optional): When true, coordinates are in the last zoom image for this pid; driver maps back to window coordinates. -- `modifier` (array of string, optional): Modifier keys held across the entire gesture: cmd/shift/option/ctrl. -- `pid` (integer, optional): Target process ID. -- `scope` (string, optional): Use desktop with no pid/window_id for native get_desktop_state screenshot coordinates. default: `"window"` +- `button` (string, optional): Mouse button. Default "left". +- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'. +- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"` +- `duration_ms` (integer, optional): Total drag duration. Default: 500. range: 0–10000 +- `from_x` (number, required) +- `from_y` (number, required) +- `from_zoom` (boolean, optional) +- `modifier` (array of string, optional): Modifier keys held during the action: cmd, shift, option/alt, ctrl. +- `pid` (integer, optional) +- `scope` (string, optional): default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. -- `steps` (integer, optional): Number of intermediate mouseDragged events linearly interpolated along the path. Default: 20. range: 1–200 +- `steps` (integer, optional): Intermediate MotionNotify events. Default: 20. range: 1–200 - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. -- `to_x` (number, required): Drag-end X in window-local screenshot pixels. -- `to_y` (number, required): Drag-end Y in window-local screenshot pixels. -- `window_id` (integer, optional): CGWindowID for the window the pixel coordinates were measured against. Optional only when pid owns exactly one eligible top-level window; otherwise the action refuses with ambiguous_window_target. +- `to_x` (number, required) +- `to_y` (number, required) +- `window_id` (integer, optional): Target window XID. Required. ```json {"from_x":100,"from_y":200,"to_x":100,"to_y":200} @@ -352,20 +312,15 @@ than silently changing coordinate spaces. ### `type_text` -Insert text into the target pid via `AXSetAttribute(kAXSelectedText)`. Works for standard Cocoa text fields and text views. No keystrokes are synthesized — special keys (Return / Escape / arrows) go through `press_key` / `hotkey`. For Chromium / Electron inputs that don't implement `kAXSelectedText`, the tool falls back to CGEvent character synthesis automatically when the estimated route stays within the daemon transport budget. Longer synthesized routes are refused before character events and return a safe chunk size; one-call AX insertion remains uncapped. - -Optional `element_index` + `window_id` (from the last `get_window_state` snapshot) directs the write to a specific field. Without `element_index`, the write goes to the pid's currently focused element. - -WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's compose box): AXValue is not independent proof that the renderer/DOM observed an AX write or synthesized keystrokes. The driver detects this at the element level (an AXWebArea ancestor) and refuses to trust AXValue-only read-back there — type_text returns effect:"unverifiable" + escalation, never a false "confirmed" (a browser's own native address bar/toolbar stays trusted). For a browser TAB the reliable path is the `page` tool (drives the DOM via CDP); for an embedded web view use this tool's px form: pass x,y (no element_index) to pixel-click the field then type, in one call. NOTE: a px focus-click won't reliably open+focus a CLOSED control; AX-press to open/activate it first (works in the background), then px-type. Always confirm via the screenshot; if px-background still drops, escalate to delivery_mode:"foreground". +Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal. **Arguments:** -- `delay_ms` (integer, optional): Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds. range: 0–200 -- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": AX insert, then CGEvent keystrokes if needed — no focus steal; native controls can be confirmed via AXValue read-back, while web-content writes remain effect:"unverifiable". "foreground": briefly front the window, type, restore the prior frontmost — the explicit last resort for focus-sensitive surfaces (e.g. WhatsApp/Catalyst) where background keystrokes don't land. Re-call with "foreground" when a background attempt remains unverifiable and a fresh snapshot shows the text did not appear. +- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"` - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. -- `pid` (integer, optional): Target process ID. -- `scope` (string, optional): Use desktop with no pid/window_id to type into the frontmost application. default: `"window"` +- `pid` (integer, optional) +- `scope` (string, optional): default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. @@ -373,8 +328,8 @@ WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's comp `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. -- `text` (string, required): Text to insert at the target's cursor. -- `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it). +- `text` (string, required) +- `window_id` (integer, optional) - `x` (number, optional): Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click. - `y` (number, optional): Screenshot-pixel Y of the field (see x). @@ -384,21 +339,17 @@ than silently changing coordinate spaces. ### `press_key` -Press and release a single key. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default: -• `background` (default): post to the pid WITHOUT fronting/raising — the auth-message path (Chromium-safe). With element_index it focuses that AX element first. `window_id` only targets; it does not raise. -• `foreground`: guard and briefly front the exact window, focus an addressed AX element when supplied, send a genuine HID key transition so Chromium content, inline editors, and native menu equivalents receive it, then restore prior frontmost. Requires window_id. - -A key press is confirmed only when a bounded native AX value/selection read-back changes on the same control. Otherwise a successfully attempted post remains effect:"unverifiable" without implying delivery failure or recommending foreground. Key names: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, plus any letter or digit. Modifiers array: cmd, shift, option/alt, ctrl, fn. +Press a key via XSendEvent to a window. No focus steal. **Arguments:** -- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. +- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"` - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. -- `key` (string, required): Key name: return, tab, escape, up, down, etc. -- `modifiers` (array of string, optional): Modifier keys: cmd, shift, option/alt, ctrl, fn. +- `key` (string, required) +- `modifiers` (array of string, optional) - `pid` (integer, optional) -- `scope` (string, optional): Use desktop with no pid/window_id to send the key to the frontmost application. default: `"window"` +- `scope` (string, optional): default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. @@ -406,7 +357,7 @@ A key press is confirmed only when a bounded native AX value/selection read-back `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. -- `window_id` (integer, optional): Target window. Required for delivery_mode:"foreground". Does NOT itself raise the window — raising is gated on delivery_mode. +- `window_id` (integer, optional) - `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index. - `y` (number, optional): Screenshot-pixel Y (see x). @@ -416,22 +367,16 @@ than silently changing coordinate spaces. ### `hotkey` -Press a key combination — e.g. `["cmd", "c"]` for Copy, `["cmd", "shift", "4"]` for screenshot selection. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default: -• `background` (default): post the combo to the target pid WITHOUT fronting or raising it — uses the macOS 14+ auth-message envelope so Chromium/Electron accept it as trusted live input. With an AX target, focus that exact element first. No top-level focus steal. `window_id` here only targets the combo; it does not raise. -• `foreground`: briefly front the window (NSMenu path, < 1 ms via SLPSSetFrontProcessWithOptions) so native menu key-equivalents (Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost — the explicit escalation for menu-bar shortcuts on non-Chromium apps that ignore a background combo. With an AX target or x,y, the focused field receives the chord through the foreground HID queue (needed by native Chromium fields such as the omnibox). Requires window_id. - -A combo is never driver-verifiable (no read-back) → effect:"unverifiable"; confirm via screenshot. NOTE: a keyboard combo does NOT focus a text field — to type into a backgrounded Electron input, establish real renderer focus with a PIXEL click first, then `type_text`. If an app only accepts paste, call `clipboard_write`, then `clipboard_read` and verify its types (and text when applicable) before selecting or replacing editor content; only then send Cmd+V. - -Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modifier keys use the same vocabulary as `press_key`. Order: modifiers first, one non-modifier last. +Press a combination of keys simultaneously, e.g. ["ctrl","c"] for Copy. Sent via XSendEvent directly to the target pid; target does NOT need to be frontmost. **Arguments:** -- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. +- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"` - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. -- `keys` (array of string, required): Modifier(s) and one non-modifier key, e.g. ["cmd", "c"]. items: 2–unbounded -- `pid` (integer, optional): Target process ID. -- `scope` (string, optional): Use desktop with no pid/window_id to send the chord to the frontmost application. default: `"window"` +- `keys` (array of string, required): Modifier(s) + one non-modifier key, e.g. ["ctrl","c"]. items: 2–unbounded +- `pid` (integer, optional) +- `scope` (string, optional): default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. @@ -439,8 +384,8 @@ Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modi `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. -- `window_id` (integer, optional): Target window. Required for delivery_mode:"foreground" (the NSMenu activation needs a window). Does NOT itself raise the window — raising is gated on delivery_mode. -- `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Cmd+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach. +- `window_id` (integer, optional) +- `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach. - `y` (number, optional): Screenshot-pixel Y (see x). ```json @@ -449,13 +394,7 @@ than silently changing coordinate spaces. ### `set_value` -Set a value on a UI element. Two modes depending on element role: - -- **AXPopUpButton / select dropdown**: finds the child option whose title or value matches `value` (case-insensitive) and AXPresses it directly — the native macOS popup menu is never opened, so focus is never stolen. Use this for HTML <select> elements in Safari or any native NSPopUpButton. - -- **All other elements**: writes AXValue directly (sliders, steppers, date pickers, native text fields that expose settable AXValue). - -For free-form text entry into web inputs, prefer `type_text_chars` which synthesises key events — AXValue writes are ignored by WebKit. +Set value of an AT-SPI element via SetValue action. **Arguments:** @@ -464,8 +403,8 @@ For free-form text entry into web inputs, prefer `type_text_chars` which synthes - `pid` (integer, required) - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. -- `value` (string, required): New value. AX will coerce to the element's native type. -- `window_id` (integer, optional): CGWindowID for the window whose get_window_state produced the element_index. Required when element_index is used; optional when element_token is supplied (the token carries it). +- `value` (string, required) +- `window_id` (integer, optional): Required when element_index is used; optional when element_token is supplied (the token carries it). ```json {"pid":844,"value":"42"} @@ -473,24 +412,19 @@ For free-form text entry into web inputs, prefer `type_text_chars` which synthes ### `scroll` -Scroll the target pid. Two paths, picked by how you address the scroll: - -• **Targeted wheel path** — when you pass a target, either `element_index`/`element_token` (preferred) or window-local `x, y` pixels: the driver synthesizes a real mouse-wheel event (CGEventCreateScrollWheelEvent, at that screen point. The renderer hit-tests the wheel at the cursor, so the scroll lands on whatever element is under the point — exactly like physically rolling the wheel over it. This is the ONLY way to scroll a nested `overflow:auto` region (e.g. a scrollable <div> with no tabindex): such regions never take keyboard focus, so the keystroke path below no-ops on them. Use this for inner/nested scrollers in web views. - -• **Keystroke path (focused region)** — when you pass NO target (just pid + direction): synthesizes PageDown/PageUp (by='page') or Down/Up arrows (by='line'); horizontal uses Left/Right arrows. Drives the focused / page scroller only. - -Mapping: by='page' → larger step; by='line' → smaller step; amount = number of wheel notches (targeted path) or keystroke repetitions (keystroke path). +Scroll the target pid's focused region via XSendEvent Button4/5. direction required; by defaults to line, amount defaults to 3. **Arguments:** -- `amount` (integer, optional): Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Default: 3. range: 1–50 -- `by` (string, optional): Scroll granularity. Default: line. -- `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. -- `direction` (string, required): Scroll direction. +- `amount` (integer, optional): range: 1–50 +- `by` (string, optional) +- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'. +- `delivery_mode` (string, optional): Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface. default: `"background"` +- `direction` (string, required) - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. - `pid` (integer, optional) -- `scope` (string, optional): Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates. default: `"window"` +- `scope` (string, optional): default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. @@ -499,8 +433,8 @@ Mapping: by='page' → larger step; by='line' → smaller step; amount = number Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. - `window_id` (integer, optional) -- `x` (number, optional): Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion. -- `y` (number, optional): Window-local screenshot Y. See `x`. +- `x` (number, optional): Window-local screenshot-pixel X of the scroll target. Pass with y and without element_index. +- `y` (number, optional): Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_index. ```json {"direction":"up"} @@ -508,11 +442,11 @@ than silently changing coordinate spaces. ### `move_cursor` -Move a cursor to (x, y). In window scope (default), moves only the agent cursor overlay. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates. +Move the synthetic agent cursor without changing the user's pointer. Only an explicit scope=desktop request moves the real OS pointer in get_desktop_state coordinates. **Arguments:** -- `cursor_id` (string, optional): Cursor instance to move. Default: 'default'. +- `cursor_id` (string, optional) - `scope` (string, optional): default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `target` (window target or desktop target, optional): Preferred per-call target. New callers should set this field. @@ -525,18 +459,18 @@ Move a cursor to (x, y). In window scope (default), moves only the agent cursor ### `zoom` -Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixel coordinates, with 20% padding added on each side. The output image is at most 500 px wide. +Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixels, with 20% padding. Output is at most 500 px wide. -After a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordinates back to full-window space. +After a zoom, pass from_zoom=true to click/type_text to auto-translate coordinates back to full-window space. **Arguments:** - `pid` (integer, optional): Target pid — required for from_zoom click/type translation. -- `window_id` (integer, required): CGWindowID from list_windows. -- `x1` (number, required): Left edge of region in screenshot pixels. -- `x2` (number, required): Right edge of region in screenshot pixels. -- `y1` (number, required): Top edge of region in screenshot pixels. -- `y2` (number, required): Bottom edge of region in screenshot pixels. +- `window_id` (integer, required) +- `x1` (number, required) +- `x2` (number, required) +- `y1` (number, required) +- `y2` (number, required) ```json {"window_id":10725,"x1":100,"x2":100,"y1":200,"y2":200} @@ -665,16 +599,21 @@ Caveats: ### `set_config` -Update cua-driver-rs configuration. Changes to max_image_dimension take effect immediately. The experimental_pip keys are persisted to ~/.cua-driver/config.json and take effect on the next daemon restart (the PiP backend is initialised once at startup). +Update cua-driver-rs configuration. capture_mode / max_image_dimension take effect immediately. + +Two input shapes (both accepted, matching Windows/Swift): +- **{key, value}** (preferred): `{"key": "max_image_dimension", "value": 800}` — single leaf write. +- **Legacy per-field**: `{"capture_mode": "som", "max_image_dimension": 0}`. -Note: capture_mode is a per-call param (on get_window_state / click), not a stored setting. Capture modality is selected by each action's target; the old capture_scope config key is retired. +The experimental_pip keys persist to ~/.cua-driver/config.json and apply on next daemon restart (the PiP backend is initialised once at startup; Linux ships only the trait stub today — see issue #1729). **Arguments:** -- `experimental_pip` (boolean, optional): Enable the experimental picture-in-picture preview window. Applies on next daemon restart. -- `experimental_pip_geometry` (string, optional): PiP window size + optional position in `WxH` or `WxH+X+Y` form (e.g. `320x200+24+24`). Applies on next daemon restart. -- `key` (string, optional): Name of a single config field to write ({key, value} shape, matching the CLI `config set` and the Windows/Linux tools). Pair with `value`. Equivalent to passing the field directly. -- `max_image_dimension` (integer, optional): Max dimension for screenshot resizing (0 = no limit). +- `capture_mode` (string, optional): Legacy per-field shape. Default capture mode for get_window_state. ("som"/"screenshot" still decode as deprecated aliases.) +- `experimental_pip` (boolean, optional): Enable the experimental PiP preview window (applies next restart; Linux backend stubbed). +- `experimental_pip_geometry` (string, optional): PiP window size + optional position in `WxH` or `WxH+X+Y` form. +- `key` (string, optional): Name of a single config field to write ({key, value} shape). Pair with `value`. +- `max_image_dimension` (integer, optional): Legacy per-field shape. Max dimension for screenshot resizing (0 = no limit). - `value` (unknown, optional): New value for `key`. JSON type depends on the key. ### `start_session` @@ -738,14 +677,9 @@ Configure only movement physics and visibility timing for a session cursor. ### `check_permissions` -Report TCC permission status for Accessibility and Screen Recording. By default also raises the system permission dialogs for any missing grants — Apple's request APIs are no-ops when the grant is already active, so this is safe to call repeatedly. Pass {"prompt": false} for a purely read-only status check. +Check required permissions for cua-driver-rs on Linux. -Returns: `accessibility` + `screen_recording` (booleans from the TCC preflight APIs), `screen_recording_capturable` (a live ScreenCaptureKit probe when `prompt` is true; null on read-only calls), `direct_capture_status` (`ready`, `unavailable`, `timed_out`, `probe_failed`, `blocked_by_screen_recording`, or `not_checked`), `direct_capture_error` (a structured timeout/probe failure when applicable), `direct_capture_verification` (validated source, UTC time, and bundle identity from an explicit grant probe), and `source` (which TCC identity the booleans reflect: the CuaDriver daemon vs the launching terminal/IDE). macOS attributes grants to the responsible process, so a standalone call from a terminal reports the terminal's grants, not the driver's. The prompt-capable ScreenCaptureKit probe never runs when `prompt` is false. Pass `probe_direct_capture:false` with `prompt:true` to register/request only the two required TCC grants before separately explaining Tahoe's direct-capture consent. - -**Arguments:** - -- `probe_direct_capture` (boolean, optional): When prompting and Screen Recording is granted, also run the live ScreenCaptureKit probe that may raise Tahoe's direct-capture consent. Default true. Set false for a staged Accessibility/Screen Recording request. -- `prompt` (boolean, optional): Raise the system permission prompts for missing grants. Default false; only a trusted host setup route may set true. default: `false` +**Arguments:** none. ### `health_report` @@ -850,6 +784,71 @@ accepts it. Omit it to use the authenticated transport's implicit lifecycle sess {"path":["example"],"pid":844,"window_id":10725} ``` +### `mouse_button_down` + +Press and hold a mouse button at (x,y) via background X11 delivery. Does not release the button; pair with mouse_drag / mouse_button_up. Returns the current held-button state. + +**Arguments:** + +- `button` (string, optional): Mouse button. Default "left". +- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'. +- `from_zoom` (boolean, optional): Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space. +- `pid` (integer, required) +- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id. +- `window_id` (integer, required) +- `x` (number, required) +- `y` (number, required) + +```json +{"pid":844,"window_id":10725,"x":100,"y":200} +``` + +### `mouse_drag` + +Move a previously-held mouse button to a new point via background X11 delivery. Requires an active mouse_button_down state; does not release the button. Returns the updated held-button state. + +**Arguments:** + +- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'. +- `duration_ms` (integer, optional): Total drag duration. Default: 500. range: 0–10000 +- `from_zoom` (boolean, optional): Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space. +- `pid` (integer, optional) +- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id. +- `steps` (integer, optional): Intermediate MotionNotify events. Default: 20. range: 1–200 +- `window_id` (integer, optional) +- `x` (number, required) +- `y` (number, required) + +```json +{"x":100,"y":200} +``` + +### `mouse_button_up` + +Release a previously-held mouse button via background X11 delivery. If x/y are omitted, releases at the last held position. Returns the current held-button state. + +**Arguments:** + +- `cursor_id` (string, optional): Optional multi-cursor instance id. Default: 'default'. +- `from_zoom` (boolean, optional): Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space. +- `pid` (integer, optional) +- `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id. +- `window_id` (integer, optional) +- `x` (number, optional) +- `y` (number, optional) + +### `parallel_mouse_drag` + +Run multiple mouse drag gestures concurrently via Linux MPX/XI2 virtual master pointers. Each drag item runs on its own session-scoped master pointer (true same-window concurrent draws on X11). Each item presses once, glides continuously through its whole path, and releases once — one smooth held drag, not a chain of clicks. A path is given either as a straight segment (from_x/from_y → to_x/to_y) or as a function `fn` = y(x) sampled over [x_from, x_to] in window-local pixels (e.g. fn:"x" is a diagonal, fn:"300+120*sin(x/40)" a sine wave). Functions support + - * / ^, sin/cos/tan, sqrt, abs, exp, ln, pi, e. + +**Arguments:** + +- `drags` (array of object, required): items: 2–unbounded + +```json +{"drags":["example"]} +``` + ### `set_agent_cursor_theme` Select an already-installed cursor theme for a session. diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs index 0c7a8a44d6..b963bcec77 100644 --- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs @@ -574,6 +574,67 @@ fn fold_max_dimension(ceiling: u32, per_call: Option) -> u32 { } } +/// Build a single structured element entry for `get_window_state`. +/// Returns `None` when the node has no `element_index` (non-actionable rows). +fn build_element_entry( + n: &crate::atspi::AtspiNode, + snapshot_id: Option, + bounds: Option<(i32, i32, u32, u32)>, +) -> Option { + let idx = n.element_index?; + // `label` mirrors what a human reading the markdown row would call this + // element: name first, then value, then description. + let label = n + .name + .clone() + .or_else(|| n.value.clone()) + .or_else(|| n.description.clone()); + let mut entry = json!({ + "element_index": idx, + "role": n.role, + "depth": n.depth, + }); + if let Some(snapshot_id) = snapshot_id { + entry["element_token"] = json!(cua_driver_core::element_token::token_for(snapshot_id, idx)); + } + if n.in_web_content { + entry["in_web_content"] = json!(true); + } + if let Some(label) = label { + entry["label"] = json!(label); + } + // Surface the element's value separately from `label` (which collapses + // name→value→description): a field with both a name AND typed text would + // otherwise hide the text from a caller reading the structured side, + // leaving it only in tree_markdown. See the macOS get_window_state builder + // for the rationale. + if let Some(value) = n.value.clone().filter(|v| !v.is_empty()) { + entry["value"] = json!(value); + } + if let Some(enabled) = n.enabled { + entry["enabled"] = json!(enabled); + } + if let Some(selected) = n.selected { + entry["selected"] = json!(selected); + } + let actions: Vec = n + .actions + .iter() + .filter(|a| !a.trim().is_empty()) + .cloned() + .collect(); + if !actions.is_empty() { + entry["actions"] = json!(actions); + } + if let Some(parent) = n.parent_element_index { + entry["parent_index"] = json!(parent); + } + if let Some((x, y, w, h)) = bounds { + entry["frame"] = json!({ "x": x, "y": y, "w": w, "h": h }); + } + Some(entry) +} + pub struct GetWindowStateTool { state: Arc, } @@ -592,7 +653,8 @@ impl Tool for GetWindowStateTool { the structured array.\n\n\ PREFERRED CONSUMERS read `structuredContent.elements` (one entry \ per indexed row with `element_index`, `role`, `label`, `value`, \ - `enabled`, `selected`, \ + `enabled`, `selected`, `actions` (names of AT-SPI actions exposed \ + by the element, omitted when empty), \ `frame: {x,y,w,h}` when AT-SPI reports usable bounds, \ `parent_index`, `depth`). The markdown `tree_markdown` stays \ available and unchanged in shape for existing text-parsing \ @@ -845,7 +907,7 @@ impl Tool for GetWindowStateTool { // Structured `elements` array: one entry per actionable node. // Shape: `{element_index, element_token, role, label, - // depth, parent_index?, frame?: {x,y,w,h}}`. Frame is + // depth, actions?, parent_index?, frame?: {x,y,w,h}}`. Frame is // included whenever AT-SPI Component.GetExtents(Screen) // reported usable bounds; omitted otherwise (some // toolkits leave bounds unset on hidden / virtual @@ -859,53 +921,11 @@ impl Tool for GetWindowStateTool { .nodes .iter() .filter_map(|n| { - let idx = n.element_index?; - // `label` mirrors what a human reading the markdown row - // would call this element: name first, then value, - // then description. - let label = n - .name - .clone() - .or_else(|| n.value.clone()) - .or_else(|| n.description.clone()); - let mut entry = json!({ - "element_index": idx, - "role": n.role, - "depth": n.depth, - }); - if let Some(snapshot_id) = snapshot_id { - entry["element_token"] = json!( - cua_driver_core::element_token::token_for(snapshot_id, idx) - ); - } - if n.in_web_content { - entry["in_web_content"] = json!(true); - } - if let Some(label) = label { - entry["label"] = json!(label); - } - // Surface the element's value separately from `label` - // (which collapses name→value→description): a field - // with both a name AND typed text would otherwise hide - // the text from a caller reading the structured side, - // leaving it only in tree_markdown. See the macOS - // get_window_state builder for the rationale. - if let Some(value) = n.value.clone().filter(|v| !v.is_empty()) { - entry["value"] = json!(value); - } - if let Some(enabled) = n.enabled { - entry["enabled"] = json!(enabled); - } - if let Some(selected) = n.selected { - entry["selected"] = json!(selected); - } - if let Some(parent) = n.parent_element_index { - entry["parent_index"] = json!(parent); - } - if let Some((x, y, w, h)) = bounds_by_idx.get(&idx).copied() { - entry["frame"] = json!({ "x": x, "y": y, "w": w, "h": h }); - } - Some(entry) + build_element_entry( + n, + snapshot_id, + bounds_by_idx.get(&n.element_index?).copied(), + ) }) .collect(); let elements = cua_driver_core::element_query::project_elements_for_query( @@ -1078,6 +1098,56 @@ mod get_window_state_capture_tests { } } +#[cfg(test)] +mod get_window_state_actions_tests { + use super::*; + use crate::atspi::AtspiNode; + + fn node(actions: Vec) -> AtspiNode { + AtspiNode { + element_index: Some(1), + role: "button".to_owned(), + name: Some("ok".to_owned()), + value: None, + checked: None, + enabled: Some(true), + selected: None, + description: None, + actions, + element_key: 1, + depth: 0, + parent_element_index: None, + in_web_content: false, + } + } + + #[test] + fn element_entry_includes_actions_when_present() { + let n = node(vec!["Press".to_owned(), "Open".to_owned()]); + let entry = build_element_entry(&n, None, None).unwrap(); + assert_eq!(entry["actions"], json!(["Press", "Open"])); + } + + #[test] + fn element_entry_omits_actions_when_empty() { + let n = node(Vec::new()); + let entry = build_element_entry(&n, None, None).unwrap(); + assert!(entry.get("actions").is_none()); + } + + #[test] + fn element_entry_filters_blank_action_names() { + let n = node(vec![ + "Press".to_owned(), + "".to_owned(), + " ".to_owned(), + "Open".to_owned(), + ]); + let entry = build_element_entry(&n, None, None).unwrap(); + assert_eq!(entry["actions"], json!(["Press", "Open"])); + } +} + pub struct LaunchAppTool; static LAUNCH_DEF: std::sync::OnceLock = std::sync::OnceLock::new(); diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs index f49892b8f1..01e56b5709 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs @@ -33,7 +33,8 @@ fn def() -> &'static ToolDef { PREFERRED CONSUMERS read `structuredContent.elements` (one entry per \ indexed row with `element_index`, `role`, `label`, `value` (the \ element's text/AXValue when present — use it to verify what a field \ - holds), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown \ + holds), `actions` (names of AX actions exposed by the element, \ + omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown \ `tree_markdown` stays available \ and unchanged in shape for existing text-parsing callers — but new \ fields will only be added to the structured side.\n\n\ @@ -565,8 +566,8 @@ impl Tool for GetWindowStateTool { // (Hermes' regex parser, Codex, Claude Code) and is signalled as // preferred-for-back-compat-only via the `_note` field below. let elements_json: Vec = match (snapshot_id, tree_result.as_ref()) { - (Some(sid), Some(r)) => build_elements_array_with_token(&r.nodes, sid), - (None, Some(r)) if scope_matched => build_elements_array(&r.nodes), + (Some(sid), Some(r)) => build_elements_array_with_token(&r.nodes, Some(sid)), + (None, Some(r)) if scope_matched => build_elements_array_with_token(&r.nodes, None), _ => Vec::new(), }; let elements_json = cua_driver_core::element_query::project_elements_for_query( @@ -841,7 +842,7 @@ fn degradation_for( /// omitted to match the contract on the tool description. pub(crate) fn build_elements_array_with_token( nodes: &[crate::ax::tree::AXNode], - snapshot_id: u32, + snapshot_id: Option, ) -> Vec { nodes .iter() @@ -861,15 +862,18 @@ pub(crate) fn build_elements_array_with_token( .map(|[x, y, w, h]| serde_json::json!({ "x": x, "y": y, "w": w, "h": h })); let mut entry = serde_json::json!({ "element_index": idx, - // Surface 6: opaque token paired to the integer index. - // Tools accept either; the token has explicit validity - // (invalidated when the next snapshot supersedes this - // one in the per-pid LRU). See cua-driver-core's - // `element_token` module. - "element_token": cua_driver_core::element_token::token_for(snapshot_id, idx), "role": node.role, "depth": node.depth, }); + // Surface 6: opaque token paired to the integer index. + // Tools accept either; the token has explicit validity + // (invalidated when the next snapshot supersedes this + // one in the per-pid LRU). See cua-driver-core's + // `element_token` module. + if let Some(sid) = snapshot_id { + entry["element_token"] = + serde_json::json!(cua_driver_core::element_token::token_for(sid, idx)); + } if let Some(label) = label { entry["label"] = serde_json::Value::String(label); } @@ -924,6 +928,9 @@ pub(crate) fn build_elements_array_with_token( if let Some(selected) = selected { entry["selected"] = serde_json::Value::Bool(selected); } + if !node.actions.is_empty() { + entry["actions"] = serde_json::json!(node.actions); + } if node.in_web_content { entry["in_web_content"] = serde_json::Value::Bool(true); } @@ -938,26 +945,6 @@ pub(crate) fn build_elements_array_with_token( .collect() } -/// Back-compat wrapper for callers that don't yet have a snapshot id -/// to pass through. Emits the same fields as the token-aware builder -/// minus `element_token`. New call sites should prefer -/// `build_elements_array_with_token`. -#[allow(dead_code)] -pub(crate) fn build_elements_array(nodes: &[crate::ax::tree::AXNode]) -> Vec { - // Use a snapshot_id of 0 only to satisfy the signature; tokens - // built from id=0 are not registered and would fail the registry's - // stale check — but since this entry point is only kept for - // pre-existing callers (none in production after Surface 6), it - // strips the token field after rendering. - let mut out = build_elements_array_with_token(nodes, 0); - for entry in &mut out { - if let Some(obj) = entry.as_object_mut() { - obj.remove("element_token"); - } - } - out -} - /// Keep the structured response aligned with a query-filtered markdown tree. /// /// The AX walker deliberately keeps the complete node/cache snapshot so the @@ -1136,6 +1123,7 @@ mod tests { use super::*; use crate::ax::tree::AXNode; use cua_driver_core::element_query::project_elements_for_query; + use serde_json::json; fn node( idx: Option, @@ -1144,6 +1132,7 @@ mod tests { depth: usize, parent: Option, frame: Option<[f64; 4]>, + actions: Vec, ) -> AXNode { AXNode { element_index: idx, @@ -1153,7 +1142,7 @@ mod tests { description: None, identifier: None, help: None, - actions: vec![], + actions, element_ptr: 0, depth, parent_element_index: parent, @@ -1179,8 +1168,9 @@ mod tests { 0, None, Some([0.0, 0.0, 800.0, 600.0]), + vec![], ), - node(None, "AXStaticText", Some("hint"), 1, Some(0), None), + node(None, "AXStaticText", Some("hint"), 1, Some(0), None, vec![]), node( Some(1), "AXButton", @@ -1188,6 +1178,7 @@ mod tests { 1, Some(0), Some([10.0, 20.0, 60.0, 24.0]), + vec![], ), node( Some(2), @@ -1196,9 +1187,10 @@ mod tests { 1, Some(0), Some([80.0, 20.0, 60.0, 24.0]), + vec![], ), ]; - let elements = build_elements_array(&nodes); + let elements = build_elements_array_with_token(&nodes, None); assert_eq!( elements.len(), 3, @@ -1218,8 +1210,16 @@ mod tests { #[test] fn query_projection_keeps_only_rendered_actionable_rows() { let nodes = vec![ - node(Some(0), "AXWindow", Some("Document"), 0, None, None), - node(Some(1), "AXMenuItem", Some("Window"), 1, Some(0), None), + node(Some(0), "AXWindow", Some("Document"), 0, None, None, vec![]), + node( + Some(1), + "AXMenuItem", + Some("Window"), + 1, + Some(0), + None, + vec![], + ), node( Some(2), "AXMenuItem", @@ -1227,11 +1227,28 @@ mod tests { 2, Some(1), None, + vec![], + ), + node( + Some(3), + "AXMenuItem", + Some("Left"), + 3, + Some(2), + None, + vec![], + ), + node( + Some(4), + "AXButton", + Some("Unrelated"), + 1, + Some(0), + None, + vec![], ), - node(Some(3), "AXMenuItem", Some("Left"), 3, Some(2), None), - node(Some(4), "AXButton", Some("Unrelated"), 1, Some(0), None), ]; - let elements = build_elements_array(&nodes); + let elements = build_elements_array_with_token(&nodes, None); let filtered_markdown = concat!( "- [0] AXWindow \"Document\"\n", " - [1] AXMenuItem \"Window\"\n", @@ -1250,8 +1267,16 @@ mod tests { #[test] fn query_projection_returns_no_elements_when_markdown_has_no_match() { - let nodes = vec![node(Some(0), "AXButton", Some("Unrelated"), 0, None, None)]; - let elements = build_elements_array(&nodes); + let nodes = vec![node( + Some(0), + "AXButton", + Some("Unrelated"), + 0, + None, + None, + vec![], + )]; + let elements = build_elements_array_with_token(&nodes, None); let projected = project_elements_for_query(elements, Some("zoomLeft"), ""); @@ -1261,10 +1286,10 @@ mod tests { #[test] fn unfiltered_projection_preserves_every_element() { let nodes = vec![ - node(Some(0), "AXButton", Some("One"), 0, None, None), - node(Some(1), "AXButton", Some("Two"), 0, None, None), + node(Some(0), "AXButton", Some("One"), 0, None, None, vec![]), + node(Some(1), "AXButton", Some("Two"), 0, None, None, vec![]), ]; - let elements = build_elements_array(&nodes); + let elements = build_elements_array_with_token(&nodes, None); let projected = project_elements_for_query(elements, None, ""); @@ -1280,8 +1305,9 @@ mod tests { 3, Some(2), Some([1.5, 2.5, 33.0, 44.0]), + vec![], )]; - let entry = &build_elements_array(&nodes)[0]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; assert_eq!(entry["element_index"], 7); assert_eq!(entry["role"], "AXButton"); assert_eq!(entry["label"], "Go"); @@ -1306,9 +1332,10 @@ mod tests { 1, None, None, + vec![], )]; nodes[0].value = Some("i love u".into()); - let entry = &build_elements_array(&nodes)[0]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; assert_eq!(entry["label"], "Compose message", "label stays the title"); assert_eq!( entry["value"], "i love u", @@ -1328,6 +1355,7 @@ mod tests { 1, None, None, + vec![], )]; nodes[0].value_state = Some("8".into()); nodes[0].value_description = Some("8 dB".into()); @@ -1335,7 +1363,7 @@ mod tests { nodes[0].max_value = Some(8.0); nodes[0].enabled = Some(true); nodes[0].selected = Some(false); - let entry = &build_elements_array(&nodes)[0]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; assert_eq!( entry["value"], "8", "numeric AXValue surfaces via value_state" @@ -1356,25 +1384,34 @@ mod tests { 2, None, None, + vec![], )]; nodes[0].in_web_content = true; - let entry = &build_elements_array(&nodes)[0]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; assert_eq!(entry["in_web_content"], true); } #[test] fn checkbox_value_state_normalizes_to_selected() { - let mut nodes = vec![node(Some(0), "AXCheckBox", Some("I agree"), 0, None, None)]; + let mut nodes = vec![node( + Some(0), + "AXCheckBox", + Some("I agree"), + 0, + None, + None, + vec![], + )]; nodes[0].value_state = Some("0".into()); - let entry = &build_elements_array(&nodes)[0]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; assert_eq!(entry["selected"], false); } #[test] fn elements_control_state_fields_omitted_when_absent() { // Stock behaviour is unchanged for elements without control state. - let nodes = vec![node(Some(0), "AXButton", Some("OK"), 0, None, None)]; - let entry = &build_elements_array(&nodes)[0]; + let nodes = vec![node(Some(0), "AXButton", Some("OK"), 0, None, None, vec![])]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; for key in ["value_description", "min", "max", "enabled", "selected"] { assert!(entry.get(key).is_none(), "{key} must be omitted"); } @@ -1384,10 +1421,18 @@ mod tests { fn elements_omit_degenerate_min_max_range() { // WebKit reports AXMinValue/AXMaxValue as 0.0/0.0 on non-range // controls (checkboxes, radios) — a degenerate range is omitted. - let mut nodes = vec![node(Some(0), "AXCheckBox", Some("On"), 0, None, None)]; + let mut nodes = vec![node( + Some(0), + "AXCheckBox", + Some("On"), + 0, + None, + None, + vec![], + )]; nodes[0].min_value = Some(0.0); nodes[0].max_value = Some(0.0); - let entry = &build_elements_array(&nodes)[0]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; assert!(entry.get("min").is_none(), "degenerate min must be omitted"); assert!(entry.get("max").is_none(), "degenerate max must be omitted"); } @@ -1395,9 +1440,9 @@ mod tests { #[test] fn elements_value_state_falls_back_to_string_value() { // String-valued elements keep their `value` even with no value_state. - let mut nodes = vec![node(Some(0), "AXComboBox", None, 0, None, None)]; + let mut nodes = vec![node(Some(0), "AXComboBox", None, 0, None, None, vec![])]; nodes[0].value = Some("Search".into()); - let entry = &build_elements_array(&nodes)[0]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; assert_eq!(entry["value"], "Search"); } @@ -1405,16 +1450,16 @@ mod tests { fn elements_omit_empty_value() { // An empty AXValue must not emit a `value` field (matches the other // optional fields' omit-when-absent contract). - let mut nodes = vec![node(Some(0), "AXButton", Some("OK"), 0, None, None)]; + let mut nodes = vec![node(Some(0), "AXButton", Some("OK"), 0, None, None, vec![])]; nodes[0].value = Some(String::new()); - let entry = &build_elements_array(&nodes)[0]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; assert!(entry.get("value").is_none(), "empty value must be omitted"); } #[test] fn elements_omit_optional_fields_when_missing() { - let nodes = vec![node(Some(0), "AXUnknown", None, 0, None, None)]; - let entry = &build_elements_array(&nodes)[0]; + let nodes = vec![node(Some(0), "AXUnknown", None, 0, None, None, vec![])]; + let entry = &build_elements_array_with_token(&nodes, None)[0]; assert!( entry.get("label").is_none(), "label must be omitted when title/value/desc/id are all empty" @@ -1435,33 +1480,53 @@ mod tests { fn elements_label_fallback_chain() { // title missing → description → value → identifier let nodes = vec![ - node(Some(0), "AXButton", None, 0, None, None), - node(Some(1), "AXButton", None, 0, None, None), - node(Some(2), "AXButton", None, 0, None, None), + node(Some(0), "AXButton", None, 0, None, None, vec![]), + node(Some(1), "AXButton", None, 0, None, None, vec![]), + node(Some(2), "AXButton", None, 0, None, None, vec![]), ]; let mut nodes = nodes; nodes[0].description = Some("from-desc".into()); nodes[1].value = Some("from-val".into()); nodes[2].identifier = Some("from-id".into()); - let elements = build_elements_array(&nodes); + let elements = build_elements_array_with_token(&nodes, None); assert_eq!(elements[0]["label"], "from-desc"); assert_eq!(elements[1]["label"], "from-val"); assert_eq!(elements[2]["label"], "from-id"); } - /// Every element entry carries a non-empty snapshot-bound - /// `element_token` alongside its numeric `element_index`. + #[test] + fn build_elements_array_with_token_emits_actions_when_present() { + let nodes = vec![node( + Some(0), + "AXButton", + Some("OK"), + 1, + None, + None, + vec!["AXPress".to_owned(), "AXShowMenu".to_owned()], + )]; + let entries = build_elements_array_with_token(&nodes, None); + assert_eq!(entries[0]["actions"], json!(["AXPress", "AXShowMenu"])); + } + + #[test] + fn build_elements_array_with_token_omits_actions_when_empty() { + let nodes = vec![node(Some(0), "AXButton", Some("OK"), 1, None, None, vec![])]; + let entries = build_elements_array_with_token(&nodes, None); + assert!(entries[0].get("actions").is_none()); + } + #[test] fn build_elements_array_with_token_emits_element_token_per_row() { let reg = cua_driver_core::element_token::global(); let pid = 0x6abc_0001_i32; let sid = reg.register_snapshot(pid, /* window_id = */ 9, 3); let nodes = vec![ - node(Some(0), "AXButton", Some("A"), 1, None, None), - node(Some(1), "AXButton", Some("B"), 1, None, None), - node(Some(2), "AXButton", Some("C"), 1, None, None), + node(Some(0), "AXButton", Some("A"), 1, None, None, vec![]), + node(Some(1), "AXButton", Some("B"), 1, None, None, vec![]), + node(Some(2), "AXButton", Some("C"), 1, None, None, vec![]), ]; - let entries = build_elements_array_with_token(&nodes, sid); + let entries = build_elements_array_with_token(&nodes, Some(sid)); assert_eq!(entries.len(), 3); // Every entry must have BOTH fields (additive contract). for e in &entries { @@ -1487,17 +1552,23 @@ mod tests { } } - /// Back-compat: `build_elements_array` (the old shim) must NOT emit - /// `element_token` — older callers that never plumb a snapshot id - /// through get a clean shape. #[test] - fn build_elements_array_shim_skips_element_token() { - let nodes = vec![node(Some(0), "AXButton", Some("A"), 1, None, None)]; - let entries = build_elements_array(&nodes); + fn build_elements_array_with_token_observation_only_has_actions_no_token() { + let nodes = vec![node( + Some(0), + "AXButton", + Some("OK"), + 1, + None, + None, + vec!["AXPress".to_owned(), "AXShowMenu".to_owned()], + )]; + let entries = build_elements_array_with_token(&nodes, None); assert_eq!(entries.len(), 1); + assert_eq!(entries[0]["actions"], json!(["AXPress", "AXShowMenu"])); assert!( entries[0].get("element_token").is_none(), - "back-compat shim must NOT emit element_token; got: {}", + "observation-only entries must not emit unregistered element_token: {}", entries[0] ); } diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs index f15bf10ae6..c4aad564c9 100644 --- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs @@ -1062,6 +1062,48 @@ mod list_windows_z_index_tests { } } +#[cfg(test)] +mod get_window_state_actions_tests { + use super::*; + use crate::uia::UiaNode; + + fn node(actions: Vec) -> UiaNode { + UiaNode { + element_index: Some(1), + control_type: "Button".to_owned(), + name: Some("OK".to_owned()), + value: None, + automation_id: None, + help_text: None, + actions, + enabled: Some(true), + selected: None, + element_ptr: 0, + center_x: 0, + center_y: 0, + rect: None, + msaa_role: None, + depth: 0, + parent_element_index: None, + in_web_content: false, + } + } + + #[test] + fn element_entry_includes_actions_when_present() { + let n = node(vec!["invoke".to_owned(), "toggle".to_owned()]); + let entry = build_element_entry(&n, None).unwrap(); + assert_eq!(entry["actions"], json!(["invoke", "toggle"])); + } + + #[test] + fn element_entry_omits_actions_when_empty() { + let n = node(Vec::new()); + let entry = build_element_entry(&n, None).unwrap(); + assert!(entry.get("actions").is_none()); + } +} + // ── get_window_state ───────────────────────────────────────────────────────── /// Fold a per-call `max_dimension` cap with the configured @@ -1076,6 +1118,65 @@ fn fold_max_dimension(ceiling: u32, per_call: Option) -> u32 { } } +/// Build a single structured element entry for `get_window_state`. +/// Returns `None` when the node has no `element_index` (non-actionable rows). +fn build_element_entry( + n: &crate::uia::UiaNode, + snapshot_id: Option, +) -> Option { + let idx = n.element_index?; + // `label`: name → value → automation_id → help_text. + let label = n + .name + .clone() + .or_else(|| n.value.clone()) + .or_else(|| n.automation_id.clone()) + .or_else(|| n.help_text.clone()); + let mut entry = json!({ + "element_index": idx, + "role": n.control_type, + "depth": n.depth, + }); + if let Some(snapshot_id) = snapshot_id { + entry["element_token"] = json!(cua_driver_core::element_token::token_for(snapshot_id, idx)); + } + if n.in_web_content { + entry["in_web_content"] = json!(true); + } + if let Some(label) = label { + entry["label"] = json!(label); + } + // Surface the element's value separately from `label` (which collapses + // name→value→automation_id→help): a control with both a name AND text + // (a ValuePattern edit holding typed content) would otherwise hide the + // text from a caller reading the structured side. See the macOS + // get_window_state builder for the rationale. + if let Some(value) = n.value.clone().filter(|v| !v.is_empty()) { + entry["value"] = json!(value); + } + if let Some(enabled) = n.enabled { + entry["enabled"] = json!(enabled); + } + if let Some(selected) = n.selected { + entry["selected"] = json!(selected); + } + if !n.actions.is_empty() { + entry["actions"] = json!(n.actions); + } + if let Some(parent) = n.parent_element_index { + entry["parent_index"] = json!(parent); + } + if let Some((l, t, r, b)) = n.rect { + entry["frame"] = json!({ + "x": l, + "y": t, + "w": (r - l).max(0), + "h": (b - t).max(0), + }); + } + Some(entry) +} + pub struct GetWindowStateTool { state: Arc, } @@ -1101,7 +1202,8 @@ impl Tool for GetWindowStateTool { the next snapshot of the same (pid, window_id).\n\n\ PREFERRED CONSUMERS read `structuredContent.elements` (one entry per \ indexed row with `element_index`, `role`, `label`, `value`, `enabled`, \ - `selected`, `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown \ + `selected`, `actions` (names of UIA patterns exposed as actions, \ + omitted when empty), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown \ `tree_markdown` stays available \ and unchanged in shape for existing text-parsing callers — but new \ fields will only be added to the structured side.\n\n\ @@ -1401,64 +1503,12 @@ impl Tool for GetWindowStateTool { // Structured `elements` array — preferred consumption // path. Shape matches the cross-platform spec: // `{element_index, element_token, role, label, depth, - // parent_index?, frame?: {x,y,w,h}}`. Frame is + // actions?, parent_index?, frame?: {x,y,w,h}}`. Frame is // included when UIA reported a usable BoundingRectangle. let elements: Vec = tr .nodes .iter() - .filter_map(|n| { - let idx = n.element_index?; - // `label`: name → value → automation_id → help_text. - let label = n - .name - .clone() - .or_else(|| n.value.clone()) - .or_else(|| n.automation_id.clone()) - .or_else(|| n.help_text.clone()); - let mut entry = json!({ - "element_index": idx, - "role": n.control_type, - "depth": n.depth, - }); - if let Some(snapshot_id) = snapshot_id { - entry["element_token"] = json!( - cua_driver_core::element_token::token_for(snapshot_id, idx) - ); - } - if n.in_web_content { - entry["in_web_content"] = json!(true); - } - if let Some(label) = label { - entry["label"] = json!(label); - } - // Surface the element's value separately from `label` - // (which collapses name→value→automation_id→help): a - // control with both a name AND text (a ValuePattern - // edit holding typed content) would otherwise hide the - // text from a caller reading the structured side. See - // the macOS get_window_state builder for the rationale. - if let Some(value) = n.value.clone().filter(|v| !v.is_empty()) { - entry["value"] = json!(value); - } - if let Some(enabled) = n.enabled { - entry["enabled"] = json!(enabled); - } - if let Some(selected) = n.selected { - entry["selected"] = json!(selected); - } - if let Some(parent) = n.parent_element_index { - entry["parent_index"] = json!(parent); - } - if let Some((l, t, r, b)) = n.rect { - entry["frame"] = json!({ - "x": l, - "y": t, - "w": (r - l).max(0), - "h": (b - t).max(0), - }); - } - Some(entry) - }) + .filter_map(|n| build_element_entry(n, snapshot_id)) .collect(); let elements = cua_driver_core::element_query::project_elements_for_query( elements,