Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
e27fb98
test(cua-driver): modality recording harness + per-action effect veri…
f-trycua Jun 27, 2026
b3e332c
fix(cua-driver/windows): guard element clicks against off-screen targets
f-trycua Jun 27, 2026
3ff50a2
docs(cua-driver): update modality findings — Windows off-screen guard…
f-trycua Jun 27, 2026
9b6b36b
docs: update modality index with Windows verifier numbers
f-trycua Jun 27, 2026
5b6df9e
test(cua-driver): cross-toolkit coverage matrix (WPF/WinUI3/WebView2/…
f-trycua Jun 27, 2026
7bf6988
fix(cua-driver/windows): scope off-screen click guard to coordinate-d…
f-trycua Jun 27, 2026
fd4813c
fix(cua-driver): start_session revives an ended id; stray actions on …
f-trycua Jun 27, 2026
9c2dffd
fix(cua-driver/test-harness): pin AppKit harness window geometry acro…
f-trycua Jun 27, 2026
a35e6f3
feat(cua-driver/linux): wrap Electron harness for Linux + record 5 mo…
f-trycua Jun 27, 2026
eb43f22
test(cua-driver/gtk3): bring GTK3 harness to control parity with WPF
f-trycua Jun 27, 2026
a93bf82
test(cua-driver/winui3): add click-target + scroll-target for 8-actio…
f-trycua Jun 27, 2026
10f93e1
test-harness(macos): add WKWebView harness + Electron macOS launch glue
f-trycua Jun 27, 2026
da39808
docs(cua-driver/windows): record WinUI3 dispatch finding (double/righ…
f-trycua Jun 27, 2026
6a5a019
test(cua-driver/web): add scroll_target to shared web harness for mat…
f-trycua Jun 27, 2026
008b7d1
test(cua-driver/harness): macOS AppKit + SwiftUI control parity with …
f-trycua Jun 27, 2026
92d8aeb
fix(cua-driver/macos): set_value writes a CFNumber for numeric AXValu…
f-trycua Jun 27, 2026
5c8eaf1
test(cua-driver/linux): exercise shared scroll-target in Electron rec…
f-trycua Jun 27, 2026
1ffc940
docs(cua-driver): record control-parity pass + scroll-target + macOS …
f-trycua Jun 27, 2026
7499fed
docs(cua-driver/macos): live host verification of SwiftUI parity + sc…
f-trycua Jun 27, 2026
1cdc897
docs(cua-driver/macos): CFNumber set_value fix VERIFIED live on AppKi…
f-trycua Jun 28, 2026
d2b2923
feat(cua-driver/macos): set_value steps AXIncrement/AXDecrement when …
f-trycua Jun 28, 2026
a470b74
docs(cua-driver): keep the demo blob URL out of the public repo
f-trycua Jun 28, 2026
d95b89a
fix(cua-driver/windows): retain cached UIA element under the lock (UAF)
f-trycua Jun 28, 2026
7d35828
fix(cua-driver/linux): land GTK left-click via AT-SPI + surface value…
f-trycua Jun 28, 2026
b249716
test(cua-driver/windows): extend matrix recorder for parity controls …
f-trycua Jun 28, 2026
dfc4464
docs(cua-driver): refresh Windows matrix + record this round's driver…
f-trycua Jun 28, 2026
79e546c
feat(cua-driver/linux): land GTK right/double/middle-click + scroll v…
f-trycua Jun 28, 2026
e7145c1
fix(cua-driver/macos): pixel right-click + NSButton click + wheel scr…
f-trycua Jun 28, 2026
698a336
docs(cua-driver): document pixel-wheel scroll + windowless button-pre…
f-trycua Jun 28, 2026
531aa6d
fix(cua-driver/windows): WinUI3 double-click + Electron contract + of…
f-trycua Jun 28, 2026
0d90503
test(cua-driver/macos): fix modality recorder verdicts for the parity…
f-trycua Jun 28, 2026
80b4e3d
fix(cua-driver/macos): coordinate-space consistency for vision agents…
f-trycua Jun 28, 2026
c3efb58
fix(cua-driver/windows): off-screen scroll-into-view falls back to an…
f-trycua Jun 28, 2026
010fdd7
test(cua-driver): vision-agent coordinate-invariant test prototype
f-trycua Jun 28, 2026
49bdb41
test(cua-driver/windows): measure the no-foreground contract against …
f-trycua Jun 28, 2026
b71d2e9
docs(cua-driver): detailed modality test-suite + harness reference
f-trycua Jun 28, 2026
5c0a1d3
test(cua-driver/linux): measure the no-foreground contract against a …
f-trycua Jun 28, 2026
d42ffa1
docs(cua-driver): correct Linux contract numbers after the recorder-b…
f-trycua Jun 28, 2026
197c605
docs(cua-driver): publish the modality test-suite reference in fumadocs
f-trycua Jun 28, 2026
2f0b77e
test(cua-driver): session lifecycle/disposal coverage + stop recorder…
f-trycua Jun 28, 2026
35d3f06
docs(cua-driver): document session lifecycle & disposal in the test-s…
f-trycua Jun 28, 2026
f512e32
docs(cua-driver): real-app edge cases + Linux-Electron ax-bg 2/8 corr…
f-trycua Jun 28, 2026
0c1d9e9
docs(cua-driver): real-app edge cases round 2 (Win11 Settings/Store/P…
f-trycua Jun 28, 2026
040caed
fix(cua-driver/windows): unhandled-protocol launch_app no longer dead…
f-trycua Jun 28, 2026
6754346
docs(cua-driver): make element_index pid requirement explicit + corre…
f-trycua Jun 28, 2026
b7f66cb
fix(cua-driver/windows): resolve UWP launches to their ApplicationFra…
f-trycua Jun 28, 2026
a348721
fix(cua-driver/test-harness): WPF recorder — panel encoding + right-c…
f-trycua Jun 28, 2026
bd9d325
fix(cua-driver/test-harness): WPF recorder drag uses dispatch:foregro…
f-trycua Jun 28, 2026
5a780ec
fix(cua-driver/test-harness): WPF recorder asserts harness foreground…
f-trycua Jun 28, 2026
f44264b
fix(cua-driver/test-harness): gate WPF fg foreground-assert to coordi…
f-trycua Jun 28, 2026
13276e7
fix(cua-driver/test-harness): macOS web recorder uses live element fr…
f-trycua Jun 28, 2026
eefcfe4
fix(cua-driver/test-harness): WPF fg foreground-assert every step + E…
f-trycua Jun 28, 2026
e8ae50d
docs(cua-driver): WKWebView modality results — recorder coord fix + h…
f-trycua Jun 28, 2026
ba5ff66
fix(cua-driver/test-harness): WPF ax-mode slider drag via RangeValue …
f-trycua Jun 28, 2026
443cb5c
fix(cua-driver/macos): agent cursor sandwiches at z+1 via SkyLight SL…
f-trycua Jun 28, 2026
6a5bfeb
Revert "fix(cua-driver/macos): agent cursor sandwiches at z+1 via Sky…
f-trycua Jun 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 15 additions & 10 deletions docs/content/docs/reference/cua-driver/mcp-tools.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -181,11 +181,12 @@ Two addressing modes:

- **Element index** (`element_index` + `window_id`): AX action path. Works on backgrounded/hidden windows. No cursor move, no focus steal. Cache is scoped per `(pid, window_id)` and is replaced by the next snapshot.
- **Pixel** (`x`, `y`): CGEvent path. Synthesizes mouse events posted to pid. Needs a visible on-screen window.
- **Screen-absolute** (`x`, `y` with no window owner, under `capture_scope: desktop`): clicks true screen coordinates even when no window owns the point (desktop background, menu-extra gaps). The button is preserved — a windowless `right_click`/middle no longer downgrades to left.

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `pid` | integer | Yes | Target process ID. |
| `element_index` | integer | No | Element index from last `get_window_state`. |
| `element_index` | integer | No | Element index from last `get_window_state`. Requires `pid` and `window_id` alongside it — `element_index` with no `pid` fails fast with `Missing required integer field: pid` (it is not a silent no-op). |
| `window_id` | integer | No | Target window ID. Required for `element_index`. |
| `x` | number | No | Window-local screenshot X coordinate. |
| `y` | number | No | Window-local screenshot Y coordinate. |
Expand All @@ -212,7 +213,7 @@ Pixel path: two down/up pairs ~80 ms apart.
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `pid` | integer | Yes | Target process ID. |
| `element_index` | integer | No | Element index from last `get_window_state`. Uses AX path. |
| `element_index` | integer | No | Element index from last `get_window_state`. Uses AX path. Requires `pid` and `window_id` alongside it — `element_index` with no `pid` fails fast with `Missing required integer field: pid` (it is not a silent no-op). |
| `window_id` | integer | No | CGWindowID. Required when `element_index` is used. |
| `x` | number | No | Screen X coordinate (pixel path). |
| `y` | number | No | Screen Y coordinate (pixel path). |
Expand All @@ -237,7 +238,7 @@ Exactly one of `element_index` or (`x` AND `y`) must be provided. `pid` always r
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `pid` | integer | Yes | Target process ID. |
| `element_index` | integer | No | Element index from last `get_window_state`. Routes through `AXShowMenu`. Requires `window_id`. |
| `element_index` | integer | No | Element index from last `get_window_state`. Routes through `AXShowMenu`. Requires `pid` and `window_id` alongside it — `element_index` with no `pid` fails fast with `Missing required integer field: pid` (it is not a silent no-op). |
| `window_id` | integer | No | CGWindowID. Required when `element_index` is used. |
| `x` | number | No | X in window-local screenshot pixels. |
| `y` | number | No | Y in window-local screenshot pixels. |
Expand Down Expand Up @@ -281,7 +282,7 @@ Insert text into the target pid via `AXSetAttribute(kAXSelectedText)`. Works for
| -------- | ---- | -------- | ----------- |
| `pid` | integer | Yes | Target process ID. |
| `text` | string | Yes | Text to insert at the target's cursor. |
| `element_index` | integer | No | Element index from last `get_window_state`. Directs the write to a specific field. Requires `window_id`. |
| `element_index` | integer | No | Element index from last `get_window_state`. Directs the write to a specific field. Requires `pid` and `window_id` alongside it — `element_index` with no `pid` fails fast with `Missing required integer field: pid` (it is not a silent no-op). |
| `window_id` | integer | No | CGWindowID. Required when `element_index` is used. |
| `delay_ms` | integer | No | Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds. |

Expand Down Expand Up @@ -350,7 +351,7 @@ Two modes:
| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `pid` | integer | Yes | Target process ID. |
| `element_index` | integer | Yes | Element index from last `get_window_state`. |
| `element_index` | integer | Yes | Element index from last `get_window_state`. Requires `pid` and `window_id` alongside it — `element_index` with no `pid` fails fast with `Missing required integer field: pid` (it is not a silent no-op). |
| `window_id` | integer | Yes | CGWindowID for the window whose `get_window_state` produced the element index. |
| `value` | string | Yes | New value. AX will coerce to the element's native type. |

Expand All @@ -362,18 +363,22 @@ Two modes:

### `scroll`

Scroll the target pid's focused region by synthesized keystrokes.
Scroll the target pid. Two addressing paths:

Mapping: `by='page'` → PageDown/PageUp × amount; `by='line'` → DownArrow/UpArrow × amount. Horizontal variants use Left/Right arrow keys.
- **Keystroke** (default — no coordinate/element target): scrolls the focused / page scroller via synthesized keys. `by='page'` → PageDown/PageUp × amount; `by='line'` → Down/Up arrow × amount. Horizontal uses Left/Right.
- **Pixel-wheel** (`element_index` / `element_token`, or window-local `x, y`) — *macOS*: synthesizes a mouse wheel at the target point so the scroll routes to the element under it. Use it to drive a **nested overflow region or an unfocused scroller** the keystroke path can't reach (e.g. a web `overflow` div). Background-safe — no cursor warp, no focus steal.

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `pid` | integer | Yes | Target process ID. |
| `direction` | string | Yes | Scroll direction: `up`, `down`, `left`, `right`. |
| `by` | string | No | Scroll granularity: `line` (default) or `page`. |
| `amount` | integer | No | Number of keystroke repetitions. Default 3. |
| `element_index` | integer | No | Pre-focuses this element before scrolling. |
| `window_id` | integer | No | Required when `element_index` is set. |
| `amount` | integer | No | Pixel-wheel: number of wheel notches. Keystroke: number of key repetitions. Default 3. |
| `element_index` | integer | No | Element from last `get_window_state`. macOS: pixel-wheel at its center (e.g. a nested overflow region located in the AX tree); other platforms: pre-focuses it before the keystroke scroll. |
| `element_token` | string | No | Per-snapshot element handle (`structuredContent.elements[].element_token`); precedence over `element_index`. macOS: pixel-wheel at its center. |
| `x` | number | No | Window-local screenshot X (top-left origin). With `y`, macOS pixel-wheel at this point — for a scrollable surface not in the AX tree. Requires `window_id`. |
| `y` | number | No | Window-local screenshot Y. See `x`. |
| `window_id` | integer | No | Required when `element_index` / `x, y` is set. |

```json
{"direction": "down", "pid": 844}
Expand Down
2 changes: 1 addition & 1 deletion docs/content/docs/reference/cua-driver/meta.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{ "title": "Cua Driver", "pages": ["cli-reference", "mcp-tools", "contracts", "limits"] }
{ "title": "Cua Driver", "pages": ["cli-reference", "mcp-tools", "contracts", "limits", "modality-test-suite"] }
Loading
Loading