Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 44 additions & 50 deletions libs/cua-driver/rust/PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ cd libs/cua-driver/rust/tests/integration
| `page` tool | ✅ (cross-platform: Apple-Events on macOS, UIA+CDP on Windows, AT-SPI+CDP on Linux) | ✅ (macOS only) |
| `--version` flag | ✅ | ✅ |
| `call check_permissions` JSON | ✅ JSON | human-readable text |
| `call screenshot` (no window_id) | ✅ full-display default | error (requires window_id) |

---

Expand Down Expand Up @@ -867,53 +866,49 @@ Windows's `click` takes `{button: enum}` instead. Rationale:

---

## MCP tool: `screenshot`
- Swift: `libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift:5-170`
- Rust: windows=`crates/platform-windows/src/tools/impl_.rs` (ScreenshotTool); macOS/linux OPEN
- Status: windows VERIFIED
- Test: `crates/platform-windows/examples/screenshot_parity.rs`

### Fixed (Windows)
1. **Text format** — was `"Screenshot (window): WxH png."`; now matches
Swift verbatim: `"✅ Window screenshot — WxH png [window_id: ID]"`
(em-dash, checkmark, window-id suffix). Display fallback uses
`"✅ Display screenshot — WxH png"` (Rust-only, see intentional below).
2. **Description** — multi-paragraph port from Swift adapted to Windows
(BitBlt + PrintWindow transport; no permission gate needed).
3. **`idempotent`** — was `true`; Swift uses `false` (a fresh pixel grab
every call). Now matches Swift.
4. **`max_image_dimension` default** — was 0 (no cap), Swift uses 1568.
Now 1568 on all 3 Rust platforms (matches
`CuaDriverConfig.defaultMaxImageDimension`). The 0 default was
producing 10MB screenshots on the Windows VM; 1568 caps the long
edge before encoding.

### Intentional Rust-only

- **Optional `window_id`** — Swift requires it; Windows allows omission
for whole-display capture (`screenshot_display_bytes`). Useful
Windows-only convenience that Swift can't easily provide because
macOS Screen Recording requires per-window grants. Schema accepts
both shapes; description explains.
- **Default `format`** — Swift defaults `png`; all 3 Rust platforms now
default `jpeg`. Rationale: agents typically want compact images for
vision-model context windows; PNG is lossless but multi-MB on screen
content. Schema still accepts both; callers wanting PNG pass
`{"format":"png"}`. Swift may follow; tracked as a follow-up parity
question.
- **Default JPEG `quality`** — Swift defaults 95; Rust defaults 85
(already Linux's default and the macOS Claude-Code-compat tool's
default). 85 is the typical sweet spot for screen content. Diverges
from Swift only when both sides actually emit JPEG.
## MCP tool: `screenshot` — REMOVED on all platforms

### Verified on Windows

`screenshot_parity.exe`:
- Window screenshot: text matches `"✅ Window screenshot — 1087x644 png
[window_id: 4464038]"` ✓
- Image content block present ✓
- structuredContent has `width`, `height`, `format` ✓
- JPEG format: `mimeType: image/jpeg`, `format: "jpeg"` ✓
- Swift: `libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift:5-170`
- Rust: **not registered on any platform** (macOS / Windows / Linux).
- Status: **removed** — superseded by `get_window_state` (`capture_mode:"vision"`).

### History

The standalone `screenshot` tool was intentionally removed from all
three Rust platforms:

- **PR #1692** removed the `screenshot` tool registration; `get_window_state`
with `capture_mode:"vision"` became the single canonical full-frame
screenshot path, and `zoom` covers region captures.
- **PR #1694** deleted the now-dead `ScreenshotTool` / `ScreenshotCompatTool`
structs on macOS and Windows.

So `cua-driver call screenshot '{}'` returning `Unknown tool: screenshot`
is the **intended, parity-consistent** behavior on every platform — there
is no Linux-specific gap here.

The underlying capture functions still exist on each platform because the
canonical paths use them internally:
- Linux: `crate::capture::screenshot_window_bytes` /
`screenshot_display_bytes` (and `wayland::screenshot_dispatch`) feed
`GetWindowStateTool` and `ZoomTool`.
- Windows: `crate::capture::screenshot_window_bytes` / `screenshot_display_bytes`
(and the WGC backend) feed `GetWindowStateTool`.

### Canonical replacement

To capture a screenshot, call `get_window_state` with
`capture_mode:"vision"` for a full-frame image (returns the screenshot as
an image content block, honoring `max_image_dimension`, default 1568), or
`zoom` for a cropped native-resolution region. The image-encoding behavior
the old `screenshot` tool documented (default `jpeg`, `max_image_dimension`
1568, JPEG quality 85) now lives in those paths.

> Historical note: this section previously documented a Windows-only
> `ScreenshotTool` as "VERIFIED" (text format `"✅ Window screenshot —
> WxH png [window_id: ID]"`, optional `window_id`, `jpeg` default,
> `max_image_dimension` 1568). That tool and its `screenshot_parity.rs`
> example predate PR #1692/#1694 and no longer exist.

---

Expand Down Expand Up @@ -1184,8 +1179,8 @@ description content.
### Verified on Windows
`list_tools_parity.exe`:
- Names sorted ascending ✓
- All 23 core tools present (click, double_click, right_click, type_text,
press_key, hotkey, scroll, screenshot, list_apps, list_windows,
- All core tools present (click, double_click, right_click, type_text,
press_key, hotkey, scroll, list_apps, list_windows,
get_cursor_position, get_screen_size, launch_app, move_cursor,
set_agent_cursor_enabled, set_agent_cursor_motion,
get_agent_cursor_state, set_value, get_config, set_config,
Expand Down Expand Up @@ -1411,7 +1406,6 @@ older clients that only read name/description still work.
{ "name": "press_key", "…": "…" },
{ "name": "replay_trajectory", "…": "…" },
{ "name": "right_click", "…": "…" },
{ "name": "screenshot", "…": "…" },
{ "name": "scroll", "…": "…" },
{ "name": "set_config", "…": "…" },
{ "name": "set_recording", "…": "…" },
Expand Down
11 changes: 6 additions & 5 deletions libs/cua-driver/rust/Skills/cua-driver/LINUX.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ description: Drive a native Linux app (X11 / Wayland) via the cua-driver CLI —
# cua-driver-rs — Linux

**Status: BETA.** The Linux backend in cua-driver-rs covers the core
tool surface (click, type_text, scroll, hotkey, screenshot,
launch_app, list_apps, list_windows, get_window_state) but several
tool surface (click, type_text, scroll, hotkey, launch_app,
list_apps, list_windows, get_window_state, zoom) but several
behaviors that the macOS / Windows skills consider table-stakes are
**not yet implemented or only partially supported**:

Expand All @@ -26,7 +26,7 @@ behaviors that the macOS / Windows skills consider table-stakes are
KDE-KWin with `org.freedesktop.portal.RemoteDesktop` enabled, some
click and key paths work. Under most other compositors, input
synthesis is denied by the security model and the tool surface
degrades to "passive" (snapshot, screenshot) only.
degrades to "passive" (snapshot via get_window_state) only.
- **UIA / AX-tree equivalent**: AT-SPI when available, otherwise
empty. Many GTK4 / Qt6 apps populate AT-SPI lazily; agents should
expect partial trees and re-snapshot.
Expand All @@ -38,8 +38,9 @@ behaviors that the macOS / Windows skills consider table-stakes are
See `SKILL.md` (macOS) and `WINDOWS.md` (Windows) for the full
patterns. This file will grow as the Linux backend reaches GA. For
now, **prefer macOS / Windows hosts** for agent-driven GUI tasks; use
the Linux daemon for read-only inspection (screenshot,
list_windows, get_window_state) when running on a Linux host.
the Linux daemon for read-only inspection (list_windows,
get_window_state — which embeds a screenshot — and zoom) when running
on a Linux host.

## Quick triage

Expand Down
Loading