diff --git a/docs/content/docs/cua-driver/guide/getting-started/installation.mdx b/docs/content/docs/cua-driver/guide/getting-started/installation.mdx index 533d1088bf..8f4f7fffbb 100644 --- a/docs/content/docs/cua-driver/guide/getting-started/installation.mdx +++ b/docs/content/docs/cua-driver/guide/getting-started/installation.mdx @@ -113,6 +113,9 @@ Cua Driver speaks MCP over stdio. Use `cua-driver mcp-config --client ` to # Claude Code (add --scope project|global as needed) claude mcp add --transport stdio cua-driver -- ~/.local/bin/cua-driver mcp +# Claude Code computer-use compatibility mode +claude mcp add --transport stdio cua-computer-use -- ~/.local/bin/cua-driver mcp --claude-code-computer-use-compat + # Codex (OpenAI) codex mcp add cua-driver -- ~/.local/bin/cua-driver mcp @@ -120,6 +123,10 @@ codex mcp add cua-driver -- ~/.local/bin/cua-driver mcp cua-driver mcp-config --client openclaw | sh ``` +The Claude Code compatibility mode keeps CuaDriver's normal MCP tools, but replaces `screenshot` with a window-only screenshot shim that requires `pid` and `window_id`. Use it when you want Claude Code's vision/computer-use-style flow to ground on CuaDriver window captures. + +Use MCP for that Claude Code vision/computer-use-style path. Shelling out to `cua-driver screenshot` can capture a window, but it does not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue. + ### Clients configured via a config file Cursor, OpenCode, and Hermes all configure MCP servers via files. Use `mcp-config` to print the exact snippet, paste it into the right path: diff --git a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx index daabda628b..a3f0715470 100644 --- a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx +++ b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx @@ -14,6 +14,8 @@ import { Callout } from 'fumadocs-ui/components/callout'; ## Claude Code +Standard MCP registration: + ```bash claude mcp add --transport stdio cua-driver -- cua-driver mcp ``` @@ -25,6 +27,22 @@ claude mcp list # cua-driver: cua-driver mcp (stdio) - ✓ Connected ``` +### Claude Code computer-use compatibility mode + +Claude Code vision/computer-use-style flows appear to use the presence of a screenshot tool as a cue for image-grounded operation. If you want that behavior, register the compatibility server instead: + +```bash +claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat +``` + +This mode still exposes the normal CuaDriver tools. The only changed tool is `screenshot`: it requires `pid` and `window_id`, captures that window only, and returns a window-local image coordinate frame. Start with `launch_app` or `list_windows`, then call `screenshot` with the target window. + +For this Claude Code vision/computer-use-style path, use MCP rather than shelling out to the CLI. CLI screenshots can still capture windows, but they do not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue. + + + This does not call Anthropic APIs or expose Anthropic's native computer-use API tool. It is a CuaDriver MCP compatibility mode for Claude Code. + + ## GitHub Copilot CLI Add to `~/.copilot/mcp-config.json`: diff --git a/docs/content/docs/cua-driver/reference/cli-reference.mdx b/docs/content/docs/cua-driver/reference/cli-reference.mdx index 7a31501cb6..a61daedff8 100644 --- a/docs/content/docs/cua-driver/reference/cli-reference.mdx +++ b/docs/content/docs/cua-driver/reference/cli-reference.mdx @@ -6,8 +6,8 @@ description: Command Line Interface reference for Cua Driver {/* AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY Generated by: npx tsx scripts/docs-generators/cua-driver.ts - Source: libs/cua-driver/Sources/**/*.swift - Version: 0.1.0 + Source: recursive Swift sources under libs/cua-driver/Sources + Version: 0.1.1 */} import { Callout } from 'fumadocs-ui/components/callout'; @@ -16,7 +16,7 @@ import { VersionHeader } from '@/components/version-selector'; @@ -36,8 +36,8 @@ provided, the tool is called with no arguments. Examples: cua-driver call list_apps - cua-driver call launch_app '{"bundle_id":"com.apple.finder"}' - echo '{"pid":844,"window_id":1234}' | cua-driver call get_window_state + cua-driver call launch_app '{"bundle_id":"com.apple.finder"}' + echo '{"pid":844,"window_id":1234}' | cua-driver call get_window_state **Arguments:** @@ -106,6 +106,12 @@ Print a tool's full description and JSON input schema. Run the stdio MCP server. +**Flags:** + +| Name | Description | +| ---- | ----------- | +| `--claude-code-computer-use-compat` | Expose normal CuaDriver tools, replacing only `screenshot` with a Claude Code-friendly window-only screenshot that establishes the vision coordinate frame. | + ### cua-driver serve Run cua-driver as a long-running daemon on a Unix domain socket. @@ -158,6 +164,12 @@ Print MCP server config or a client-specific install command. | ---- | ---- | ------- | ----------- | | `--client` | String | — | Client to print the install command for: claude \| codex \| cursor \| openclaw \| opencode \| hermes \| pi. Omit for the generic JSON snippet. | +**Flags:** + +| Name | Description | +| ---- | ----------- | +| `--claude-code-computer-use-compat` | Print config for Claude Code's window-scoped screenshot compatibility mode registered as `cua-computer-use`. | + ## Trajectory recording ### cua-driver recording diff --git a/docs/content/docs/cua-driver/reference/mcp-tools.mdx b/docs/content/docs/cua-driver/reference/mcp-tools.mdx index b16542adbd..b41588de26 100644 --- a/docs/content/docs/cua-driver/reference/mcp-tools.mdx +++ b/docs/content/docs/cua-driver/reference/mcp-tools.mdx @@ -6,8 +6,8 @@ description: Reference for every MCP tool cua-driver exposes {/* AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY Generated by: npx tsx scripts/docs-generators/cua-driver.ts - Source: libs/cua-driver/Sources/**/*.swift - Version: 0.1.0 + Source: recursive Swift sources under libs/cua-driver/Sources + Version: 0.1.1 */} import { Callout } from 'fumadocs-ui/components/callout'; @@ -25,7 +25,7 @@ Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a te Report TCC permission status for Accessibility and Screen Recording. By default also raises the system permission dialogs for any missing grants — Apple's request APIs are no-ops when the grant is already -active, so this is safe to call repeatedly. Pass {"prompt": false} +active, so this is safe to call repeatedly. Pass {"prompt": false} for a purely read-only status check. **Arguments:** @@ -208,18 +208,18 @@ at startup. Sibling to `set_config` / `cua-driver config`. Current schema: - { + { "schema_version": 1, "capture_mode": "vision" | "ax" | "som", - "agent_cursor": { + "agent_cursor": { "enabled": true, - "motion": { + "motion": { "start_handle": 0.3, "end_handle": 0.3, "arc_size": 0.25, "arc_flow": 0.0, "spring": 0.72 - } - } - } + } + } + } **Arguments:** none. @@ -436,10 +436,10 @@ later to resolve a target. - `additional_arguments` (array of string, optional): Extra command-line arguments passed to the launched process. Passed directly as argv entries — no shell expansion. Example: ["--user-data-dir=/tmp/cua-session", "--no-first-run"] for an isolated Chrome session. - `bundle_id` (string, optional): App bundle identifier, e.g. com.apple.calculator. - `creates_new_application_instance` (boolean, optional): Force a brand-new process even if the app is already running. Useful for isolated browser sessions: pass creates_new_application_instance=true together with additional_arguments=["--user-data-dir=/tmp/session-a", "--no-first-run", "--no-default-browser-check"] to launch a sandboxed Chrome that cannot see the user's real profile, cookies, or extensions. Each session gets its own pid and window identity and can be controlled independently. -- `electron_debugging_port` (integer, optional): Launch an Electron app with --remote-debugging-port= so the `page` tool gets full renderer/DOM access. Use 9222 unless running multiple Electron apps. Ignored for non-Electron apps. +- `electron_debugging_port` (integer, optional): Launch an Electron app with --remote-debugging-port=<N> so the `page` tool gets full renderer/DOM access. Use 9222 unless running multiple Electron apps. Ignored for non-Electron apps. - `name` (string, optional): App display name. Used only when bundle_id is absent. - `urls` (array of string, optional): Optional file:// or http(s):// URLs (or plain paths with ~ expansion) to hand to the launched app via application(_:open:). For Finder, pass a folder URL or path to open a backgrounded Finder window rooted at that folder — no activation. Apps that don't implement application(_:open:) launch normally and ignore these. -- `webkit_inspector_port` (integer, optional): Launch a Tauri/WKWebView app with WEBKIT_INSPECTOR_SERVER=127.0.0.1: so the `page` tool can reach its WebKit inspector. Use 9226 (reserved WebKit range: 9226–9228, distinct from Electron's 9222–9225). Requires developerExtrasEnabled=true in the WKWebView config (default in Tauri debug builds). +- `webkit_inspector_port` (integer, optional): Launch a Tauri/WKWebView app with WEBKIT_INSPECTOR_SERVER=127.0.0.1:<N> so the `page` tool can reach its WebKit inspector. Use 9226 (reserved WebKit range: 9226–9228, distinct from Electron's 9222–9225). Requires developerExtrasEnabled=true in the WKWebView config (default in Tauri debug builds). ### list_apps @@ -457,7 +457,7 @@ apps come from scanning /Applications, /Applications/Utilities, Use this for "is X installed?" as well as "is X running?". For per-window state — on-screen, on-current-Space, minimized, window titles — call list_windows instead. For just opening an -app — running or not — call launch_app({bundle_id: ...}) directly; +app — running or not — call launch_app({bundle_id: ...}) directly; list_apps is not a prerequisite. **Arguments:** none. @@ -696,10 +696,9 @@ propagate modifier keys). ### screenshot Capture a screenshot using ScreenCaptureKit. Returns base64-encoded -image data in the requested format (default png). +image data for a single window in the requested format (default png). -Without `window_id`, captures the full main display. With `window_id`, -captures just that window (get the id from `list_windows`). +`window_id` is required. Get window ids from `list_windows`. Requires the Screen Recording TCC grant — call `check_permissions` first if unsure. @@ -708,7 +707,11 @@ first if unsure. - `format` (string, optional): Image format. Default: png. - `quality` (integer, optional): JPEG quality 1-95; ignored for png. -- `window_id` (integer, optional): Optional CGWindowID / kCGWindowNumber to capture just that window. +- `window_id` (integer, required): Required CGWindowID / kCGWindowNumber to capture. + +```json +{"window_id":10725} +``` ### scroll @@ -843,13 +846,13 @@ optional; omitted fields keep their current value. string) to revert to the procedural arrow. Example — brand-colored arrow: - {"gradient_colors": ["#A855F7", "#6366F1"], "bloom_color": "#A855F7"} + {"gradient_colors": ["#A855F7", "#6366F1"], "bloom_color": "#A855F7"} Example — custom PNG cursor: - {"image_path": "~/cursors/my-cursor.png"} + {"image_path": "~/cursors/my-cursor.png"} Example — revert to default: - {"gradient_colors": [], "bloom_color": "", "image_path": ""} + {"gradient_colors": [], "bloom_color": "", "image_path": ""} **Arguments:** @@ -931,7 +934,7 @@ Set a value on a UI element. Two modes depending on element role: - **AXPopUpButton / select dropdown**: finds the child option whose title or value matches `value` (case-insensitive) and AXPresses it directly — the native macOS popup menu is never opened, so focus - is never stolen. Use this for HTML