Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -113,13 +113,20 @@ Cua Driver speaks MCP over stdio. Use `cua-driver mcp-config --client <name>` to
# Claude Code (add --scope project|global as needed)
claude mcp add --transport stdio cua-driver -- ~/.local/bin/cua-driver mcp

# Claude Code computer-use compatibility mode
claude mcp add --transport stdio cua-computer-use -- ~/.local/bin/cua-driver mcp --claude-code-computer-use-compat

# Codex (OpenAI)
codex mcp add cua-driver -- ~/.local/bin/cua-driver mcp

# OpenClaw
cua-driver mcp-config --client openclaw | sh
```

The Claude Code compatibility mode keeps CuaDriver's normal MCP tools, but replaces `screenshot` with a window-only screenshot shim that requires `pid` and `window_id`. Use it when you want Claude Code's vision/computer-use-style flow to ground on CuaDriver window captures.

Use MCP for that Claude Code vision/computer-use-style path. Shelling out to `cua-driver screenshot` can capture a window, but it does not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue.

### Clients configured via a config file

Cursor, OpenCode, and Hermes all configure MCP servers via files. Use `mcp-config` to print the exact snippet, paste it into the right path:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ import { Callout } from 'fumadocs-ui/components/callout';

## Claude Code

Standard MCP registration:

```bash
claude mcp add --transport stdio cua-driver -- cua-driver mcp
```
Expand All @@ -25,6 +27,22 @@ claude mcp list
# cua-driver: cua-driver mcp (stdio) - ✓ Connected
```

### Claude Code computer-use compatibility mode

Claude Code vision/computer-use-style flows appear to use the presence of a screenshot tool as a cue for image-grounded operation. If you want that behavior, register the compatibility server instead:

```bash
claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat
```

This mode still exposes the normal CuaDriver tools. The only changed tool is `screenshot`: it requires `pid` and `window_id`, captures that window only, and returns a window-local image coordinate frame. Start with `launch_app` or `list_windows`, then call `screenshot` with the target window.

For this Claude Code vision/computer-use-style path, use MCP rather than shelling out to the CLI. CLI screenshots can still capture windows, but they do not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue.

<Callout type="info">
This does not call Anthropic APIs or expose Anthropic's native computer-use API tool. It is a CuaDriver MCP compatibility mode for Claude Code.
</Callout>

## GitHub Copilot CLI

Add to `~/.copilot/mcp-config.json`:
Expand Down
22 changes: 17 additions & 5 deletions docs/content/docs/cua-driver/reference/cli-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ description: Command Line Interface reference for Cua Driver
{/*
AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
Generated by: npx tsx scripts/docs-generators/cua-driver.ts
Source: libs/cua-driver/Sources/**/*.swift
Version: 0.1.0
Source: recursive Swift sources under libs/cua-driver/Sources
Version: 0.1.1
*/}

import { Callout } from 'fumadocs-ui/components/callout';
Expand All @@ -16,7 +16,7 @@ import { VersionHeader } from '@/components/version-selector';
<VersionHeader
versions={[{"version":"0.1","href":"/cua-driver/reference/cli-reference","isCurrent":true}]}
currentVersion="0.1"
fullVersion="0.1.0"
fullVersion="0.1.1"
packageName="cua-driver"
installCommand="curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash"
/>
Expand All @@ -36,8 +36,8 @@ provided, the tool is called with no arguments.

Examples:
cua-driver call list_apps
cua-driver call launch_app '{"bundle_id":"com.apple.finder"}'
echo '{"pid":844,"window_id":1234}' | cua-driver call get_window_state
cua-driver call launch_app '&#123;"bundle_id":"com.apple.finder"&#125;'
echo '&#123;"pid":844,"window_id":1234&#125;' | cua-driver call get_window_state

**Arguments:**

Expand Down Expand Up @@ -106,6 +106,12 @@ Print a tool's full description and JSON input schema.

Run the stdio MCP server.

**Flags:**

| Name | Description |
| ---- | ----------- |
| `--claude-code-computer-use-compat` | Expose normal CuaDriver tools, replacing only `screenshot` with a Claude Code-friendly window-only screenshot that establishes the vision coordinate frame. |

### cua-driver serve

Run cua-driver as a long-running daemon on a Unix domain socket.
Expand Down Expand Up @@ -158,6 +164,12 @@ Print MCP server config or a client-specific install command.
| ---- | ---- | ------- | ----------- |
| `--client` | String | — | Client to print the install command for: claude \| codex \| cursor \| openclaw \| opencode \| hermes \| pi. Omit for the generic JSON snippet. |

**Flags:**

| Name | Description |
| ---- | ----------- |
| `--claude-code-computer-use-compat` | Print config for Claude Code's window-scoped screenshot compatibility mode registered as `cua-computer-use`. |

## Trajectory recording

### cua-driver recording
Expand Down
43 changes: 23 additions & 20 deletions docs/content/docs/cua-driver/reference/mcp-tools.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ description: Reference for every MCP tool cua-driver exposes
{/*
AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
Generated by: npx tsx scripts/docs-generators/cua-driver.ts
Source: libs/cua-driver/Sources/**/*.swift
Version: 0.1.0
Source: recursive Swift sources under libs/cua-driver/Sources
Version: 0.1.1
*/}

import { Callout } from 'fumadocs-ui/components/callout';
Expand All @@ -25,7 +25,7 @@ Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a te
Report TCC permission status for Accessibility and Screen Recording.
By default also raises the system permission dialogs for any missing
grants — Apple's request APIs are no-ops when the grant is already
active, so this is safe to call repeatedly. Pass {"prompt": false}
active, so this is safe to call repeatedly. Pass &#123;"prompt": false&#125;
for a purely read-only status check.

**Arguments:**
Expand Down Expand Up @@ -208,18 +208,18 @@ at startup. Sibling to `set_config` / `cua-driver config`.

Current schema:

{
&#123;
"schema_version": 1,
"capture_mode": "vision" | "ax" | "som",
"agent_cursor": {
"agent_cursor": &#123;
"enabled": true,
"motion": {
"motion": &#123;
"start_handle": 0.3, "end_handle": 0.3,
"arc_size": 0.25, "arc_flow": 0.0,
"spring": 0.72
}
}
}
&#125;
&#125;
&#125;

**Arguments:** none.

Expand Down Expand Up @@ -436,10 +436,10 @@ later to resolve a target.
- `additional_arguments` (array of string, optional): Extra command-line arguments passed to the launched process. Passed directly as argv entries — no shell expansion. Example: ["--user-data-dir=/tmp/cua-session", "--no-first-run"] for an isolated Chrome session.
- `bundle_id` (string, optional): App bundle identifier, e.g. com.apple.calculator.
- `creates_new_application_instance` (boolean, optional): Force a brand-new process even if the app is already running. Useful for isolated browser sessions: pass creates_new_application_instance=true together with additional_arguments=["--user-data-dir=/tmp/session-a", "--no-first-run", "--no-default-browser-check"] to launch a sandboxed Chrome that cannot see the user's real profile, cookies, or extensions. Each session gets its own pid and window identity and can be controlled independently.
- `electron_debugging_port` (integer, optional): Launch an Electron app with --remote-debugging-port=<N> so the `page` tool gets full renderer/DOM access. Use 9222 unless running multiple Electron apps. Ignored for non-Electron apps.
- `electron_debugging_port` (integer, optional): Launch an Electron app with --remote-debugging-port=&lt;N&gt; so the `page` tool gets full renderer/DOM access. Use 9222 unless running multiple Electron apps. Ignored for non-Electron apps.
- `name` (string, optional): App display name. Used only when bundle_id is absent.
- `urls` (array of string, optional): Optional file:// or http(s):// URLs (or plain paths with ~ expansion) to hand to the launched app via application(_:open:). For Finder, pass a folder URL or path to open a backgrounded Finder window rooted at that folder — no activation. Apps that don't implement application(_:open:) launch normally and ignore these.
- `webkit_inspector_port` (integer, optional): Launch a Tauri/WKWebView app with WEBKIT_INSPECTOR_SERVER=127.0.0.1:<N> so the `page` tool can reach its WebKit inspector. Use 9226 (reserved WebKit range: 9226–9228, distinct from Electron's 9222–9225). Requires developerExtrasEnabled=true in the WKWebView config (default in Tauri debug builds).
- `webkit_inspector_port` (integer, optional): Launch a Tauri/WKWebView app with WEBKIT_INSPECTOR_SERVER=127.0.0.1:&lt;N&gt; so the `page` tool can reach its WebKit inspector. Use 9226 (reserved WebKit range: 9226–9228, distinct from Electron's 9222–9225). Requires developerExtrasEnabled=true in the WKWebView config (default in Tauri debug builds).

### list_apps

Expand All @@ -457,7 +457,7 @@ apps come from scanning /Applications, /Applications/Utilities,
Use this for "is X installed?" as well as "is X running?". For
per-window state — on-screen, on-current-Space, minimized,
window titles — call list_windows instead. For just opening an
app — running or not — call launch_app({bundle_id: ...}) directly;
app — running or not — call launch_app(&#123;bundle_id: ...&#125;) directly;
list_apps is not a prerequisite.

**Arguments:** none.
Expand Down Expand Up @@ -696,10 +696,9 @@ propagate modifier keys).
### screenshot

Capture a screenshot using ScreenCaptureKit. Returns base64-encoded
image data in the requested format (default png).
image data for a single window in the requested format (default png).

Without `window_id`, captures the full main display. With `window_id`,
captures just that window (get the id from `list_windows`).
`window_id` is required. Get window ids from `list_windows`.

Requires the Screen Recording TCC grant — call `check_permissions`
first if unsure.
Expand All @@ -708,7 +707,11 @@ first if unsure.

- `format` (string, optional): Image format. Default: png.
- `quality` (integer, optional): JPEG quality 1-95; ignored for png.
- `window_id` (integer, optional): Optional CGWindowID / kCGWindowNumber to capture just that window.
- `window_id` (integer, required): Required CGWindowID / kCGWindowNumber to capture.

```json
{"window_id":10725}
```

### scroll

Expand Down Expand Up @@ -843,13 +846,13 @@ optional; omitted fields keep their current value.
string) to revert to the procedural arrow.

Example — brand-colored arrow:
{"gradient_colors": ["#A855F7", "#6366F1"], "bloom_color": "#A855F7"}
&#123;"gradient_colors": ["#A855F7", "#6366F1"], "bloom_color": "#A855F7"&#125;

Example — custom PNG cursor:
{"image_path": "~/cursors/my-cursor.png"}
&#123;"image_path": "~/cursors/my-cursor.png"&#125;

Example — revert to default:
{"gradient_colors": [], "bloom_color": "", "image_path": ""}
&#123;"gradient_colors": [], "bloom_color": "", "image_path": ""&#125;

**Arguments:**

Expand Down Expand Up @@ -931,7 +934,7 @@ Set a value on a UI element. Two modes depending on element role:
- **AXPopUpButton / select dropdown**: finds the child option whose
title or value matches `value` (case-insensitive) and AXPresses it
directly — the native macOS popup menu is never opened, so focus
is never stolen. Use this for HTML <select> elements in Safari or
is never stolen. Use this for HTML &lt;select&gt; elements in Safari or
any native NSPopUpButton. Pass the option's display label as `value`
(e.g. "Blue", not "blue").

Expand Down
2 changes: 1 addition & 1 deletion docs/content/docs/cua/guide/sandbox/lifecycle.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -143,4 +143,4 @@ async with Sandbox.ephemeral(Image.linux(), local=True) as sb: ...
async with Sandbox.ephemeral(Image.macos(), local=True) as sb: ...
```

See [Self-Hosted Sandboxes](/cua/guide/get-started/self-hosted-sandboxes) for local setup instructions.
See [Set Up a Sandbox](/cua/guide/get-started/set-up-sandbox) for local setup instructions.
18 changes: 18 additions & 0 deletions libs/cua-driver/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,21 @@
Background computer-use driver for any agents. Speaks MCP over stdio; drives native macOS apps without stealing focus.

**[Documentation](https://cua.ai/docs/cua-driver)** - Installation, guides, and API reference.

## Claude Code computer-use compatibility

Standard Claude Code MCP registration:

```bash
claude mcp add --transport stdio cua-driver -- cua-driver mcp
```

If you want Claude Code's vision/computer-use-style flow to ground on CuaDriver window screenshots, register the compatibility mode:

```bash
claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat
```

This keeps CuaDriver's normal MCP tools and changes only `screenshot`, which requires `pid` and `window_id` and captures that window only.

Use MCP for this Claude Code vision/computer-use-style path. CLI screenshots still work as CuaDriver calls, but they do not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue.
12 changes: 12 additions & 0 deletions libs/cua-driver/Skills/cua-driver/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,18 @@ also invoke it explicitly:
/cua-driver
```

## Claude Code MCP compatibility mode

For normal skill-driven use, prefer the CLI or the standard MCP server. If you want Claude Code's vision/computer-use-style flow to ground on CuaDriver screenshots, register the compatibility server:

```bash
claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat
```

This mode exposes the normal CuaDriver tools and changes only `screenshot`. The compatibility screenshot requires `pid` and `window_id`, captures that window only, and establishes a window-local pixel coordinate frame. It does not call Anthropic APIs or expose Anthropic's native computer-use API tool.

Use MCP for this Claude Code vision/computer-use-style path. CLI screenshots still work as CuaDriver calls, but they do not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue.

## Files

- `SKILL.md` — the main skill body (~500 lines). Loaded on first
Expand Down
12 changes: 12 additions & 0 deletions libs/cua-driver/Skills/cua-driver/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,18 @@ Every reference to `click(...)`, `get_window_state(...)` etc. in this
skill means `cua-driver click '{...}'` — translate to MCP form only
when MCP is requested.

### Claude Code computer-use compatibility mode

For normal Claude Code use, keep the default CLI or `cua-driver` MCP server path above. If the user explicitly wants Claude Code's vision/computer-use-style flow, they can register:

```bash
claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat
```

Observation: Claude Code vision flows appear to treat a screenshot MCP tool as the image-grounding anchor. This compatibility mode keeps the normal CuaDriver tools and changes only `screenshot`. The compatibility `screenshot` requires `pid` and `window_id`, captures only that target window, and returns the window-local pixel coordinate frame. Start with `launch_app` or `list_windows`, then call `screenshot({pid, window_id})`; do not assume desktop coordinates or a full-screen capture.

Use MCP for this Claude Code vision/computer-use-style path. Do not shell out to `cua-driver screenshot` as a substitute: CLI screenshots still work as CuaDriver calls, but they do not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue.

Intent → tool mapping. If you find yourself reaching for the right
column, something has gone wrong — re-read "The no-foreground
contract" above:
Expand Down
31 changes: 31 additions & 0 deletions libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import AppKit
import ArgumentParser
import CuaDriverServer
import Foundation
Expand Down Expand Up @@ -171,6 +172,7 @@ struct CallCommand: AsyncParsableCommand {

let result: CallTool.Result
do {
await bootstrapAppKitForInProcessCallIfNeeded(toolName: toolName)
// Route through `registry.call(...)` so the recording hook
// (and any future cross-cutting wrapper) fires consistently
// with the MCP and daemon paths. The in-process one-shot
Expand Down Expand Up @@ -264,6 +266,35 @@ struct CallCommand: AsyncParsableCommand {
}
}

private func bootstrapAppKitForInProcessCallIfNeeded(toolName: String) async {
let appKitBackedTools: Set<String> = [
"check_permissions",
"click",
"double_click",
"drag",
"get_accessibility_tree",
"get_cursor_position",
"get_window_state",
"hotkey",
"launch_app",
"list_apps",
"list_windows",
"move_cursor",
"press_key",
"right_click",
"screenshot",
"scroll",
"set_value",
"type_text",
"type_text_chars",
"zoom",
]
guard appKitBackedTools.contains(toolName) else { return }
await MainActor.run {
_ = NSApplication.shared.setActivationPolicy(.accessory)
}
}

struct ListToolsCommand: AsyncParsableCommand {
static let configuration = CommandConfiguration(
commandName: "list-tools",
Expand Down
8 changes: 8 additions & 0 deletions libs/cua-driver/Sources/CuaDriverCLI/ConfigCommand.swift
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ import MCP
///
/// Keys are dotted snake_case paths:
/// - `schema_version`
/// - `capture_mode`
/// - `max_image_dimension`
/// - `agent_cursor.enabled`
/// - `agent_cursor.motion.{start_handle,end_handle,arc_size,arc_flow,spring}`
struct ConfigCommand: AsyncParsableCommand {
Expand All @@ -31,6 +33,8 @@ struct ConfigCommand: AsyncParsableCommand {

Examples:
cua-driver config # print full config
cua-driver config get capture_mode
cua-driver config set capture_mode vision
cua-driver config get agent_cursor.enabled
cua-driver config set agent_cursor.enabled false
cua-driver config set agent_cursor.motion.arc_size 0.4
Expand Down Expand Up @@ -147,6 +151,10 @@ struct ConfigGetCommand: AsyncParsableCommand {
switch key {
case "schema_version":
print(config.schemaVersion)
case "capture_mode":
print(config.captureMode.rawValue)
case "max_image_dimension":
print(config.maxImageDimension)
case "agent_cursor.enabled":
print(config.agentCursor.enabled)
case "agent_cursor.motion.start_handle":
Expand Down
Loading
Loading