diff --git a/.github/workflows/ci-check-docs-links.yml b/.github/workflows/ci-check-docs-links.yml index c7cf48045e..de6851202c 100644 --- a/.github/workflows/ci-check-docs-links.yml +++ b/.github/workflows/ci-check-docs-links.yml @@ -5,7 +5,9 @@ on: paths: - "docs/content/**" - "docs/src/**" + - "docs/package.json" - "docs/scripts/check-links.ts" + - "docs/scripts/check-hygiene.ts" jobs: check-internal-links: @@ -31,6 +33,10 @@ jobs: run: pnpm docs:check-links working-directory: docs + - name: Check docs hygiene + run: pnpm docs:check-hygiene + working-directory: docs + - name: Show help if check failed if: failure() run: | diff --git a/docs/README.md b/docs/README.md index 97b8a8ac82..055d2c9e94 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,39 +2,24 @@ Production docs are served from https://cua.ai/docs; this app is now a local MDX preview only. -This is a Next.js application generated with -[Create Fumadocs](https://github.com/fuma-nama/fumadocs). - -Run development server: +Run the local preview: ```bash -npm run dev -# or pnpm dev -# or -yarn dev ``` -Open http://localhost:3000 with your browser to see the result. - -## Explore - -In the project, you can see: - -- `lib/source.ts`: Code for content source adapter, [`loader()`](https://fumadocs.dev/docs/headless/source-api) provides the interface to access your content. -- `app/layout.config.tsx`: Shared options for layouts, optional but preferred to keep. +Open http://localhost:8090 with your browser to see the result. -| Route | Description | -| ------------------------- | ------------------------------------------------------ | -| `app/(home)` | The route group for your landing page and other pages. | -| `app/docs` | The documentation layout and pages. | -| `app/api/search/route.ts` | The Route Handler for search. | +## Docs conventions -### Fumadocs MDX +Public docs live in `content/docs/` and follow the Diátaxis modes: -A `source.config.ts` config file has been included, you can customise different options like frontmatter schema. +- `tutorials/` teach a guided first success. +- `how-to-guides/` give steps for a specific goal. +- `explanation/` explains concepts, constraints, and tradeoffs. +- `reference/` is dry lookup: commands, APIs, contracts, limits. -Read the [Introduction](https://fumadocs.dev/docs/mdx) for further details. +Place content by what the reader is trying to do, not by topic. Do not mix modes in one page; move reference tables to reference pages and link to them from how-to guides or explanations. ## Setup Telemetry diff --git a/docs/content/docs/concepts/capture-and-delivery-modalities.mdx b/docs/content/docs/concepts/capture-and-delivery-modalities.mdx new file mode 100644 index 0000000000..906421bcff --- /dev/null +++ b/docs/content/docs/concepts/capture-and-delivery-modalities.mdx @@ -0,0 +1,75 @@ +--- +title: "Capture and Delivery Modalities" +description: "How Cua Driver observes and acts on an app. Perception returns both the accessibility tree and a screenshot; the action call chooses the ax or px rung, delivery mode, and scope." +--- + +# Capture and Delivery Modalities + +Every Cua Driver action is shaped by four things: **what the agent observes**, **which rung delivers the action**, **how input is delivered**, and **what coordinate space the action targets**. Most callers use the defaults: background, per-window, accessibility-first automation. The key change from earlier versions is that perception is no longer a mode you pick. `get_window_state` returns *both* the accessibility tree and a screenshot in one call. The action call chooses `ax` or `px` by how it addresses the target. + +## The Axes + +### 1. Perception: what the agent observes + +`get_window_state(pid, window_id)` returns **both the accessibility tree and a screenshot by default**, in one call. There is no capture mode to pick: you ground on the tree and the screenshot together and cross-check one against the other. This matters because the tree *lies* on some surfaces: it can expose useful structure while still echoing a write the app did not apply, omitting the rendered value, or reporting geometry that disagrees with the pixels. A grounding screenshot is always present, so when the tree looks wrong you check the pixels in the *same* response. + +The accessibility tree is the ground truth for *what is clickable*: roles, labels, advertised actions, and an `element_index` handle on every actionable element. The screenshot tells you *which one*. It disambiguates repeated or empty labels and shows captions, colors, and layout the tree omits, which is common in Chromium and Electron. They come back together because each catches what the other misses. + +> **Performance opt-out: `include_screenshot`.** `include_screenshot` (boolean, default `true`) is the one performance knob. The default returns both. Pass `include_screenshot: false` to skip the screen grab and get the tree only when you are re-indexing before an element ax action and do not need fresh pixels. The `ax`-versus-`px` decision still lives at action time. + +> **`capture_mode` is deprecated and ignored.** `get_window_state` still accepts it so old callers do not error, but both the tree and the screenshot come back regardless of what you pass. The old `ax` / `vision` / `som` / `screenshot` values all decode (`som` mapped to `ax`, `screenshot` to `vision`) but none changes what is captured. Perception is always both. + +### 2. Action rung: how the target is addressed + +You don't pick a capture mode; you pick **how you address the target** on the action call, and that one choice selects the rung: + +| Rung | Address with | Delivered through | Properties | +|---|---|---|---| +| **element ax action** | `element_index` / `element_token` | the accessibility rung: UIA Invoke (Windows), `AXPerformAction` (macOS), AT-SPI `doAction` (Linux) | Backgroundable, z-order-independent, and the only **driver-verifiable** rung. | +| **element px action** | `x, y` | the pixel rung, reading the coordinate straight off the screenshot already in the `get_window_state` response | Best-effort; the caller confirms the effect off the screenshot. | + +Default to the element ax action because the driver can verify it and often keep it in the background. Drop to an element px action when the tree cannot disambiguate repeated or empty labels, when it is empty (`degraded`, a non-AX surface), when an action came back `suspected_noop`, or when the tree disagrees with the pixels. You never re-capture to switch rungs. The screenshot is already in the snapshot, so you only change *how you address* the target. + +Both rungs apply to the **keyboard family** (`type_text`, `press_key`, `hotkey`) as well as the pointer tools. Address by `element_index` (ax) to target a field with no pre-click. Address by `x, y` (px) to pixel-click at `(x, y)`, establish real renderer focus, and deliver the keystroke(s) to the now-focused element. The px form is the one-call path for Chromium/Electron inputs the AX layer cannot focus: `type_text({ pid, window_id, x, y, text })` focuses and types in a single call. The two forms are mutually exclusive. `set_value` is the exception: it stays ax-only because it sets the value of a non-text control like a dropdown, checkbox, or slider. + +### 3. Delivery: how input is delivered + +Set `delivery_mode` per call on the input family (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`). The same two values work on Windows, macOS, and Linux. + +| `delivery_mode` | Behavior | +|---|---| +| `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched when the target surface supports background delivery. See [Best-effort background](/concepts/the-no-foreground-contract). | +| `foreground` | The target is briefly fronted (pair with `bring_to_front` to avoid a per-call flash), input lands on the now-active window, then the prior frontmost is restored. Use this when a background attempt did not land, or when the app only accepts events while foregrounded (DirectInput games, raw-input canvases). | + +Only `background` and `foreground` are valid; the historical `auto` heuristic is removed. At runtime, omitted or unknown values fall back to `background` for safety. Element ax actions (`element_index`) address an element instead of the focused window, so they hold the background path without any `delivery_mode` flag. The delivery axis matters most for the pixel rung (`x, y`), where `background` routes the event to the target and `foreground` raises the window first. + +### 4. Capture scope: what coordinate space the action targets + +Set with `set_config capture_scope=…`. + +| `capture_scope` | Coordinate space | Capture surface | +|---|---|---| +| `window` (default) | Per-window. Actions carry `pid` + `window_id`; coordinates are window-relative or addressed by `element_index`. | `get_window_state` | +| `desktop` | Screen-absolute. Window-less actions (no `pid`/`window_id`) land at absolute screen coordinates via hit-testing (`WindowFromPoint`). | `get_desktop_state` with full display, no downscale | + +Desktop scope is the "Computer-Use 1.0" loop: the agent reads the whole screen and clicks absolute coordinates, the way a screenshot-only model expects. Window scope is the default because it is what makes background, concurrent automation possible. + +## Response signals: verification, effect, and escalation + +Delivery success differs from application state change. The driver can verify an effect only when it can read the changed state back through the accessibility layer. That is why `verified: true` is reserved for AX read-back: it means the driver observed the effect after sending the event. Pixel input, foreground input, and echo-prone AX surfaces can be routed correctly while still leaving confirmation to the caller. + +`effect` is the confidence signal that separates those cases. `"confirmed"` means the driver verified the result through AX read-back. `"unverifiable"` means the delivery path ran, but the driver cannot prove the application applied it. `"suspected_noop"` means an AX action ran but almost certainly did not change the target. Treat `effect` as the action outcome. + +`escalation` is the machine-readable climb-the-ladder hint. When present, it tells the caller which surface to try next: `"px"` for acting off the screenshot, `"foreground"` for explicitly fronting the target, or `"page"` for the browser-tab DOM path through the `page` tool. See [Agent action policy](/reference/cua-driver/action-selection-policy) for the agent behavior and [MCP tool notes](/reference/cua-driver/mcp-tool-notes#action-response-shape) for the field table. + +### When the tree lies + +Some accessibility layers echo writes they did not apply. Electron can report an AX value change through its shim while the renderer stays unchanged. Catalyst controls can expose null `AXValue`s. Chromium/WebKit web content can reflect a write through the accessibility bridge without proving the DOM or rendered view changed. + +The driver treats those as surface-aware verification cases. It probes at the element level for a web-content surface, including an `AXWebArea` ancestor, so native chrome such as a browser address bar stays trusted while browser-tab content does not get a false confirmation. On those surfaces the driver refuses false `verified: true` responses and returns `verified: false`, `effect: "unverifiable"`, and an `escalation` object instead. Electron app surfaces recommend `"px"` so the caller can act by pixel off the screenshot in the same response; browser-tab web content recommends `"page"` so the caller can switch to DOM/CDP via the `page` tool. + +## Where the exact matrix lives + +Perception is no longer an axis: every `get_window_state` returns both the tree and a screenshot by default. The enforceable constraints are the combination of **capture scope**, **action rung**, and **delivery mode**. + +The exact validity matrix and platform-support table live in [Interface contracts](/reference/cua-driver/contracts). Keep this page as the mental model: observe both tree and pixels, start with the accessibility rung, use the screenshot-backed pixel rung when the tree is insufficient, and escalate to foreground only when the target app requires it. diff --git a/docs/content/docs/explanation/how-sandboxes-work.mdx b/docs/content/docs/concepts/how-sandboxes-work.mdx similarity index 100% rename from docs/content/docs/explanation/how-sandboxes-work.mdx rename to docs/content/docs/concepts/how-sandboxes-work.mdx diff --git a/docs/content/docs/concepts/index.mdx b/docs/content/docs/concepts/index.mdx new file mode 100644 index 0000000000..f298e56070 --- /dev/null +++ b/docs/content/docs/concepts/index.mdx @@ -0,0 +1,8 @@ +--- +title: "Concepts" +description: "Understand the design ideas behind Cua and how its main pieces fit together." +--- + +Use these pages when you want the model behind Cua rather than a step-by-step guide or API table. They explain what computer use means in Cua, how Cua Driver keeps the desktop usable while it acts, and how Cua Sandbox gives an agent a disposable computer. + +Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers. diff --git a/docs/content/docs/concepts/meta.json b/docs/content/docs/concepts/meta.json new file mode 100644 index 0000000000..ba09bd3452 --- /dev/null +++ b/docs/content/docs/concepts/meta.json @@ -0,0 +1 @@ +{ "title": "Concepts", "icon": "Lightbulb", "pages": ["index", "what-is-computer-use", "the-no-foreground-contract", "capture-and-delivery-modalities", "how-sandboxes-work"] } diff --git a/docs/content/docs/concepts/the-no-foreground-contract.mdx b/docs/content/docs/concepts/the-no-foreground-contract.mdx new file mode 100644 index 0000000000..0459bfa44a --- /dev/null +++ b/docs/content/docs/concepts/the-no-foreground-contract.mdx @@ -0,0 +1,54 @@ +--- +title: Best-effort background +description: How Cua Driver tries to operate apps without taking focus, moving the cursor, or raising windows, and when it must fall back to foreground. +--- + +# Best-effort background + +Best-effort background means Cua Driver tries to operate a target app while preserving the user's active desktop. The default paths do not move the real pointer, do not raise the target window, and do not switch the user's frontmost app. + +This is a best effort rather than an absolute promise. Most app automation can stay in the background through accessibility actions, routed input, and window-specific capture. A small set of apps and OS surfaces only accept real foreground input, so the driver reports that limit and lets the caller choose a foreground escalation for that specific action. + +## Why it matters + +Traditional GUI automation assumes the automated app owns the desktop. It activates a window, moves the pointer, and repeats. That is fine for unattended jobs or disposable desktops, but it breaks down when a person is using the same machine. + +Cua Driver's default path lets the agent operate an app in the background while the developer keeps coding, reading logs, or using another app. The visible agent cursor is an overlay; the real mouse pointer stays where the user left it. + +## Platform mechanisms + +Each OS splits accessibility, input delivery, capture, and focus policy differently. Cua Driver chooses the most background-capable path the platform and app expose. + +### macOS + +The Accessibility API can press buttons, set values, and read semantic state even when the app is not frontmost. ScreenCaptureKit can capture a specific window without requiring that window to be raised or visible on the active Space. Cua Driver also uses scoped CoreGraphics and SkyLight delivery for routed input when an app responds better to pointer-like events than accessibility actions. + +Some macOS surfaces still need foreground. SwiftUI windows parked on another Space can lose their detailed accessibility tree, and game/canvas surfaces may reject routed input. Those cases are documented in [Known limits](/reference/cua-driver/limits). + +### Windows + +UI Automation can inspect and operate controls by window handle and automation element while another app is active. For input-like behavior, Cua Driver can post messages to a target window or use foreground escalation when the app only listens to active device input. + +Windows also has session boundaries. A daemon running in the interactive user session can see and operate the desktop; a process launched from OpenSSH in Session 0 cannot. See [Process model](/reference/cua-driver/process-model) and [Drive a Windows app over SSH](/how-to-guides/driver/windows-ssh). + +### Linux + +AT-SPI provides the semantic path on Linux. Element actions call the toolkit's own accessibility action (`Action.DoAction`) and do not need pointer injection or foreground focus. On X11, window-addressable input and capture can also route to a target window. On Wayland, synthetic input is intentionally constrained by the compositor, so the background path depends more heavily on AT-SPI and reconstructed element frames. + +The remaining Linux gap is raw keyboard injection into native Wayland apps. Typing into accessible fields can still work through AT-SPI, but shortcuts or raw key events may need XWayland or foreground/user-granted compositor paths. See [Known limits](/reference/cua-driver/limits#native-wayland-apps-cant-receive-synthetic-keystrokes). + +## The agent cursor + +Cua Driver does not move the user's real pointer to show agent activity. It renders a synthetic cursor overlay for supervision. The user can see where the agent is acting while their own cursor and active app stay untouched. + +## How fallback works + +The safest ladder is: + +1. Act by `element_index` in the background. +2. If the element path is unavailable or unverifiable, act by `x, y` from the same window screenshot. +3. If the app still rejects the action, retry that one action with `delivery_mode: "foreground"`. + +Foreground escalation is explicit. It is the right answer for apps that only accept focused input, but callers should use it narrowly and only when interrupting the user's desktop is acceptable. + +For the agent-side action behavior, see [Agent action policy](/reference/cua-driver/action-selection-policy). For the reference matrix, see [Interface contracts](/reference/cua-driver/contracts). diff --git a/docs/content/docs/explanation/what-is-computer-use.mdx b/docs/content/docs/concepts/what-is-computer-use.mdx similarity index 84% rename from docs/content/docs/explanation/what-is-computer-use.mdx rename to docs/content/docs/concepts/what-is-computer-use.mdx index 9c347c750e..067c6bf0bb 100644 --- a/docs/content/docs/explanation/what-is-computer-use.mdx +++ b/docs/content/docs/concepts/what-is-computer-use.mdx @@ -1,11 +1,11 @@ --- title: "Computer-Use 2.0" -description: "Computer-Use 2.0 is how an agent operates a real computer through code, structured tool calls, and the graphical interface." +description: "What Cua refers to as Computer-Use 2.0: an agent operating a real computer through code, structured tool calls, and the graphical interface." --- # Computer-Use 2.0 -Computer-Use 2.0 is the way an AI agent operates a real computer by choosing among three action surfaces: writing and running code, calling tools and APIs, and driving the graphical interface a person would use. The older use of the term, where an agent looks at screenshots and clicks through a GUI, still matters as the UI surface inside a broader model of how agents get work done on computers. +What we refer to as **Computer-Use 2.0** is an AI agent operating a real computer by choosing among three action surfaces: writing and running code, calling tools and APIs, and driving the graphical interface a person would use. The older screenshot-and-click loop still matters as the UI surface inside a broader model of how agents get work done on computers. ## Three action surfaces @@ -27,7 +27,7 @@ The observe, decide, act loop belongs specifically to the UI automation surface. Planning carries the task across changing interface states. A click may open a dialog, a page may reflow after loading, or an application may show an error that changes the next useful action, so the model has to keep track of the goal while the computer responds. Frontier models such as Claude can handle understanding, grounding, and planning together in one call, while grounding-specialist models such as UI-TARS and Moondream can help when coordinate accuracy is the limiting factor. -The narrow origin of computer-use came in October 2024, when Anthropic introduced an agent that operated a GUI through screenshots and input events. Through 2025, coding agents were increasingly recognized as computer-use agents too, with CoAct-1 making the connection explicit, and the field began to converge on Computer-Use 2.0 as a wider concept. Francesco Bonacci traces that arc in [A Story of Computer-Use](https://github.com/trycua/cua/blob/main/blog/clawdbot-computer-use-history.md). +The narrow origin of computer-use came in October 2024, when Anthropic introduced an agent that operated a GUI through screenshots and input events. Through 2025, coding agents were increasingly recognized as computer-use agents too, with CoAct-1 making the connection explicit. Cua uses **Computer-Use 2.0** as shorthand for that wider model. Francesco Bonacci traces that arc in [A Story of Computer-Use](https://github.com/trycua/cua/blob/main/blog/clawdbot-computer-use-history.md). ## Where Cua fits diff --git a/docs/content/docs/explanation/architecture.mdx b/docs/content/docs/explanation/architecture.mdx deleted file mode 100644 index a07fd10dce..0000000000 --- a/docs/content/docs/explanation/architecture.mdx +++ /dev/null @@ -1,36 +0,0 @@ ---- -title: Architecture -description: "How Cua Driver and Cua Sandbox give an external agent a computer-use interface: one on a real machine, one in a disposable VM." ---- - -Cua has two separate ways to expose a computer to an external agent. In both cases, the agent sends computer-use operations and receives observations such as screenshots, window lists, accessibility trees, and command results. The operating-system details stay behind a driver or server. The agent, model, or orchestration code remains outside that boundary and decides what to do next. - -## Two paths to a computer - -The real-machine path uses Cua Driver. The computer is an existing macOS, Windows, or Linux machine. Cua Driver runs on that machine and exposes OS-level operations through MCP over stdio and through a CLI. This path is for operating real native applications in the user's environment. - -The disposable-computer path uses Cua Sandbox. The computer is a cloud or local VM or container created for the task. Cua Sandbox starts and controls that environment, then talks to a `computer-server` running inside it over HTTP and WebSocket. This path is for isolated workspaces that can be created, used, and discarded. - -These paths are not two backends behind one shared SDK. Cua Driver and Cua Sandbox are separate products. They both provide a computer-use surface, but they use different transports, runtimes, and deployment targets. - -## Cua Driver: the real machine - -Cua Driver is a single Rust binary that operates a real machine. It exposes operations for window discovery, screenshots, accessibility-tree walking, input dispatch, and background operation that does not take over the user's active workspace. Tools can call it through MCP over stdio, or users can call it directly through the CLI. - -Under the hood, Cua Driver uses the native platform APIs for each operating system. On macOS it uses AX plus CoreGraphics. On Windows it uses UIA. On Linux it uses AT-SPI. Those APIs provide the bridge from a model-agnostic computer-use request to concrete OS behavior, such as finding windows, reading accessibility state, capturing pixels, and sending input. - -Because it runs on the real machine, Cua Driver inherits that machine's installed applications, user session, permissions, displays, and files. That is the main reason to use it. The tradeoff is that the computer is not disposable, so the caller is responsible for choosing operations that are appropriate for the user's active environment. - -## Cua Sandbox: a disposable computer - -Cua Sandbox creates a disposable cloud or local computer. The environment can be an isolated VM or container, depending on the backend. The Python Sandbox SDK creates and controls that sandbox, then communicates with the `computer-server` running inside it over REST and WebSocket APIs. - -The `computer-server` is the component inside the sandbox that exposes the computer-use surface for that disposable machine. It receives operations over HTTP and WebSocket, performs them inside the VM or container, and returns observations. The Python sandbox stack talks to this server only over HTTP and WebSocket. It does not invoke Cua Driver and it does not use MCP. - -Cua Sandbox can target several backends, including Lume and Lumier for macOS VMs, cloud environments, Windows Sandbox, QEMU, Hyper-V, and Docker. The backend determines where the disposable computer runs and what isolation boundary it uses, but the external shape remains the same: create a sandbox, connect to its `computer-server`, operate the computer, then tear it down when the task is finished. - -## The model boundary - -Cua Driver and `computer-server` are model-agnostic. They receive operations and return observations. They do not call language models, plan tasks, decide intent, or choose the next action. Cua does not ship a model. - -That boundary keeps OS interaction separate from reasoning. The user brings an agent or model, and that system decides what operation to send next based on the observations it receives. Cua provides the computer-use interface on either a real machine through Cua Driver, or a disposable machine through Cua Sandbox and `computer-server`. diff --git a/docs/content/docs/explanation/capture-and-dispatch-modalities.mdx b/docs/content/docs/explanation/capture-and-dispatch-modalities.mdx deleted file mode 100644 index 4798bd8fc6..0000000000 --- a/docs/content/docs/explanation/capture-and-dispatch-modalities.mdx +++ /dev/null @@ -1,97 +0,0 @@ ---- -title: "Capture and Dispatch Modalities" -description: "How Cua Driver observes and acts on an app — perception always returns both the accessibility tree and a screenshot, the action rung (ax or px) is chosen at action time, plus dispatch and capture scope — and which combinations are valid." ---- - -# Capture and Dispatch Modalities - -Every Cua Driver action is shaped by four things: **what the agent observes**, **which rung dispatches the action**, **how input is delivered**, and **what coordinate space the action targets**. Most callers never set any of them — the defaults give background, per-window, accessibility-first automation. The key change from earlier versions: perception is no longer a mode you pick. `get_window_state` returns *both* the accessibility tree and a screenshot in one call, and the `ax`-versus-`px` choice is made at action time, by how you address the target. This page documents the full set so you can reason about the combinations that are valid and the ones the driver rejects by design. - -## The Axes - -### 1. Perception — what the agent observes - -`get_window_state(pid, window_id)` returns **both the accessibility tree and a screenshot by default**, in one call. There is no capture mode to pick: you ground on the tree and the screenshot together and cross-check one against the other. This matters because the tree *lies* on some surfaces: it can expose useful structure while still echoing a write the app did not apply, omitting the rendered value, or reporting geometry that disagrees with the pixels. A grounding screenshot is always present, so when the tree looks wrong you check the pixels in the *same* response. - -The accessibility tree is the ground truth for *what is clickable* — roles, labels, advertised actions, and an `element_index` handle on every actionable element. The screenshot tells you *which one*, disambiguates repeated or empty labels, and surfaces captions, colors, and layout the tree omits entirely (common in Chromium and Electron). They are complementary, which is why both come back together. - -> **Perf opt-out — `include_screenshot`.** `include_screenshot` (boolean, default `true`) is the one knob, and it is a **performance** knob, not a modality choice. The default returns both. Pass `include_screenshot: false` to skip the screen grab and get the tree only — the cheap path when you are re-indexing before an element ax action and don't need fresh pixels. The `ax`-versus-`px` decision still lives at action time, not here. - -> **`capture_mode` is deprecated and ignored.** It is still *accepted* on `get_window_state` so old callers don't error, but it has **no effect** — both the tree and the screenshot come back regardless of what you pass. The old `ax` / `vision` / `som` / `screenshot` values all decode (`som` mapped to `ax`, `screenshot` to `vision`) but none changes what is captured. There is no capture-mode choice anymore; perception is always both. - -### 2. Action rung — how the target is addressed - -You don't pick a capture mode; you pick **how you address the target** on the action call, and that one choice selects the rung: - -| Rung | Address with | Dispatches through | Properties | -|---|---|---|---| -| **element ax action** | `element_index` / `element_token` | the accessibility rung — UIA Invoke (Windows), `AXPerformAction` (macOS), AT-SPI `doAction` (Linux) | Backgroundable, z-order-independent, and the only **driver-verifiable** rung. | -| **element px action** | `x, y` | the pixel rung, reading the coordinate straight off the screenshot already in the `get_window_state` response | Best-effort; the caller confirms the effect off the screenshot. | - -Default to the element ax action — it is verifiable and backgroundable. Drop to an element px action when the tree can't disambiguate (repeated or empty labels), when it's empty (`degraded` — a non-AX surface), when an action came back `suspected_noop`, or when the tree disagrees with the pixels. You never re-capture to switch rungs: the screenshot is already in the snapshot, so you only change *how you address* the target. - -Both rungs apply to the **keyboard family** (`type_text`, `press_key`, `hotkey`), not just the pointer tools. Address by `element_index` (ax) to target a field with no pre-click, or by `x, y` (px) — which **pixel-clicks at `(x, y)` to establish real renderer focus, then delivers the keystroke(s)** to the now-focused element. The px form is the one-call path for Chromium/Electron inputs the AX layer can't focus: `type_text({ pid, window_id, x, y, text })` focuses and types in a single call. The two forms are mutually exclusive. (`set_value` is the exception — it stays ax-only, because it sets the value of a non-text control like a dropdown, checkbox, or slider; its pixel counterpart is a `click`/`drag` on the control, not a "set value at a pixel.") - -### 3. Dispatch — how input is delivered - -Set per call on the input family (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`) with the `delivery_mode` field. This field is a shared cross-platform parameter — the same two values, accepted uniformly on Windows, macOS, and Linux. - -| `delivery_mode` | Behavior | -|---|---| -| `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched. This is the [no-foreground contract](/explanation/the-no-foreground-contract). | -| `foreground` | The target is briefly fronted (pair with `bring_to_front` to avoid a per-call flash), input lands on the now-active window, then the prior frontmost is restored. The explicit last resort when a background attempt did not land — and the only path for apps that accept events solely when foregrounded (DirectInput games, raw-input canvases). | - -Only `background` and `foreground` are valid; the historical `auto` heuristic is removed. At runtime, omitted or unknown values fall back to `background` for safety. Element ax actions (`element_index`) are inherently background — they address an element, not the focused window — so they hold the contract without any `delivery_mode` flag. The dispatch axis matters most for the pixel rung (`x, y`), where `background` routes the event to the target and `foreground` raises the window first. - -### 4. Capture scope — what coordinate space the action targets - -Set with `set_config capture_scope=…`. - -| `capture_scope` | Coordinate space | Capture surface | -|---|---|---| -| `window` (default) | Per-window. Actions carry `pid` + `window_id`; coordinates are window-relative or addressed by `element_index`. | `get_window_state` | -| `desktop` | Screen-absolute. Window-less actions (no `pid`/`window_id`) land at absolute screen coordinates via hit-testing (`WindowFromPoint`). | `get_desktop_state` — full display, no downscale | - -Desktop scope is the "Computer-Use 1.0" loop: the agent reads the whole screen and clicks absolute coordinates, the way a screenshot-only model expects. Window scope is the default because it is what makes background, concurrent automation possible. - -## Response signals: verification, effect, and escalation - -Dispatch success is not the same thing as application state change. The driver can verify an effect only when it can read the changed state back through the accessibility layer. That is why `verified: true` is reserved for AX read-back: it means the driver observed the effect, not merely that it sent an event. Pixel input, foreground input, and echo-prone AX surfaces can be routed correctly while still leaving confirmation to the caller. - -`effect` is the confidence signal that separates those cases. `"confirmed"` means the driver verified the result through AX read-back. `"unverifiable"` means the dispatch path ran, but the driver cannot prove the application applied it. `"suspected_noop"` means an AX action dispatched but almost certainly did not change the target. Callers should treat `effect`, not the transport-level success status, as the action outcome. - -`escalation` is the machine-readable climb-the-ladder hint. When present, it tells the caller which surface to try next: `"px"` for acting off the screenshot, `"foreground"` for explicitly fronting the target, or `"page"` for the browser-tab DOM path through the `page` tool. See [Choose an action rung and dispatch mode](/how-to-guides/driver/choose-a-modality) for the procedural ladder and [MCP tool notes](/reference/cua-driver/mcp-tool-notes#action-response-shape) for the field table. - -### When the tree lies - -Some accessibility layers echo writes they did not apply. Electron can report an AX value change through its shim while the renderer stays unchanged. Catalyst controls can expose null `AXValue`s. Chromium/WebKit web content can reflect a write through the accessibility bridge without proving the DOM or rendered view changed. - -The driver treats those as surface-aware verification cases. It probes at the element level for a web-content surface, including an `AXWebArea` ancestor, so native chrome such as a browser address bar stays trusted while browser-tab content does not get a false confirmation. On those surfaces the driver refuses false `verified: true` responses and returns `verified: false`, `effect: "unverifiable"`, and an `escalation` object instead. Electron app surfaces recommend `"px"` so the caller can act by pixel off the screenshot in the same response; browser-tab web content recommends `"page"` so the caller can switch to DOM/CDP via the `page` tool. - -## The Validity Matrix - -Perception is no longer an axis in this matrix — every `get_window_state` returns both the tree and a screenshot, so there is nothing to cross here. The real constraints are **capture scope** (window or desktop) against the **action rung** (`ax` or `px`) and **delivery** (background or foreground). Capture scope is the constraining axis: window scope supports **any** combination of rung and delivery; desktop scope supports **only** the `px` rung with `foreground` delivery. - -| `capture_scope` | action rung | `delivery_mode` | Valid? | Why | -|---|---|---|---|---| -| `window` | `ax` (element_index) | `background` | ✅ | The default. Semantic actions on a backgrounded window. | -| `window` | `ax` (element_index) | `foreground` | ✅ | Activate, then act by element. | -| `window` | `px` (x, y) | `background` | ✅ | Click a coordinate off the window's screenshot without raising it. | -| `window` | `px` (x, y) | `foreground` | ✅ | Activate, then click by coordinate. | -| `desktop` | `px` (x, y) | `foreground` | ✅ | The only desktop combination. Read the whole screen, click absolute coordinates on the active desktop. | -| `desktop` | `ax` (element_index) | any | ❌ | A desktop-absolute action has no `window_id`, so there is no element tree to resolve an `element_index` against — there is no `ax` rung. | -| `desktop` | any | `background` | ❌ | Screen-absolute input hits whatever owns those pixels on the active desktop; there is no per-process route, so it cannot be backgrounded. | -| `window` | (window-less) | — | ❌ | A window-less (no `pid`/`window_id`) action while scope is `window` is rejected with the structured `desktop_scope_disabled` error. | - -The two rejections are enforced, not advisory. A window-less click while `capture_scope=window` returns `desktop_scope_disabled`, pointing the caller at `set_config capture_scope=desktop`. Desktop scope inherently foregrounds and works on pixels, so it cannot honor the background contract or reach the `ax` rung — which is exactly why background, per-window automation is the default. - -## Platform Support - -| Axis / value | Windows | macOS | Linux | -|---|---|---|---| -| action rung: `ax` (element_index) / `px` (x, y) | ✅ | ✅ | ✅ | -| `dispatch: background` (the contract) | ✅ | ✅ | ✅ (X11 and Wayland via AT-SPI; raw keyboard into native-Wayland apps is the [residual gap](/explanation/the-no-foreground-contract#linux)) | -| `dispatch: foreground` + `bring_to_front` | ✅ explicit activation | ✅ explicit activation (`NSRunningApplication.activate`) | ✅ X11 EWMH activation (`_NET_ACTIVE_WINDOW` + input focus); Wayland raise is compositor-constrained | -| `capture_scope: desktop` (full screen-absolute loop) | ✅ | rolling out | rolling out | - -Both action rungs work on all three platforms, so window-scope automation — the four window-scope rows of the matrix — works everywhere. The desktop-scope loop (`get_desktop_state` plus window-less screen-absolute input via hit-testing) is complete on Windows and rolling out to macOS and Linux; on those platforms a window-less action under window scope is still rejected. See the [MCP tool reference](/reference/cua-driver/mcp-tools) for per-tool parameters and the [no-foreground contract](/explanation/the-no-foreground-contract) for how background dispatch is implemented on each OS. diff --git a/docs/content/docs/explanation/demonstrations-skills-and-trajectories.mdx b/docs/content/docs/explanation/demonstrations-skills-and-trajectories.mdx deleted file mode 100644 index 0d74b642d1..0000000000 --- a/docs/content/docs/explanation/demonstrations-skills-and-trajectories.mdx +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: 'Demonstrations, skills, and trajectories' -description: 'Cua has three different recording systems that get conflated. This page explains what each one records, where it lives, and when to use it.' ---- - -Several cua features use the words "record" and "trajectory", but they do not record the same thing. One system records a human demonstration and turns it into a reusable skill. Another records the agent's action-tool calls at the driver boundary. A third records the agent loop itself, including model outputs and computer observations. - -The **actor being recorded** is the central distinction. Demonstration skills capture a human and produce a **generalizing document**. Cua Driver recordings and agent trajectories capture the agent, but at different layers: Cua Driver records the tools sent to the desktop, while `TrajectorySaverCallback` records the agent run around those tools. - -## Demonstration Skills - -Demonstration skills are exposed through the `cua skills record`, `list`, `read`, `replay`, `delete`, and `clean` CLI commands. The implementation lives in `cua-cli`. - -This system records a human demonstrating a task by hand over VNC. During recording, the human uses the remote desktop normally. Afterward, cua extracts frames at each input event and uses a vision-language model to caption the step. Each caption describes the observation, intent, action, and expected result for that moment in the demonstration. - -The output is a reusable skill under `~/.cua/skills//`. The main artifact is `SKILL.md`, which contains frontmatter with the skill name and description, a `Steps` section made from the captioned steps, and an `Agent Prompt`. The same skill directory also contains a `trajectory/` folder with the MP4 video, `events.json`, `trajectory.json`, and per-step screenshots. - -Replay has two different meanings here. `cua skills replay` opens the MP4 so a person can watch the demonstration. It does not re-drive the original mouse and keyboard input. Actual reuse happens by reading `SKILL.md`, prefixing it as context to a `ComputerAgent` run, and then giving the agent a new task with new inputs. The agent follows the demonstrated pattern and adapts the parameters. This is the cua analog of OpenAI Codex record-and-replay: show a workflow once, get a reusable skill, not a verbatim playback script. - -Use demonstration skills when the valuable thing is human know-how. They fit multi-step GUI workflows that should generalize across similar tasks, especially when the important knowledge is the order of operations, the visual cues, and the decision points rather than exact screen coordinates. - -See [Record a demonstration](/how-to-guides/skills/record-a-demonstration) for the how-to guide. - -## Cua Driver Trajectory Recording - -Cua Driver trajectory recording is exposed through the `cua-driver recording start`, `stop`, `status`, and `render` subcommands, and through the equivalent MCP tools `start_recording`, `stop_recording`, `get_recording_state`, and `replay_trajectory`. It requires a running Cua Driver daemon. Recording state is per-process and kept in memory, so restarting the daemon resets it. - -This system records the agent's action-tool calls. While recording is enabled, every action tool call writes a numbered `turn-NNNNN` folder under the selected `output_dir`. Recording fires for every non-read-only action tool, such as `click`, `right_click`, `scroll`, `type_text`, `press_key`, `hotkey`, and `set_value`. Read-only tools such as `get_window_state` and `list_windows` are not recorded. - -Each turn folder contains `action.json`, which stores the tool name, full arguments, result, process id, click point, and timestamp. It also contains `screenshot.png`, a post-action capture of the target window, and `app_state.json`, a post-action AX or UIA tree snapshot that is omitted on Linux. Click-family actions also write `click.png`, a screenshot with a red dot at the click point. Video capture is off by default. Pass `record_video: true` to also capture the main display as `recording.mp4`, written as H.264 at 30 fps. - -Replay is literal at the driver boundary. `replay_trajectory` walks the turn folders in order and re-invokes each recorded tool with its recorded arguments. The main controls are `delay_ms`, which spaces out actions, and `stop_on_error`, which decides whether replay stops on the first failed action. - -The important caveat is **element addressing**. `element_index` values do not survive across sessions because indices are assigned per `get_window_state` snapshot and are keyed on `pid` and `window_id`. Element-indexed actions therefore do not resolve reliably on replay. Pixel clicks and keyboard tools replay cleanly. For that reason, Cua Driver recording is best treated as a regression and evidence artifact rather than a guaranteed durable automation script. - -Use Cua Driver recording when the relevant question is exactly what the agent sent to the desktop and what the desktop looked like afterward. It is useful for demos, comparing a future run against a saved run across builds, and collecting tool-level training data. - -See the Cua Driver [trajectory recording CLI reference](/reference/cua-driver/cli-reference#trajectory-recording) and [recording MCP tools](/reference/cua-driver/mcp-tools#start_recording) for the command and tool surfaces. - -## Agent Trajectories - -Agent trajectories are produced by `TrajectorySaverCallback` in the `cua-agent` package. The callback is attached to a `ComputerAgent`, so recording is wired into the agent run itself rather than managed by a separate daemon. The CLI surface for saved sessions is `cua trajectory ls`, `view`, and `clean`, implemented across `cua-agent` and `cua-cli`. - -This system records the agent loop's `(state, action, next_state)` data as the run proceeds. The saved files include messages, model outputs, computer calls, call outputs, and screenshots. It captures the reasoning and observation context around the desktop operations, not only the driver calls themselves. - -Inspection is session-oriented. `cua trajectory ls` lists saved sessions. `cua trajectory view` zips a session and opens it in the cua.ai trajectory viewer in the browser. `cua trajectory clean` removes old sessions. - -Use agent trajectories when the relevant question is *why* an agent behaved as it did. They are the right artifact for debugging model decisions, reviewing the chain of observations and actions in a run, and collecting training or evaluation data from real agent sessions. - -## Which One Do I Want - -The systems differ by the actor recorded and the layer where the recording happens. Demonstration skills record a human and produce `SKILL.md`, which is meant to generalize. Cua Driver recordings and agent trajectories record the agent. Cua Driver records at the tool-call boundary for replay and regression. Agent trajectories record at the agent-loop boundary for debugging and data analysis. - -| System | Records | Artifact | Replay means | Use for | -| --- | --- | --- | --- | --- | -| Demonstration skills | Human demo over VNC | `~/.cua/skills//SKILL.md` plus `trajectory/` assets | Watch the MP4 with `cua skills replay`, or reuse by giving `SKILL.md` to a `ComputerAgent` as context | Teaching a reusable GUI workflow from one human demonstration | -| Cua Driver recording | Agent action-tool calls | `turn-NNNNN` folders under `output_dir`, plus `recording.mp4` when `record_video` is set | Re-invoke recorded driver tools in order with `replay_trajectory` | Demos, regression diffs, and tool-level data collection | -| Agent trajectories | Agent loop state, actions, next states, messages, model outputs, computer calls, outputs, and screenshots | Saved trajectory session files | Inspect sessions with `cua trajectory view` in the trajectory viewer | Debugging agent behavior and collecting training or eval data | diff --git a/docs/content/docs/explanation/index.mdx b/docs/content/docs/explanation/index.mdx deleted file mode 100644 index 3c0e03a24e..0000000000 --- a/docs/content/docs/explanation/index.mdx +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: "Explanation" -description: "Understand the design ideas behind Cua and how its main pieces fit together." ---- - -Explanation is for the parts of Cua that are easier to use once the model is clear. These pages are not recipes or API listings. They describe the **concepts, constraints, and tradeoffs** behind Cua's two main ways of giving an agent a computer: driving an existing machine with Cua Driver, or running an isolated cloud desktop with Cua Sandbox. Read this section when you want to understand why the system behaves the way it does, what guarantees it is trying to preserve, and where each component sits in the stack. - -The section starts with [What is computer use?](/explanation/what-is-computer-use), which defines computer-use agents and where Cua fits. [The no foreground contract](/explanation/the-no-foreground-contract) explains the central invariant in Cua Driver: agent actions must not steal focus, move the real pointer, or interrupt the person using the machine. [Capture and dispatch modalities](/explanation/capture-and-dispatch-modalities) lays out the four axes of every action — what the agent observes, which rung dispatches it, how input is delivered, and the coordinate space it targets — and which combinations are valid. [Process model](/explanation/process-model) describes how Cua Driver runs as an MCP stdio server, daemon, and CLI. [How sandboxes work](/explanation/how-sandboxes-work) covers cloud desktop sandboxes, their lifecycle, and snapshots. [Architecture](/explanation/architecture) ties the pieces together: Cua Driver, Cua Sandbox, and the computer-server. diff --git a/docs/content/docs/explanation/linux-and-wayland.mdx b/docs/content/docs/explanation/linux-and-wayland.mdx deleted file mode 100644 index a10ede7fa5..0000000000 --- a/docs/content/docs/explanation/linux-and-wayland.mdx +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: "Linux and Wayland" -description: "Why Cua Driver leans on the accessibility tree on Linux, how it recovers real screen coordinates GTK4 hides, and how a coordinate click lands on Wayland where there is no global coordinate space." ---- - -# Linux and Wayland - -Linux is not one target. The same desktop app runs under two different session stacks — X11 and Wayland — and they expose input and geometry on opposite terms. This page explains how Cua Driver reaches apps on both, and why the accessibility tree, not synthetic input, is the through-line. - -## Background - -X11 was built when one program reading or driving another's window was normal; window IDs are global, synthetic events can be addressed to any window, and pixels can be read from any mapped surface. Wayland was built to end exactly that: a client cannot see or inject into another client's surface, and there is no global coordinate space to address. Both models are deliberate. The driver does not fight either — it routes around the parts that are closed. - -## The semantic path is the through-line - -The one mechanism that behaves the same on X11 and Wayland is **AT-SPI** (the Linux accessibility bus). When the driver clicks a control by `element_index`, it calls that control's own accessibility action — `Action.DoAction` — which the toolkit (GTK, Qt) executes internally. No coordinates, no synthetic pointer event, no window focus, no raised window. Because the action is dispatched toolkit-side rather than as input, it crosses the Wayland client boundary that blocks synthetic input. This is why background, by-element automation is the default on Linux: it is the rung that works everywhere and the one the driver can *verify without a screenshot* (it reads the result back from the same tree). Pixel actions still get an honest verdict — the driver cross-checks the screenshot and reports `effect`/`escalation` — they just aren't confirmable from the tree alone. - -## Why coordinates need reconstructing - -Agents that work from a screenshot think in pixels, so the driver also has to answer "where is this element on screen?" — and Linux makes that harder than it sounds. - -On GTK4, the accessibility bridge reports `Component.GetExtents` in *screen* coordinates as `(0, 0)` for **every** widget (GNOME/gtk issues #1564 / #1739). Taken at face value, every element collapses onto the window's corner — the agent cursor, element frames, and any coordinate click all pile up in one place. - -The fix is to ask a different question. GTK4 *does* report each widget's position correctly relative to its **window** (`CoordType::Window`); what's missing is only the window's origin on screen. So the driver reconstructs the real coordinate as `window-relative position + window origin`, and sources the origin per session stack: - -- **X11** — the window's `_GTK_FRAME_EXTENTS` (the client-side-decoration inset) plus the X11 window origin. -- **Wayland** — a small bundled GNOME Shell helper (`org.cua.WinRects`) that reports each window's frame from inside the compositor, which is the only privileged vantage point that can see it. A normal client cannot. - -The tradeoff is honest: this is a workaround for a toolkit regression, gated on the GTK markers being present, so non-GTK toolkits (Qt reports screen coordinates correctly) keep their native path untouched. - -## How a coordinate click lands on Wayland - -On Wayland there is no global coordinate space and the compositor discards synthetic pointer events, so "click at (x, y)" cannot be delivered as input at all. Rather than give up the vision modality, the driver inverts the problem: it takes the reconstructed frames above, finds the accessible element whose frame contains the target pixel, and fires *that* element's action — back on the semantic path that already works. - -The subtlety is which element. GTK4 nests a `label` inside every `button` with a near-identical, slightly smaller frame; picking the smallest covering element lands on the inert label, whose action does nothing — a click that "succeeds" and changes nothing. So the selection is role-aware: it prefers the smallest covering *real actuator* and only falls back to a passive label when nothing else covers the point. The net effect is that "click pixel (x, y)" becomes "activate the control the agent sees there," with no pointer injection. - -The same shell helper that supplies window origins also draws the **agent cursor** on Wayland — for the same reason a client can't position a surface at an arbitrary screen coordinate, but the compositor can. - -## What still needs X11 - -The residual gap is **raw keyboard injection**. Typing into an accessible text field works (AT-SPI writes the field directly), but synthetic key events — and keys that some apps only read as raw device input — have no free path on native Wayland: the cooperative routes (`libei`, virtual-keyboard protocols) need a portal grant or a compositor that Mutter/KWin aren't. Running the app under XWayland restores the X11 `XTEST` keyboard path, which is why a GTK4 app launched with `GDK_BACKEND=x11` types fine. - -## A related quirk: finding windows - -One more place X11 assumptions leak: native Wayland apps have no X11 window ID, and neither GNOME Mutter nor KDE KWin implement the `wlr-foreign-toplevel` protocol (that is wlroots-only), so the usual window list comes back empty. The driver falls back to enumerating top-level frames from the AT-SPI registry and hands back a synthetic but stable `window_id`. Everything downstream walks the tree by process, not by that ID, so the full `list_windows → get_window_state → click` flow works unchanged. - -## Further reading - -- **Decide what to call**: [Choose an action rung and dispatch mode](/how-to-guides/driver/choose-a-modality) -- **The contract this all serves**: [Background computer-use](/explanation/the-no-foreground-contract) -- **The three axes in full**: [Capture and dispatch modalities](/explanation/capture-and-dispatch-modalities) -- **The exact limits**: [Known limits](/reference/cua-driver/limits) · [Modality test suite](/reference/cua-driver/modality-test-suite) diff --git a/docs/content/docs/explanation/meta.json b/docs/content/docs/explanation/meta.json deleted file mode 100644 index d02e298b0b..0000000000 --- a/docs/content/docs/explanation/meta.json +++ /dev/null @@ -1 +0,0 @@ -{ "title": "Explanation", "icon": "Lightbulb", "pages": ["index", "what-is-computer-use", "the-no-foreground-contract", "capture-and-dispatch-modalities", "linux-and-wayland", "process-model", "how-sandboxes-work", "demonstrations-skills-and-trajectories", "architecture"] } diff --git a/docs/content/docs/explanation/the-no-foreground-contract.mdx b/docs/content/docs/explanation/the-no-foreground-contract.mdx deleted file mode 100644 index 3c8d6fed8e..0000000000 --- a/docs/content/docs/explanation/the-no-foreground-contract.mdx +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: "Background Computer-Use" -description: "How Cua Driver operates an app in the background without taking focus, moving the cursor, or raising windows." ---- - -# Background Computer-Use - -Background computer-use means an agent operates an application without taking over the desktop. It does not change the user's frontmost app, move the real cursor, or raise the target window. - -The **no-foreground contract** is the invariant and guarantee that makes background computer-use possible. Cua Driver addresses the target application through background-capable input, accessibility, and capture paths while preserving the developer's active session. An agent can operate one application while the developer keeps coding, reading logs, and using the same machine. - -## Why It Matters - -Most GUI automation assumes the automated app owns the desktop. It activates a window, moves the pointer, and repeats. That works for unattended jobs, but fails when **a human is using the same machine**. - -If an agent raised the target window before every action, the developer's editor would lose focus every time the agent acted. If the agent moved the real pointer, the developer would lose their cursor position. If screenshots required the target window to be visible on the active desktop or Space, the workspace would jump to follow the target. - -That limits the tool to fully automated, unattended scenarios: scheduled tasks, CI-style jobs, or disposable desktops. Preserving desktop state enables concurrent work instead. The agent operates an app in the background while the developer keeps working in the foreground. - -## Platform Mechanisms - -Each OS splits accessibility, input delivery, capture, and focus policy differently. Cua Driver upholds the contract with APIs that address a process, window, element, or message queue directly. - -### macOS - -On macOS, scoped CoreGraphics event routing lets synthetic events be posted to a target process. `CGEvent.postToPid` routes input directly to a process, bypassing the window server's focus check. The normal event path is focus-based. - -The Accessibility API provides semantic action dispatch. `AXUIElementPerformAction` operates on an accessibility element, such as a button or menu item, regardless of whether the owning app is frontmost. When an app exposes useful structure, the driver can act on the element rather than replay motion. - -ScreenCaptureKit can capture a specific window without requiring it to be visible, raised, or on the active Space. Cua Driver also uses SkyLight pid-routed mouse delivery for apps that respond better to routed pointer events than accessibility actions. - -### Windows - -Windows separates windows, handles, message queues, and foreground ownership. UI Automation can inspect and operate controls by window handle and automation element even when the target window is behind another window. - -For input-like behavior, Cua Driver posts messages directly to a window's message queue with `PostMessage`, and uses synthetic pointer-device injection for cases UIPI blocks. That differs from global input injection, where the OS routes input to the focused window. Many standard Win32 controls can respond while the active window stays active. - -Only a narrow set of operations still require foreground. Some DirectInput games and raw-input canvases poll device state or reject background messages. UAC dialogs live on a protected desktop where normal automation paths are restricted. - -### Linux - -Linux depends on the session stack. AT-SPI provides accessibility-tree access and action dispatch, giving the driver a semantic path for controls that expose accessibility metadata. - -On X11, synthetic input and capture are window-addressable. `XSendEvent` can target a window ID without activating that window, and X11 window capture can read pixels from any mapped window. XWayland bridges help when an app still exposes an X11 surface. - -**Wayland** narrows the path but does not close it. Its security model intentionally blocks one client from synthesizing arbitrary *input* into another — that is the point of the design — so synthetic pointer and key events do not cross client boundaries. Actions still land, because the driver does not depend on synthetic input there. It dispatches through AT-SPI, where `Action.DoAction` invokes a control's own action toolkit-side, with no coordinates and no focus. Coordinate (vision) clicks resolve the same way: the target screen pixel is mapped to the accessible element under it, and that element's action is fired — the driver reconstructs each element's on-screen frame, since Wayland exposes no global coordinate space for a client to query. - -The residual gap on native Wayland is **raw keyboard injection**. Typing into accessible text fields works through AT-SPI, but synthetic key events — and the keys some apps only read as raw input — require an X11 / XWayland surface or a compositor-cooperative input portal, rather than coming for free. - -## Agent Cursor Overlay - -The real pointer is part of the user's working state, so Cua Driver does not move it to show agent activity. Instead, the driver can render a *synthetic cursor overlay*: a visible marker for agent activity. - -This is useful for supervision. The user can glance at the screen and see the agent clicking a button, selecting a field, or dragging inside a target window, while their own pointer stays where they left it. The overlay is an observation aid, not the input device. - -## Honest Limits - -The contract has exceptions, and they come from OS and application input models. - -Canvas and game-style applications often poll raw input state rather than accepting normal window messages or accessibility actions. DirectInput games are a common Windows example. These apps may only respond when foregrounded because their event loop is built around active device state. - -On macOS, SwiftUI windows that are off the active Space can lose their accessibility tree by OS design. The window may still exist, and pixels may still be capturable, but the semantic tree can disappear until the OS considers that UI present. - -These cases are not bugs in Cua Driver. They are boundaries set by the host platform. The contract is the default wherever the OS and target application provide background-addressable input, accessibility, and capture paths. diff --git a/docs/content/docs/how-to-guides/driver/choose-a-modality.mdx b/docs/content/docs/how-to-guides/driver/choose-a-modality.mdx deleted file mode 100644 index 3605fd857b..0000000000 --- a/docs/content/docs/how-to-guides/driver/choose-a-modality.mdx +++ /dev/null @@ -1,136 +0,0 @@ ---- -title: "Choose an Action Rung and Dispatch Mode" -description: "Pick how each action addresses its target — the accessibility (ax) rung or the pixel (px) rung — plus dispatch and scope. Start on the accessibility path, fall back to a pixel action, and escalate to foreground only when you must." ---- - -# Choose an Action Rung and Dispatch Mode - -This guide shows you how to pick how the agent acts on a target, so each action lands the first time and stays in the background when it can. - -## When to use this guide - -Use this when you are wiring an agent to Cua Driver and deciding what to pass to `click`, `type_text`, `get_window_state`, and friends — `element_index` versus `x, y`, `delivery_mode: background` versus `foreground`, and window versus desktop scope. - -## Before you start - -You should already be connected to the driver and able to launch an app and read a window. Perception is no longer a mode you pick: `get_window_state` returns both the accessibility tree and a screenshot in one call by default, and you choose the rung at action time. For the concepts behind the axes, see [Capture and dispatch modalities](/explanation/capture-and-dispatch-modalities); this guide only covers what to call. - -## Start on the accessibility path (the default) - -Default to the **element ax action with `background` dispatch — act by `element_index`.** It is the only rung the driver can verify, it never steals focus, and it works on Windows, macOS, and Linux (X11 and Wayland). - -Read the window once, then act on an element from that snapshot. The snapshot already carries the screenshot too, so you never re-capture to switch how you address the target: - -```jsonc -// 1. snapshot — returns the accessibility tree AND a screenshot by default -get_window_state({ pid, window_id }) -// → elements[], each with an element_index and a frame, plus a grounding screenshot - -// 2. act by element_index — the element ax action, background, no focus steal -click({ pid, window_id, element_index: 12 }) -type_text({ pid, text: "hello" }) -``` - -When you are only re-indexing before an element ax action and don't need fresh pixels, pass `include_screenshot: false` to skip the screen grab and get the tree alone — a cheap perf opt-out, not a modality choice. The `ax` versus `px` decision still happens at action time, by how you address the target. To pin the rendered frame to disk instead of inlining it, set `screenshot_out_file`. - -## Follow the escalation ladder - -You don't have to guess when the AX path failed — every action response carries the signals that tell you the next rung. Walk the ladder in order, and only step down when the response says to: - -1. **Element ax action, background (the default).** Act by `element_index`. If the response shows `effect: "confirmed"`, you're done — the driver read the result back. If `get_window_state` came back `degraded` (empty AX tree), an action returns `effect: "suspected_noop"` (the AX action dispatched but likely no-op'd), an action returns `effect: "unverifiable"` on an echo-prone surface, or the tree disagrees with the screenshot (an `h:1` or off-viewport row), follow `escalation.recommended`. - -2. **Element px action, background.** When `escalation.recommended` is `"px"`, pick the target pixel from the screenshot already in the `get_window_state` response and click it — no re-capture to switch rung, because the screenshot was always there. Coordinates are window-relative for a windowed target. - - ```jsonc - // the screenshot is already in the snapshot above — just read a pixel off it - click({ pid, window_id, x: 320, y: 210 }) // → { path: "cgevent", effect: "unverifiable" } - ``` - - Use the same px form for keyboard fallback. If AX `type_text`, `press_key`, or `hotkey` returns `effect: "unverifiable"` on Electron/Chromium, retry with `x, y`: the tool pixel-clicks to focus, then sends the keys. - - ```jsonc - type_text({ pid, window_id, element_index: 18, text: "hello" }) - // → { effect: "unverifiable", escalation: { recommended: "px", reason: "..." } } - - type_text({ pid, window_id, x: 320, y: 210, text: "hello" }) - press_key({ pid, window_id, x: 320, y: 210, key: "return" }) - hotkey({ pid, window_id, x: 320, y: 210, keys: ["cmd", "a"] }) - ``` - - On Linux this still avoids synthetic input where it can: the driver resolves the pixel to the element under it and fires that element's action via AT-SPI `doAction` at that point. See [Linux and Wayland](/explanation/linux-and-wayland) for why. - -3. **Browser-tab DOM.** When `escalation.recommended` is `"page"`, switch to the `page` tool for browser-tab DOM work instead of retrying the AX write. - - ```jsonc - type_text({ pid, window_id, element_index: 18, text: "hello" }) - // → { effect: "unverifiable", escalation: { recommended: "page", reason: "..." } } - - page({ - pid, - window_id, - action: "execute_javascript", - javascript: "document.querySelector('#search').value = 'hello'" - }) - ``` - -4. **Foreground.** If the response recommends `"foreground"` or the pixel click still doesn't land — DirectInput games, raw-input canvases (Blender, Unity), focus-polling apps — retry with **`delivery_mode: "foreground"`**, which activates the window first. - - ```jsonc - click({ pid, window_id, x: 320, y: 210, delivery_mode: "foreground" }) - // → { path: "cgevent_fg", effect: "unverifiable" } - ``` - - Foreground only for the action that needs it, and only when the user isn't actively working on the machine — it raises the window. See [Known limits](/reference/cua-driver/limits) for the specific apps. - -### The escalation signal on the response - -Two additive fields make the ladder explicit, so you escalate on data rather than a hunch: - -- **`effect`** — `"confirmed"` (the driver verified the result through AX read-back), `"unverifiable"` (the rung fired but only the caller can confirm), or `"suspected_noop"` (an AX action dispatched but almost certainly did nothing). -- **`escalation`** — present when there's a next rung: `{ recommended: "px" | "foreground" | "page", reason }`. A `degraded` `get_window_state` carries the same hint (recommending `px`). - -```jsonc -click({ pid, window_id, element_index: 12 }) -// → { effect: "suspected_noop", escalation: { recommended: "px", reason: "..." } } -``` - -**Wayland exception.** On a native Wayland session an unfocused window can't be pixel-targeted in the background — there is no global coordinate space and the compositor drops synthetic pointer events. So when an AX action no-ops there, the escalation skips the pixel rung and recommends **`foreground`** directly. See [Linux and Wayland](/explanation/linux-and-wayland). - -## Switch to desktop scope only for screen-absolute work - -Reach for **desktop scope** only when the action has no single window — dragging between windows, or clicking absolute screen coordinates. It inherently foregrounds and works on pixels, so it cannot honor the background contract. - -```jsonc -set_config({ capture_scope: "desktop" }) -get_desktop_state() // full-screen screenshot -click({ x: 1280, y: 40 }) // screen-absolute, no window_id -``` - -A window-less click while scope is still `window` is rejected with `desktop_scope_disabled` — that error is the prompt to switch scope. - -## Confirm the action landed - -Only AX read-back can produce `verified: true` (the driver read the result back). Echo-prone AX surfaces, pixel actions, and foreground actions return `verified: false` or omit it; use `effect` and `escalation` to decide the next call. After an unverifiable action, re-read and check (the re-read returns both the tree and the screenshot): - -```jsonc -click({ pid, window_id, x: 320, y: 210 }) // → { verified: false, path: "cgevent", effect: "unverifiable" } -get_window_state({ pid, window_id }) // confirm the change against tree + screenshot -``` - -## Troubleshooting - -**Problem: the call returned success but nothing changed (false success).** -Don't trust the status code on a `verified: false` action. Re-read the window — the snapshot carries both the tree and the screenshot — and confirm the effect; if it didn't land, switch rung (element ax action → element px action off the same screenshot) or escalate dispatch (background → foreground). - -**Problem: `desktop_scope_disabled` on a window-less click.** -You're in `window` scope. Either pass a `window_id` (+ `element_index` or `x, y`), or `set_config({ capture_scope: "desktop" })` for genuinely screen-absolute work. - -**Problem: keystrokes don't land on a Linux app.** -On native Wayland, raw keys have no background path. Type into accessible fields with `type_text`, drive controls by `element_index`, or run the app under XWayland. See [Known limits](/reference/cua-driver/limits#native-wayland-apps-cant-receive-synthetic-keystrokes). - -## Related - -- [Capture and dispatch modalities](/explanation/capture-and-dispatch-modalities) — the axes and the validity matrix -- [MCP tools](/reference/cua-driver/mcp-tools) — full parameters for every tool -- [Known limits](/reference/cua-driver/limits) — targets that need foreground -- [Linux and Wayland](/explanation/linux-and-wayland) — how the Linux rungs work diff --git a/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx b/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx index c7512286f5..d66dd8e772 100644 --- a/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx +++ b/docs/content/docs/how-to-guides/driver/connect-your-agent.mdx @@ -1,99 +1,82 @@ --- -title: Connect your agent -description: Register Cua Driver as an MCP server with Claude Code, Codex, Cursor, Antigravity, OpenClaw, OpenCode, Hermes, Pi, Qwen Code, Factory Droid, and ZCode. +title: Connect your agent to Cua Driver +description: Register Cua Driver as an MCP server so an agent can drive the host desktop. --- import { Callout } from 'fumadocs-ui/components/callout'; -`cua-driver mcp` runs as a stdio MCP server. Register it with an MCP-capable agent when you want that agent to **control the host desktop** through cua-driver. +`cua-driver mcp` runs Cua Driver as a stdio MCP server. Register it when you want an MCP-capable agent to drive the **host desktop**: installed apps, signed-in browser sessions, local files opened in apps, and the current OS user session. - - This verb wires the agent to Cua Driver, which is control of the HOST machine ONLY. It does NOT connect an agent to a cua sandbox. Sandbox access is a separate, SDK-first path: use the Computer SDK, or the separate cua-agent MCP server. Do not imply that mcp-config gives sandbox access. + + This connects an agent to Cua Driver on the current machine. To create disposable cloud desktops, use [Cua Sandbox](/tutorials/your-first-cloud-sandbox) instead. - - Grant the required permissions before you connect an agent. On macOS, grant Accessibility and Screen Recording. Verify them with `cua-driver permissions status`. - +## Before you start -The helper command prints the exact registration snippet for each supported client: +Install Cua Driver and verify it can see the host desktop: ```bash -cua-driver mcp-config --client +cua-driver --version +cua-driver call list_apps ``` - - The `claude` client can register a computer-use compatibility surface through `--claude-code-computer-use-compat`. Every client gets the same driver tools. There is no standalone screenshot tool, so window capture comes from `get_window_state`, which returns a screenshot alongside the tree by default. - - -## Claude Code - -Register the plain stdio server: +On macOS, grant Accessibility and Screen Recording before connecting an agent: ```bash -claude mcp add --transport stdio cua-driver -- cua-driver mcp +cua-driver permissions status ``` -Verify: +See [Install Cua Driver](/how-to-guides/driver/install) and [macOS permissions](/reference/cua-driver/macos-permissions) for setup details. + +## Generate the client config + +Use `mcp-config` to print the registration command or JSON for a supported client: ```bash -claude mcp list -# cua-driver: cua-driver mcp (stdio) - connected +cua-driver mcp-config --client ``` -Claude Code also supports a computer-use compatibility mode. Generate the command: +The generated reference is the source of truth for the complete roster and exact command shapes: [`cua-driver mcp-config`](/reference/cua-driver/cli-reference#cua-driver-mcp-config). + +## Claude Code + +Register the plain stdio server: ```bash -cua-driver mcp-config --client claude +claude mcp add --transport stdio cua-driver -- cua-driver mcp +claude mcp list ``` -It emits: +Claude Code can also use Cua Driver's computer-use compatibility profile. Generate the current command: ```bash -claude mcp add-json --scope user cua-computer-use '{"command":"cua-driver","args":["mcp"]}' +cua-driver mcp-config --client claude ``` -This registers the server under the key `cua-computer-use` at user scope, so it is available across your projects. Every client gets the same driver tools. There is no standalone screenshot tool, so capture the target window with `get_window_state`, which returns a screenshot alongside the tree by default. - - - This is NOT Anthropic's native computer-use API. It is Cua Driver's MCP compatibility mode for Claude Code. Start with `launch_app` or `list_windows`, then capture the target window with `get_window_state`. - +That profile exposes the same driver tools under the compatibility server name. It still runs Cua Driver over MCP; Anthropic's native computer-use API is separate. ## Codex -Print the Codex registration command: +Print the Codex command: ```bash cua-driver mcp-config --client codex ``` -It emits a command with the absolute installed binary path, which avoids PATH issues in app-launched Codex sessions: +It emits a registration using the absolute installed binary path, which avoids `PATH` issues in app-launched Codex sessions: ```bash codex mcp add cua-driver -- /Users/you/.local/bin/cua-driver mcp -``` - -Verify: - -```bash codex mcp list ``` -Restart Codex or open a fresh Codex session after adding the MCP server so the new tools appear. - -For the best experience, also install the Cua Driver skill: +Restart Codex or open a fresh session after adding the server. For richer agent guidance, install the Cua Driver skill: ```bash cua-driver skills install cua-driver skills status ``` -If the Codex agent skills directory is missing, create it and install again: - -```bash -mkdir -p ~/.agents/skills -cua-driver skills install -``` - ## Cursor Generate the Cursor snippet: @@ -116,157 +99,24 @@ Paste the JSON into `~/.cursor/mcp.json`, or `.cursor/mcp.json` for project scop } ``` -Verify: restart Cursor and confirm `cua-driver` appears in the MCP server list. - -## Antigravity - -Generate the Antigravity snippet: - -```bash -cua-driver mcp-config --client antigravity -``` - -Antigravity CLI, the `agy` binary and successor to Gemini CLI, has no `agy mcp add` subcommand. Paste the printed JSON into `~/.gemini/config/mcp_config.json`, merging it under the top-level `mcpServers` object. On Windows, use `%USERPROFILE%\.gemini\config\mcp_config.json`. Restart `agy` after saving. The same file is shared with the Antigravity IDE. `--client gemini` is a legacy alias. - -```json -{ - "mcpServers": { - "cua-driver": { - "command": "cua-driver", - "args": ["mcp"] - } - } -} -``` - -Verify: restart `agy` or the Antigravity IDE and confirm `cua-driver` is listed as an MCP server. - -## OpenClaw - -Register the stdio server: - -```bash -openclaw mcp set cua-driver '{"command":"cua-driver","args":["mcp"]}' -``` - -This registers the driver as a normal gateway-spawned MCP server; on macOS it does **not** inherit OpenClaw.app's permission grants. For that, the app process must spawn `cua-driver --embedded` directly; see [Embedding](/reference/cua-driver/embedding). - -Verify: restart OpenClaw and confirm `cua-driver` is available in the MCP server list. - -## OpenCode - -Generate the OpenCode snippet: - -```bash -cua-driver mcp-config --client opencode -``` - -Paste it under `mcp` in `opencode.json`, or in `~/.config/opencode/config.json` for global config: - -```json -{ - "$schema": "https://opencode.ai/config.json", - "mcp": { - "cua-driver": { - "type": "local", - "command": ["cua-driver", "mcp"], - "enabled": true - } - } -} -``` - - - Configure it as a real MCP server. If MCP is not configured, OpenCode runs Cua Driver as a shell subprocess, the screenshot image block is dropped, and the model gets only the AX tree with no visual context. - - -Verify: restart OpenCode and confirm `cua-driver` appears under MCP servers. - -## Hermes - -Generate the Hermes snippet: - -```bash -cua-driver mcp-config --client hermes -``` - -Paste it under `mcp_servers` in `~/.hermes/config.yaml`: - -```yaml -mcp_servers: - cua-driver: - command: "cua-driver" - args: ["mcp"] -``` - -Reload MCP servers inside Hermes: - -```text -/reload-mcp -``` - -Verify: confirm Hermes reports `cua-driver` after `/reload-mcp`. +Restart Cursor and confirm `cua-driver` appears in the MCP server list. -## Pi +## Other supported clients -Pi, `badlogic/pi-mono`, does not support MCP natively. The author has stated MCP support will not be added for context-budget reasons, so this client is an honest caveat rather than an MCP registration path. - -Run: - -```bash -cua-driver mcp-config --client pi -``` - -The output tells you to use Cua Driver as a plain CLI from inside Pi instead: - -```bash -cua-driver call list_apps -cua-driver call click '{"pid": 1234, "x": 100, "y": 200}' -cua-driver --help # full tool catalog -``` - -Each call is one-shot and returns JSON or text on stdout, which is the shape Pi is built around. - -Verify: run `cua-driver --help` from the same Pi shell context. - -## Qwen Code - -Alibaba's open-source coding CLI. It has a CLI add command and a settings file. - -- CLI: `qwen mcp add cua-driver cua-driver mcp` (also `cua-driver mcp-config --client qwen` prints this) -- Or paste under top-level `mcpServers` in `~/.qwen/settings.json` (user) or `.qwen/settings.json` (project): - -```json -{ "mcpServers": { "cua-driver": { "command": "cua-driver", "args": ["mcp"] } } } -``` - -- Verify inside Qwen Code with `/mcp`. - -## Factory Droid - -Factory.ai's Droid CLI. - -- CLI: `droid mcp add cua-driver "cua-driver mcp"` (also `cua-driver mcp-config --client droid`) -- Or paste under top-level `mcpServers` in `~/.factory/mcp.json` (user) or `.factory/mcp.json` (project): - -```json -{ "mcpServers": { "cua-driver": { "type": "stdio", "command": "cua-driver", "args": ["mcp"] } } } -``` - -## ZCode - -ZCode by Z.ai (a GLM coding GUI app). Configure in the GUI — there is no CLI add command. - -- In ZCode: Settings → MCP Servers → New MCP Server (type: stdio, command `cua-driver`, arguments `mcp`), or paste under "Full configuration": - -```json -{ "mcpServers": { "cua-driver": { "type": "stdio", "command": "cua-driver", "args": ["mcp"] } } } -``` +`mcp-config` also prints the right shape for clients that use a config file or a different add command. -- `cua-driver mcp-config --client zcode` prints this JSON to paste. -- If you use Z.ai's separate `zai` CLI instead of the ZCode app, run: `zai mcp add cua-driver --transport stdio --command "cua-driver" --args mcp` +| Client | Generate with | Notes | +|---|---|---| +| Antigravity | `cua-driver mcp-config --client antigravity` | Paste into `~/.gemini/config/mcp_config.json`; `--client gemini` is a legacy alias. | +| OpenClaw | `cua-driver mcp-config --client openclaw` | Normal gateway-spawned MCP does not inherit OpenClaw.app's macOS permission grants; embedded hosts should use [Embedding](/reference/cua-driver/embedding). | +| OpenCode | `cua-driver mcp-config --client opencode` | Configure a real MCP server so screenshots are preserved in image blocks. | +| Hermes | `cua-driver mcp-config --client hermes` | Paste under `mcp_servers` and reload MCP servers in Hermes. | +| Pi | `cua-driver mcp-config --client pi` | Pi does not support MCP natively; use one-shot `cua-driver call …` commands from its shell. | +| Qwen Code | `cua-driver mcp-config --client qwen` | Supports both a CLI add command and `~/.qwen/settings.json`. | +| Factory Droid | `cua-driver mcp-config --client droid` | Supports CLI and JSON config forms. | +| ZCode | `cua-driver mcp-config --client zcode` | Configure MCP in the GUI, or use `zai mcp add` for Z.ai's separate CLI. | -## Any other client +## Generic MCP JSON For any client that accepts the standard `mcpServers` shape, print the generic config: @@ -274,7 +124,7 @@ For any client that accepts the standard `mcpServers` shape, print the generic c cua-driver mcp-config ``` -Output: +It returns: ```json { @@ -287,10 +137,10 @@ Output: } ``` -Verify: restart the client and confirm `cua-driver` appears in its MCP server list. +After saving the config, restart the client and confirm the `cua-driver` server is connected. ## Next steps -- [Install Cua Driver / grant permissions](/how-to-guides/driver/install): install Cua Driver and approve host permissions. -- [Keep Cua Driver running](/how-to-guides/driver/keep-running): configure autostart so the daemon comes back after reboots. -- [Update Cua Driver](/how-to-guides/driver/update): check for new releases and apply them. +- [Keep Cua Driver running](/how-to-guides/driver/keep-running): keep the daemon alive across reboots and sessions. +- [Agent action policy](/reference/cua-driver/action-selection-policy): behavior agent wrappers should follow for `element_index`, `x,y`, and foreground escalation. +- [MCP tools](/reference/cua-driver/mcp-tools): inspect every exposed tool and parameter. diff --git a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx index aa26fc5a78..b77e934dce 100644 --- a/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx +++ b/docs/content/docs/how-to-guides/driver/drive-a-web-page.mdx @@ -1,35 +1,17 @@ --- title: 'Drive a Web Page' -description: 'Read and act on the page already loaded in a browser or Electron app with the page tool — get text, query the DOM, click, insert text, type keystrokes, run JavaScript.' +description: 'Read and act on the page already loaded in a browser or Electron app with the page tool: get text, query the DOM, click, insert text, type keystrokes, run JavaScript.' --- # Drive a Web Page -Use the `page` tool on the page already loaded in a running browser or Electron app. It does not navigate; open the URL first, either through `launch_app` or by having the user open it. This is the DOM/CDP path, the same rung that honest-verification `escalation.recommended: 'page'` points to when AX typing echoes but the DOM does not observe it. For the surrounding ladder, see [Choose an action rung and dispatch mode](/how-to-guides/driver/choose-a-modality) and [Capture and dispatch modalities](/explanation/capture-and-dispatch-modalities). +Use the `page` tool on the page already loaded in a running browser or Electron app. Open the URL first, either through `launch_app` or by having the user open it. This is the DOM/CDP path. Honest-verification `escalation.recommended: 'page'` points here when AX typing echoes but the DOM does not observe it. For the agent-side escalation behavior, see [Agent action policy](/reference/cua-driver/action-selection-policy) and [Capture and delivery modalities](/concepts/capture-and-delivery-modalities). -## Actions +## When to use it -| Action | What it does | -|---|---| -| `get_text` | Extract visible text from the page. Params: `pid`, `window_id`. | -| `query_dom` | Find elements by CSS selector. Params: `css_selector`, `attributes` (array of attribute names to return). | -| `click_element` | Click a CSS-selected element (animates the agent cursor). Params: `selector`. | -| `insert_text` | Set text into the focused field via CDP (fast DOM insert). Params: `text` (required), `cdp_port` (optional), `target_url_contains` (optional). | -| `type_keystrokes` | Type text as real per-character key events via CDP — fires JS keydown/keyup handlers. Use when `insert_text` does not trigger the app's input logic. Params: `text` (required), `cdp_port` (optional), `target_url_contains` (optional). | -| `execute_javascript` | Run JS and return the result. Params: `javascript`. | -| `enable_javascript_apple_events` | macOS only. One-time patch: edits the browser Preferences to allow JS from Apple Events. Requires a browser restart. Params: `bundle_id`, `user_has_confirmed_enabling` (must be true). | +Use `page` when the target is browser-tab content and the accessibility layer cannot prove the result. It is especially useful after `type_text` returns `effect: "unverifiable"` with `escalation.recommended: "page"`. -## Backend and browser support - -| Backend | Support | -|---|---| -| Chrome, Brave, Edge | Browser page support, with CDP for JavaScript and typing actions. | -| Safari | Uses AppleScript on macOS. | -| Electron | Uses CDP. | -| Chromium/Firefox on Windows | UIA for read actions; CDP for `execute_javascript` when started with `--remote-debugging-port`. | -| WKWebView, Tauri, AT-SPI | Fallback paths for page reads where available. | - -Read actions (`get_text`, `query_dom`) work broadly, and `execute_javascript` works cross-platform given a CDP endpoint. **`insert_text` and `type_keystrokes` are macOS-only for now** — Windows and Linux don't yet implement the CDP `Input.insertText`/`Input.dispatchKeyEvent` calls these actions need, and return a clear "not implemented" error rather than a silent no-op. Tracked in [trycua/cua#2084](https://github.com/trycua/cua/issues/2084). Where a CDP endpoint is available (macOS), launch Chromium with `--remote-debugging-port=` and pass `cdp_port`, or target a tab with `target_url_contains`. +Read actions (`get_text`, `query_dom`) work broadly, and `execute_javascript` works cross-platform given a CDP endpoint. `insert_text` and `type_keystrokes` are macOS-only for now; Windows and Linux return a clear "not implemented" error rather than a silent no-op. See the [`page` tool reference](/reference/cua-driver/mcp-tools#page) for the full action list, parameters, and platform notes. ## insert_text vs type_keystrokes @@ -84,5 +66,5 @@ page({ ## Related - [page tool reference](/reference/cua-driver/mcp-tools#page) -- [Choose an action rung and dispatch mode](/how-to-guides/driver/choose-a-modality) -- [Capture and dispatch modalities](/explanation/capture-and-dispatch-modalities) +- [Agent action policy](/reference/cua-driver/action-selection-policy) +- [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) diff --git a/docs/content/docs/how-to-guides/driver/meta.json b/docs/content/docs/how-to-guides/driver/meta.json index 7cf0c546a8..256d587f5b 100644 --- a/docs/content/docs/how-to-guides/driver/meta.json +++ b/docs/content/docs/how-to-guides/driver/meta.json @@ -1 +1 @@ -{ "title": "Driver", "pages": ["install", "connect-your-agent", "choose-a-modality", "drive-a-web-page", "keep-running", "personalize-cursor", "update", "windows-ssh"] } +{ "title": "Driver", "pages": ["install", "connect-your-agent", "drive-a-web-page", "keep-running", "personalize-cursor", "update", "windows-ssh"] } diff --git a/docs/content/docs/how-to-guides/driver/personalize-cursor.mdx b/docs/content/docs/how-to-guides/driver/personalize-cursor.mdx index 20ac778670..0a162bb9e3 100644 --- a/docs/content/docs/how-to-guides/driver/personalize-cursor.mdx +++ b/docs/content/docs/how-to-guides/driver/personalize-cursor.mdx @@ -1,15 +1,14 @@ --- title: Personalize the Cua cursor -description: Swap the agent cursor shape and palette — pick a built-in silhouette, override at runtime, or bring your own SVG/PNG/ICO. +description: "Swap the agent cursor shape and palette: pick a built-in silhouette, override at runtime, or bring your own SVG/PNG/ICO." --- -import { Tabs, Tab } from 'fumadocs-ui/components/tabs'; import { Callout } from 'fumadocs-ui/components/callout'; -The cua agent cursor ships with two built-in silhouettes: +The Cua Driver agent cursor ships with two built-in silhouettes: -- **`teardrop`** (the default) — an embedded `cursor-up` SVG (upward teardrop with notched bottom, gradient body, white outline) rasterised once into a 52 px buffer, sized 2× the display target for a clean 2:1 retina downscale. -- **`arrow`** — a procedural gradient diamond drawn from vector primitives each frame. Sharp at any backing scale. +- **`teardrop`** (the default) +- **`arrow`** `teardrop` is the default. Opt back into the arrow with `--cursor-shape arrow`. You can also replace the silhouette entirely with your own SVG / PNG / JPEG / ICO file via `--cursor-icon `, swap the body palette at runtime, or scale the bloom halo independently. @@ -20,7 +19,7 @@ cua-driver serve --cursor-shape teardrop # explicit; same as the default cua-driver serve --cursor-shape arrow # opt into the procedural arrow ``` -`--cursor-shape` is parsed on `serve` and `mcp`. It's a no-op on one-shot CLI calls like `cua-driver call` — those don't keep the long-lived UI runloop the overlay needs. +`--cursor-shape` is parsed on `serve` and `mcp`. It has no effect on one-shot CLI calls like `cua-driver call`, because those do not keep the long-lived UI runloop the overlay needs. `--cursor-icon ` always wins over `--cursor-shape`: if you pass both, the custom file is what renders. @@ -39,7 +38,7 @@ Every per-instance cursor accepts a runtime style override via the `set_agent_cu - `bloom_color`: hex string for the radial halo behind the cursor. Empty string reverts. - `image_path`: path to a PNG / JPEG / SVG / ICO file. When set, replaces the built-in silhouette with your asset. Empty string clears the override, reverting to the default silhouette. -Switching between the built-in `arrow` and `teardrop` silhouettes at runtime is exposed via `set_agent_cursor_motion`'s `cursor_icon` field — pass the built-in name (or a file path), exactly the vocabulary the CLI `--cursor-shape` / `--cursor-icon` flags accept. Both the CLI and MCP resolve names through the same `BuiltinShape` table, so they can never drift. +Switch between the built-in `arrow` and `teardrop` silhouettes at runtime with `set_agent_cursor_motion`'s `cursor_icon` field. Pass the built-in name or a file path, using the same vocabulary the CLI `--cursor-shape` / `--cursor-icon` flags accept. Both the CLI and MCP resolve names through the same `BuiltinShape` table, so they can never drift. ## Use your own cursor asset @@ -50,15 +49,15 @@ cua-driver serve --cursor-icon ~/my-cursor.svg ``` -Custom cursor assets are rendered with no rotation compensation — the driver assumes your asset's tip points to the **right** at rest. If your SVG has a tip pointing up, up-left, or anywhere else, the cursor will appear rotated off-axis during motion. Two options: +Custom cursor assets are rendered with no rotation compensation. The driver assumes your asset's tip points to the **right** at rest. If your SVG has a tip pointing up, up-left, or anywhere else, the cursor will appear rotated off-axis during motion. Two options: 1. **Re-author your SVG** so the tip points right at the unrotated default. The driver's rotation logic then aligns the tip with motion direction automatically. -2. **Stick with PNG / static art** if you don't care about motion-aligned rotation — the cursor will render at a fixed orientation regardless of motion direction. This is what most OS cursor packs do. +2. **Stick with PNG / static art** if you do not care about motion-aligned rotation. The cursor will render at a fixed orientation regardless of motion direction. This is what most OS cursor packs do. ## Recolour at launch -`--cursor-palette ` picks a built-in colour palette for the cursor at launch — the launch-time counterpart to the runtime `set_agent_cursor_style` palette. Like the other cursor flags it is parsed on `serve` and `mcp` only. +`--cursor-palette ` picks a built-in colour palette for the cursor at launch. It is the launch-time counterpart to the runtime `set_agent_cursor_style` palette. Like the other cursor flags it is parsed on `serve` and `mcp` only. ```bash cua-driver serve --cursor-palette @@ -66,31 +65,13 @@ cua-driver serve --cursor-palette Run `cua-driver serve --help` for the available palette names. -## How the built-ins render - -### `arrow` - -- **Path**: procedural gradient diamond. 4-vertex polygon `(14, 0) → (−8, −9) → (−3, 0) → (−8, 9)` rebuilt each frame. -- **Body**: linear gradient from the palette's `cursor_start` to `cursor_end` (runtime `gradient_colors` overrides). -- **Rotation**: tip at +x at rest; `heading + π` aligns the tip with motion direction. -- **Resolution**: drawn from vector primitives every frame, so it stays sharp at any backing scale without rasterisation artifacts. - -### `teardrop` (default) - -- **Path**: classic upward teardrop with a notched bottom (Streamline Iconoir `cursor-up`). -- **Body**: linear gradient from `#F0FBFF` at the tip to `#35C6D8` at the tail. -- **Outline**: white 1.5 px stroke, rounded caps and joins. -- **Bloom**: soft radial halo behind the cursor, tinted from the palette's bloom color. -- **Rotation**: the SVG points up at rest; a `+90°` paint-time offset aligns the tip with motion direction so the cursor faces where it's heading. -- **Resolution**: source rasterised at 2× the display target (52 px) and rendered at backing-aware physical resolution on retina — the 2:1 ratio maps 1:1 to physical pixels on 2× displays for a clean downscale. Switch to `arrow` if you want a fully procedural, rasterisation-free silhouette. - ## Things that aren't currently personalizable -- **Multiple agent cursors with distinct palettes.** Only one built-in palette today. If you want per-agent distinct cursors, file an issue with the use case. -- **Cursor display size.** Fixed at 26 logical pixels (52 physical on retina). The runtime `cursor_size` field controls the **dot-style** cursor's radius and doesn't apply to the shape-based render path. -- **Motion-path curve shape.** Glide duration, post-click dwell, idle-hide delay, and spring damping are tunable via `set_agent_cursor_motion`, but the underlying bezier shape of the path is hardcoded. +- **Multiple built-in palettes per agent.** Runtime overrides can recolour a cursor, but built-in named palettes are shared. +- **Shape render size.** `cursor_size` controls the dot-style cursor radius. The shape-based render path uses its own fixed size. +- **Motion-path curve shape.** Glide duration, post-click dwell, idle-hide delay, and spring damping are tunable via `set_agent_cursor_motion`, but the underlying path shape is fixed. ## See also -- [Connect your agent](./connect-your-agent.mdx) — register cua-driver with Claude Code, Codex, Hermes, and others. -- [`set_agent_cursor_style` MCP tool reference](../../reference/cua-driver/mcp-tools.mdx) — full parameter list and return shape. +- [Connect your agent](./connect-your-agent): register Cua Driver with Claude Code, Codex, Hermes, and others. +- [MCP tools](/reference/cua-driver/mcp-tools): full parameter list and return shape. diff --git a/docs/content/docs/how-to-guides/driver/windows-ssh.mdx b/docs/content/docs/how-to-guides/driver/windows-ssh.mdx index 9e3c2a25fe..0c3b1b2f50 100644 --- a/docs/content/docs/how-to-guides/driver/windows-ssh.mdx +++ b/docs/content/docs/how-to-guides/driver/windows-ssh.mdx @@ -23,7 +23,7 @@ cua-driver call list_windows ``` [warn] interactive session: running in Session 0 (services); window-driving tools (list_windows, click, type_text, get_window_state) - will return empty results — these APIs need an attached interactive + will return empty results. These APIs need an attached interactive desktop. ``` @@ -34,7 +34,7 @@ Run a `cua-driver serve` daemon in your **interactive session**, Session 1 or hi ``` ┌───────────────────────────────────────────────────────────────┐ -│ Session 1+ (RDP / console — has interactive desktop) │ +│ Session 1+ (RDP / console, has interactive desktop) │ │ │ │ cua-driver-serve (autostart Scheduled Task) │ │ ↑ │ @@ -43,7 +43,7 @@ Run a `cua-driver serve` daemon in your **interactive session**, Session 1 or hi └──────┼────────────────────────────────────────────────────────┘ │ ┌──────┼────────────────────────────────────────────────────────┐ -│ Session 0 (services / SSH — no desktop) │ +│ Session 0 (services / SSH, no desktop) │ │ │ │ │ cua-driver mcp │ │ cua-driver call list_windows │ @@ -106,7 +106,7 @@ Each MCP tool call starts `cua-driver mcp` on the SSH side. That process detects Check these items before opening an issue: -1. Confirm that `cua-driver --version` on the SSH side reports `0.2.7` or later. Earlier Windows builds do not proxy `mcp`. Upgrade with `irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex`. +1. Confirm that `cua-driver --version` on the SSH side reports the same current install you expect. Upgrade if needed with `irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex`. 2. Run `cua-driver status` from SSH and confirm it reports a running daemon. If it does not, use `cua-driver autostart status` to see whether the Scheduled Task is registered. 3. Run `query session` and confirm your user has a row in `Active` or `Disc` state. 4. Run `cua-driver doctor` from RDP and confirm it reports `[ok] interactive session: session N has an attached interactive desktop`. diff --git a/docs/content/docs/how-to-guides/lume/install-lume.mdx b/docs/content/docs/how-to-guides/lume/install-lume.mdx index ae0aa8756e..10d6eb08dd 100644 --- a/docs/content/docs/how-to-guides/lume/install-lume.mdx +++ b/docs/content/docs/how-to-guides/lume/install-lume.mdx @@ -1,11 +1,11 @@ --- -title: Install Lume +title: Install and run Lume description: Install Lume, Cua's local Apple Silicon VM manager for macOS and Linux guests. --- import { Callout } from 'fumadocs-ui/components/callout'; -Lume is Cua's local VM manager for Apple Silicon Macs. Use it when you want to create, run, or serve macOS and Linux VMs on your own machine. If you want disposable desktops hosted for you, start with [Cua Sandbox](/tutorials/your-first-cloud-sandbox) instead. +Lume is Cua's local VM manager for Apple Silicon Macs. Use it when you want to create, run, or serve macOS and Linux VMs on your own machine. It can provide local VM substrate for Cua workflows, but the primary product choice is still [Cua Driver](/tutorials/drive-your-first-app) for an existing machine or [Cua Sandbox](/tutorials/your-first-cloud-sandbox) for a fresh isolated desktop. ## Requirements diff --git a/docs/content/docs/how-to-guides/meta.json b/docs/content/docs/how-to-guides/meta.json index 9f12de433e..a82e2f23e8 100644 --- a/docs/content/docs/how-to-guides/meta.json +++ b/docs/content/docs/how-to-guides/meta.json @@ -4,9 +4,9 @@ "pages": [ "index", "driver", - "recipes", "sandbox", "lume", + "recipes", "skills", "agent-context" ] diff --git a/docs/content/docs/how-to-guides/recipes/automate-a-legacy-windows-app-behind-a-vpn.mdx b/docs/content/docs/how-to-guides/recipes/automate-a-legacy-windows-app-behind-a-vpn.mdx index ae21471569..2c108bf285 100644 --- a/docs/content/docs/how-to-guides/recipes/automate-a-legacy-windows-app-behind-a-vpn.mdx +++ b/docs/content/docs/how-to-guides/recipes/automate-a-legacy-windows-app-behind-a-vpn.mdx @@ -8,6 +8,10 @@ import { Steps, Step } from 'fumadocs-ui/components/steps'; When a desktop app has **no API** and only runs **behind the corporate VPN**, drive it where it already runs, on the Windows box inside the network. **Cua Driver** runs on that machine and exposes the app through MCP stdio tools, so the desktop session, app traffic, and staged files stay inside the VPN boundary. **No data leaves the VPN boundary.** + + Before you start: [install Cua Driver](/how-to-guides/driver/install), [connect your agent](/how-to-guides/driver/connect-your-agent), and configure [Keep Cua Driver running](/how-to-guides/driver/keep-running). If you connect over SSH, also read [Drive a Windows app over SSH](/how-to-guides/driver/windows-ssh). + + ### Start from the Windows machine inside the VPN @@ -23,47 +27,6 @@ When a desktop app has **no API** and only runs **behind the corporate VPN**, dr Confirm the app is installed or reachable from that desktop session before you install anything. For this recipe, assume the payroll client opens normally when you launch it by hand. - - ### Install Cua Driver - - Run the Windows installer in PowerShell on that same machine: - - ```powershell - irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex - ``` - - Open a new PowerShell window after the installer updates your user `Path`, then verify the CLI resolves: - - ```powershell - cua-driver --version - ``` - - - - ### Keep the daemon in the interactive session - - Windows OpenSSH and services run in Session 0, which has no desktop. Register Cua Driver as an interactive-session daemon so window tools can see the RDP or console desktop: - - ```powershell - cua-driver autostart enable - cua-driver autostart kick - ``` - - Verify that the daemon is running and reports a real session number: - - ```powershell - cua-driver status - # Cua Driver daemon is running - # socket: \\.\pipe\cua-driver - # pid: 12345 - # session: 2 - ``` - - - For the full daemon setup, see [Keep Cua Driver running](/how-to-guides/driver/keep-running). If you connect to this box over SSH, also read [Drive a Windows app over SSH](/how-to-guides/driver/windows-ssh). - - - ### Check that Cua Driver sees the real GUI @@ -83,26 +46,6 @@ When a desktop app has **no API** and only runs **behind the corporate VPN**, dr Empty output usually means the command is running outside the interactive desktop. Recheck `cua-driver status`, `query session`, and the autostart task. - - ### Register Cua Driver with your agent - - Add Cua Driver as an MCP stdio server. For Claude Code on Windows: - - ```powershell - claude mcp add --transport stdio cua-driver -- cua-driver.exe mcp - ``` - - Then verify the server entry: - - ```powershell - claude mcp list - ``` - - - For other MCP clients, see [Connect Cua Driver to an MCP client](/how-to-guides/driver/connect-your-agent). - - - ### Drive the legacy app @@ -120,6 +63,6 @@ When a desktop app has **no API** and only runs **behind the corporate VPN**, dr ## Scale this out -When the same workflow must run across many employees, regions, or tenants at once, move it onto **Cua Sandbox** cloud Windows desktops and fan out the jobs. Start with [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) and [Drive a sandbox with the SDK](/tutorials/run-an-agent-in-a-sandbox). +When the same workflow must run across many employees, regions, or tenants at once, move it onto **Cua Sandbox** cloud Windows desktops and fan out the jobs. Start with [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) and [Your first cloud sandbox](/tutorials/your-first-cloud-sandbox). The driving logic stays the same. The host changes from your VPN-connected Windows box to a fleet of cloud desktops with the VPN configuration baked into the image. See [Choose and build a sandbox image](/how-to-guides/sandbox/images). diff --git a/docs/content/docs/how-to-guides/recipes/build-a-report-in-a-native-app.mdx b/docs/content/docs/how-to-guides/recipes/build-a-report-in-a-native-app.mdx index b1f622ad98..c16832a178 100644 --- a/docs/content/docs/how-to-guides/recipes/build-a-report-in-a-native-app.mdx +++ b/docs/content/docs/how-to-guides/recipes/build-a-report-in-a-native-app.mdx @@ -12,6 +12,10 @@ Numbers is a **macOS-only** app with **no clean automation API** for building a This recipe is macOS-only. Numbers, Keynote, and Pages do not exist on Linux or Windows, so you need a real Mac now, or a cloud macOS desktop later. + + Before you start: [install Cua Driver](/how-to-guides/driver/install), grant [macOS permissions](/reference/cua-driver/macos-permissions), and [connect your agent](/how-to-guides/driver/connect-your-agent). For long-running jobs, configure [Keep Cua Driver running](/how-to-guides/driver/keep-running). + + ### Check Numbers.app @@ -25,49 +29,6 @@ Numbers is a **macOS-only** app with **no clean automation API** for building a Numbers ships with macOS. If you removed it, install it again from the App Store before continuing. - - ### Install Cua Driver - - Install `cua-driver` with the one-line installer: - - ```bash - /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" - ``` - - Verify the CLI is available: - - ```bash - cua-driver --version - # cua-driver 0.7.0 - ``` - - - - ### Grant macOS permissions - - Grant Accessibility and Screen Recording to the driver, then verify both grants: - - ```bash - cua-driver permissions status - ``` - - If either permission is missing, approve it in macOS System Settings, then run the check again. - - For long-running report jobs, configure the daemon so macOS attributes the permissions to the driver process. See [Keep Cua Driver running](/how-to-guides/driver/keep-running). - - - - ### Register the MCP server - - Register `cua-driver` as a stdio MCP server in your agent: - - ```bash - claude mcp add --transport stdio cua-driver -- cua-driver mcp - ``` - - Other MCP clients use the same `cua-driver mcp` command with their own config format. See [Connect Cua Driver to an MCP client](/how-to-guides/driver/connect-your-agent). - - ### Ask the agent to build the report @@ -111,6 +72,6 @@ Numbers is a **macOS-only** app with **no clean automation API** for building a One Mac builds one Numbers report at a time. To generate many reports at once, for example one per ticker, client, or reporting period, move the same workflow onto **Cua Sandbox** cloud macOS desktops and fan them out. -Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run the jobs concurrently, [Choose and build a sandbox image](/how-to-guides/sandbox/images) to bake Numbers and any helper tools into the macOS image, and [Drive a sandbox with the SDK](/tutorials/run-an-agent-in-a-sandbox) to run an agent in each desktop. +Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run the jobs concurrently, [Choose and build a sandbox image](/how-to-guides/sandbox/images) to bake Numbers and any helper tools into the macOS image, and [Your first cloud sandbox](/tutorials/your-first-cloud-sandbox) for the basic SDK lifecycle. A cloud macOS desktop is the only way to scale Numbers automation, since Numbers cannot run on Linux containers. diff --git a/docs/content/docs/how-to-guides/recipes/export-contacts-overnight.mdx b/docs/content/docs/how-to-guides/recipes/export-contacts-overnight.mdx index fb8599e367..80aa407b52 100644 --- a/docs/content/docs/how-to-guides/recipes/export-contacts-overnight.mdx +++ b/docs/content/docs/how-to-guides/recipes/export-contacts-overnight.mdx @@ -8,6 +8,10 @@ import { Steps, Step } from 'fumadocs-ui/components/steps'; After a networking event, use Cua Driver on the machine where you already have a logged-in browser. The agent inherits that **real authenticated session** from the browser profile, so there are no credentials in the script and no anti-bot fight. Let it run overnight, then wake up to a CSV. + + Before you start: [install Cua Driver](/how-to-guides/driver/install), grant host permissions, [connect your agent](/how-to-guides/driver/connect-your-agent), and configure [Keep Cua Driver running](/how-to-guides/driver/keep-running). + + @@ -24,52 +28,6 @@ On your own machine, open the browser profile that Cua Driver will drive and log -### Install Cua Driver - -Install `cua-driver` on the same machine: - -```bash -/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" -``` - -Verify that the CLI is on your PATH: - -```bash -cua-driver --version -``` - - - - - -### Grant macOS permissions - -On macOS, grant Accessibility and Screen Recording. Then verify the driver can see its permission state: - -```bash -cua-driver permissions status -``` - -For overnight runs, keep the daemon running instead of depending on a terminal session. See [Keep Cua Driver running](/how-to-guides/driver/keep-running). - - - - - -### Register it with your agent - -Add `cua-driver` as a stdio MCP server in your agent. For Claude Code: - -```bash -claude mcp add --transport stdio cua-driver -- cua-driver mcp -``` - -For other MCP clients, see [Connect Cua Driver to an MCP client](/how-to-guides/driver/connect-your-agent). - - - - - ### Give the overnight task Ask the agent for the exact extraction you want. For LinkedIn, start from the connections page and cap the run with a clear `N` while you test: @@ -109,4 +67,4 @@ Kick off the task before you leave. In the morning, open `~/contacts.csv` and sp One logged-in browser profile handles one account at a time. To process many accounts, events, or platforms in parallel overnight, move each authenticated session into its own **Cua Sandbox** cloud desktop. Log in once per sandbox, persist the session in the image, then fan the jobs out. -Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run many desktops at once, [Images](/how-to-guides/sandbox/images) to bake the logged-in session into an image, and [Run an agent in a sandbox](/tutorials/run-an-agent-in-a-sandbox) for the agent loop. +Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run many desktops at once, [Images](/how-to-guides/sandbox/images) to bake the logged-in session into an image, and [Your first cloud sandbox](/tutorials/your-first-cloud-sandbox) for the basic SDK lifecycle. diff --git a/docs/content/docs/how-to-guides/recipes/fill-a-form-from-a-local-file.mdx b/docs/content/docs/how-to-guides/recipes/fill-a-form-from-a-local-file.mdx index 0174626daa..a6c2c8f16e 100644 --- a/docs/content/docs/how-to-guides/recipes/fill-a-form-from-a-local-file.mdx +++ b/docs/content/docs/how-to-guides/recipes/fill-a-form-from-a-local-file.mdx @@ -8,6 +8,10 @@ import { Steps, Step } from 'fumadocs-ui/components/steps'; Use this recipe when the data already lives on your machine, for example in a PDF resume or a CSV file, and the form lives in a browser on the same desktop. **Cua Driver** drives both, so the **data never has to be uploaded anywhere** because the agent reads it locally and types it into the form. + + Before you start: [install Cua Driver](/how-to-guides/driver/install), grant host permissions, and [connect your agent](/how-to-guides/driver/connect-your-agent). + + ### Put the source file on the machine @@ -23,55 +27,6 @@ Use this recipe when the data already lives on your machine, for example in a PD For a CSV, open the file in a text editor, spreadsheet app, or browser tab. If the file contains many rows, start with one row and name the row or record the agent should use. - - ### Install Cua Driver - - Run the one-line installer: - - ```bash - /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" - ``` - - Verify the CLI is available: - - ```bash - cua-driver --version - # cua-driver 0.7.0 - ``` - - - - ### Grant desktop permissions - - On macOS, grant Accessibility and Screen Recording so Cua Driver can inspect windows, click controls, and type into apps. - - ```bash - cua-driver permissions grant - ``` - - Approve both prompts, then run the check again: - - ```bash - cua-driver permissions status - # Accessibility: granted. - # Screen Recording: granted. - ``` - - For a driver process that survives reboots and keeps the right macOS TCC attribution, see [Keep Cua Driver running](/how-to-guides/driver/keep-running). - - - - ### Register Cua Driver with your MCP client - - For Claude Code, add Cua Driver as a stdio MCP server: - - ```bash - claude mcp add --transport stdio cua-driver -- cua-driver mcp - ``` - - Other MCP clients use the same `cua-driver mcp` command. See [Connect Cua Driver to an MCP client](/how-to-guides/driver/connect-your-agent) for Cursor, Codex, Gemini CLI, and other clients. - - ### Give the agent the task @@ -102,4 +57,4 @@ Use this recipe when the data already lives on your machine, for example in a PD One desktop fills one form at a time. To submit many forms in parallel, for example one per CSV row or one per applicant, move the same workflow onto **Cua Sandbox** cloud desktops. Drop each source file into its own sandbox, start one agent job per sandbox, and fan the jobs out. -Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run many jobs concurrently. Use [Choose and build a sandbox image](/how-to-guides/sandbox/images) to stage the source files in an image or mount them into each sandbox. Use [Run an agent in a sandbox](/tutorials/run-an-agent-in-a-sandbox) when you want the same MCP-driven workflow to run inside a disposable cloud desktop. +Use [Run sandboxes in parallel](/how-to-guides/sandbox/scale-out) to run many jobs concurrently. Use [Choose and build a sandbox image](/how-to-guides/sandbox/images) to stage the source files in an image or mount them into each sandbox. Use [Your first cloud sandbox](/tutorials/your-first-cloud-sandbox) for the basic SDK lifecycle. diff --git a/docs/content/docs/how-to-guides/sandbox/interactive-shell.mdx b/docs/content/docs/how-to-guides/sandbox/interactive-shell.mdx index bd83b2d39f..3be4ec9c1a 100644 --- a/docs/content/docs/how-to-guides/sandbox/interactive-shell.mdx +++ b/docs/content/docs/how-to-guides/sandbox/interactive-shell.mdx @@ -54,11 +54,6 @@ Pass a `command` to run a specific program instead of the default login shell: session = await sb.terminal.create(command='python3', cols=80, rows=24) ``` -## Terminal methods - -| Method | Description | -|--------|-------------| -| create(command=None, cols=80, rows=24) | Open a PTY session; returns a dict with the session `pid`, `cols`, and `rows` | -| send_input(pid, data) | Send input text to the session | -| info(pid) | Return session info, or `None` if the session is gone | -| close(pid) | Kill the session; returns a bool | +## Reference + +See [Sandbox SDK interfaces](/reference/sandbox-sdk/interfaces) for the full terminal method signatures and return shapes. diff --git a/docs/content/docs/how-to-guides/sandbox/tunneling.mdx b/docs/content/docs/how-to-guides/sandbox/tunneling.mdx index 59ddacef82..c8c0d7e832 100644 --- a/docs/content/docs/how-to-guides/sandbox/tunneling.mdx +++ b/docs/content/docs/how-to-guides/sandbox/tunneling.mdx @@ -58,14 +58,9 @@ async with sb.tunnel.forward('chrome_devtools_remote') as t: Abstract socket names are Linux-only (Android). On Linux and macOS sandboxes use integer port numbers. -## TunnelInfo fields - -| Field | Type | Description | -|-------|------|-------------| -| host | str | Always 'localhost' | -| port | int | The host-side port assigned | -| sandbox_port | int or str | The original port/socket inside the sandbox | -| url | str | Shorthand `http://{host}:{port}` | +## Reference + +See [Sandbox SDK interfaces](/reference/sandbox-sdk/interfaces) for the full `TunnelInfo` shape. ## Example: Chrome DevTools Protocol diff --git a/docs/content/docs/how-to-guides/skills/record-a-demonstration.mdx b/docs/content/docs/how-to-guides/skills/record-a-demonstration.mdx index 0f0f1c707b..edf72dbdec 100644 --- a/docs/content/docs/how-to-guides/skills/record-a-demonstration.mdx +++ b/docs/content/docs/how-to-guides/skills/record-a-demonstration.mdx @@ -1,13 +1,13 @@ --- -title: 'Record a demonstration -> create a skill' +title: 'Record a demonstration as a skill' description: 'Demonstrate a task once over VNC and turn it into a reusable cua skill you can read, replay, and apply with new inputs.' --- import { Callout } from 'fumadocs-ui/components/callout'; -Use `cua skills record` to demonstrate a task once in a VNC session and save it as a **reusable skill**. The result is `~/.cua/skills//SKILL.md` plus the recorded trajectory assets, which you can inspect, replay as a video, and pass to `ComputerAgent` for similar tasks with new inputs. This is *record-and-replay* in the sense of show-once-get-a-reusable-skill, the cua analog of OpenAI Codex record-and-replay, not verbatim input playback. +Use `cua skills record` to demonstrate a task once in a VNC session and save it as a **reusable skill**. The result is `~/.cua/skills//SKILL.md` plus the recorded trajectory assets, which you can inspect, replay as a video, and pass to `ComputerAgent` for similar tasks with new inputs. This is *record-and-replay* in the sense of show-once-get-a-reusable-skill. It does not replay the original input verbatim. -This records a HUMAN demonstration into a reusable skill. It is not the Cua Driver agent-trajectory recorder. See [Demonstrations, skills, and trajectories](/explanation/demonstrations-skills-and-trajectories) for how the three recording systems differ. +This records a human demonstration into a reusable skill. The Cua Driver agent-trajectory recorder is separate. ## Prerequisites @@ -71,7 +71,7 @@ Recording will start automatically when you connect. 3. Stop recording. - Click `Stop Recording` in the VNC panel. If you did not pass `--name` or `--description`, enter them when prompted. Skill names can contain only letters, numbers, hyphens, and underscores. + Click `Stop Recording` in the VNC panel. If you omitted `--name` or `--description`, enter them when prompted. Skill names can contain only letters, numbers, hyphens, and underscores. 4. Inspect the skill. diff --git a/docs/content/docs/index.mdx b/docs/content/docs/index.mdx index 3aa594a6e1..499edf4a1f 100644 --- a/docs/content/docs/index.mdx +++ b/docs/content/docs/index.mdx @@ -3,7 +3,7 @@ title: 'Cua documentation' description: 'Open-source computer-use automation for real machines and disposable cloud desktops.' --- -Cua is an open-source MIT-licensed platform ([github.com/trycua/cua](https://github.com/trycua/cua)) for **computer-use automation**: letting AI agents operate real computers by clicking, typing, and reading the screen and accessibility tree. The main decision is whether to drive a machine you already have with Cua Driver, or spin up a fresh, isolated cloud desktop with Cua Sandbox. +Cua is an open-source MIT-licensed platform ([github.com/trycua/cua](https://github.com/trycua/cua)) for **computer-use automation**: letting AI agents operate real computers by clicking, typing, and reading the screen and accessibility tree. What we refer to as **Computer-Use 2.0** treats the GUI as one tool surface in an agent loop, alongside code, shell, files, and APIs. The main decision is whether to drive a machine you already have with Cua Driver, or spin up a fresh, isolated cloud desktop with Cua Sandbox. import { Card, Cards } from 'fumadocs-ui/components/card'; @@ -26,10 +26,10 @@ import { Card, Cards } from 'fumadocs-ui/components/card'; Tutorials: learn by doing. Start with [Tutorials](/tutorials). -Explanation: understand how and why Cua works. Start with [Explanation](/explanation). +Concepts: understand how and why Cua works. Start with [Concepts](/concepts). How-to guides: recipes for a goal. Start with [How-to guides](/how-to-guides). Reference: precise technical description. Start with [Reference](/reference). -Cua also has an Agent SDK, a Computer SDK, CLIs, an MCP server, and Lume for local Apple Silicon VMs. +For real local apps and signed-in state, start with **Cua Driver**. For fresh isolated desktops, start with **Cua Sandbox**. Lume is the local Apple Silicon VM substrate behind part of the stack. diff --git a/docs/content/docs/meta.json b/docs/content/docs/meta.json index e77424b694..57df55db85 100644 --- a/docs/content/docs/meta.json +++ b/docs/content/docs/meta.json @@ -1,5 +1,5 @@ { "title": "Documentation", "root": true, - "pages": ["index", "---Learn---", "tutorials", "explanation", "---Do---", "how-to-guides", "reference"] + "pages": ["index", "---Learn---", "tutorials", "concepts", "---Do---", "how-to-guides", "reference"] } diff --git a/docs/content/docs/reference/cua-driver/action-selection-policy.mdx b/docs/content/docs/reference/cua-driver/action-selection-policy.mdx new file mode 100644 index 0000000000..e048890803 --- /dev/null +++ b/docs/content/docs/reference/cua-driver/action-selection-policy.mdx @@ -0,0 +1,134 @@ +--- +title: "Agent action policy" +description: "Behavior an agent should follow when choosing element, pixel, page, and foreground actions in Cua Driver." +--- + +# Agent action policy + +This policy describes how an agent should choose Cua Driver action parameters. It is meant for agent prompts, wrappers, and evaluators. It is not a user walkthrough. + +Use it when the agent must decide what to pass to `click`, `type_text`, `get_window_state`, and related tools: `element_index` versus `x, y`, `delivery_mode: background` versus `foreground`, and window versus desktop scope. + +## Preconditions + +The agent should already be connected to the driver and able to launch an app and read a window. Perception is no longer a mode to pick: `get_window_state` returns both the accessibility tree and a screenshot in one call by default, and the agent chooses the rung at action time. For the concepts behind the axes, see [Capture and delivery modalities](/concepts/capture-and-delivery-modalities). + +## Start on the accessibility path (the default) + +The agent should default to the **element ax action with background delivery**. Act by `element_index`. It is the only rung the driver can verify, it avoids foregrounding when the target surface supports it, and it works on Windows, macOS, and Linux (X11 and Wayland). + +Read the window once, then act on an element from that snapshot. The snapshot already carries the screenshot too, so the agent does not re-capture to switch how it addresses the target: + +```jsonc +// 1. snapshot: returns the accessibility tree AND a screenshot by default +get_window_state({ pid, window_id }) +// → elements[], each with an element_index and a frame, plus a grounding screenshot + +// 2. act by element_index: the element ax action, background by default +click({ pid, window_id, element_index: 12 }) +type_text({ pid, text: "hello" }) +``` + +When the agent is only re-indexing before an element ax action and does not need fresh pixels, pass `include_screenshot: false` to skip the screen grab and get the tree alone. The `ax` versus `px` decision still happens at action time, by how the agent addresses the target. To pin the rendered frame to disk instead of inlining it, set `screenshot_out_file`. + +## Follow the escalation ladder + +Every action response carries the signals that tell the agent the next rung. Walk the ladder in order, and only step down when the response says to: + +1. **Element ax action, background (the default).** Act by `element_index`. If the response shows `effect: "confirmed"`, the driver read the result back. If `get_window_state` came back `degraded` (empty AX tree), an action returns `effect: "suspected_noop"` (the AX action ran but likely no-op'd), an action returns `effect: "unverifiable"` on an echo-prone surface, or the tree disagrees with the screenshot (an `h:1` or off-viewport row), follow `escalation.recommended`. + +2. **Element px action, background.** When `escalation.recommended` is `"px"`, pick the target pixel from the screenshot already in the `get_window_state` response and click it. Coordinates are window-relative for a windowed target. + + ```jsonc + // the screenshot is already in the snapshot above; read a pixel off it + click({ pid, window_id, x: 320, y: 210 }) // → { path: "cgevent", effect: "unverifiable" } + ``` + + Use the same px form for keyboard fallback. If AX `type_text`, `press_key`, or `hotkey` returns `effect: "unverifiable"` on Electron/Chromium, retry with `x, y`: the tool pixel-clicks to focus, then sends the keys. + + ```jsonc + type_text({ pid, window_id, element_index: 18, text: "hello" }) + // → { effect: "unverifiable", escalation: { recommended: "px", reason: "..." } } + + type_text({ pid, window_id, x: 320, y: 210, text: "hello" }) + press_key({ pid, window_id, x: 320, y: 210, key: "return" }) + hotkey({ pid, window_id, x: 320, y: 210, keys: ["cmd", "a"] }) + ``` + + On Linux this still avoids synthetic input where it can: the driver resolves the pixel to the element under it and fires that element's action via AT-SPI `doAction` at that point. See [Known limits](/reference/cua-driver/limits) for the remaining Wayland keyboard gap. + +3. **Browser-tab DOM.** When `escalation.recommended` is `"page"`, switch to the `page` tool for browser-tab DOM work instead of retrying the AX write. + + ```jsonc + type_text({ pid, window_id, element_index: 18, text: "hello" }) + // → { effect: "unverifiable", escalation: { recommended: "page", reason: "..." } } + + page({ + pid, + window_id, + action: "execute_javascript", + javascript: "document.querySelector('#search').value = 'hello'" + }) + ``` + +4. **Foreground.** If the response recommends `"foreground"` or the pixel click still does not land, retry with **`delivery_mode: "foreground"`**. This activates the window first. Common cases include DirectInput games, raw-input canvases (Blender, Unity), and focus-polling apps. + + ```jsonc + click({ pid, window_id, x: 320, y: 210, delivery_mode: "foreground" }) + // → { path: "cgevent_fg", effect: "unverifiable" } + ``` + + Use foreground only for the action that needs it, and only when the user is not actively working on the machine. It raises the window. See [Known limits](/reference/cua-driver/limits) for the specific apps. + +### The escalation signal on the response + +Two additive fields make the ladder explicit, so the agent escalates from data rather than a hunch: + +- **`effect`**: `"confirmed"` (the driver verified the result through AX read-back), `"unverifiable"` (the rung fired but only the caller can confirm), or `"suspected_noop"` (an AX action ran but almost certainly did nothing). +- **`escalation`**: present when there is a next rung: `{ recommended: "px" | "foreground" | "page", reason }`. A `degraded` `get_window_state` carries the same hint (recommending `px`). + +```jsonc +click({ pid, window_id, element_index: 12 }) +// → { effect: "suspected_noop", escalation: { recommended: "px", reason: "..." } } +``` + +**Wayland exception.** On a native Wayland session, raw keyboard input has no universal background path. When an AX keyboard action no-ops there, prefer an accessible field action or run the app under XWayland; otherwise escalate only that action to foreground. See [Known limits](/reference/cua-driver/limits#native-wayland-apps-cant-receive-synthetic-keystrokes). + +## Switch to desktop scope only for screen-absolute work + +Reach for **desktop scope** only when the action has no single window, such as dragging between windows or clicking absolute screen coordinates. It foregrounds and works on pixels, so it cannot provide best-effort background behavior. + +```jsonc +set_config({ capture_scope: "desktop" }) +get_desktop_state() // full-screen screenshot +click({ x: 1280, y: 40 }) // screen-absolute, no window_id +``` + +A window-less click while scope is still `window` is rejected with `desktop_scope_disabled`. That error is the prompt to switch scope. + +## Confirm the action landed + +Only AX read-back can produce `verified: true` (the driver read the result back). Echo-prone AX surfaces, pixel actions, and foreground actions return `verified: false` or omit it; use `effect` and `escalation` to decide the next call. After an unverifiable action, re-read and check (the re-read returns both the tree and the screenshot): + +```jsonc +click({ pid, window_id, x: 320, y: 210 }) // → { verified: false, path: "cgevent", effect: "unverifiable" } +get_window_state({ pid, window_id }) // confirm the change against tree + screenshot +``` + +## Troubleshooting + +**Problem: the call returned success but nothing changed (false success).** +Do not trust the status code on a `verified: false` action. Re-read the window. The snapshot carries both the tree and the screenshot. Confirm the effect; if it did not land, switch rung (element ax action -> element px action off the same screenshot) or escalate delivery (background -> foreground). + +**Problem: `desktop_scope_disabled` on a window-less click.** +The agent is in `window` scope. Either pass a `window_id` (+ `element_index` or `x, y`), or `set_config({ capture_scope: "desktop" })` for genuinely screen-absolute work. + +**Problem: keystrokes don't land on a Linux app.** +On native Wayland, raw keys have no background path. Type into accessible fields with `type_text`, drive controls by `element_index`, or run the app under XWayland. See [Known limits](/reference/cua-driver/limits#native-wayland-apps-cant-receive-synthetic-keystrokes). + +## Related + +- [Capture and delivery modalities](/concepts/capture-and-delivery-modalities): the axes and the validity matrix +- [MCP tools](/reference/cua-driver/mcp-tools): full parameters for every tool +- [Known limits](/reference/cua-driver/limits): targets that need foreground +- [Interface contracts](/reference/cua-driver/contracts): valid combinations and platform support diff --git a/docs/content/docs/reference/cua-driver/cli-reference.mdx b/docs/content/docs/reference/cua-driver/cli-reference.mdx index d7f3988659..42f63f0bb4 100644 --- a/docs/content/docs/reference/cua-driver/cli-reference.mdx +++ b/docs/content/docs/reference/cua-driver/cli-reference.mdx @@ -7,7 +7,7 @@ description: Command-line interface specification for Cua Driver AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY Generated by: npx tsx scripts/docs-generators/cua-driver.ts Source: cua-driver dump-docs - Version: 0.7.0 + Version: 0.7.1 */} Cross-platform computer-use automation driver. Install via the official script: @@ -16,7 +16,7 @@ Cross-platform computer-use automation driver. Install via the official script: curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash ``` -Documented against Cua Driver **0.7.0**. Run `cua-driver --version` for your installed version. +Documented against Cua Driver **0.7.1**. Run `cua-driver --version` for your installed version. The macOS-only `cua-driver permissions` command is documented separately in [macOS permissions](/reference/cua-driver/macos-permissions). diff --git a/docs/content/docs/reference/cua-driver/contracts.mdx b/docs/content/docs/reference/cua-driver/contracts.mdx index 0c717eaee7..58c5955182 100644 --- a/docs/content/docs/reference/cua-driver/contracts.mdx +++ b/docs/content/docs/reference/cua-driver/contracts.mdx @@ -1,11 +1,11 @@ --- title: Interface Contracts -description: The cross-cutting contracts behind the CLI and MCP surfaces — transport state, config persistence, capture scope, and how an action is routed. +description: "The contracts behind the CLI and MCP surfaces: transport state, config persistence, capture scope, and action routing." --- import { Callout } from 'fumadocs-ui/components/callout'; -The CLI (`cua-driver call …`) and the MCP server (`cua-driver serve` / `mcp`) run the same tool code, but they differ in **what state survives between calls**, **where configuration lands**, and **which parameters a call must carry**. This page is the contract map. For the *why* behind the process shapes, see [Process model](/explanation/process-model); for the modality axes, see [Capture and dispatch modalities](/explanation/capture-and-dispatch-modalities). +The CLI (`cua-driver call …`) and the MCP server (`cua-driver serve` / `mcp`) run the same tool code, but they differ in **what state survives between calls**, **where configuration lands**, and **which parameters a call must carry**. This page is the contract map. For the *why* behind the process shapes, see [Process model](/reference/cua-driver/process-model); for the modality axes, see [Capture and delivery modalities](/concepts/capture-and-delivery-modalities). --- @@ -20,7 +20,7 @@ The CLI (`cua-driver call …`) and the MCP server (`cua-driver serve` / `mcp`) | Agent cursor | none | shown when a `session` is declared | -When a daemon is already listening, `cua-driver call` **proxies to it**. A freshly built binary's behavior will not appear through the CLI until that daemon restarts, because the call runs in the daemon, not in the new process. Integration tests avoid this by spawning their own MCP server. +When a daemon is already listening, `cua-driver call` **proxies to it**. A freshly built binary's behavior appears through the CLI after that daemon restarts, because the call runs in the daemon process. Integration tests avoid this by spawning their own MCP server. --- @@ -32,7 +32,7 @@ When a daemon is already listening, `cua-driver call` **proxies to it**. A fresh | Caller | `_session_id` | Effect | |---|---|---| | `cua-driver config set …`, one-shot `cua-driver call` | absent (anonymous) | writes the global `DriverConfig` and persists to `~/.cua-driver/config.json` | -| MCP call with a `session` | present | in-memory override for that session only — no disk write, no clobber of the default | +| MCP call with a `session` | present | in-memory override for that session only; no disk write, no clobber of the default | Every tool then reads the **effective** value with this precedence: @@ -40,7 +40,7 @@ Every tool then reads the **effective** value with this precedence: effective = call-argument > session override > global default (disk) ``` -Keys that flow through this: `capture_scope`, `max_image_dimension`. (`capture_mode` is deprecated and ignored — it is still accepted for back-compat but has no effect; `get_window_state` always returns both the tree and a screenshot.) See the [`set_config` reference](/reference/cua-driver/mcp-tools) for the per-session isolation details. +Keys that flow through this: `capture_scope`, `max_image_dimension`. `capture_mode` is deprecated and ignored. It is still accepted for back-compat but has no effect; `get_window_state` always returns both the tree and a screenshot. See the [`set_config` reference](/reference/cua-driver/mcp-tools) for the per-session isolation details. --- @@ -65,7 +65,7 @@ Keys that flow through this: `capture_scope`, `max_image_dimension`. (`capture_m | Coordinate space | window-local (the PNG `get_window_state` returns) | true screen pixels | | Required params | `pid` (+ `window_id` for `element_index`) | only `x`, `y` | | Capture surface | `get_window_state` (tree + screenshot) | `get_desktop_state` (screenshot) | -| Dispatch | background (default) or foreground | foreground only | +| Delivery | background (default) or foreground | foreground only | | Action rung | element ax action (`element_index`) or element px action (`x,y`) | element px action (`x,y`) only | Desktop scope is the screen-absolute "Computer-Use 1.0" loop: read the whole screen, click an absolute coordinate, the way a screenshot-only model expects. Window scope is the default because it is what makes background, concurrent automation possible. @@ -74,28 +74,47 @@ Desktop scope is the screen-absolute "Computer-Use 1.0" loop: read the whole scr ## How an action is routed -Every input tool — `click`, `scroll`, and the keyboard family (`type_text`, `press_key`, `hotkey`) — picks its path from the arguments present: +Every input tool (`click`, `scroll`, and the keyboard family: `type_text`, `press_key`, `hotkey`) picks its path from the arguments present: | Arguments | Path | Behavior | |---|---|---| -| `element_index` + `window_id` | accessibility action | UIA Invoke / `AXPerformAction` / AT-SPI — background, no cursor move, no focus steal | -| `x`, `y` + `pid` | window-local pixel | coordinates are relative to that window's screenshot. For the **keyboard family** this px form pixel-clicks `(x, y)` to establish real renderer focus, then delivers the keystroke(s) to the now-focused element — the one-call path for Chromium/Electron inputs the AX layer can't focus | +| `element_index` + `window_id` | accessibility action | UIA Invoke / `AXPerformAction` / AT-SPI. Background, no cursor move, no focus steal. | +| `x`, `y` + `pid` | window-local pixel | coordinates are relative to that window's screenshot. For the **keyboard family** this px form pixel-clicks `(x, y)` to establish real renderer focus, then delivers the keystroke(s) to the now-focused element. Use it for Chromium/Electron inputs the AX layer cannot focus. | | `x`, `y`, **no** `pid`/`window_id`, scope `desktop` | screen-absolute | true screen pixels, lands on whatever is frontmost there | | `x`, `y`, **no** `pid`/`window_id`, scope `window` | rejected | structured `desktop_scope_disabled` error | -The keyboard family's `x, y` (px) form is mutually exclusive with `element_index` (ax) — pass one or the other, not both. +The keyboard family's `x, y` (px) form is mutually exclusive with `element_index` (ax). Pass one or the other. A window-less action is **never silently reinterpreted**. Under window scope it is rejected with a structured `desktop_scope_disabled` error that points the caller at `set_config capture_scope=desktop`, rather than treating screen pixels as window-local pixels. --- +## Valid combinations + +`get_window_state` always returns both the accessibility tree and a screenshot by default, so perception is not a matrix axis. The enforced combinations are `capture_scope`, action rung, and delivery mode: + +| `capture_scope` | Action rung | `delivery_mode` | Valid? | Why | +|---|---|---|---|---| +| `window` | `ax` (`element_index`) | `background` | ✅ | The default. Semantic actions on a background-capable window. | +| `window` | `ax` (`element_index`) | `foreground` | ✅ | Activate, then act by element. | +| `window` | `px` (`x`, `y`) | `background` | ✅ | Click a coordinate off the window screenshot without raising the target when the platform/app supports routed delivery. | +| `window` | `px` (`x`, `y`) | `foreground` | ✅ | Activate, then click by coordinate. | +| `desktop` | `px` (`x`, `y`) | `foreground` | ✅ | The screen-absolute loop: read the whole desktop, click absolute coordinates on the active desktop. | +| `desktop` | `ax` (`element_index`) | any | ❌ | A desktop-absolute action has no `window_id`, so there is no element tree to resolve an `element_index` against. | +| `desktop` | any | `background` | ❌ | Screen-absolute input hits whatever owns those pixels on the active desktop; there is no per-process background route. | +| `window` | window-less | any | ❌ | A window-less action while scope is `window` returns `desktop_scope_disabled`. | + +The rejected combinations are enforced. Desktop scope is foreground and pixel-only by design. + +--- + ## Platform support | Capability | Windows | macOS | Linux | |---|---|---|---| | `get_window_state` returns both tree + screenshot (element ax / px actions) | ✅ | ✅ | ✅ | -| `dispatch: background` (the [no-foreground contract](/explanation/the-no-foreground-contract)) | ✅ | ✅ | ✅ (X11/AT-SPI; native Wayland input is a gap) | -| `dispatch: foreground` + `bring_to_front` | ✅ | ✅ explicit activation (input is already background-safe; activation is for focus-proxy surfaces like RDP) | stubbed | +| `delivery_mode: "background"` (best-effort background) | ✅ | ✅ | ✅ for semantic AT-SPI actions on X11 and Wayland; native-Wayland raw keyboard input remains limited | +| `delivery_mode: "foreground"` + `bring_to_front` | ✅ explicit activation | ✅ explicit activation (input is already background-capable for most surfaces; activation is for focus-proxy surfaces like RDP) | ✅ X11 EWMH activation (`_NET_ACTIVE_WINDOW` + input focus); Wayland raise is compositor-constrained | | `get_desktop_state` (desktop capture) | ✅ | ✅ | ✅ | | Window-less desktop click (`click{x,y}`, no `pid`) | ✅ | rolling out | rolling out | diff --git a/docs/content/docs/reference/cua-driver/limits.mdx b/docs/content/docs/reference/cua-driver/limits.mdx index d1ff7d97c2..30654104a2 100644 --- a/docs/content/docs/reference/cua-driver/limits.mdx +++ b/docs/content/docs/reference/cua-driver/limits.mdx @@ -5,7 +5,7 @@ description: Documented behavioral limits of Cua Driver and available workaround import { Callout } from 'fumadocs-ui/components/callout'; -Cua Driver's *no-foreground contract* holds for every app it reaches via AX or a routed pixel click. A handful of targets and platform quirks fall outside that envelope. The macOS cases come first, followed by the Linux session-stack limits. +Cua Driver uses best-effort background delivery for every app it can reach via accessibility or routed pixel input. A handful of targets and platform quirks fall outside that envelope. The macOS cases come first, followed by the Linux session-stack limits. --- @@ -17,8 +17,8 @@ Cua Driver's *no-foreground contract* holds for every app it reaches via AX or a **Workarounds, in order of preference:** -1. Use `right_click({pid, element_index})` on AX-addressable targets (links, buttons, toolbar items). AX dispatch sidesteps the renderer filter entirely. -2. For context menus on pure web content (nothing in the AX tree), activate Chrome briefly and fall back to a HID-tap right-click. This breaks the no-foreground-steal promise for that one click. +1. Use `right_click({pid, element_index})` on AX-addressable targets (links, buttons, toolbar items). AX delivery sidesteps the renderer filter entirely. +2. For context menus on pure web content (nothing in the AX tree), activate Chrome briefly and fall back to a HID-tap right-click. This interrupts best-effort background behavior for that one click. Element-indexed right-click (`right_click` with `element_index`) works fine. The limit is @@ -33,12 +33,12 @@ Cua Driver's *no-foreground contract* holds for every app it reaches via AX or a **Symptom:** `click({pid, x, y})` on a Blender viewport silently no-ops. The window is visible and `launch_app` works, but clicks vanish. -**Cause:** These apps only accept events from `cghidEventTap` with a leading `mouseMoved`. They explicitly filter out per-pid-routed events, which is the path Cua Driver uses for backgrounded dispatch. There is no per-pid recipe that reaches them. +**Cause:** These apps only accept events from `cghidEventTap` with a leading `mouseMoved`. They explicitly filter out per-pid-routed events, which is the path Cua Driver uses for background delivery. There is no per-pid recipe that reaches them. **Workaround:** Bring the app to the foreground before clicking, then use pixel `click({pid, x, y})`. Where the target exposes AX-addressable controls, prefer `right_click` or element actions, which sidestep the renderer filter without foregrounding. - When automating Blender or a native game, the no-foreground-steal contract does not apply: these apps must be foregrounded to receive clicks, so do this only when the user is not actively working on the machine. + When automating Blender or a native game, best-effort background delivery does not apply: these apps must be foregrounded to receive clicks, so do this only when the user is not actively working on the machine. --- @@ -63,13 +63,13 @@ Cua Driver's *no-foreground contract* holds for every app it reaches via AX or a **Symptom:** `press_key({pid, element_index, key: "return"})` on a text field in a minimized window returns success, but the field doesn't commit. The macOS system-alert beep fires, or nothing happens. -**Cause:** AX reads and AX-dispatched clicks propagate through to minimized windows normally, but keyboard-commit events (Return, Space, Tab) require renderer focus, which AX focus does not confer on a minimized window. This is a macOS-wide behavior. +**Cause:** AX reads and AX clicks propagate through to minimized windows normally, but keyboard-commit events (Return, Space, Tab) require renderer focus, which AX focus does not confer on a minimized window. This is a macOS-wide behavior. **Workarounds:** 1. Use `set_value({pid, element_index, value: "..."})` to write the field's value directly. No keyboard event involved; no focus handoff required. 2. AX-click a commit-equivalent button (Go, Submit, Send, OK) rather than relying on Return. -3. Un-minimize the window (`hotkey({pid, keys: ["cmd", "m"]})` or click the Dock icon). This breaks the background contract for that window. +3. Un-minimize the window (`hotkey({pid, keys: ["cmd", "m"]})` or click the Dock icon). This interrupts best-effort background behavior for that window. `set_value` is the correct approach 90% of the time. It sidesteps both the minimized-focus issue and the general "which event commits this field" ambiguity. @@ -81,13 +81,13 @@ Cua Driver's *no-foreground contract* holds for every app it reaches via AX or a **Affected:** GTK/Qt apps running as native Wayland clients on GNOME Mutter or KDE KWin (no X11 surface). -**Symptom:** `press_key` / `hotkey` — and `type_text` into a field that isn't AT-SPI-editable — return success but the keystroke never reaches the app. Clicks and element actions on the same app work fine. +**Symptom:** `press_key` / `hotkey`, and `type_text` into a field that is not AT-SPI-editable, return success but the keystroke never reaches the app. Clicks and element actions on the same app work fine. **Cause:** Wayland blocks one client from synthesizing input into another by design. The compositor-cooperative paths that would re-enable it are not free here: `libei` requires a one-time RemoteDesktop-portal grant, and `wtype` (virtual-keyboard) is wlroots-only, so it fails on Mutter/KWin. **Workarounds:** -1. Type into accessible text fields with `type_text` — AT-SPI `insertText` writes the field directly, no synthetic key event involved. +1. Type into accessible text fields with `type_text`. AT-SPI `insertText` writes the field directly, with no synthetic key event involved. 2. Drive controls by `element_index` (`click`, `set_value`) instead of keyboard shortcuts where an equivalent control exists. 3. Run the app under XWayland (`GDK_BACKEND=x11` / `QT_QPA_PLATFORM=xcb`). It then exposes an X11 surface and the X11 `XTEST` keyboard path applies. @@ -99,11 +99,11 @@ Cua Driver's *no-foreground contract* holds for every app it reaches via AX or a ## GTK4 reports (0,0) screen coordinates over AT-SPI (handled) -**Symptom:** none in normal use — element `frame`s, the agent cursor, and vision clicks are correct on GTK4. Documented here because the underlying toolkit bug is real and visible in raw AT-SPI. +**Symptom:** none in normal use. Element `frame`s, the agent cursor, and vision clicks are correct on GTK4. Documented here because the underlying toolkit bug is real and visible in raw AT-SPI. **Cause:** GTK4's AT-SPI bridge returns `Component.GetExtents(SCREEN)` as `(0,0)` for every widget (GNOME/gtk issues #1564 / #1739). A naive consumer would collapse every element to the window's top-left corner. -**How Cua Driver handles it:** it queries `CoordType::Window` (which GTK4 *does* report correctly per-widget) and adds the window's screen origin — from `_GTK_FRAME_EXTENTS` on X11, or the `org.cua.WinRects` shell helper on Wayland — to reconstruct true screen coordinates. So no caller action is required; GTK4 frames are reliable. +**How Cua Driver handles it:** it queries `CoordType::Window` (which GTK4 *does* report correctly per-widget) and adds the window's screen origin from `_GTK_FRAME_EXTENTS` on X11 or the `org.cua.WinRects` shell helper on Wayland. That reconstructs true screen coordinates, so no caller action is required. --- diff --git a/docs/content/docs/reference/cua-driver/mcp-tool-notes.mdx b/docs/content/docs/reference/cua-driver/mcp-tool-notes.mdx index 6b536e5d8d..c317167696 100644 --- a/docs/content/docs/reference/cua-driver/mcp-tool-notes.mdx +++ b/docs/content/docs/reference/cua-driver/mcp-tool-notes.mdx @@ -1,6 +1,6 @@ --- title: MCP Tool Notes -description: Cross-cutting contracts behind the MCP tools — shared parameters, required-parameter rules, platform-specific parameters, and the action response shape. +description: "Cross-cutting MCP tool contracts: shared parameters, required-parameter rules, platform-specific parameters, and the action response shape." --- These notes are hand-maintained companions to the auto-generated [MCP Tools](/reference/cua-driver/mcp-tools) reference. They document the cross-cutting parameter contract and response shape that span multiple tools and are not derivable from any single tool's schema. @@ -11,15 +11,15 @@ Several parameters are a **shared cross-platform contract**: the same JSON shape | Parameter | Where | Notes | | --------- | ----- | ----- | -| `session` | every action and cursor tool | Optional run identity for the agent cursor and per-session state; the same id works over MCP, the CLI, or the raw socket and follows the run across apps and windows. Accepted on all three platforms — earlier Windows and Linux builds rejected it via `additionalProperties:false`. | -| `delivery_mode` | the input family (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`) | `"background"` (default) injects without fronting or raising the target — the [no-foreground contract](/explanation/the-no-foreground-contract). `"foreground"` briefly fronts the target, acts, then restores the prior frontmost — the explicit last resort when a background attempt did not land. Legacy `"auto"` is removed; omitted or unknown values fall back to `"background"` for safety. | -| `capture_mode` | `get_window_state` | **Deprecated and ignored.** Still accepted for back-compat so old callers don't error, but it has no effect — `get_window_state` always returns both the accessibility tree and a screenshot by default. There is no `ax`/`vision`/`som` capture choice; the modality (`ax` vs `px`) is chosen at action time by how you address the target. | -| `include_screenshot` | `get_window_state` | Boolean, default `true` (returns the tree **and** a screenshot). Set `false` to skip the screenshot grab and return the tree only — a **perf** opt-out for the cheap re-index-before-an-element-`ax`-action path, not a modality choice. | +| `session` | every action and cursor tool | Optional run identity for the agent cursor and per-session state; the same id works over MCP, the CLI, or the raw socket and follows the run across apps and windows. Accepted on all three platforms. Earlier Windows and Linux builds rejected it via `additionalProperties:false`. | +| `delivery_mode` | the input family (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`) | `"background"` (default) tries to inject without fronting or raising the target. See [Best-effort background](/concepts/the-no-foreground-contract). `"foreground"` briefly fronts the target, acts, then restores the prior frontmost. Use it when a background attempt did not land. Legacy `"auto"` is removed; omitted or unknown values fall back to `"background"` for safety. | +| `capture_mode` | `get_window_state` | **Deprecated and ignored.** Still accepted for back-compat so old callers do not error, but it has no effect. `get_window_state` always returns both the accessibility tree and a screenshot by default. There is no `ax`/`vision`/`som` capture choice; the modality (`ax` vs `px`) is chosen at action time by how you address the target. | +| `include_screenshot` | `get_window_state` | Boolean, default `true` (returns the tree **and** a screenshot). Set `false` to skip the screenshot grab and return the tree only when re-indexing before an element-`ax` action. | | `modifier`, `button`, `element_index`, `element_token` | pointer and element tools | Held modifier keys, mouse button, and the two element-addressing handles. | ### Required parameters -The `required` set is uniform across platforms: `click` requires nothing, `scroll` requires `direction`, and `zoom` requires `window_id` plus `x1`/`y1`/`x2`/`y2`. `pid` is **conditionally** required — needed for every per-window action, omitted for a windowless desktop-scope call — so it is validated in code with a clear error rather than pinned in the schema. A client that omits `pid` for a desktop-scope action is therefore not schema-rejected. +The `required` set is uniform across platforms: `click` requires nothing, `scroll` requires `direction`, and `zoom` requires `window_id` plus `x1`/`y1`/`x2`/`y2`. `pid` is **conditionally** required. Every per-window action needs it; a windowless desktop-scope call omits it. Code validates that rule with a clear error rather than pinning it in the schema, so a client that omits `pid` for a desktop-scope action is not schema-rejected. ### Platform-specific parameters @@ -40,7 +40,7 @@ Action tools (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_te | Field | Type | Meaning | Value set / presence | | ----- | ---- | ------- | -------------------- | | `path` | string | Delivery rung that ran. | `"ax"`, `"cgevent"`, `"cgevent_fg"`, `"key_events"`, `"key_events_fg"`, `"pixel"`, `"x11_atspi"`, `"x11_pixel"`, `"x11_pixel_fg"`, `"msaa"`. | -| `verified` | boolean or absent | AX read-back verification result. `true` means the driver read the effect back through AX; `false` means the action dispatched but is unconfirmed; absent means the tool does not carry this field. | `true`, `false`, or absent. | +| `verified` | boolean or absent | AX read-back verification result. `true` means the driver read the effect back through AX; `false` means the action ran but is unconfirmed; absent means the tool does not carry this field. | `true`, `false`, or absent. | | `effect` | string | Action confidence signal. | `"confirmed"`, `"unverifiable"`, `"suspected_noop"`. | | `escalation` | object or absent | Machine-readable next-rung recommendation. Present only when the driver recommends climbing the ladder. | `{ recommended: "px" \| "foreground" \| "page", reason: string }`, or absent. | @@ -48,7 +48,7 @@ Action tools (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_te ### `get_window_state` degraded results -On Linux (and macOS/Windows), the structured result may include `degraded: true` alongside a `degraded_reason` string when the accessibility walk completed but found no actionable elements. This distinguishes "a11y bridge not up, daemon not on the session D-Bus, or non-AX surface" from a window that genuinely has no controls — do not treat `elements: []` as authoritative when `degraded: true` is set. When `degraded: true`, act by `px` off the screenshot returned in the same response. +On Linux (and macOS/Windows), the structured result may include `degraded: true` alongside a `degraded_reason` string when the accessibility walk completed but found no actionable elements. This distinguishes "a11y bridge not up, daemon not on the session D-Bus, or non-AX surface" from a window that genuinely has no controls. Treat `elements: []` as incomplete when `degraded: true` is set, and act by `px` off the screenshot returned in the same response. ### `page` platform support diff --git a/docs/content/docs/reference/cua-driver/mcp-tools.mdx b/docs/content/docs/reference/cua-driver/mcp-tools.mdx index 2817733de0..f9ce56287f 100644 --- a/docs/content/docs/reference/cua-driver/mcp-tools.mdx +++ b/docs/content/docs/reference/cua-driver/mcp-tools.mdx @@ -7,7 +7,7 @@ description: Reference for every MCP tool Cua Driver exposes AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY Generated by: npx tsx scripts/docs-generators/cua-driver.ts Source: cua-driver dump-docs - Version: 0.7.0 + Version: 0.7.1 */} import { Callout } from 'fumadocs-ui/components/callout'; @@ -23,7 +23,7 @@ For the cross-cutting parameter contract (shared parameters, required-parameter - **TCC auto-delegation.** When an MCP client spawns `cua-driver mcp` from an IDE terminal (Claude Code, Cursor, VS Code, Warp), macOS attributes the subprocess to the parent terminal — not `CuaDriver.app` — so AX probes fail against the wrong bundle id. `mcp` detects this and auto-launches a `cua-driver serve` daemon via `open -n -g -a CuaDriver --args serve`, then proxies every tool call through the daemon's Unix socket. Tool semantics are identical to the in-process path; no Python bridge is needed. Pass `--no-daemon-relaunch` (or set `CUA_DRIVER_MCP_NO_RELAUNCH=1`) to force in-process execution. See the [process model](/explanation/process-model) for the full lifecycle, failure modes, and wrapper-author guidance. + **TCC auto-delegation.** When an MCP client spawns `cua-driver mcp` from an IDE terminal (Claude Code, Cursor, VS Code, Warp), macOS attributes the subprocess to the parent terminal — not `CuaDriver.app` — so AX probes fail against the wrong bundle id. `mcp` detects this and auto-launches a `cua-driver serve` daemon via `open -n -g -a CuaDriver --args serve`, then proxies every tool call through the daemon's Unix socket. Tool semantics are identical to the in-process path; no Python bridge is needed. Pass `--no-daemon-relaunch` (or set `CUA_DRIVER_MCP_NO_RELAUNCH=1`) to force in-process execution. See the [process model](/reference/cua-driver/process-model) for the full lifecycle, failure modes, and wrapper-author guidance. ## Inspection tools @@ -67,14 +67,14 @@ Always returns BOTH the element tree AND a screenshot — ground on both and cro Optional `query` filters the tree_markdown to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged — filtering only trims the rendered Markdown. -Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees (#22865). When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25). +Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25). **Arguments:** - `capture_mode` (string, optional): DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_index/element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. - `include_screenshot` (boolean, optional): Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk. -- `max_depth` (integer, optional): Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees (#22865). -- `max_elements` (integer, optional): Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows (#22865). +- `max_depth` (integer, optional): Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees. +- `max_elements` (integer, optional): Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows. - `pid` (integer, required): Target process ID. - `query` (string, optional): Case-insensitive filter for tree_markdown. - `screenshot_out_file` (string, optional): When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output will contain screenshot_file_path instead. @@ -678,7 +678,7 @@ Returns: `accessibility` + `screen_recording` (booleans from the TCC preflight A ### `health_report` -Single-call end-to-end driver diagnostics. Designed to let downstream consumers (Hermes Agent and similar) ship one stable call instead of stitching together check_permissions, doctor, version, bundle attribution, and a screenshot probe. cua-driver owns the health model; consumers stay thin. +Single-call end-to-end driver diagnostics. Designed to let downstream consumers ship one stable call instead of stitching together check_permissions, doctor, version, bundle attribution, and a screenshot probe. cua-driver owns the health model; consumers stay thin. Input — all optional: { @@ -719,7 +719,7 @@ Output — stable contract, schema_version="1": - `degraded` — at least one non-core check fails (binary is still usable) - `failed` — any core check fails (binary_version, platform_supported, session_active) -Stability: schema_version="1" is the contract. Future breaking changes will be `"2"`. Adding new check names under the same schema_version is non-breaking; consumers must tolerate unknown check names. Downstream consumer: NousResearch/hermes-agent#47065. +Stability: schema_version="1" is the contract. Future breaking changes will be `"2"`. Adding new check names under the same schema_version is non-breaking; consumers must tolerate unknown check names. **Arguments:** diff --git a/docs/content/docs/reference/cua-driver/meta.json b/docs/content/docs/reference/cua-driver/meta.json index c0804e3322..d37f264b67 100644 --- a/docs/content/docs/reference/cua-driver/meta.json +++ b/docs/content/docs/reference/cua-driver/meta.json @@ -1 +1 @@ -{ "title": "Cua Driver", "pages": ["cli-reference", "macos-permissions", "embedding", "mcp-tools", "mcp-tool-notes", "contracts", "limits", "modality-test-suite"] } +{ "title": "Cua Driver", "pages": ["cli-reference", "macos-permissions", "embedding", "mcp-tools", "mcp-tool-notes", "action-selection-policy", "contracts", "process-model", "limits"] } diff --git a/docs/content/docs/explanation/process-model.mdx b/docs/content/docs/reference/cua-driver/process-model.mdx similarity index 100% rename from docs/content/docs/explanation/process-model.mdx rename to docs/content/docs/reference/cua-driver/process-model.mdx diff --git a/docs/content/docs/reference/docs-code-mcp/index.mdx b/docs/content/docs/reference/docs-code-mcp/index.mdx index cf99b48c30..304148e06c 100644 --- a/docs/content/docs/reference/docs-code-mcp/index.mdx +++ b/docs/content/docs/reference/docs-code-mcp/index.mdx @@ -3,7 +3,7 @@ title: Docs & Code MCP Server description: "Reference for the hosted Cua Docs & Code MCP Server: endpoint, the four read-only query tools, database tables, and version indexing." --- -The Docs & Code MCP Server is a hosted MCP server exposing **read-only** search over Cua's documentation and versioned source code. The endpoint is `https://vk-mcp.cua.ai/mcp` over the streamable-HTTP transport. The Docs & Code MCP Server is backed by SQLite FTS5 for keyword search and LanceDB embeddings using `all-MiniLM-L6-v2`, 384-dimensional vectors, for semantic search. It is re-crawled daily via Modal. All tools are read-only. SQL tools accept SELECT queries only. +The Docs & Code MCP Server is a hosted MCP server exposing **read-only** search over Cua's documentation and versioned source code. The endpoint is `https://vk-mcp.cua.ai/mcp` over the streamable-HTTP transport. The Docs & Code MCP Server is backed by SQLite FTS5 for keyword search and LanceDB embeddings using `all-MiniLM-L6-v2`, 384-dimensional vectors, for semantic search. It is re-crawled daily. All tools are read-only. SQL tools accept SELECT queries only. ## Tools diff --git a/docs/content/docs/reference/index.mdx b/docs/content/docs/reference/index.mdx index 67807c8149..310e8fcf36 100644 --- a/docs/content/docs/reference/index.mdx +++ b/docs/content/docs/reference/index.mdx @@ -3,10 +3,11 @@ title: 'Reference' description: 'Index of reference material for Cua Driver, Lume, and the Sandbox SDK.' --- -Reference documents the full technical specification for Cua's subsystems. It covers `cua-driver`, Lume, and `sandbox-sdk`. The pages define commands, APIs, exposed tools, types, and stated limits. +Reference documents the full technical specification for Cua's subsystems. It covers `cua-driver`, Lume, the Sandbox SDK, and the Docs & Code MCP server. The pages define commands, APIs, exposed tools, types, and stated limits. | Section | What it documents | |---------|-------------------| -| [`cua-driver`](/reference/cua-driver/cli-reference) | CLI commands, 35 MCP tools exposed via the stdio MCP server, and known behavioral limits, including platform-specific constraints on the no-foreground-steal contract. | +| [`cua-driver`](/reference/cua-driver/cli-reference) | CLI commands, the generated MCP tool reference for the stdio MCP server, and known behavioral limits for best-effort background automation. | | [Lume](/reference/lume/cli-reference) | CLI commands and local HTTP API for creating, running, and managing macOS and Linux VMs on Apple Silicon Macs. | -| [`sandbox-sdk`](/reference/sandbox-sdk) | Sandbox SDK API, documented by a separate agent. | +| [Sandbox SDK](/reference/sandbox-sdk) | Python API reference for building images, creating sandboxes, and driving their interfaces. | +| [Docs & Code MCP](/reference/docs-code-mcp) | Hosted read-only search over Cua docs and versioned source code. | diff --git a/docs/content/docs/reference/lume/cli-reference.mdx b/docs/content/docs/reference/lume/cli-reference.mdx index fdaad4c50b..044ea402a2 100644 --- a/docs/content/docs/reference/lume/cli-reference.mdx +++ b/docs/content/docs/reference/lume/cli-reference.mdx @@ -7,31 +7,15 @@ description: Command Line Interface reference for Lume AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY Generated by: npx tsx scripts/docs-generators/lume.ts Source: lume dump-docs --type cli - Version: 0.3.10 + Version: 0.3.11 */} A lightweight CLI and local API server to build, run and manage macOS VMs. -Documented against Lume **0.3.10**. Run `lume --version` for your installed version. +Documented against Lume **0.3.11**. Run `lume --version` for your installed version. For installation steps, see [Install Lume](/how-to-guides/lume/install-lume). -## Update Lume - -```bash -# Check whether a newer release is available -lume check-update - -# Apply an available update -lume update --apply -``` - -`lume check-update --json` returns the current version, latest version, update availability, -release notes URL, and install command. `lume update` only applies an update when `--apply` is -provided. - -The MCP server also exposes a read-only `check_for_update` tool with the same update-state payload. - ## VM Management ### lume create @@ -360,6 +344,28 @@ View lume serve logs - `lume logs all` - View both info and error logs - `-n, --lines` - Number of lines to display +### lume check-update + +Check whether a newer Lume release is available + +**Flags:** + +| Name | Default | Description | +| ---- | ------- | ----------- | +| `--json` | false | Emit the structured update-state payload as JSON | +| `--no-cache` | false | Bypass the local update-check cache | + +### lume update + +Check for a Lume update and optionally apply it + +**Flags:** + +| Name | Default | Description | +| ---- | ------- | ----------- | +| `--apply` | false | Apply the update by re-running the official installer | +| `--json` | false | Emit the structured update-state payload as JSON | + ## Global Options These options are available for all commands: diff --git a/docs/content/docs/reference/lume/http-api.mdx b/docs/content/docs/reference/lume/http-api.mdx index bec78d3ea6..a886e00267 100644 --- a/docs/content/docs/reference/lume/http-api.mdx +++ b/docs/content/docs/reference/lume/http-api.mdx @@ -7,14 +7,14 @@ description: HTTP API reference for Lume server AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY Generated by: npx tsx scripts/docs-generators/lume.ts Source: lume dump-docs --type api - Version: 0.3.10 + Version: 0.3.11 */} import { Tabs, Tab } from 'fumadocs-ui/components/tabs'; HTTP API for managing macOS and Linux virtual machines -Documented against Lume **0.3.10**. Run `lume --version` for your installed version. +Documented against Lume **0.3.11**. Run `lume --version` for your installed version. ## Default URL diff --git a/docs/content/docs/tutorials/drive-your-first-app.mdx b/docs/content/docs/tutorials/drive-your-first-app.mdx index 40e6b3e847..881720a5c6 100644 --- a/docs/content/docs/tutorials/drive-your-first-app.mdx +++ b/docs/content/docs/tutorials/drive-your-first-app.mdx @@ -22,7 +22,7 @@ Use the same one-line installer on every platform; it picks the right path for t /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" ``` - Start the daemon through the app bundle so macOS attributes permission prompts to CuaDriver.app — this is what makes the TCC grant stick to the driver: + Start the daemon through the app bundle so macOS attributes permission prompts to CuaDriver.app. This is what makes the TCC grant stick to the driver: ```bash open -n -g -a CuaDriver --args serve @@ -174,11 +174,11 @@ You can now prompt your agent in plain English, and it will drive Cua Driver for Everything runs in the background, and the agent picks the right calculator for your OS (Calculator on macOS and Windows, the system calculator on Linux), so the same prompt works on all three platforms. -To explore capture and dispatch modes, see [Choose an action rung and dispatch mode](/how-to-guides/driver/choose-a-modality). +To understand capture and delivery modes, see [Capture and delivery modalities](/concepts/capture-and-delivery-modalities). ## 5. Confirm what happened -The agent reports 42. Cua Driver kept the calculator in the background the whole time. It never stole your keyboard focus and never moved your cursor. That is the *no-foreground contract*: the agent drives the app while you keep working. See [The no-foreground contract](/explanation/the-no-foreground-contract). +The agent reports 42. Cua Driver kept the calculator in the background the whole time. It never stole your keyboard focus and never moved your cursor. That is best-effort background: the agent drives the app while you keep working. See [Best-effort background](/concepts/the-no-foreground-contract). ## What you did @@ -187,5 +187,5 @@ You installed Cua Driver, verified it could see your desktop, connected your age ## Next steps - [Connect your agent](/how-to-guides/driver/connect-your-agent): register Cua Driver with Cursor, Antigravity, OpenCode, OpenClaw, Pi, and more. -- [The no-foreground contract](/explanation/the-no-foreground-contract): why the agent never steals focus or moves your cursor. +- [Best-effort background](/concepts/the-no-foreground-contract): how Cua Driver avoids focus and cursor disruption when the target app supports it. - [How-to guides](/how-to-guides): keep the driver running, update it, and more. diff --git a/docs/content/docs/tutorials/index.mdx b/docs/content/docs/tutorials/index.mdx index bb9e396cd9..fb25b0ca2f 100644 --- a/docs/content/docs/tutorials/index.mdx +++ b/docs/content/docs/tutorials/index.mdx @@ -7,6 +7,5 @@ Tutorials teach Cua by building something end to end. Start here if you are new - [Drive your first app](/tutorials/drive-your-first-app) - operate a native app in the background with Cua Driver. - [Your first cloud sandbox](/tutorials/your-first-cloud-sandbox) - spin up a disposable cloud computer. -- [Run code and drive the GUI in a sandbox](/tutorials/run-an-agent-in-a-sandbox) - run code and drive the GUI in a cloud sandbox via the SDK. Once you are comfortable, [How-to guides](/how-to-guides) cover specific goals and [Reference](/reference) has the exact APIs. diff --git a/docs/content/docs/tutorials/meta.json b/docs/content/docs/tutorials/meta.json index 5d4fa1c0ea..9175cf80f5 100644 --- a/docs/content/docs/tutorials/meta.json +++ b/docs/content/docs/tutorials/meta.json @@ -1 +1 @@ -{ "title": "Tutorials", "icon": "GraduationCap", "pages": ["index", "drive-your-first-app", "your-first-cloud-sandbox", "run-an-agent-in-a-sandbox"] } +{ "title": "Tutorials", "icon": "GraduationCap", "pages": ["index", "drive-your-first-app", "your-first-cloud-sandbox"] } diff --git a/docs/content/docs/tutorials/run-an-agent-in-a-sandbox.mdx b/docs/content/docs/tutorials/run-an-agent-in-a-sandbox.mdx deleted file mode 100644 index a6f0bf0dcc..0000000000 --- a/docs/content/docs/tutorials/run-an-agent-in-a-sandbox.mdx +++ /dev/null @@ -1,99 +0,0 @@ ---- -title: Run code and drive the GUI in a sandbox -description: Spin up a cloud Linux sandbox, run code and a GUI action against it with the Python SDK, then let it tear down. ---- - -import { Callout } from 'fumadocs-ui/components/callout'; - -# Run code and drive the GUI in a sandbox - -In this tutorial, you create an *ephemeral* cloud Linux sandbox, run Python code inside it, set and read the sandbox clipboard, and save a screenshot from the sandbox to your current directory. - - - Prerequisites: Python 3.12 or 3.13. A Cua API key from cua.ai under **Dashboard > API Keys > New API Key**. No Anthropic key is needed. - - -## Sign in and create an API key - -Sign in at cua.ai. - -Open **Dashboard > API Keys > New API Key**. - -Copy the API key immediately. You will use it in your terminal in the next step. - -## Install the SDK - -Open a terminal and install the Python SDK: - -```bash -pip install cua -``` - -## Set your API key - -Set `CUA_API_KEY` in the same terminal: - -```bash -export CUA_API_KEY=sk_cua-... -``` - -Replace `sk_cua-...` with the API key you copied from the dashboard. - -## Create the script - -Create a file named `drive_sandbox.py`: - -```python -import asyncio -from cua import Sandbox, Image - -async def main(): - async with Sandbox.ephemeral(Image.linux()) as sb: - result = await sb.shell.run("python3 -c 'print(1 + 1)'") - print(result.stdout) - - await sb.clipboard.set("Hello from the sandbox clipboard") - value = await sb.clipboard.get() - print(value) - - screenshot = await sb.screenshot() - with open("screenshot.png", "wb") as f: - f.write(screenshot) - print("Screenshot saved to screenshot.png") - -asyncio.run(main()) -``` - -## Run it - -Run the script from the same directory: - -```bash -python drive_sandbox.py -``` - -You should see output like this: - -```text -2 - -Hello from the sandbox clipboard -Screenshot saved to screenshot.png -``` - -You should also see a new file named `screenshot.png` in the current directory. - -## What just happened - -`Sandbox.ephemeral(Image.linux())` created a cloud Linux sandbox. `sb.shell.run("python3 -c 'print(1 + 1)'")` executed code inside the cloud container and returned a `CommandResult` with `stdout`, `stderr`, `returncode`, and the `success` boolean property. The script printed `result.stdout`. - -`sb.clipboard.set(...)` changed the sandbox desktop clipboard state. `sb.clipboard.get()` read that GUI state back as a string. `sb.screenshot()` captured the sandbox display as PNG bytes and wrote them to `screenshot.png`. - -When the `async with` block exited, **the sandbox destroyed itself**. No agent was involved. - -## Next steps - -- [Manage sandbox lifecycle](/how-to-guides/sandbox/lifecycle) -- [Run an interactive shell](/how-to-guides/sandbox/interactive-shell) -- [Forward a sandbox port](/how-to-guides/sandbox/tunneling) -- [Sandbox SDK API reference](/reference/sandbox-sdk) diff --git a/docs/content/docs/tutorials/your-first-cloud-sandbox.mdx b/docs/content/docs/tutorials/your-first-cloud-sandbox.mdx index 2af9213247..62992b1d35 100644 --- a/docs/content/docs/tutorials/your-first-cloud-sandbox.mdx +++ b/docs/content/docs/tutorials/your-first-cloud-sandbox.mdx @@ -7,7 +7,7 @@ import { Callout } from 'fumadocs-ui/components/callout'; # Your first cloud sandbox -In this tutorial, you create an *ephemeral* cloud Linux sandbox, run `uname -a` inside it, and save a screenshot from the sandbox to your current directory. +In this tutorial, you create an *ephemeral* cloud Linux sandbox, run `uname -a` inside it, touch the sandbox clipboard, and save a screenshot from the sandbox to your current directory. Prerequisites: Python 3.12 or 3.13. A free account at cua.ai for cloud sandboxes. @@ -52,6 +52,10 @@ async def main(): result = await sb.shell.run("uname -a") print(result.stdout) + await sb.clipboard.set("Hello from the sandbox clipboard") + value = await sb.clipboard.get() + print(value) + screenshot = await sb.screenshot() with open("screenshot.png", "wb") as f: f.write(screenshot) @@ -68,13 +72,13 @@ Run the script from the same directory: python first_sandbox.py ``` -You should see a Linux kernel string printed in your terminal. +You should see a Linux kernel string and the clipboard value printed in your terminal. You should also see a new file named `screenshot.png` in the current directory. ## What just happened -`Sandbox.ephemeral(Image.linux())` created a cloud Linux container. The script ran `uname -a` inside that container, printed the command output, took a screenshot, and wrote it to `screenshot.png`. +`Sandbox.ephemeral(Image.linux())` created a cloud Linux container. The script ran `uname -a` inside that container, printed the command output, changed and read the sandbox desktop clipboard, took a screenshot, and wrote it to `screenshot.png`. When the `async with` block exited, **the sandbox destroyed itself**. No cleanup is needed. diff --git a/docs/next.config.mjs b/docs/next.config.mjs index 85708b8d45..3aa8a0ee67 100644 --- a/docs/next.config.mjs +++ b/docs/next.config.mjs @@ -1,10 +1,16 @@ import { createMDX } from 'fumadocs-mdx/next'; +import { dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; const withMDX = createMDX(); +const docsRoot = dirname(fileURLToPath(import.meta.url)); /** @type {import('next').NextConfig} */ const config = { reactStrictMode: true, + turbopack: { + root: docsRoot, + }, trailingSlash: false, basePath: '/docs', assetPrefix: '/docs', @@ -60,6 +66,136 @@ const config = { destination: '/cua/reference/agent-sdk', permanent: true, }, + { + source: '/reference/cua-driver/modality-test-suite', + destination: '/reference/cua-driver/limits', + permanent: true, + }, + { + source: '/reference/cua-driver/modality-test-suite.mdx', + destination: '/reference/cua-driver/limits', + permanent: true, + }, + { + source: '/explanation/linux-and-wayland', + destination: '/reference/cua-driver/limits', + permanent: true, + }, + { + source: '/explanation/linux-and-wayland.mdx', + destination: '/reference/cua-driver/limits', + permanent: true, + }, + { + source: '/explanation/capture-and-dispatch-modalities', + destination: '/concepts/capture-and-delivery-modalities', + permanent: true, + }, + { + source: '/explanation/capture-and-dispatch-modalities.mdx', + destination: '/concepts/capture-and-delivery-modalities', + permanent: true, + }, + { + source: '/explanation/demonstrations-skills-and-trajectories', + destination: '/concepts', + permanent: true, + }, + { + source: '/explanation/demonstrations-skills-and-trajectories.mdx', + destination: '/concepts', + permanent: true, + }, + { + source: '/explanation/process-model', + destination: '/reference/cua-driver/process-model', + permanent: true, + }, + { + source: '/explanation/process-model.mdx', + destination: '/reference/cua-driver/process-model', + permanent: true, + }, + { + source: '/concepts/process-model', + destination: '/reference/cua-driver/process-model', + permanent: true, + }, + { + source: '/concepts/process-model.mdx', + destination: '/reference/cua-driver/process-model', + permanent: true, + }, + { + source: '/explanation/architecture', + destination: '/concepts', + permanent: true, + }, + { + source: '/explanation/architecture.mdx', + destination: '/concepts', + permanent: true, + }, + { + source: '/concepts/architecture', + destination: '/concepts', + permanent: true, + }, + { + source: '/concepts/architecture.mdx', + destination: '/concepts', + permanent: true, + }, + { + source: '/concepts/how-cua-fits-together', + destination: '/concepts', + permanent: true, + }, + { + source: '/concepts/how-cua-fits-together.mdx', + destination: '/concepts', + permanent: true, + }, + { + source: '/how-to-guides/driver/choose-a-modality', + destination: '/reference/cua-driver/action-selection-policy', + permanent: true, + }, + { + source: '/how-to-guides/driver/choose-a-modality.mdx', + destination: '/reference/cua-driver/action-selection-policy', + permanent: true, + }, + { + source: '/explanation', + destination: '/concepts', + permanent: true, + }, + { + source: '/explanation.mdx', + destination: '/concepts', + permanent: true, + }, + { + source: '/explanation/:path*.mdx', + destination: '/concepts/:path*', + permanent: true, + }, + { + source: '/explanation/:path*', + destination: '/concepts/:path*', + permanent: true, + }, + { + source: '/tutorials/run-an-agent-in-a-sandbox', + destination: '/tutorials/your-first-cloud-sandbox', + permanent: true, + }, + { + source: '/tutorials/run-an-agent-in-a-sandbox.mdx', + destination: '/tutorials/your-first-cloud-sandbox', + permanent: true, + }, ]; }, images: { diff --git a/docs/package.json b/docs/package.json index 1ea9233186..52fa59a170 100644 --- a/docs/package.json +++ b/docs/package.json @@ -16,7 +16,8 @@ "docs:generate:changelog": "tsx ../scripts/docs-generators/generate-changelog.ts", "docs:generate:versions": "tsx ../scripts/docs-generators/generate-versioned-docs.ts", "docs:check-links": "tsx scripts/check-links.ts", - "docs:check-links:external": "tsx scripts/check-links.ts --external" + "docs:check-links:external": "tsx scripts/check-links.ts --external", + "docs:check-hygiene": "tsx scripts/check-hygiene.ts" }, "dependencies": { "fumadocs-core": "16.0.8", @@ -43,39 +44,5 @@ "tailwindcss": "^4.1.8", "tsx": "^4.7.0", "typescript": "^5.8.3" - }, - "pnpm": { - "onlyBuiltDependencies": [ - "@tailwindcss/oxide", - "esbuild", - "sharp" - ], - "overrides": { - "@ai-sdk/google-vertex": "4.0.148", - "@hono/node-server": "1.19.13", - "@modelcontextprotocol/sdk": "1.26.0", - "ajv": "8.18.0", - "diff": "5.2.2", - "dompurify": "3.4.11", - "fast-uri": "3.1.2", - "fast-xml-builder": "1.1.7", - "fast-xml-parser": "5.7.0", - "form-data": "4.0.6", - "hono": "4.12.25", - "js-yaml": "4.2.0", - "js-yaml@>=4.0.0 <4.1.1": ">=4.1.1", - "langsmith": "0.7.10", - "lodash-es": "4.18.0", - "mdast-util-to-hast": "13.2.1", - "path-to-regexp": "8.4.0", - "picomatch": "4.0.4", - "postcss": "8.5.10", - "preact": "10.27.3", - "prismjs": "1.30.0", - "qs": "6.15.2", - "tar": "7.5.16", - "ts-deepmerge": "8.0.0", - "uuid": "11.1.1" - } } } diff --git a/docs/pnpm-workspace.yaml b/docs/pnpm-workspace.yaml new file mode 100644 index 0000000000..c478b84cff --- /dev/null +++ b/docs/pnpm-workspace.yaml @@ -0,0 +1,34 @@ +packages: + - . + +onlyBuiltDependencies: + - '@tailwindcss/oxide' + - esbuild + - sharp + +overrides: + '@ai-sdk/google-vertex': 4.0.148 + '@hono/node-server': 1.19.13 + '@modelcontextprotocol/sdk': 1.26.0 + ajv: 8.18.0 + diff: 5.2.2 + dompurify: 3.4.11 + fast-uri: 3.1.2 + fast-xml-builder: 1.1.7 + fast-xml-parser: 5.7.0 + form-data: 4.0.6 + hono: 4.12.25 + js-yaml: 4.2.0 + 'js-yaml@>=4.0.0 <4.1.1': '>=4.1.1' + langsmith: 0.7.10 + lodash-es: 4.18.0 + mdast-util-to-hast: 13.2.1 + path-to-regexp: 8.4.0 + picomatch: 4.0.4 + postcss: 8.5.10 + preact: 10.27.3 + prismjs: 1.30.0 + qs: 6.15.2 + tar: 7.5.16 + ts-deepmerge: 8.0.0 + uuid: 11.1.1 diff --git a/docs/scripts/check-hygiene.ts b/docs/scripts/check-hygiene.ts new file mode 100644 index 0000000000..0ce25ddf50 --- /dev/null +++ b/docs/scripts/check-hygiene.ts @@ -0,0 +1,52 @@ +#!/usr/bin/env npx tsx + +import * as fs from 'node:fs/promises'; +import * as path from 'node:path'; +import fg from 'fast-glob'; + +const DOCS_DIR = path.resolve(__dirname, '..'); +const CONTENT_DIR = path.join(DOCS_DIR, 'content/docs'); + +const bannedPatterns: Array<[RegExp, string]> = [ + [/\bSTOLE FOCUS\b/, 'internal modality recorder verdict'], + [/\bax-bg\b|\bpx-bg\b|\bpx-fg\b|\bax-fg\b/, 'internal modality recorder lane'], + [/\bderec\.sh\b/, 'internal test harness script'], + [/\baz exec\b/, 'internal CI/container access detail'], + [/\bTEST_SUITE\.md\b|\bFINDINGS\.md\b/, 'repo-side contributor document reference'], + [/\bNousResearch\b|#47065\b|#22865\b/, 'private partner or issue reference'], + [/\bdocumented by a separate agent\b/i, 'authoring note'], + [/\bDo not imply\b/i, 'authoring instruction'], + [/\bdiorama\b/i, 'misnamed docs framework'], +]; + +async function main() { + const files = await fg('**/*.mdx', { cwd: CONTENT_DIR }); + const failures: string[] = []; + + for (const file of files) { + const abs = path.join(CONTENT_DIR, file); + const content = await fs.readFile(abs, 'utf8'); + const lines = content.split(/\r?\n/); + + for (const [lineIndex, line] of lines.entries()) { + for (const [pattern, reason] of bannedPatterns) { + if (pattern.test(line)) { + failures.push(`${file}:${lineIndex + 1}: ${reason}: ${line.trim()}`); + } + } + } + } + + if (failures.length > 0) { + console.error('Public docs hygiene check failed:\n'); + console.error(failures.join('\n')); + process.exit(1); + } + + console.log('Public docs hygiene check passed.'); +} + +main().catch((error) => { + console.error('Fatal error:', error); + process.exit(1); +}); diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/health_report.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/health_report.rs index 84dd090bdd..634248a35e 100644 --- a/libs/cua-driver/rust/crates/cua-driver-core/src/health_report.rs +++ b/libs/cua-driver/rust/crates/cua-driver-core/src/health_report.rs @@ -1,7 +1,6 @@ //! `health_report` — single-call end-to-end driver diagnostics. //! -//! The point of this tool is to let downstream consumers (Hermes Agent -//! and similar — see NousResearch/hermes-agent#47065) ship one stable +//! The point of this tool is to let downstream consumers ship one stable //! diagnostic call and never have to know cua-driver internals: specific //! MCP tool names, TCC field names, bundle IDs, per-platform check //! matrix. cua-driver owns the health model entirely; consumers stay @@ -338,10 +337,9 @@ fn def() -> &'static ToolDef { DEF.get_or_init(|| ToolDef { name: "health_report".into(), // The description is part of the public contract — downstream - // consumers (Hermes Agent / `hermes computer-use doctor`) - // depend on it spelling out `schema_version="1"` and the per- + // consumers depend on it spelling out `schema_version="1"` and the per- // platform check matrix. A test pins this commitment. - description: r#"Single-call end-to-end driver diagnostics. Designed to let downstream consumers (Hermes Agent and similar) ship one stable call instead of stitching together check_permissions, doctor, version, bundle attribution, and a screenshot probe. cua-driver owns the health model; consumers stay thin. + description: r#"Single-call end-to-end driver diagnostics. Designed to let downstream consumers ship one stable call instead of stitching together check_permissions, doctor, version, bundle attribution, and a screenshot probe. cua-driver owns the health model; consumers stay thin. Input — all optional: { @@ -382,7 +380,7 @@ Output — stable contract, schema_version="1": - `degraded` — at least one non-core check fails (binary is still usable) - `failed` — any core check fails (binary_version, platform_supported, session_active) -Stability: schema_version="1" is the contract. Future breaking changes will be `"2"`. Adding new check names under the same schema_version is non-breaking; consumers must tolerate unknown check names. Downstream consumer: NousResearch/hermes-agent#47065."#.into(), +Stability: schema_version="1" is the contract. Future breaking changes will be `"2"`. Adding new check names under the same schema_version is non-breaking; consumers must tolerate unknown check names."#.into(), input_schema: json!({ "type": "object", "properties": { diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs index 6dd7eec873..5f3c058d31 100644 --- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs @@ -444,7 +444,7 @@ impl Tool for GetWindowStateTool { deprecated and ignored.\n\n\ Optional `max_elements` / `max_depth` bound the AT-SPI walk to \ mitigate context-window blow-up on Electron / large web apps \ - that produce 10k+ element trees (#22865). When applied, BOTH \ + that produce 10k+ element trees. When applied, BOTH \ the markdown and the structured elements are truncated \ identically. Omit both for current default behaviour.".into(), input_schema: json!({"type":"object","required":["pid","window_id"],"properties":{ @@ -457,8 +457,8 @@ impl Tool for GetWindowStateTool { "screenshot_out_file":{"type":"string", "description":"When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead."}, "query":{"type":"string"}, - "max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees (#22865)."}, - "max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps (#22865)."} + "max_elements":{"type":"integer","minimum":1,"description":"Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees."}, + "max_depth":{"type":"integer","minimum":1,"description":"Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps."} },"additionalProperties":false}), read_only: true, destructive: false, idempotent: true, open_world: false, }) @@ -627,7 +627,7 @@ impl Tool for GetWindowStateTool { structured["_note"] = json!( "Prefer `elements` — `tree_markdown` will continue to work \ but new fields will only be added to the structured side. \ - Issue #22865: use `max_elements` / `max_depth` to bound the \ + Use `max_elements` / `max_depth` to bound the \ AT-SPI walk on apps with very large trees." ); // Best-effort-background ladder parity with macOS/Windows: an @@ -4799,7 +4799,7 @@ impl Tool for KillAppTool { } } -// ── bring_to_front (Linux stub) ────────────────────────────────────────────── +// ── bring_to_front (Linux) ─────────────────────────────────────────────────── pub struct BringToFrontTool; @@ -4935,8 +4935,7 @@ pub fn build_registry(compat: bool) -> ToolRegistry { r.register(Box::new(SetAgentCursorStyleTool { state: state.clone() })); r.register(Box::new(CheckPermissionsTool)); // `health_report` — single-call cross-platform driver diagnostics. - // Stable schema_version="1" contract for downstream consumers - // (Hermes Agent, NousResearch/hermes-agent#47065). Linux skips + // Stable schema_version="1" contract for downstream consumers. Linux skips // tcc_* and bundle_identity with "not applicable on Linux". r.register(Box::new(cua_driver_core::health_report::HealthReportTool::new( std::sync::Arc::new(crate::health_report::LinuxHealthProvider), diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs index 953c8f8147..ddff5a15cd 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs @@ -47,7 +47,7 @@ fn def() -> &'static ToolDef { filtering only trims the rendered Markdown.\n\n\ Optional `max_elements` / `max_depth` bound the AX walk to mitigate \ context-window blow-up on Electron / Obsidian / large web apps that \ - produce 10k+ element trees (#22865). When applied, BOTH the markdown \ + produce 10k+ element trees. When applied, BOTH the markdown \ and the structured elements are truncated identically. Omit both for \ current default behaviour (≤2 000 elements, depth ≤25).".into(), input_schema: serde_json::json!({ @@ -70,12 +70,12 @@ fn def() -> &'static ToolDef { "max_elements": { "type": "integer", "minimum": 1, - "description": "Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows (#22865)." + "description": "Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows." }, "max_depth": { "type": "integer", "minimum": 1, - "description": "Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees (#22865)." + "description": "Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees." } }, "additionalProperties": false diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs index c55ca22f4b..ba90c36015 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs @@ -425,10 +425,8 @@ pub fn register_all(registry: &mut ToolRegistry, compat: bool) { registry.register(Box::new(cursor_tools::GetAgentCursorStateTool::new(state.clone()))); registry.register(Box::new(check_permissions::CheckPermissionsTool)); // `health_report` — single-call end-to-end diagnostics. Stable - // schema_version="1" contract aimed at downstream consumers - // (Hermes Agent's `hermes computer-use doctor`, NousResearch/ - // hermes-agent#47065) who must NOT have to know cua-driver - // internals. Provider is platform-specific; tool plumbing is in + // schema_version="1" contract aimed at downstream consumers who must + // not have to know cua-driver internals. Provider is platform-specific; tool plumbing is in // `cua_driver_core::health_report`. registry.register(Box::new(cua_driver_core::health_report::HealthReportTool::new( Arc::new(health_report::MacosHealthProvider), diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs index 4d7dd96c62..f1b9b84bd3 100644 --- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs @@ -716,7 +716,7 @@ impl Tool for GetWindowStateTool { timing out at 4s with per-property RPCs).\n\n\ Optional `max_elements` / `max_depth` bound the UIA walk to mitigate \ context-window blow-up on Electron / large web apps that produce 10k+ \ - element trees (#22865). When applied, BOTH the markdown and the structured \ + element trees. When applied, BOTH the markdown and the structured \ elements are truncated identically. Omit both for current default behaviour \ (≤5 000 elements, depth ≤25).\n\n\ Windows requires no special permissions.".into(), @@ -728,8 +728,8 @@ impl Tool for GetWindowStateTool { "include_screenshot":{"type":"boolean","description":"Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action)."}, "screenshot_out_file":{"type":"string","description":"When set, write the PNG to this file path instead of embedding base64 in the response. The structured output will contain `screenshot_file_path` instead."}, "query":{"type":"string","description":"Optional case-insensitive substring. When set, `tree_markdown` only contains lines that match plus their ancestor chain; element indices and `element_count` are unchanged."}, - "max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees (#22865)."}, - "max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees (#22865)."} + "max_elements":{"type":"integer","minimum":1,"description":"Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (5 000). Lower for Electron / large web apps that produce 10k+ element trees."}, + "max_depth":{"type":"integer","minimum":1,"description":"Cap on the UIA-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower for deep menu / Electron trees."} },"additionalProperties":false}), // Swift annotation: idempotent: false (each call is a fresh snapshot). read_only: true, destructive: false, idempotent: false, open_world: false, @@ -7042,8 +7042,7 @@ pub fn build_registry(compat: bool) -> ToolRegistry { r.register(Box::new(SetAgentCursorStyleTool { state: state.clone() })); r.register(Box::new(CheckPermissionsTool)); // `health_report` — single-call cross-platform driver diagnostics. - // Stable schema_version="1" contract for downstream consumers - // (Hermes Agent, NousResearch/hermes-agent#47065). Windows skips + // Stable schema_version="1" contract for downstream consumers. Windows skips // tcc_* and bundle_identity with "not applicable on Windows". r.register(Box::new(cua_driver_core::health_report::HealthReportTool::new( std::sync::Arc::new(crate::health_report::WindowsHealthProvider), diff --git a/docs/content/docs/reference/cua-driver/modality-test-suite.mdx b/libs/cua-driver/test-harness/MODALITY_TEST_SUITE.md similarity index 99% rename from docs/content/docs/reference/cua-driver/modality-test-suite.mdx rename to libs/cua-driver/test-harness/MODALITY_TEST_SUITE.md index 1d7f956a85..8594983a7b 100644 --- a/docs/content/docs/reference/cua-driver/modality-test-suite.mdx +++ b/libs/cua-driver/test-harness/MODALITY_TEST_SUITE.md @@ -1,7 +1,10 @@ ---- -title: Modality Test Suite & Harnesses -description: How the Cua Driver modality test suite and per-OS app harnesses work — action rungs, scopes, the action matrix, and per-platform results. ---- +# Modality test suite and harnesses + +This is an internal contributor runbook for the Cua Driver modality test suite. +It used to live in the public docs reference tree, but it describes the harness, +CI lanes, recorder scripts, and regression matrices rather than public product +behavior. User-facing limits harvested from this page belong in +`docs/content/docs/reference/cua-driver/limits.mdx`. This document describes the cua-driver test-harness: a cross-OS, cross-toolkit rig that exercises every driver action against a controlled application and diff --git a/libs/cua-driver/test-harness/TEST_SUITE.md b/libs/cua-driver/test-harness/TEST_SUITE.md index 72bbd767b8..c128ceba28 100644 --- a/libs/cua-driver/test-harness/TEST_SUITE.md +++ b/libs/cua-driver/test-harness/TEST_SUITE.md @@ -88,18 +88,18 @@ mirror pairs into single `cfg!`-branching tests, sharing one `RawDriver`. - **Transport** is now a first-class axis: `transport_config_persistence` exercises CLI (disk) vs MCP (session) directly; most other tests run over MCP. -### The modality matrix: action target × dispatch × `capture_scope` +### The modality matrix: action target × delivery mode × `capture_scope` The user-facing matrix is documented in -`docs/content/docs/explanation/capture-and-dispatch-modalities.mdx`. `get_window_state` +`docs/content/docs/concepts/capture-and-delivery-modalities.mdx`. `get_window_state` always returns both the accessibility tree and a screenshot; AX vs pixel modality is chosen at action time by using `element_index` vs `x,y`. Coverage of its valid cells, per platform: -| Cell (`scope`/action target/`dispatch`) | Windows | macOS | Linux | +| Cell (`scope`/action target/`delivery_mode`) | Windows | macOS | Linux | |---|---|---|---| | `window`/`element_index`/`background` (default) | `harness_*`, `modality_background` | `harness_{appkit,swiftui}`, `modality_focus`, `modality_capture_mode` | `harness_gtk3`, `modality_capture_mode` | -| `window`/`element_index`/`foreground` | `harness_wpf` (`dispatch:"foreground"`) | activation differs | activation differs | +| `window`/`element_index`/`foreground` | `harness_wpf` (`delivery_mode:"foreground"`) | activation differs | activation differs | | `window`/`x,y`/`background` | `modality_background`, `modality_capture_mode`, `modality_input_e2e` | `modality_capture_mode` | `modality_capture_mode` | | `window`/`x,y`/`foreground` | gap | n/a (no `bring_to_front`) | n/a (stubbed) | | `desktop`/`x,y`/`foreground` | `modality_desktop_scope` | rolling out | rolling out | diff --git a/scripts/docs-generators/config.json b/scripts/docs-generators/config.json index 70b6e4ac27..ce22432186 100644 --- a/scripts/docs-generators/config.json +++ b/scripts/docs-generators/config.json @@ -123,7 +123,8 @@ "extractCommand": "python3 scripts/docs-generators/extract_python_docs.py" } ], - "enabled": true + "enabled": false, + "notes": "Disabled while these SDK docs are not part of the public docs nav; prevents docs:generate from resurrecting the old docs/content/docs/cua/reference tree." }, "computer-sdk-typescript": { "name": "Computer SDK (TypeScript)", @@ -178,7 +179,8 @@ "extractCommand": null } ], - "enabled": true + "enabled": false, + "notes": "Disabled while cuabot docs are not part of the public docs nav; prevents docs:generate from resurrecting docs/content/docs/cuabot/reference." } } } diff --git a/scripts/docs-generators/cua-driver.ts b/scripts/docs-generators/cua-driver.ts index 55ce861447..09f37a6ecf 100644 --- a/scripts/docs-generators/cua-driver.ts +++ b/scripts/docs-generators/cua-driver.ts @@ -593,7 +593,7 @@ export function generateMCPToolsMDX(docs: MCPDocumentation, releasedVersion: str lines.push(''); lines.push(''); lines.push( - " **TCC auto-delegation.** When an MCP client spawns `cua-driver mcp` from an IDE terminal (Claude Code, Cursor, VS Code, Warp), macOS attributes the subprocess to the parent terminal — not `CuaDriver.app` — so AX probes fail against the wrong bundle id. `mcp` detects this and auto-launches a `cua-driver serve` daemon via `open -n -g -a CuaDriver --args serve`, then proxies every tool call through the daemon's Unix socket. Tool semantics are identical to the in-process path; no Python bridge is needed. Pass `--no-daemon-relaunch` (or set `CUA_DRIVER_MCP_NO_RELAUNCH=1`) to force in-process execution. See the [process model](/explanation/process-model) for the full lifecycle, failure modes, and wrapper-author guidance." + " **TCC auto-delegation.** When an MCP client spawns `cua-driver mcp` from an IDE terminal (Claude Code, Cursor, VS Code, Warp), macOS attributes the subprocess to the parent terminal — not `CuaDriver.app` — so AX probes fail against the wrong bundle id. `mcp` detects this and auto-launches a `cua-driver serve` daemon via `open -n -g -a CuaDriver --args serve`, then proxies every tool call through the daemon's Unix socket. Tool semantics are identical to the in-process path; no Python bridge is needed. Pass `--no-daemon-relaunch` (or set `CUA_DRIVER_MCP_NO_RELAUNCH=1`) to force in-process execution. See the [process model](/reference/cua-driver/process-model) for the full lifecycle, failure modes, and wrapper-author guidance." ); lines.push(''); lines.push('');