Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/ci-check-docs-links.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ on:
paths:
- "docs/content/**"
- "docs/src/**"
- "docs/package.json"
- "docs/scripts/check-links.ts"
- "docs/scripts/check-hygiene.ts"

jobs:
check-internal-links:
Expand All @@ -31,6 +33,10 @@ jobs:
run: pnpm docs:check-links
working-directory: docs

- name: Check docs hygiene
run: pnpm docs:check-hygiene
working-directory: docs

- name: Show help if check failed
if: failure()
run: |
Expand Down
33 changes: 9 additions & 24 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,39 +2,24 @@

Production docs are served from https://cua.ai/docs; this app is now a local MDX preview only.

This is a Next.js application generated with
[Create Fumadocs](https://github.com/fuma-nama/fumadocs).

Run development server:
Run the local preview:

```bash
npm run dev
# or
pnpm dev
# or
yarn dev
```

Open http://localhost:3000 with your browser to see the result.

## Explore

In the project, you can see:

- `lib/source.ts`: Code for content source adapter, [`loader()`](https://fumadocs.dev/docs/headless/source-api) provides the interface to access your content.
- `app/layout.config.tsx`: Shared options for layouts, optional but preferred to keep.
Open http://localhost:8090 with your browser to see the result.

| Route | Description |
| ------------------------- | ------------------------------------------------------ |
| `app/(home)` | The route group for your landing page and other pages. |
| `app/docs` | The documentation layout and pages. |
| `app/api/search/route.ts` | The Route Handler for search. |
## Docs conventions

### Fumadocs MDX
Public docs live in `content/docs/` and follow the Diátaxis modes:

A `source.config.ts` config file has been included, you can customise different options like frontmatter schema.
- `tutorials/` teach a guided first success.
- `how-to-guides/` give steps for a specific goal.
- `explanation/` explains concepts, constraints, and tradeoffs.
- `reference/` is dry lookup: commands, APIs, contracts, limits.

Read the [Introduction](https://fumadocs.dev/docs/mdx) for further details.
Place content by what the reader is trying to do, not by topic. Do not mix modes in one page; move reference tables to reference pages and link to them from how-to guides or explanations.

## Setup Telemetry

Expand Down
75 changes: 75 additions & 0 deletions docs/content/docs/concepts/capture-and-delivery-modalities.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
---
title: "Capture and Delivery Modalities"
description: "How Cua Driver observes and acts on an app. Perception returns both the accessibility tree and a screenshot; the action call chooses the ax or px rung, delivery mode, and scope."
---

# Capture and Delivery Modalities

Every Cua Driver action is shaped by four things: **what the agent observes**, **which rung delivers the action**, **how input is delivered**, and **what coordinate space the action targets**. Most callers use the defaults: background, per-window, accessibility-first automation. The key change from earlier versions is that perception is no longer a mode you pick. `get_window_state` returns *both* the accessibility tree and a screenshot in one call. The action call chooses `ax` or `px` by how it addresses the target.

## The Axes

### 1. Perception: what the agent observes

`get_window_state(pid, window_id)` returns **both the accessibility tree and a screenshot by default**, in one call. There is no capture mode to pick: you ground on the tree and the screenshot together and cross-check one against the other. This matters because the tree *lies* on some surfaces: it can expose useful structure while still echoing a write the app did not apply, omitting the rendered value, or reporting geometry that disagrees with the pixels. A grounding screenshot is always present, so when the tree looks wrong you check the pixels in the *same* response.

The accessibility tree is the ground truth for *what is clickable*: roles, labels, advertised actions, and an `element_index` handle on every actionable element. The screenshot tells you *which one*. It disambiguates repeated or empty labels and shows captions, colors, and layout the tree omits, which is common in Chromium and Electron. They come back together because each catches what the other misses.

> **Performance opt-out: `include_screenshot`.** `include_screenshot` (boolean, default `true`) is the one performance knob. The default returns both. Pass `include_screenshot: false` to skip the screen grab and get the tree only when you are re-indexing before an element ax action and do not need fresh pixels. The `ax`-versus-`px` decision still lives at action time.

> **`capture_mode` is deprecated and ignored.** `get_window_state` still accepts it so old callers do not error, but both the tree and the screenshot come back regardless of what you pass. The old `ax` / `vision` / `som` / `screenshot` values all decode (`som` mapped to `ax`, `screenshot` to `vision`) but none changes what is captured. Perception is always both.

### 2. Action rung: how the target is addressed

You don't pick a capture mode; you pick **how you address the target** on the action call, and that one choice selects the rung:

| Rung | Address with | Delivered through | Properties |
|---|---|---|---|
| **element ax action** | `element_index` / `element_token` | the accessibility rung: UIA Invoke (Windows), `AXPerformAction` (macOS), AT-SPI `doAction` (Linux) | Backgroundable, z-order-independent, and the only **driver-verifiable** rung. |
| **element px action** | `x, y` | the pixel rung, reading the coordinate straight off the screenshot already in the `get_window_state` response | Best-effort; the caller confirms the effect off the screenshot. |

Default to the element ax action because the driver can verify it and often keep it in the background. Drop to an element px action when the tree cannot disambiguate repeated or empty labels, when it is empty (`degraded`, a non-AX surface), when an action came back `suspected_noop`, or when the tree disagrees with the pixels. You never re-capture to switch rungs. The screenshot is already in the snapshot, so you only change *how you address* the target.

Both rungs apply to the **keyboard family** (`type_text`, `press_key`, `hotkey`) as well as the pointer tools. Address by `element_index` (ax) to target a field with no pre-click. Address by `x, y` (px) to pixel-click at `(x, y)`, establish real renderer focus, and deliver the keystroke(s) to the now-focused element. The px form is the one-call path for Chromium/Electron inputs the AX layer cannot focus: `type_text({ pid, window_id, x, y, text })` focuses and types in a single call. The two forms are mutually exclusive. `set_value` is the exception: it stays ax-only because it sets the value of a non-text control like a dropdown, checkbox, or slider.

### 3. Delivery: how input is delivered

Set `delivery_mode` per call on the input family (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`). The same two values work on Windows, macOS, and Linux.

| `delivery_mode` | Behavior |
|---|---|
| `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched when the target surface supports background delivery. See [Best-effort background](/concepts/the-no-foreground-contract). |
| `foreground` | The target is briefly fronted (pair with `bring_to_front` to avoid a per-call flash), input lands on the now-active window, then the prior frontmost is restored. Use this when a background attempt did not land, or when the app only accepts events while foregrounded (DirectInput games, raw-input canvases). |

Only `background` and `foreground` are valid; the historical `auto` heuristic is removed. At runtime, omitted or unknown values fall back to `background` for safety. Element ax actions (`element_index`) address an element instead of the focused window, so they hold the background path without any `delivery_mode` flag. The delivery axis matters most for the pixel rung (`x, y`), where `background` routes the event to the target and `foreground` raises the window first.

### 4. Capture scope: what coordinate space the action targets

Set with `set_config capture_scope=…`.

| `capture_scope` | Coordinate space | Capture surface |
|---|---|---|
| `window` (default) | Per-window. Actions carry `pid` + `window_id`; coordinates are window-relative or addressed by `element_index`. | `get_window_state` |
| `desktop` | Screen-absolute. Window-less actions (no `pid`/`window_id`) land at absolute screen coordinates via hit-testing (`WindowFromPoint`). | `get_desktop_state` with full display, no downscale |

Desktop scope is the "Computer-Use 1.0" loop: the agent reads the whole screen and clicks absolute coordinates, the way a screenshot-only model expects. Window scope is the default because it is what makes background, concurrent automation possible.

## Response signals: verification, effect, and escalation

Delivery success differs from application state change. The driver can verify an effect only when it can read the changed state back through the accessibility layer. That is why `verified: true` is reserved for AX read-back: it means the driver observed the effect after sending the event. Pixel input, foreground input, and echo-prone AX surfaces can be routed correctly while still leaving confirmation to the caller.

`effect` is the confidence signal that separates those cases. `"confirmed"` means the driver verified the result through AX read-back. `"unverifiable"` means the delivery path ran, but the driver cannot prove the application applied it. `"suspected_noop"` means an AX action ran but almost certainly did not change the target. Treat `effect` as the action outcome.

`escalation` is the machine-readable climb-the-ladder hint. When present, it tells the caller which surface to try next: `"px"` for acting off the screenshot, `"foreground"` for explicitly fronting the target, or `"page"` for the browser-tab DOM path through the `page` tool. See [Agent action policy](/reference/cua-driver/action-selection-policy) for the agent behavior and [MCP tool notes](/reference/cua-driver/mcp-tool-notes#action-response-shape) for the field table.

### When the tree lies

Some accessibility layers echo writes they did not apply. Electron can report an AX value change through its shim while the renderer stays unchanged. Catalyst controls can expose null `AXValue`s. Chromium/WebKit web content can reflect a write through the accessibility bridge without proving the DOM or rendered view changed.

The driver treats those as surface-aware verification cases. It probes at the element level for a web-content surface, including an `AXWebArea` ancestor, so native chrome such as a browser address bar stays trusted while browser-tab content does not get a false confirmation. On those surfaces the driver refuses false `verified: true` responses and returns `verified: false`, `effect: "unverifiable"`, and an `escalation` object instead. Electron app surfaces recommend `"px"` so the caller can act by pixel off the screenshot in the same response; browser-tab web content recommends `"page"` so the caller can switch to DOM/CDP via the `page` tool.

## Where the exact matrix lives

Perception is no longer an axis: every `get_window_state` returns both the tree and a screenshot by default. The enforceable constraints are the combination of **capture scope**, **action rung**, and **delivery mode**.

The exact validity matrix and platform-support table live in [Interface contracts](/reference/cua-driver/contracts). Keep this page as the mental model: observe both tree and pixels, start with the accessibility rung, use the screenshot-backed pixel rung when the tree is insufficient, and escalate to foreground only when the target app requires it.
8 changes: 8 additions & 0 deletions docs/content/docs/concepts/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
title: "Concepts"
description: "Understand the design ideas behind Cua and how its main pieces fit together."
---

Use these pages when you want the model behind Cua rather than a step-by-step guide or API table. They explain what computer use means in Cua, how Cua Driver keeps the desktop usable while it acts, and how Cua Sandbox gives an agent a disposable computer.

Start with [What is computer use?](/concepts/what-is-computer-use) for the basic model. Read [Best-effort background](/concepts/the-no-foreground-contract) to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities](/concepts/capture-and-delivery-modalities) for the action axes. Read [How sandboxes work](/concepts/how-sandboxes-work) when you need the model for disposable computers.
1 change: 1 addition & 0 deletions docs/content/docs/concepts/meta.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{ "title": "Concepts", "icon": "Lightbulb", "pages": ["index", "what-is-computer-use", "the-no-foreground-contract", "capture-and-delivery-modalities", "how-sandboxes-work"] }
54 changes: 54 additions & 0 deletions docs/content/docs/concepts/the-no-foreground-contract.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
title: Best-effort background
description: How Cua Driver tries to operate apps without taking focus, moving the cursor, or raising windows, and when it must fall back to foreground.
---

# Best-effort background

Best-effort background means Cua Driver tries to operate a target app while preserving the user's active desktop. The default paths do not move the real pointer, do not raise the target window, and do not switch the user's frontmost app.

This is a best effort rather than an absolute promise. Most app automation can stay in the background through accessibility actions, routed input, and window-specific capture. A small set of apps and OS surfaces only accept real foreground input, so the driver reports that limit and lets the caller choose a foreground escalation for that specific action.

## Why it matters

Traditional GUI automation assumes the automated app owns the desktop. It activates a window, moves the pointer, and repeats. That is fine for unattended jobs or disposable desktops, but it breaks down when a person is using the same machine.

Cua Driver's default path lets the agent operate an app in the background while the developer keeps coding, reading logs, or using another app. The visible agent cursor is an overlay; the real mouse pointer stays where the user left it.

## Platform mechanisms

Each OS splits accessibility, input delivery, capture, and focus policy differently. Cua Driver chooses the most background-capable path the platform and app expose.

### macOS

The Accessibility API can press buttons, set values, and read semantic state even when the app is not frontmost. ScreenCaptureKit can capture a specific window without requiring that window to be raised or visible on the active Space. Cua Driver also uses scoped CoreGraphics and SkyLight delivery for routed input when an app responds better to pointer-like events than accessibility actions.

Some macOS surfaces still need foreground. SwiftUI windows parked on another Space can lose their detailed accessibility tree, and game/canvas surfaces may reject routed input. Those cases are documented in [Known limits](/reference/cua-driver/limits).

### Windows

UI Automation can inspect and operate controls by window handle and automation element while another app is active. For input-like behavior, Cua Driver can post messages to a target window or use foreground escalation when the app only listens to active device input.

Windows also has session boundaries. A daemon running in the interactive user session can see and operate the desktop; a process launched from OpenSSH in Session 0 cannot. See [Process model](/reference/cua-driver/process-model) and [Drive a Windows app over SSH](/how-to-guides/driver/windows-ssh).

### Linux

AT-SPI provides the semantic path on Linux. Element actions call the toolkit's own accessibility action (`Action.DoAction`) and do not need pointer injection or foreground focus. On X11, window-addressable input and capture can also route to a target window. On Wayland, synthetic input is intentionally constrained by the compositor, so the background path depends more heavily on AT-SPI and reconstructed element frames.

The remaining Linux gap is raw keyboard injection into native Wayland apps. Typing into accessible fields can still work through AT-SPI, but shortcuts or raw key events may need XWayland or foreground/user-granted compositor paths. See [Known limits](/reference/cua-driver/limits#native-wayland-apps-cant-receive-synthetic-keystrokes).

## The agent cursor

Cua Driver does not move the user's real pointer to show agent activity. It renders a synthetic cursor overlay for supervision. The user can see where the agent is acting while their own cursor and active app stay untouched.

## How fallback works

The safest ladder is:

1. Act by `element_index` in the background.
2. If the element path is unavailable or unverifiable, act by `x, y` from the same window screenshot.
3. If the app still rejects the action, retry that one action with `delivery_mode: "foreground"`.

Foreground escalation is explicit. It is the right answer for apps that only accept focused input, but callers should use it narrowly and only when interrupting the user's desktop is acceptable.

For the agent-side action behavior, see [Agent action policy](/reference/cua-driver/action-selection-policy). For the reference matrix, see [Interface contracts](/reference/cua-driver/contracts).
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: "Computer-Use 2.0"
description: "Computer-Use 2.0 is how an agent operates a real computer through code, structured tool calls, and the graphical interface."
description: "What Cua refers to as Computer-Use 2.0: an agent operating a real computer through code, structured tool calls, and the graphical interface."
---

# Computer-Use 2.0

Computer-Use 2.0 is the way an AI agent operates a real computer by choosing among three action surfaces: writing and running code, calling tools and APIs, and driving the graphical interface a person would use. The older use of the term, where an agent looks at screenshots and clicks through a GUI, still matters as the UI surface inside a broader model of how agents get work done on computers.
What we refer to as **Computer-Use 2.0** is an AI agent operating a real computer by choosing among three action surfaces: writing and running code, calling tools and APIs, and driving the graphical interface a person would use. The older screenshot-and-click loop still matters as the UI surface inside a broader model of how agents get work done on computers.

## Three action surfaces

Expand All @@ -27,7 +27,7 @@ The observe, decide, act loop belongs specifically to the UI automation surface.

Planning carries the task across changing interface states. A click may open a dialog, a page may reflow after loading, or an application may show an error that changes the next useful action, so the model has to keep track of the goal while the computer responds. Frontier models such as Claude can handle understanding, grounding, and planning together in one call, while grounding-specialist models such as UI-TARS and Moondream can help when coordinate accuracy is the limiting factor.

The narrow origin of computer-use came in October 2024, when Anthropic introduced an agent that operated a GUI through screenshots and input events. Through 2025, coding agents were increasingly recognized as computer-use agents too, with CoAct-1 making the connection explicit, and the field began to converge on Computer-Use 2.0 as a wider concept. Francesco Bonacci traces that arc in [A Story of Computer-Use](https://github.com/trycua/cua/blob/main/blog/clawdbot-computer-use-history.md).
The narrow origin of computer-use came in October 2024, when Anthropic introduced an agent that operated a GUI through screenshots and input events. Through 2025, coding agents were increasingly recognized as computer-use agents too, with CoAct-1 making the connection explicit. Cua uses **Computer-Use 2.0** as shorthand for that wider model. Francesco Bonacci traces that arc in [A Story of Computer-Use](https://github.com/trycua/cua/blob/main/blog/clawdbot-computer-use-history.md).

## Where Cua fits

Expand Down
Loading
Loading