Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/cd-rust-cua-driver.yml
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,22 @@ jobs:
# current version's baked default — not the previous version's.
cp libs/cua-driver-rs/scripts/install.sh release-upload/install.sh
cp libs/cua-driver-rs/scripts/install.ps1 release-upload/install.ps1

# Skill pack — single platform-agnostic tarball fetched by
# `cua-driver skills install`. The .md files are identical across
# OS so we publish one asset per release tag. Naming matches the
# versioned binary tarballs.
VERSION="${{ steps.version.outputs.version }}"
SKILLS_STAGE="cua-driver-rs-v${VERSION}-skills"
mkdir -p "${SKILLS_STAGE}/cua-driver-rs"
if [ -d libs/cua-driver-rs/Skills/cua-driver-rs ]; then
cp -R libs/cua-driver-rs/Skills/cua-driver-rs/* "${SKILLS_STAGE}/cua-driver-rs/"
tar -czf "release-upload/${SKILLS_STAGE}.tar.gz" "${SKILLS_STAGE}"
echo "Packaged skill pack: release-upload/${SKILLS_STAGE}.tar.gz"
else
echo "Note: libs/cua-driver-rs/Skills/cua-driver-rs not present; skipping skill-pack asset."
fi

echo "Release files:"
ls -lh release-upload/

Expand Down
87 changes: 87 additions & 0 deletions libs/cua-driver-rs/Skills/cua-driver-rs/LINUX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
name: cua-driver-rs-linux
description: Drive a native Linux app (X11 / Wayland) via the cua-driver CLI — snapshot the AT-SPI tree, click by element_index or pixel, verify via re-snapshot. Linux backend is BETA in cua-driver-rs and the no-foreground contract has open issues (see this doc).
---

# cua-driver-rs — Linux

**Status: BETA.** The Linux backend in cua-driver-rs covers the core
tool surface (click, type_text, scroll, hotkey, screenshot,
launch_app, list_apps, list_windows, get_window_state) but several
behaviors that the macOS / Windows skills consider table-stakes are
**not yet implemented or only partially supported**:

- **No-foreground contract**: limited. X11's input model has no clean
per-pid event routing equivalent of macOS `CGEventPostToPSN` or
Windows `PostMessage(WM_LBUTTONDOWN)`. `XTestFakeKeyEvent` /
`XTestFakeButtonEvent` synthesize input but route to the focused
window — similar focus-stealing behavior to Windows `SendInput`,
which the Windows backend avoids by using `PostMessage` instead.
Linux has no equivalent per-window-message channel that bypasses
focus, which is why XTest's focus-stealing is the binding
limitation here. AT-SPI `accDoDefaultAction` works for accessible
elements but requires the user's accessibility bus to be running,
which is not the default on every distro.
- **Wayland support**: depends on compositor. Under GNOME-Mutter and
KDE-KWin with `org.freedesktop.portal.RemoteDesktop` enabled, some
click and key paths work. Under most other compositors, input
synthesis is denied by the security model and the tool surface
degrades to "passive" (snapshot, screenshot) only.
- **UIA / AX-tree equivalent**: AT-SPI when available, otherwise
empty. Many GTK4 / Qt6 apps populate AT-SPI lazily; agents should
expect partial trees and re-snapshot.
- **launch_app**: backed by `xdg-open` / `gtk-launch` / `dbus-send`
with display-environment scrubbing to avoid stealing the user's
workspace. Not yet equivalent to macOS `FocusRestoreGuard`.
- **Recording**: not supported.

See `SKILL.md` (macOS) and `WINDOWS.md` (Windows) for the full
patterns. This file will grow as the Linux backend reaches GA. For
now, **prefer macOS / Windows hosts** for agent-driven GUI tasks; use
the Linux daemon for read-only inspection (screenshot,
list_windows, get_window_state) when running on a Linux host.

## Quick triage

If you're agent-driving on Linux and a tool call surprises you:

1. Run `cua-driver doctor` — reports display server (X11 / Wayland),
AT-SPI bus reachability, XTest availability.
2. Check `XDG_SESSION_TYPE` — `wayland` means most input synthesis
is gated by portals; `x11` means XTest works but routes via focus.
3. If Wayland: confirm `org.freedesktop.portal.RemoteDesktop` is
present (`gdbus introspect --session --dest
org.freedesktop.portal.Desktop --object-path
/org/freedesktop/portal/desktop`). Without it, input synthesis
is denied.

## Forbidden vectors

Same idea as macOS / Windows — don't shell out to anything that
foregrounds a target:

- `wmctrl -a <window>` — activates the named window.
- `xdotool windowactivate <wid>` — activates.
- `wmctrl -R <window>` — raises and activates.
- `xdotool key --window <wid> alt+Tab` — same problem as Windows
Alt+Tab.

Prefer cua-driver tools with explicit `window_id`. When in doubt,
ask the user.

## What to expect today

| Intent | Status |
|---|---|
| Snapshot UIA tree | ✅ AT-SPI when available, often partial for GTK4/Qt6 |
| Pixel click | ⚠️ X11 only, focus-stealing semantics |
| Element-indexed click | ⚠️ AT-SPI `accDoDefaultAction` when supported |
| Type text | ⚠️ XTest, focus-sensitive |
| Hotkey | ⚠️ XTest, focus-sensitive |
| Screenshot full-display | ✅ X11 (xshm); ⚠️ Wayland (portal-gated) |
| Screenshot per-window | ⚠️ X11 with composite extension; Wayland TBD |
| launch_app | ⚠️ xdg-open / gtk-launch; no FocusRestoreGuard yet |
| Recording | ❌ not implemented |

Until Linux reaches GA, treat this doc as a planning placeholder
rather than a contract.
205 changes: 205 additions & 0 deletions libs/cua-driver-rs/Skills/cua-driver-rs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
# cua-driver-rs — Claude Code skill

A [Claude Code](https://code.claude.com) skill that teaches Claude to
drive native GUI apps on **macOS, Windows, and Linux** via the
[`cua-driver`](https://github.com/trycua/cua/tree/main/libs/cua-driver-rs)
CLI — snapshot an app's accessibility tree (AX on macOS, UIA on
Windows, AT-SPI on Linux), click/type/scroll by `element_index` or
pixel coords, and verify via re-snapshot. Backgrounded-first on every
platform: no focus steal, no cursor warp.

## Platform reading order

- `SKILL.md` — shared core + macOS-specific patterns (read this
first on macOS).
- `WINDOWS.md` — Windows-specific carve-out (UIA tree, UWP /
ApplicationFrameHost, layered UIA+PostMessage click chain,
Session 0 isolation, Windows web-apps section). Read this when
driving on Windows.
- `LINUX.md` — Linux status + carve-out (X11/Wayland, AT-SPI,
BETA-level). Read this when driving on Linux.

## What the skill covers

- The snapshot-before-AND-after invariant that keeps the agent honest
about whether an action actually landed.
- The backgrounded-click recipe that lets synthetic clicks land on
web content without raising the window or pulling the user across
virtual desktops. Per-platform mechanisms:
- **macOS**: yabai focus-without-raise + stamped `SLEventPostToPid`.
- **Windows**: layered UIA `Invoke` + `PostMessage` fallback, both
z-order-independent and focus-steal-free.
- **Linux** (BETA): AT-SPI `accDoDefaultAction` when available;
XTest as a focus-stealing last resort.
- Web-app quirks — macOS specifics in `WEB_APPS.md` (Chromium / WebKit
/ Electron / Tauri, minimized-Chrome keyboard-commit caveat,
`set_value` workaround). Windows web-apps coverage lives in
`WINDOWS.md`'s "Web apps on Windows" section.
- Trajectory recording (`RECORDING.md`) — optional per-session
recording + replay for demos and regressions. macOS-only today.
- Canvas/viewport apps (Blender, Unity, GHOST, Qt, wxWidgets) —
fallback paths when the AX/UIA/AT-SPI tree is empty.

See `SKILL.md` for the main body and platform-specific carve-outs for
forbidden-list / launch / click details.

## Prerequisites

### macOS

1. **macOS 14 or newer** — the driver depends on SkyLight private SPIs
that were stabilized in Sonoma.
2. **`cua-driver` CLI + `CuaDriver.app`** — installable one-liner:
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)"
```
Or from a clone of `trycua/cua`:
```bash
cd libs/cua-driver
scripts/install-local.sh # builds + installs + symlinks for dev use
```
The driver runs as an `.app` bundle because macOS TCC grants are
tied to a stable bundle id (`com.trycua.driver`). The CLI symlink
lets Claude invoke tools via plain shell.
3. **TCC grants on `CuaDriver.app`** — **Accessibility** and
**Screen Recording** in System Settings → Privacy & Security.
Verify with:
```bash
cua-driver check_permissions
```
Both fields must be `true`. If not, the app appears in the
relevant panes of System Settings after first use; toggle it on
there.
Comment on lines +46 to +72

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Prerequisites section contradicts cross-platform claims.

The introduction (lines 3-9) describes this skill as supporting macOS, Windows, and Linux. However, the Prerequisites section (lines 39-63) presents macOS-only requirements as if they're universal:

  • Line 41: "macOS 14 or newer" (excludes Windows/Linux)
  • Lines 43-63: CuaDriver.app, TCC grants (macOS-specific)

Windows and Linux users reading this will incorrectly conclude the skill doesn't support their platform.

📋 Proposed fix to clarify platform-specific prerequisites
 ## Prerequisites
 
+### macOS Prerequisites
+
 1. **macOS 14 or newer** — the driver depends on SkyLight private SPIs
    that were stabilized in Sonoma.
 2. **`cua-driver` CLI + `CuaDriver.app`** — installable one-liner:
    ```bash
    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)"
    ```
+
+### Windows Prerequisites
+
+See `WINDOWS.md` for Windows-specific installation and permissions.
+
+### Linux Prerequisites
+
+See `LINUX.md` for Linux-specific installation and setup (BETA).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/cua-driver-rs/Skills/cua-driver-rs/README.md` around lines 39 - 63, The
Prerequisites section currently lists macOS-only requirements under the general
"Prerequisites" heading which contradicts the README intro claiming
macOS/Windows/Linux support; update the README.md Prerequisites block to clearly
separate platform-specific instructions by adding distinct subsections (e.g.,
"macOS Prerequisites", "Windows Prerequisites", "Linux Prerequisites") and move
the macOS items (macOS 14 requirement, CuaDriver.app install steps, TCC grants
and cua-driver check_permissions) under "macOS Prerequisites", then add brief
notes under "Windows Prerequisites" and "Linux Prerequisites" pointing to new
platform docs (e.g., WINDOWS.md, LINUX.md) or to BETA caveats so Windows/Linux
users aren’t misled and can find platform-specific installation/permissions
guidance.


### Windows

1. **Windows 10/11** (any edition with PowerShell 5.1+).
2. **`cua-driver` CLI (Rust port `cua-driver-rs`)** — one-liner:
```powershell
irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver-rs/scripts/install.ps1 | iex
```
No TCC equivalent; no UAC elevation required for the default
per-user install. See `WINDOWS.md` for Session 0 vs Session 1+
caveats — the daemon MUST run in an interactive session, not via
SSH-into-Windows.
3. **Verify** with `cua-driver doctor`.

### Linux

**Status: BETA.** Limited feature parity — see `LINUX.md` for what
works today. Install:

```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver-rs/scripts/install.sh)"
```

## Install

The skill is a drop-in directory. Same shape on every platform.

**Personal scope** (all Claude Code sessions on your machine):

```bash
mkdir -p ~/.claude/skills
cp -R libs/cua-driver-rs/Skills/cua-driver-rs ~/.claude/skills/
```

Or symlink if you want edits-in-place:

```bash
ln -s "$PWD/libs/cua-driver-rs/Skills/cua-driver-rs" ~/.claude/skills/cua-driver-rs
```

**Project scope** (committed alongside a specific repo):

```bash
mkdir -p .claude/skills
cp -R /path/to/cua/libs/cua-driver-rs/Skills/cua-driver-rs .claude/skills/
```
Comment on lines +96 to +118

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Correct installation paths to use cua-driver-rs directory name.

All installation commands reference Skills/cua-driver but the actual directory structure (per PR objectives and file paths) is Skills/cua-driver-rs. Users following these instructions will attempt to copy from a non-existent path or create incorrectly-named skill directories.

🔧 Proposed fix for installation paths
 **Personal scope** (all Claude Code sessions on your machine):
 
 ```bash
 mkdir -p ~/.claude/skills
-cp -R Skills/cua-driver ~/.claude/skills/
+cp -R Skills/cua-driver-rs ~/.claude/skills/

Or symlink if you want edits-in-place:

-ln -s "$PWD/Skills/cua-driver" ~/.claude/skills/cua-driver
+ln -s "$PWD/Skills/cua-driver-rs" ~/.claude/skills/cua-driver-rs

Project scope (committed alongside a specific repo):

mkdir -p .claude/skills
-cp -R /path/to/cua/libs/cua-driver/Skills/cua-driver .claude/skills/
+cp -R /path/to/cua/libs/cua-driver-rs/Skills/cua-driver-rs .claude/skills/
</details>

<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

```suggestion
## Install

The skill is two drop-in directories.

**Personal scope** (all Claude Code sessions on your machine):

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/cua-driver-rs/Skills/cua-driver-rs/README.md` around lines 65 - 87,
Update the README install paths that incorrectly reference "Skills/cua-driver"
to the correct "Skills/cua-driver-rs" variants: replace occurrences in the
Personal scope, symlink example, and Project scope so cp and ln commands point
to Skills/cua-driver-rs and create ~/.claude/skills/cua-driver-rs (or
.claude/skills/cua-driver-rs for project scope) instead of the non-existent
cua-driver directory; ensure the symlink target and copied source paths
consistently use "cua-driver-rs".


Or run the verb that does this automatically + fetches the matching
release version from GitHub:

```bash
cua-driver skills install
```

See `cua-driver skills --help` for the full subcommand list
(`install` / `update` / `uninstall` / `status` / `path`).

## Invoking the skill

Claude Code auto-invokes the skill when you ask for GUI automation —
e.g. "open the Downloads folder in Finder", "click the Save button in
Numbers", "open Calculator and compute 17×23", "navigate to
trycua.com in Edge". You can also invoke it explicitly:

```
/cua-driver-rs
```
Comment on lines +130 to +139

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Update platform references and invocation command to match cross-platform scope.

Line 92 mentions "macOS GUI automation" exclusively, contradicting the cross-platform introduction. Line 97 shows /cua-driver but the skill's frontmatter name (SKILL.md line 2) is cua-driver-rs.

📝 Proposed fixes
 ## Invoking the skill
 
-Claude Code auto-invokes the skill when you ask for macOS GUI
-automation — e.g. "open the Downloads folder in Finder", "click the
-Save button in Numbers", "navigate to trycua.com in Chrome". You can
-also invoke it explicitly:
+Claude Code auto-invokes the skill when you ask for native GUI
+automation — e.g. "open the Downloads folder in Finder" (macOS),
+"click the Save button in Notepad" (Windows), "open System Settings"
+(Linux). You can also invoke it explicitly:
 

-/cua-driver
+/cua-driver-rs

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/cua-driver-rs/Skills/cua-driver-rs/README.md` around lines 89 - 98,
Update the "Invoking the skill" section in README.md to reflect the
cross-platform scope (replace the phrase "macOS GUI automation" with a neutral
term like "GUI automation" or "cross-platform GUI automation") and change the
explicit invocation command from "/cua-driver" to match the skill frontmatter
name "cua-driver-rs" (use "/cua-driver-rs"); locate the section titled "Invoking
the skill" in README.md and the SKILL.md frontmatter name "cua-driver-rs" to
ensure the invocation string and platform description are consistent across
docs.


## Claude Code MCP compatibility mode

For normal skill-driven use, prefer the CLI or the standard MCP server. If you want Claude Code's vision/computer-use-style flow to ground on CuaDriver screenshots, register the compatibility server:

```bash
claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat
```

This mode exposes the normal CuaDriver tools and changes only `screenshot`. The compatibility screenshot requires `pid` and `window_id`, captures that window only, and establishes a window-local pixel coordinate frame. It does not call Anthropic APIs or expose Anthropic's native computer-use API tool.

Use MCP for this Claude Code vision/computer-use-style path. CLI screenshots still work as CuaDriver calls, but they do not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue.

## Files

- `SKILL.md` — main skill body, shared core + macOS-specific (~900
lines). Loaded on first invocation; stays in context.
- `WINDOWS.md` — Windows-specific carve-out (UIA, UWP, layered
click chain, Session 0, Windows web-apps). Loaded when driving
on Windows.
- `LINUX.md` — Linux-specific carve-out, BETA. Loaded when driving
on Linux.
- `WEB_APPS.md` — browser patterns on macOS (Chromium + WebKit,
Electron, Tauri, minimized-Chrome keyboard-commit caveat).
Loaded on demand from `SKILL.md`. **Note**: Windows web-apps
coverage lives in `WINDOWS.md`'s "Web apps on Windows" section.
- `RECORDING.md` — trajectory recording / replay (macOS-only
today; Windows / Linux not yet supported).
- `TESTS.md` — manual test scripts for end-to-end skill verification.

## Troubleshooting

- `cua-driver: command not found` → re-run the installer or add
`.build/CuaDriver.app/Contents/MacOS/` to `$PATH`.
- `No cached AX state for pid X window_id W` → element_index was
reused across turns, or across different windows of the same app.
Call `get_window_state({pid, window_id})` first in the same turn,
with the same window_id you're about to act against.
- Empty `tree_markdown` → `capture_mode` is set to `vision`, which
skips the AX walk by design. Flip back to the default `som`
(`cua-driver config set capture_mode som`) to get the tree.
Tiny screenshot → likely a stale window capture. See "Behavior
matrix" in SKILL.md for the full mode table.
- System-alert beep when pressing Return on a minimized Chrome
omnibox → the keyboard-commit-on-minimized limitation. Use
`set_value` on the field instead, or AX-click a Go/Submit button.
See `WEB_APPS.md`.

## Updates

The skill evolves alongside the driver. To update:

```bash
# Easiest — fetch from the matching release tag on GitHub:
cua-driver skills update

# Or, from a clone of trycua/cua:
cd /path/to/cua && git pull
# if you copied: re-copy
cp -R libs/cua-driver-rs/Skills/cua-driver-rs ~/.claude/skills/
# if you symlinked: nothing needed
```

## License

MIT. Same license as the parent `trycua/cua` repo.
Comment on lines +188 to +205

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Correct update instructions to use cua-driver-rs path.

Line 154 references the old libs/cua-driver/Skills/cua-driver path instead of libs/cua-driver-rs/Skills/cua-driver-rs.

🔧 Proposed fix
 ## Updates
 
 The skill evolves alongside the driver. To update:
 
 ```bash
 cd /path/to/cua && git pull
 # if you copied: re-copy
-cp -R libs/cua-driver/Skills/cua-driver ~/.claude/skills/
+cp -R libs/cua-driver-rs/Skills/cua-driver-rs ~/.claude/skills/
 # if you symlinked: nothing needed
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @libs/cua-driver-rs/Skills/cua-driver-rs/README.md around lines 147 - 160,
Update the README instruction that copies the skill: replace the old path string
"libs/cua-driver/Skills/cua-driver" with the correct
"libs/cua-driver-rs/Skills/cua-driver-rs" so the cp command copies from the
cua-driver-rs directory; modify the line in README.md under the "Updates"
section where the copy instruction appears to use the corrected path.


</details>

<!-- fingerprinting:phantom:triton:puma -->

<!-- This is an auto-generated comment by CodeRabbit -->

Loading
Loading