-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
feat(skills): port Skills/ to cua-driver-rs + add WINDOWS.md/LINUX.md + deploy at install time #1553
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(skills): port Skills/ to cua-driver-rs + add WINDOWS.md/LINUX.md + deploy at install time #1553
Changes from all commits
e7d3098
ae3a14d
0af9be9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| --- | ||
| name: cua-driver-rs-linux | ||
| description: Drive a native Linux app (X11 / Wayland) via the cua-driver CLI — snapshot the AT-SPI tree, click by element_index or pixel, verify via re-snapshot. Linux backend is BETA in cua-driver-rs and the no-foreground contract has open issues (see this doc). | ||
| --- | ||
|
|
||
| # cua-driver-rs — Linux | ||
|
|
||
| **Status: BETA.** The Linux backend in cua-driver-rs covers the core | ||
| tool surface (click, type_text, scroll, hotkey, screenshot, | ||
| launch_app, list_apps, list_windows, get_window_state) but several | ||
| behaviors that the macOS / Windows skills consider table-stakes are | ||
| **not yet implemented or only partially supported**: | ||
|
|
||
| - **No-foreground contract**: limited. X11's input model has no clean | ||
| per-pid event routing equivalent of macOS `CGEventPostToPSN` or | ||
| Windows `PostMessage(WM_LBUTTONDOWN)`. `XTestFakeKeyEvent` / | ||
| `XTestFakeButtonEvent` synthesize input but route to the focused | ||
| window — similar focus-stealing behavior to Windows `SendInput`, | ||
| which the Windows backend avoids by using `PostMessage` instead. | ||
| Linux has no equivalent per-window-message channel that bypasses | ||
| focus, which is why XTest's focus-stealing is the binding | ||
| limitation here. AT-SPI `accDoDefaultAction` works for accessible | ||
| elements but requires the user's accessibility bus to be running, | ||
| which is not the default on every distro. | ||
| - **Wayland support**: depends on compositor. Under GNOME-Mutter and | ||
| KDE-KWin with `org.freedesktop.portal.RemoteDesktop` enabled, some | ||
| click and key paths work. Under most other compositors, input | ||
| synthesis is denied by the security model and the tool surface | ||
| degrades to "passive" (snapshot, screenshot) only. | ||
| - **UIA / AX-tree equivalent**: AT-SPI when available, otherwise | ||
| empty. Many GTK4 / Qt6 apps populate AT-SPI lazily; agents should | ||
| expect partial trees and re-snapshot. | ||
| - **launch_app**: backed by `xdg-open` / `gtk-launch` / `dbus-send` | ||
| with display-environment scrubbing to avoid stealing the user's | ||
| workspace. Not yet equivalent to macOS `FocusRestoreGuard`. | ||
| - **Recording**: not supported. | ||
|
|
||
| See `SKILL.md` (macOS) and `WINDOWS.md` (Windows) for the full | ||
| patterns. This file will grow as the Linux backend reaches GA. For | ||
| now, **prefer macOS / Windows hosts** for agent-driven GUI tasks; use | ||
| the Linux daemon for read-only inspection (screenshot, | ||
| list_windows, get_window_state) when running on a Linux host. | ||
|
|
||
| ## Quick triage | ||
|
|
||
| If you're agent-driving on Linux and a tool call surprises you: | ||
|
|
||
| 1. Run `cua-driver doctor` — reports display server (X11 / Wayland), | ||
| AT-SPI bus reachability, XTest availability. | ||
| 2. Check `XDG_SESSION_TYPE` — `wayland` means most input synthesis | ||
| is gated by portals; `x11` means XTest works but routes via focus. | ||
| 3. If Wayland: confirm `org.freedesktop.portal.RemoteDesktop` is | ||
| present (`gdbus introspect --session --dest | ||
| org.freedesktop.portal.Desktop --object-path | ||
| /org/freedesktop/portal/desktop`). Without it, input synthesis | ||
| is denied. | ||
|
|
||
| ## Forbidden vectors | ||
|
|
||
| Same idea as macOS / Windows — don't shell out to anything that | ||
| foregrounds a target: | ||
|
|
||
| - `wmctrl -a <window>` — activates the named window. | ||
| - `xdotool windowactivate <wid>` — activates. | ||
| - `wmctrl -R <window>` — raises and activates. | ||
| - `xdotool key --window <wid> alt+Tab` — same problem as Windows | ||
| Alt+Tab. | ||
|
|
||
| Prefer cua-driver tools with explicit `window_id`. When in doubt, | ||
| ask the user. | ||
|
|
||
| ## What to expect today | ||
|
|
||
| | Intent | Status | | ||
| |---|---| | ||
| | Snapshot UIA tree | ✅ AT-SPI when available, often partial for GTK4/Qt6 | | ||
| | Pixel click | ⚠️ X11 only, focus-stealing semantics | | ||
| | Element-indexed click | ⚠️ AT-SPI `accDoDefaultAction` when supported | | ||
| | Type text | ⚠️ XTest, focus-sensitive | | ||
| | Hotkey | ⚠️ XTest, focus-sensitive | | ||
| | Screenshot full-display | ✅ X11 (xshm); ⚠️ Wayland (portal-gated) | | ||
| | Screenshot per-window | ⚠️ X11 with composite extension; Wayland TBD | | ||
| | launch_app | ⚠️ xdg-open / gtk-launch; no FocusRestoreGuard yet | | ||
| | Recording | ❌ not implemented | | ||
|
|
||
| Until Linux reaches GA, treat this doc as a planning placeholder | ||
| rather than a contract. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,205 @@ | ||
| # cua-driver-rs — Claude Code skill | ||
|
|
||
| A [Claude Code](https://code.claude.com) skill that teaches Claude to | ||
| drive native GUI apps on **macOS, Windows, and Linux** via the | ||
| [`cua-driver`](https://github.com/trycua/cua/tree/main/libs/cua-driver-rs) | ||
| CLI — snapshot an app's accessibility tree (AX on macOS, UIA on | ||
| Windows, AT-SPI on Linux), click/type/scroll by `element_index` or | ||
| pixel coords, and verify via re-snapshot. Backgrounded-first on every | ||
| platform: no focus steal, no cursor warp. | ||
|
|
||
| ## Platform reading order | ||
|
|
||
| - `SKILL.md` — shared core + macOS-specific patterns (read this | ||
| first on macOS). | ||
| - `WINDOWS.md` — Windows-specific carve-out (UIA tree, UWP / | ||
| ApplicationFrameHost, layered UIA+PostMessage click chain, | ||
| Session 0 isolation, Windows web-apps section). Read this when | ||
| driving on Windows. | ||
| - `LINUX.md` — Linux status + carve-out (X11/Wayland, AT-SPI, | ||
| BETA-level). Read this when driving on Linux. | ||
|
|
||
| ## What the skill covers | ||
|
|
||
| - The snapshot-before-AND-after invariant that keeps the agent honest | ||
| about whether an action actually landed. | ||
| - The backgrounded-click recipe that lets synthetic clicks land on | ||
| web content without raising the window or pulling the user across | ||
| virtual desktops. Per-platform mechanisms: | ||
| - **macOS**: yabai focus-without-raise + stamped `SLEventPostToPid`. | ||
| - **Windows**: layered UIA `Invoke` + `PostMessage` fallback, both | ||
| z-order-independent and focus-steal-free. | ||
| - **Linux** (BETA): AT-SPI `accDoDefaultAction` when available; | ||
| XTest as a focus-stealing last resort. | ||
| - Web-app quirks — macOS specifics in `WEB_APPS.md` (Chromium / WebKit | ||
| / Electron / Tauri, minimized-Chrome keyboard-commit caveat, | ||
| `set_value` workaround). Windows web-apps coverage lives in | ||
| `WINDOWS.md`'s "Web apps on Windows" section. | ||
| - Trajectory recording (`RECORDING.md`) — optional per-session | ||
| recording + replay for demos and regressions. macOS-only today. | ||
| - Canvas/viewport apps (Blender, Unity, GHOST, Qt, wxWidgets) — | ||
| fallback paths when the AX/UIA/AT-SPI tree is empty. | ||
|
|
||
| See `SKILL.md` for the main body and platform-specific carve-outs for | ||
| forbidden-list / launch / click details. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| ### macOS | ||
|
|
||
| 1. **macOS 14 or newer** — the driver depends on SkyLight private SPIs | ||
| that were stabilized in Sonoma. | ||
| 2. **`cua-driver` CLI + `CuaDriver.app`** — installable one-liner: | ||
| ```bash | ||
| /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" | ||
| ``` | ||
| Or from a clone of `trycua/cua`: | ||
| ```bash | ||
| cd libs/cua-driver | ||
| scripts/install-local.sh # builds + installs + symlinks for dev use | ||
| ``` | ||
| The driver runs as an `.app` bundle because macOS TCC grants are | ||
| tied to a stable bundle id (`com.trycua.driver`). The CLI symlink | ||
| lets Claude invoke tools via plain shell. | ||
| 3. **TCC grants on `CuaDriver.app`** — **Accessibility** and | ||
| **Screen Recording** in System Settings → Privacy & Security. | ||
| Verify with: | ||
| ```bash | ||
| cua-driver check_permissions | ||
| ``` | ||
| Both fields must be `true`. If not, the app appears in the | ||
| relevant panes of System Settings after first use; toggle it on | ||
| there. | ||
|
|
||
| ### Windows | ||
|
|
||
| 1. **Windows 10/11** (any edition with PowerShell 5.1+). | ||
| 2. **`cua-driver` CLI (Rust port `cua-driver-rs`)** — one-liner: | ||
| ```powershell | ||
| irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver-rs/scripts/install.ps1 | iex | ||
| ``` | ||
| No TCC equivalent; no UAC elevation required for the default | ||
| per-user install. See `WINDOWS.md` for Session 0 vs Session 1+ | ||
| caveats — the daemon MUST run in an interactive session, not via | ||
| SSH-into-Windows. | ||
| 3. **Verify** with `cua-driver doctor`. | ||
|
|
||
| ### Linux | ||
|
|
||
| **Status: BETA.** Limited feature parity — see `LINUX.md` for what | ||
| works today. Install: | ||
|
|
||
| ```bash | ||
| /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver-rs/scripts/install.sh)" | ||
| ``` | ||
|
|
||
| ## Install | ||
|
|
||
| The skill is a drop-in directory. Same shape on every platform. | ||
|
|
||
| **Personal scope** (all Claude Code sessions on your machine): | ||
|
|
||
| ```bash | ||
| mkdir -p ~/.claude/skills | ||
| cp -R libs/cua-driver-rs/Skills/cua-driver-rs ~/.claude/skills/ | ||
| ``` | ||
|
|
||
| Or symlink if you want edits-in-place: | ||
|
|
||
| ```bash | ||
| ln -s "$PWD/libs/cua-driver-rs/Skills/cua-driver-rs" ~/.claude/skills/cua-driver-rs | ||
| ``` | ||
|
|
||
| **Project scope** (committed alongside a specific repo): | ||
|
|
||
| ```bash | ||
| mkdir -p .claude/skills | ||
| cp -R /path/to/cua/libs/cua-driver-rs/Skills/cua-driver-rs .claude/skills/ | ||
| ``` | ||
|
Comment on lines
+96
to
+118
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct installation paths to use cua-driver-rs directory name. All installation commands reference 🔧 Proposed fix for installation paths **Personal scope** (all Claude Code sessions on your machine):
```bash
mkdir -p ~/.claude/skills
-cp -R Skills/cua-driver ~/.claude/skills/
+cp -R Skills/cua-driver-rs ~/.claude/skills/Or symlink if you want edits-in-place: -ln -s "$PWD/Skills/cua-driver" ~/.claude/skills/cua-driver
+ln -s "$PWD/Skills/cua-driver-rs" ~/.claude/skills/cua-driver-rsProject scope (committed alongside a specific repo): mkdir -p .claude/skills
-cp -R /path/to/cua/libs/cua-driver/Skills/cua-driver .claude/skills/
+cp -R /path/to/cua/libs/cua-driver-rs/Skills/cua-driver-rs .claude/skills/🤖 Prompt for AI Agents |
||
|
|
||
| Or run the verb that does this automatically + fetches the matching | ||
| release version from GitHub: | ||
|
|
||
| ```bash | ||
| cua-driver skills install | ||
| ``` | ||
|
|
||
| See `cua-driver skills --help` for the full subcommand list | ||
| (`install` / `update` / `uninstall` / `status` / `path`). | ||
|
|
||
| ## Invoking the skill | ||
|
|
||
| Claude Code auto-invokes the skill when you ask for GUI automation — | ||
| e.g. "open the Downloads folder in Finder", "click the Save button in | ||
| Numbers", "open Calculator and compute 17×23", "navigate to | ||
| trycua.com in Edge". You can also invoke it explicitly: | ||
|
|
||
| ``` | ||
| /cua-driver-rs | ||
| ``` | ||
|
Comment on lines
+130
to
+139
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Update platform references and invocation command to match cross-platform scope. Line 92 mentions "macOS GUI automation" exclusively, contradicting the cross-platform introduction. Line 97 shows 📝 Proposed fixes ## Invoking the skill
-Claude Code auto-invokes the skill when you ask for macOS GUI
-automation — e.g. "open the Downloads folder in Finder", "click the
-Save button in Numbers", "navigate to trycua.com in Chrome". You can
-also invoke it explicitly:
+Claude Code auto-invokes the skill when you ask for native GUI
+automation — e.g. "open the Downloads folder in Finder" (macOS),
+"click the Save button in Notepad" (Windows), "open System Settings"
+(Linux). You can also invoke it explicitly:
-/cua-driver 🤖 Prompt for AI Agents |
||
|
|
||
| ## Claude Code MCP compatibility mode | ||
|
|
||
| For normal skill-driven use, prefer the CLI or the standard MCP server. If you want Claude Code's vision/computer-use-style flow to ground on CuaDriver screenshots, register the compatibility server: | ||
|
|
||
| ```bash | ||
| claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat | ||
| ``` | ||
|
|
||
| This mode exposes the normal CuaDriver tools and changes only `screenshot`. The compatibility screenshot requires `pid` and `window_id`, captures that window only, and establishes a window-local pixel coordinate frame. It does not call Anthropic APIs or expose Anthropic's native computer-use API tool. | ||
|
|
||
| Use MCP for this Claude Code vision/computer-use-style path. CLI screenshots still work as CuaDriver calls, but they do not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue. | ||
|
|
||
| ## Files | ||
|
|
||
| - `SKILL.md` — main skill body, shared core + macOS-specific (~900 | ||
| lines). Loaded on first invocation; stays in context. | ||
| - `WINDOWS.md` — Windows-specific carve-out (UIA, UWP, layered | ||
| click chain, Session 0, Windows web-apps). Loaded when driving | ||
| on Windows. | ||
| - `LINUX.md` — Linux-specific carve-out, BETA. Loaded when driving | ||
| on Linux. | ||
| - `WEB_APPS.md` — browser patterns on macOS (Chromium + WebKit, | ||
| Electron, Tauri, minimized-Chrome keyboard-commit caveat). | ||
| Loaded on demand from `SKILL.md`. **Note**: Windows web-apps | ||
| coverage lives in `WINDOWS.md`'s "Web apps on Windows" section. | ||
| - `RECORDING.md` — trajectory recording / replay (macOS-only | ||
| today; Windows / Linux not yet supported). | ||
| - `TESTS.md` — manual test scripts for end-to-end skill verification. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| - `cua-driver: command not found` → re-run the installer or add | ||
| `.build/CuaDriver.app/Contents/MacOS/` to `$PATH`. | ||
| - `No cached AX state for pid X window_id W` → element_index was | ||
| reused across turns, or across different windows of the same app. | ||
| Call `get_window_state({pid, window_id})` first in the same turn, | ||
| with the same window_id you're about to act against. | ||
| - Empty `tree_markdown` → `capture_mode` is set to `vision`, which | ||
| skips the AX walk by design. Flip back to the default `som` | ||
| (`cua-driver config set capture_mode som`) to get the tree. | ||
| Tiny screenshot → likely a stale window capture. See "Behavior | ||
| matrix" in SKILL.md for the full mode table. | ||
| - System-alert beep when pressing Return on a minimized Chrome | ||
| omnibox → the keyboard-commit-on-minimized limitation. Use | ||
| `set_value` on the field instead, or AX-click a Go/Submit button. | ||
| See `WEB_APPS.md`. | ||
|
|
||
| ## Updates | ||
|
|
||
| The skill evolves alongside the driver. To update: | ||
|
|
||
| ```bash | ||
| # Easiest — fetch from the matching release tag on GitHub: | ||
| cua-driver skills update | ||
|
|
||
| # Or, from a clone of trycua/cua: | ||
| cd /path/to/cua && git pull | ||
| # if you copied: re-copy | ||
| cp -R libs/cua-driver-rs/Skills/cua-driver-rs ~/.claude/skills/ | ||
| # if you symlinked: nothing needed | ||
| ``` | ||
|
|
||
| ## License | ||
|
|
||
| MIT. Same license as the parent `trycua/cua` repo. | ||
|
Comment on lines
+188
to
+205
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct update instructions to use cua-driver-rs path. Line 154 references the old 🔧 Proposed fix ## Updates
The skill evolves alongside the driver. To update:
```bash
cd /path/to/cua && git pull
# if you copied: re-copy
-cp -R libs/cua-driver/Skills/cua-driver ~/.claude/skills/
+cp -R libs/cua-driver-rs/Skills/cua-driver-rs ~/.claude/skills/
# if you symlinked: nothing neededVerify each finding against current code. Fix only still-valid issues, skip the In |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prerequisites section contradicts cross-platform claims.
The introduction (lines 3-9) describes this skill as supporting macOS, Windows, and Linux. However, the Prerequisites section (lines 39-63) presents macOS-only requirements as if they're universal:
Windows and Linux users reading this will incorrectly conclude the skill doesn't support their platform.
📋 Proposed fix to clarify platform-specific prerequisites
🤖 Prompt for AI Agents