diff --git a/.github/workflows/cd-rust-cua-driver.yml b/.github/workflows/cd-rust-cua-driver.yml index fd86dde53b..47d159dd04 100644 --- a/.github/workflows/cd-rust-cua-driver.yml +++ b/.github/workflows/cd-rust-cua-driver.yml @@ -446,14 +446,15 @@ jobs: cp libs/cua-driver/scripts/install.ps1 release-upload/install.ps1 cp libs/cua-driver/scripts/_install-rust.sh release-upload/_install-rust.sh - # Uninstaller scripts — mirror the install side. The canonical - # uninstall.sh dispatches to the Swift uninstall path by default - # and to _uninstall-rust.sh on --experimental-rust / non-macOS - # auto-detect. uninstall.ps1 is the canonical Windows - # uninstaller (cua-driver-rs only — no Swift Windows build). + # Uninstaller scripts — single canonical file per shell (no + # `_uninstall-rust.sh` helper). uninstall.sh handles both Swift + # (macOS default) and Rust (Linux auto-select, macOS opt-in via + # --experimental-rust / --backend=rust) inline. uninstall.ps1 + # is the Windows uninstaller (cua-driver-rs only — no Swift + # Windows build) and self-elevates via UAC when an -AutoStart + # install (RunLevel=Highest task) needs to be torn down. cp libs/cua-driver/scripts/uninstall.sh release-upload/uninstall.sh cp libs/cua-driver/scripts/uninstall.ps1 release-upload/uninstall.ps1 - cp libs/cua-driver/scripts/_uninstall-rust.sh release-upload/_uninstall-rust.sh # Skill pack — single platform-agnostic tarball fetched by # `cua-driver skills install`. The .md files are identical across diff --git a/docs/content/docs/cua-driver/guide/getting-started/installation.mdx b/docs/content/docs/cua-driver/guide/getting-started/installation.mdx index 5e87086e59..33cabec9e3 100644 --- a/docs/content/docs/cua-driver/guide/getting-started/installation.mdx +++ b/docs/content/docs/cua-driver/guide/getting-started/installation.mdx @@ -607,7 +607,9 @@ One canonical uninstall URL per platform mirrors the install side. Flag handling | Linux | Rust port (auto-detected — Swift binary is macOS-only) | `bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/uninstall.sh)"` | | Windows | Rust port | `irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/uninstall.ps1 \| iex` | -`uninstall.sh` parses the same `--experimental-rust` / `--backend=rust` / `--backend=swift` flag set as `install.sh` and delegates to a colocated private helper, `_uninstall-rust.sh` (under `libs/cua-driver/scripts/`), when the Rust backend is selected. On Linux (and any non-macOS `bash` host) it auto-detects and runs the Rust uninstall without any flag. +`uninstall.sh` parses the same `--experimental-rust` / `--backend=rust` / `--backend=swift` flag set as `install.sh` and handles both backends inline — single file, no `_uninstall-rust.sh` helper (mirrors the one-file-per-shell shape of `uninstall.ps1`). On Linux (and any non-macOS `bash` host) it auto-detects and runs the Rust uninstall without any flag. + +`uninstall.ps1` self-elevates via UAC when it detects an `-AutoStart` install — the `cua-driver-serve` Scheduled Task is registered at `RunLevel=Highest` (see [Autostart](../autostart)), so the daemon spawned by it runs at High IL and a non-elevated process (even the same user that installed it) can't terminate it or delete the task. The script detects either condition upfront and re-spawns itself elevated; the in-place path runs without UAC when no autostart artifacts are present. **What each uninstall removes** @@ -616,10 +618,10 @@ One canonical uninstall URL per platform mirrors the install side. Flag handling | Swift (macOS) | `~/.local/bin/cua-driver`, `/usr/local/bin/cua-driver` (legacy) | `/Applications/CuaDriver.app`, `~/.cua-driver`, `~/Library/Application Support/Cua Driver`, `~/Library/Caches/cua-driver` | `~/Library/LaunchAgents/com.trycua.cua_driver_updater.plist` (legacy ≤ 0.0.5) | `cua-driver` symlinks under `~/.claude/skills`, `~/.agents/skills`, `~/.openclaw/skills`, `~/.config/opencode/skills` | Scrubbed from `~/.claude.json` | | Rust — macOS | `~/.local/bin/cua-driver` *(only when it resolves into `CuaDriver.app` and `~/.cua-driver-rs/` exists)* | `/Applications/CuaDriver.app` (current), `/Applications/CuaDriverRs.app` (legacy, pre-rename), `~/.cua-driver-rs/` (entire tree) | `~/Library/LaunchAgents/com.trycua.cua-driver-rs.plist` | `cua-driver-rs` symlinks under the same agent dirs | Scrubbed from `~/.claude.json` | | Rust — Linux | `~/.local/bin/cua-driver` *(when it resolves into `~/.cua-driver-rs/`)* | `~/.cua-driver-rs/` (entire tree) | `~/.config/systemd/user/cua-driver-rs.service` (stop + disable + remove) | Same as above | Scrubbed from `~/.claude.json` | -| Rust — Windows | `%LOCALAPPDATA%\Programs\trycua\cua-driver-rs\bin` *(directory junction)* | `%USERPROFILE%\.cua-driver-rs\` (entire tree, including `packages\current` junction) | Scheduled Task `cua-driver-serve` (`schtasks /Delete`) | Junctions under `%USERPROFILE%\.claude\skills`, `.agents\skills`, `.openclaw\skills`, `%APPDATA%\opencode\skills` | Not auto-edited — closing message prints `claude mcp remove cua-driver-rs` | +| Rust — Windows | `%LOCALAPPDATA%\Programs\Cua\cua-driver\bin` *(directory junction; legacy `Programs\trycua\cua-driver-rs\bin` also swept)* | `%USERPROFILE%\.cua-driver\` (entire tree, including `packages\current` junction; legacy `~\.cua-driver-rs\` also swept) | Scheduled Task `cua-driver-serve` (`schtasks /Delete`; needs UAC — script self-elevates) | Junctions under `%USERPROFILE%\.claude\skills`, `.agents\skills`, `.openclaw\skills`, `%APPDATA%\opencode\skills` | Not auto-edited — closing message prints `claude mcp remove cua-driver-rs` | -**Safety invariants** — every uninstall script (`uninstall.sh`, `_uninstall-rust.sh`, `uninstall.ps1`) refuses to clobber a real directory at a path where the matching installer could have only created a symlink / junction (Linux/macOS use `[[ -L ]]`; Windows checks `IO_REPARSE_TAG_MOUNT_POINT`). A user who hand-managed any of these dirs keeps theirs untouched. Re-running an uninstall on an already-clean system prints `nothing to remove` per item — never errors. +**Safety invariants** — every uninstall script (`uninstall.sh`, `uninstall.ps1`) refuses to clobber a real directory at a path where the matching installer could have only created a symlink / junction (Linux/macOS use `[[ -L ]]`; Windows checks `IO_REPARSE_TAG_MOUNT_POINT`). A user who hand-managed any of these dirs keeps theirs untouched. Re-running an uninstall on an already-clean system prints `nothing to remove` per item — never errors. diff --git a/libs/cua-driver-rs/Cargo.lock b/libs/cua-driver-rs/Cargo.lock index d50f03a564..bd21ee23c8 100644 --- a/libs/cua-driver-rs/Cargo.lock +++ b/libs/cua-driver-rs/Cargo.lock @@ -242,7 +242,7 @@ dependencies = [ [[package]] name = "cua-driver" -version = "0.2.7" +version = "0.2.18" dependencies = [ "anyhow", "async-trait", @@ -270,7 +270,7 @@ dependencies = [ [[package]] name = "cua-driver-uia" -version = "0.2.7" +version = "0.2.18" dependencies = [ "anyhow", "embed-manifest", @@ -285,7 +285,7 @@ dependencies = [ [[package]] name = "cursor-overlay" -version = "0.2.7" +version = "0.2.18" dependencies = [ "anyhow", "image", @@ -428,7 +428,7 @@ checksum = "98de4bbd547a563b716d8dfa9aad1cb19bfab00f4fa09a6a4ed21dbcf44ce9c4" [[package]] name = "focus-monitor-win" -version = "0.2.7" +version = "0.2.18" dependencies = [ "windows", ] @@ -865,7 +865,7 @@ dependencies = [ [[package]] name = "mcp-server" -version = "0.2.7" +version = "0.2.18" dependencies = [ "anyhow", "async-trait", @@ -1151,7 +1151,7 @@ checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" [[package]] name = "platform-linux" -version = "0.2.7" +version = "0.2.18" dependencies = [ "anyhow", "async-trait", @@ -1171,7 +1171,7 @@ dependencies = [ [[package]] name = "platform-macos" -version = "0.2.7" +version = "0.2.18" dependencies = [ "anyhow", "async-trait", @@ -1201,7 +1201,7 @@ dependencies = [ [[package]] name = "platform-windows" -version = "0.2.7" +version = "0.2.18" dependencies = [ "anyhow", "async-trait", diff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md b/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md new file mode 100644 index 0000000000..cf2734ce64 --- /dev/null +++ b/libs/cua-driver-rs/Skills/cua-driver-rs/MACOS.md @@ -0,0 +1,475 @@ +# cua-driver-rs — macOS specifics + +This file holds macOS-only material that used to live in `SKILL.md`. +The cross-platform core (snapshot invariant, CLI/MCP defaults, +behavior matrix, canonical loop, pixel-click contract, common error +patterns) is in `SKILL.md`. Read this in addition to `SKILL.md` when +you're driving an app on macOS. + +## The no-foreground contract + +**The user's frontmost app MUST NOT change.** This is the whole +reason cua-driver exists. Users pay for the right to keep typing in +their editor while an agent drives another app in the background. +Violate this rule and every other nice property the driver gives +you (no cursor warp, no Space switch, no window raise) stops +mattering — you just shipped the Accessibility Inspector with extra +steps. + +Before running any shell command, ask: **"does this raise, +activate, foreground, or make-key any app?"** If yes, don't run it. +Every one of the commands below activates the target on macOS and +is therefore forbidden unless the user **explicitly** asked for +frontmost state: + +- **Every form of the `open` CLI — `open -a `, `open -b + `, `open `, `open `, `open + ` — always activates.** macOS routes all forms through + LaunchServices, which unhides and foregrounds the target + regardless of whether you passed an app name, a bundle id, a + document, a URL, or the bundle path itself. The activation + happens even when the only intent was "start the process." + **Never use `open` for any app launch.** This includes launching + a just-built .app from a local build dir (e.g. `open + build/Build/Products/Debug/MyApp.app`) — resolve the + `CFBundleIdentifier` from `Info.plist` and use `launch_app` + with that id. See "The narrow carve-out" below for why + `launch_app` is safe even when the app internally calls + `NSApp.activate`. +- `osascript -e 'tell application "X" to activate'` — + activates by design. Same for `... to open `, + `... to launch`, and anything with `activate` in the tell block. +- `osascript -e 'tell application "System Events" to ... frontmost'` + in a mutating form (setting `frontmost` rather than reading it). +- AppleScript files that invoke `activate`, `launch`, or `open` + against the target app. +- `cliclick` (moves the user's real cursor to the target coords + before clicking — a focus-steal-equivalent even if the app's + window state is unchanged). +- `CGEventPost` with `cghidEventTap` targeting a coordinate over + a different app's window (warps the cursor, possibly activates + on hit). +- `AppleScriptTask`, `NSAppleScript`, `Process` wrapping `osascript` + that contains any of the above. +- `NSRunningApplication.activate(options:)` called from your own + helper binary — same class. +- Dock clicks and any `open` invocation (see the first bullet — + every form of `open` goes through LaunchServices which + activates, full stop). +- **Keyboard shortcuts that semantically mean "focus here" — + most notably Chrome / Safari / Arc's `⌘L` (focus omnibox) and + Finder's `⌘⇧G` (Go to Folder).** These aren't pure key events — + the receiving app interprets "user wants to type here" as + activation intent and raises its window to be key. Even when + delivered to a backgrounded pid via `hotkey`, the downstream app + pulls focus. **For omnibox navigation specifically**, the correct + path is `launch_app({bundle_id: "com.google.Chrome", urls: + ["https://…"]})` — no omnibox dance, no `⌘L`, no focus-steal. Do + NOT try `set_value` on the omnibox: Chrome's commit logic requires + a "user-typed" signal that neither an AX value write nor + `CGEvent.postToPid` keystrokes supply from a backgrounded pid — + the URL lands in the field but Return fires as a no-op. See + `WEB_APPS.md` → "Navigate to a URL" for the full pattern. The + general principle: a shortcut that says "put my cursor inside this + app" is a focus-steal; a shortcut that says "do this thing" (copy, + save, quit) is fine. +- **Tab-switching shortcuts in browsers (`⌘1..⌘9`, `⌘]`, `⌘[`, + `⌘⇧[`, `⌘⇧]`) are visibly disruptive even when delivered to a + backgrounded pid.** The app's key handler processes the shortcut, + the window re-renders the new tab's content, the user sees their + tabs flipping. There is no AX-only workaround: page content (HTML, + form state, `AXWebArea`) populates only for the focused tab; + inspecting a background tab requires activating it, which is the + visible flip. Observed with Dia; the same mechanic applies to every + Chromium-family browser (Chrome, Arc, Brave, Edge). + + **Prefer the windows-over-tabs pattern**: for each URL you need to + drive backgrounded, use `launch_app({bundle_id, urls: [url]})` — + browsers open each URL in a new **window**. Each window has its own + `window_id`, its own AX tree, and can be inspected / interacted with + via `element_index` without activating or switching anything. Tabs + are a UX grouping for humans; cua-driver workflows should default to + windows. See `WEB_APPS.md` → "Tabs vs windows" for the full pattern. + + Tab-title enumeration (read-only) IS safe — walk a window's toolbar + AX tree for `AXTab` / `AXRadioButton` children and read their + `AXTitle`s. Tab switching (activating one) is not. + +Reading frontmost state is fine (`osascript -e 'tell application +"System Events" to get name of first application process whose +frontmost is true'`). Mutating it is not. + +**Corollary — the AXMenuBar rule.** `AXMenuBarItem` + AXPick +dispatches at the AX layer regardless of which app is frontmost, +but macOS's on-screen menu bar always belongs to the frontmost +app. If you drive a *backgrounded* app's menu bar, the AX call +succeeds but the viewer sees the dispatch rendered over the +*frontmost* app's menu bar — confusing in any observed session and +routinely a silent no-op too, because action menu items go +`DISABLED` when their owning app isn't the key window. **So: only +use menu-bar navigation when the target is already frontmost.** For +backgrounded targets, read state via in-window AX (window title, +toolbar `AXStaticText`) and dispatch via in-window `element_index` +or pixel clicks — both paths are frontmost-insensitive. Full +rationale in "Navigating native menu bars" below. + +**"Open \" in user speech means launch, not activate.** +`cua-driver launch_app` is the one correct path for process +startup — it's idempotent (no-op on a running app), returns the +pid, and has an internal `FocusRestoreGuard` that catches +`NSApp.activate(ignoringOtherApps:)` calls the target makes during +`application(_:open:)` and clobbers the frontmost back to what it +was before the launch. That guard is why `launch_app` with `urls` +(e.g. `{"bundle_id": "com.colliderli.iina", "urls": ["~/video.mp4"]}`) +is safe even for apps that normally foreground on media-load +(Chrome, Electron, media players). + +## Intent → tool mapping (macOS-specific) + +| Intent | Use | Don't use | +|---|---|---| +| Open / launch an app | `launch_app({bundle_id})` or `launch_app({bundle_id, urls:[...]})` | `open -a`, `osascript 'tell app … to launch/activate/open'` | +| Find a pid | `list_apps` or `launch_app`'s return | `pgrep`, `ps`, `osascript frontmost` | +| Enumerate an app's windows | `list_windows({pid})` — or read the `windows` array `launch_app` already returns | `osascript 'every window of app …'` | +| Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open ` | +| Drag / drag-and-drop / marquee select | `drag({pid, from_x, from_y, to_x, to_y})` (pixel-only — macOS AX has no semantic drag) | `cliclick dd:`, `osascript drag` | +| Screenshot | `screenshot` or the PNG in `get_window_state` | `screencapture` | +| Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` | +| Hand a file/URL to an app | `launch_app({bundle_id, urls:[]})` | `open -a `, `open ` | + +### The narrow carve-out + +The **only** legitimate use of `osascript -e 'tell app X to +activate'` is when the user **explicitly** asked for frontmost +state ("bring Chrome to the front", "make it frontmost", "I want +to see X"). Reaching for it because a tool call returned something +confusing is wrong — that's the skill's classic foot-in-the-door +failure mode and it steals focus every time. + +When a cua-driver call surprises you, diagnose cua-driver first: + +- **Tiny screenshot / empty `tree_markdown`?** Check + `cua-driver get_config` → `capture_mode`. Default `"som"` returns + both the AX tree and screenshot. `"vision"` omits the AX tree + (PNG only), `"ax"` omits the PNG. If a snapshot lacks a tree, + `capture_mode` is almost certainly `"vision"` — either reason + purely from the PNG or flip to `"som"` / `"ax"` via `set_config`. +- **`has_screenshot: false`?** The window capture failed (transient + race against a close, or the window has no backing store yet). + Re-snapshot; if persistent, pick a different `window_id` via + `list_windows`. +- **`Invalid element_index` / `No cached AX state`?** You either + skipped `get_window_state` this turn or passed a different + `window_id` than the one the snapshot cached against. The cache + is keyed on `(pid, window_id)` — indices don't carry across + windows of the same app. Re-snapshot with the same window_id + you're about to click in. +- **Sparse Chromium AX tree?** Retry `get_window_state` once — the + tree populates on second call. + +Only after those are ruled out, and only if the user's action +genuinely needs frontmost state, fall through to the activate +fallback. Always name the focus steal in your response ("I'll +briefly bring Chrome to the front because …"). + +### Self-check pattern + +Before every `Bash` call whose command line touches any macOS app +(launching, opening, clicking, typing, scripting, screenshotting), +run the self-check: + +1. **Does this command foreground the target?** If yes — stop and + translate to the cua-driver equivalent from the mapping table. +2. **Does this command move the user's real cursor?** (`cliclick`, + any `CGEventPost` at `cghidEventTap` over another app's window). + If yes — stop; use `click({pid, x, y})` which routes per-pid + via SkyLight and never warps the cursor. +3. **Does this command bypass cua-driver entirely?** (`osascript` + mutating GUI state, AppleScript files, external helpers.) If + yes — stop; find the cua-driver tool that does the intent. + +If all three are "no," the command is safe. If you can't answer, +default to stop and ask rather than proceed. A single `open -a` +run by accident kills the demo, the trust, and the user's in-flight +editor state. + +## Prerequisites — macOS + +1. `cua-driver` is on `$PATH` (`which cua-driver`). If not, point the + user at `scripts/install-local.sh` and stop. +2. Run `cua-driver check_permissions` (with the daemon up — see step 3). + The default behavior also raises the system permission dialogs for + any missing grants, so the user can grant on the spot. If either + grant still reads `false` after that (user dismissed the dialog), + tell them to open System Settings → Privacy & Security and grant + Accessibility and Screen Recording to `CuaDriver.app`, then stop. + Pass `'{"prompt":false}'` for a purely read-only status check that + won't steal focus. +3. Start the daemon with `open -n -g -a CuaDriver --args serve` (the + recommended form — goes through LaunchServices so TCC attributes + the process to CuaDriver.app). `cua-driver serve &` also works; + the CLI auto-relaunches through `open -n -g -a CuaDriver` when it + detects a wrong-TCC context (any IDE-spawned shell: Claude Code, + Cursor, VS Code, Conductor). Verify with `cua-driver status`. + +## Resolve target pid — always via `launch_app` + +**Always start with `launch_app`**, whether or not the target is already +running. It's idempotent (relaunching returns the existing pid with no +side effects) and gives you the pid in one call — no `list_apps` hop. + +- `launch_app({bundle_id: "com.apple.finder"})` — preferred, unambiguous. +- `launch_app({name: "Calculator"})` — when bundle_id isn't known. + +`launch_app` is a **hidden-launch primitive by design** — that's the +entire point of cua-driver: agents drive apps in the background while +the user keeps typing in their real foreground app. The target's +window is initialized (AX tree fully populated, clickable via +`element_index`, the pid appears in `list_apps`) but not drawn on +screen. The driver never activates or unhides apps on its own; that +would violate the no-foreground contract the whole driver exists to +protect. + +If the user explicitly wants the window visible (usually for a demo +or recording), they unhide it themselves — Dock click, Cmd-Tab, or +Spotlight. Do not reach for `open` / `osascript activate` as a +shortcut to make the window visible; those paths break the backgrounded +invariant on every call, not just the call that "needed" the +foreground. Say out loud what the user needs to do ("click the +Todo app in your Dock to bring it forward") and let them do it. + +Never shell out to **any** form of `open` (including `open +` for a just-built binary — resolve the bundle id +from `Info.plist` and use `launch_app` with that), `osascript 'tell +app … to launch/open'`, or similar. Those paths activate the target, +bypass the driver's focus-restore guard, and require a Bash +permission prompt the agent loop shouldn't be burning on app launch. + +## Pixel-click dispatch (macOS) + +The pixel click is routed through SkyLight's per-pid event path +(`SLEventPostToPid`), not the system HID stream. The dispatch recipe +is the backgrounded "noraise" sequence: yabai's focus-without-raise +SLPS event records followed by an off-screen user-activation primer +and the real click. The target app becomes AppKit-active for event +routing but its window does **not** rise to the front of the +z-stack, and macOS's "switch to Space with windows for app" follow +is suppressed. Full mechanics in +`Sources/CuaDriverCore/Input/MouseInput.swift` (`clickViaAuthSignedPost`) +and the companion `FocusWithoutRaise.swift`. + +### Canvases, viewports, games (Blender, Unity, GHOST, Qt, wxWidgets) + +Apps whose main surface is an OpenGL / Metal / Qt / wxWidgets +viewport expose **no useful AX tree** — the whole surface is one +opaque `AXGroup` or `AXWindow` from AX's perspective. Per-pid event +paths (`SLEventPostToPid`, `CGEvent.postToPid`) are filtered by the +viewport's own event-source check and silently dropped — the event +loop wants "real HID origin". + +The working pattern: + +1. Bring the target frontmost (a brief `osascript activate` is + acceptable here — this is the carve-out the skill's osascript + gate allows). +2. `CGEvent.post(tap: .cghidEventTap)` with a leading `mouseMoved` + event (~30 ms before the click). `cua-driver click` when the + target is frontmost automatically takes this path. +3. Accept that the real cursor visibly moves — `cghidEventTap` is + the system HID stream, the cursor warps to the click point. + +There is no backgrounded path that reaches these apps today. + +### Known pixel-click limits + +- **Chromium `