Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/content/docs/cua-driver/reference/mcp-tools.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -521,7 +521,9 @@ Update cua-driver-rs configuration. Changes to capture_mode and max_image_dimens

### set_agent_cursor_enabled

Show or hide the agent cursor overlay for a cursor instance. With no `cursor_id`, this targets the calling session's own cursor (see the per-session note above).
Show or hide the agent cursor overlay for a cursor instance. The overlay is **ON by default** and each MCP session automatically owns its own cursor — you do not need to call this to make the cursor appear; use it only to hide (`enabled:false`) or re-show (`enabled:true`) it. With no `cursor_id`, this targets the calling session's own cursor (see the per-session note above).

> **Visibility caveat (AX runs).** On a pure accessibility-action run (clicking by `element_index`), the session cursor seeds on-screen and pulses on its very first action rather than playing a long glide, so it is easy to miss in a screen recording. For a clearly *gliding* cursor in a demo, issue a pixel `click({pid,x,y})` or a `move_agent_cursor` first to put the cursor on-screen; subsequent AX clicks then glide normally.

**Arguments:**

Expand Down
42 changes: 32 additions & 10 deletions libs/cua-driver/rust/Skills/cua-driver/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,12 +124,31 @@ cua-driver stop
## Agent cursor overlay

Visual cursor overlay for demos and screen recordings. Default:
enabled. Toggle with `cua-driver set_agent_cursor_enabled
'{"enabled":true|false}'`. A triangle pointer Bezier-glides to each
click target, ring-ripples on landing, idle-hides after ~1.5s.
Motion knobs: `set_agent_cursor_motion` takes any subset of
`start_handle`, `end_handle`, `arc_size`, `arc_flow`, `spring` —
tuneable at runtime, persisted to config.
enabled — you do NOT need to enable it. Toggle with
`cua-driver set_agent_cursor_enabled '{"enabled":true|false}'` only to
hide or re-show it. A triangle pointer Bezier-glides to each click
target, ring-ripples on landing, idle-hides after ~1.5s. Motion knobs:
`set_agent_cursor_motion` takes any subset of `start_handle`,
`end_handle`, `arc_size`, `arc_flow`, `spring` — tuneable at runtime,
persisted to config.

**Per-session cursors.** Each MCP session automatically owns its own
cursor, keyed by the session's id (the proxy mints one session id per
MCP connection and the daemon scopes the cursor, config overrides, and
recording to it). You normally pass nothing — the session key is wired
through for you. Pass an explicit `cursor_id` only to *deliberately
share* one cursor across sessions. When a session ends (the MCP client
disconnects) its cursor is removed automatically.

**Visibility caveat (AX runs).** On a pure accessibility-action run
(clicking by `element_index`), the first action **seeds the cursor
on-screen a short distance from the target and plays a brief glide +
pulse** — not the long Bezier sweep a cursor already on-screen would
trace from its previous spot. It's subtle and easy to miss in a
recording. If you want a clearly *gliding* cursor for a demo or screen
recording, do a pixel click (`click({pid,x,y})`) or a `move_agent_cursor`
first to put the cursor on-screen; subsequent AX actions then glide the
full path normally.

Requires the daemon process's UI runloop, which `cua-driver serve` /
`mcp` bootstraps. One-shot CLI invocations skip the overlay entirely.
Expand Down Expand Up @@ -474,10 +493,13 @@ respective companion files.
is for visual disambiguation, not coordinates. Use the
`element_index`.
- **Prefer accessibility actions over pixels.** `click({pid, x, y})`
works for canvas / WebView regions, but it lands blindly and skips
the agent-cursor overlay. Exhaust accessibility paths (menu bars,
cmd-k palettes, toolbar items, keyboard shortcuts) before dropping
to coordinates.
works for canvas / WebView regions, but it lands blindly on raw
coordinates. Exhaust accessibility paths (menu bars, cmd-k palettes,
toolbar items, keyboard shortcuts) before dropping to coordinates.
(The AX path does **not** skip the agent-cursor overlay — it seeds and
pulses the session cursor and draws a focus rect on the targeted
element; it just doesn't play a long glide on the very first action.
See "Agent cursor overlay" for the demo-recording caveat.)
- **Never** drive destructive actions (delete files, close unsaved
documents, send messages, submit forms) without explicit user
intent for that specific destructive step.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ Workflow per turn:
4. click/type_text/press_key using element_index from step 3
5. get_window_state(pid, window_id) again → verify the action landed

Agent cursor: set_agent_cursor_* tools visualise where the agent is acting without affecting the real mouse pointer.
Agent cursor: a per-session overlay cursor (ON by default, one per MCP session) visualises where the agent is acting without affecting the real mouse pointer; it is removed when the session ends. set_agent_cursor_* tools hide/show/customise it. Note: a pure accessibility-action (element_index) click snaps the cursor with a brief pulse on its very first action rather than a long glide, so it can be easy to miss — issue a pixel click or move_agent_cursor first for a visibly gliding demo/recording.

If a `cua-driver` skill is loaded in your harness (Claude Code / Codex / OpenClaw / OpenCode dirs), prefer its detailed workflow — SKILL.md plus {platform_skill_pointer}. Install with `cua-driver skills install` if not yet present."#
)
Expand Down
62 changes: 58 additions & 4 deletions libs/cua-driver/rust/crates/cua-driver/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,13 @@ pub enum Command {
/// (checked inside the gate itself), so the flag is only one of
/// two opt-out signals.
no_permissions_gate: bool,
/// True when `--claude-code-computer-use-compat` is on argv. The MCP
/// proxy forwards this flag to the daemon it auto-launches (see
/// `launch_daemon_and_wait`) so the proxy path registers the compat
/// `screenshot` surface, not just the in-process path. Without it the
/// flag was a no-op for `cua-driver mcp --claude-code-computer-use-compat`,
/// which always routes through the proxy on an installed bundle.
claude_code_compat: bool,
},
Stop { socket: Option<String> },
Status { socket: Option<String> },
Expand Down Expand Up @@ -182,6 +189,28 @@ pub fn parse_command() -> Command {
println!(" --no-daemon-relaunch Stay in-process; skip auto-launching the CuaDriver daemon.");
println!(" Also: CUA_DRIVER_RS_MCP_NO_RELAUNCH=1");
println!(" --socket <path> Override the daemon UDS path used by the proxy fallback.");
println!(" --claude-code-computer-use-compat");
println!(" Select the Claude Code computer-use compat surface.");
println!(" Now forwarded to the proxy-launched daemon (was a no-op");
println!(" on the proxy path — the path you actually run — because");
println!(" the daemon hardcoded compat=false). Note: the compat");
println!(" screenshot tool itself was removed in #1692, so the flag");
println!(" has no tool-surface effect today; the wiring is in place");
println!(" for any future compat-gated tool.");
println!();
println!("agent cursor overlay (serve / mcp only — needs the daemon UI runloop):");
println!(" The overlay is ON by default: every MCP session automatically gets its own");
println!(" cursor (keyed by session id) that shows where the agent acts without moving the");
println!(" real pointer. It is removed when the session ends. A pure accessibility (AX)");
println!(" action snaps the cursor with a brief pulse on its first action instead of a long");
println!(" glide, so it can be easy to miss — do a pixel click or move_agent_cursor first");
println!(" for a visibly gliding demo. These flags tune the overlay on `serve`/`mcp`:");
println!(" --no-overlay Disable the cursor overlay entirely for this daemon.");
println!(" --cursor-id <id> Name the default cursor instance (default: 'default').");
println!(" --cursor-icon <path> Use a custom PNG cursor icon.");
println!(" --cursor-palette <name> Pick a built-in colour palette for the cursor.");
println!(" (These are no-ops for one-shot CLI calls like `cua-driver call` — the overlay");
println!(" needs the long-lived AppKit runloop that only `serve` / `mcp` keep alive.)");
println!();
println!("doctor options:");
println!(" --json Emit the probe report as JSON for scripting.");
Expand Down Expand Up @@ -260,6 +289,7 @@ pub fn parse_command() -> Command {
socket,
// Bare flag — present anywhere on argv counts as "skip the gate".
no_permissions_gate: args.iter().any(|a| a == "--no-permissions-gate"),
claude_code_compat,
},
Some("stop") => Command::Stop { socket },
Some("status") => Command::Status { socket },
Expand Down Expand Up @@ -562,7 +592,11 @@ pub fn should_use_daemon_proxy(no_daemon_relaunch: bool) -> bool {
/// `waitForDaemon`. Split into one Rust function because we don't
/// need the post-launch probe separation Swift has.
#[cfg(target_os = "macos")]
pub fn launch_daemon_and_wait(socket_path: &str, timeout_secs: u64) -> anyhow::Result<()> {
pub fn launch_daemon_and_wait(
socket_path: &str,
timeout_secs: u64,
claude_code_compat: bool,
) -> anyhow::Result<()> {
use std::process::{Command as Cmd, Stdio};
use std::time::{Duration, Instant};

Expand All @@ -579,6 +613,20 @@ pub fn launch_daemon_and_wait(socket_path: &str, timeout_secs: u64) -> anyhow::R
open_args.push("--socket");
open_args.push(socket_path);
}
// Thread the Claude-Code compat flag through to the daemon. Without this
// the proxy-spawned daemon always called build_macos_registry() (compat
// hardcoded false), so `cua-driver mcp --claude-code-computer-use-compat`
// SILENTLY DROPPED the flag on the proxy path — the path users actually
// run on an installed bundle. Today this is latent: the compat screenshot
// tool was removed in #1692, so `register_all(compat)` ignores the flag and
// the served surface is identical either way. But the flag was being lost
// before reaching the daemon at all, so the moment any compat-gated tool is
// re-introduced the proxy path would not honour it. This makes the flag
// travel end-to-end. Only honoured on a freshly-launched daemon — a
// pre-existing daemon keeps whatever surface it launched with.
if claude_code_compat {
open_args.push("--claude-code-computer-use-compat");
}

let status = Cmd::new("/usr/bin/open")
// `-n` forces a new instance: CuaDriver.app might already be
Expand Down Expand Up @@ -630,7 +678,10 @@ pub fn launch_daemon_and_wait(socket_path: &str, timeout_secs: u64) -> anyhow::R
/// `open` if needed), then `crate::proxy::run_proxy` against its
/// socket. Builds its own tokio runtime — same shape as the other
/// `run_*` helpers in this file that own their event loop.
pub fn run_mcp_via_daemon_proxy(socket: Option<String>) -> anyhow::Result<()> {
pub fn run_mcp_via_daemon_proxy(
socket: Option<String>,
claude_code_compat: bool,
) -> anyhow::Result<()> {
// Windows: prefer the uiAccess'd worker pipe over the regular daemon pipe
// when both are running, so MCP tool calls land in a process that can
// bypass UIPI for UWP apps. The protocol on both pipes is identical so
Expand Down Expand Up @@ -678,8 +729,10 @@ pub fn run_mcp_via_daemon_proxy(socket: Option<String>) -> anyhow::Result<()> {
auto-launching the daemon via `open -n -g -a CuaDriver --args serve{socket_suffix}` \
and proxying MCP requests through it. Pass --no-daemon-relaunch to stay in-process."
);
launch_daemon_and_wait(&socket_path, 10)?;
launch_daemon_and_wait(&socket_path, 10, claude_code_compat)?;
}
#[cfg(not(target_os = "macos"))]
let _ = claude_code_compat;
// On Linux / Windows there's no equivalent `open -a CuaDriver`
// mechanism to spawn a daemon attributed to the user's
// interactive session. The caller is expected to have one
Expand Down Expand Up @@ -1603,7 +1656,8 @@ fn run_permissions_grant() {
"A dialog titled \u{201c}Cua Driver\u{201d} will appear — approve Accessibility \
and Screen Recording in System Settings, then this command continues."
);
if let Err(e) = launch_daemon_and_wait(&socket, 180) {
// Permissions-grant launch never needs the compat screenshot surface.
if let Err(e) = launch_daemon_and_wait(&socket, 180, false) {
eprintln!("\nDidn't detect the CuaDriver daemon: {e}");
eprintln!(
"If you haven't yet, grant Accessibility + Screen Recording to CuaDriver \
Expand Down
20 changes: 14 additions & 6 deletions libs/cua-driver/rust/crates/cua-driver/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ fn main() {
cli::run_call(reg, &tool, json_args, screenshot_out_file, socket);
return;
}
cli::Command::Serve { socket, no_permissions_gate } => {
cli::Command::Serve { socket, no_permissions_gate, claude_code_compat } => {
// Long-running daemon — kick off the background update check
// before any blocking work so the banner can land on stderr
// early in the serve lifecycle.
Expand Down Expand Up @@ -259,7 +259,12 @@ fn main() {
None => pip_preview::PipConfig::from_args(),
};
maybe_init_pip();
let reg = Arc::new(build_macos_registry());
// Honour the compat flag forwarded by the MCP proxy
// (launch_daemon_and_wait passes `serve
// --claude-code-computer-use-compat`). The Serve arm is the daemon
// the proxy talks to, so without this the proxy path always served
// the full screenshot tool regardless of the client's request.
let reg = Arc::new(build_macos_registry_with_compat(claude_code_compat));
reg.init_self_weak();
let sp = socket.unwrap_or_else(serve::default_socket_path);
let pid_path = serve::default_pid_file_path();
Expand Down Expand Up @@ -401,7 +406,7 @@ fn main() {
// attribution and forwards stdio MCP through its socket.
// Issue #1525 / mirror of Swift PR #1479.
if cli::should_use_daemon_proxy(no_daemon_relaunch) {
if let Err(e) = cli::run_mcp_via_daemon_proxy(socket) {
if let Err(e) = cli::run_mcp_via_daemon_proxy(socket, claude_code_compat) {
eprintln!("cua-driver-rs: {e}");
std::process::exit(1);
}
Expand Down Expand Up @@ -540,14 +545,17 @@ fn main() -> anyhow::Result<()> {
}).join().ok();
return Ok(());
}
cli::Command::Serve { socket, no_permissions_gate } => {
cli::Command::Serve { socket, no_permissions_gate, claude_code_compat } => {
// Long-running daemon — kick off the background update check
// before any blocking work so the banner can land on stderr.
version_check::maybe_announce_update();
// The Rust permissions gate is macOS-only (TCC concept).
// On Windows / Linux the flag is silently accepted for
// CLI uniformity and ignored.
// CLI uniformity and ignored. The Claude-Code compat screenshot
// surface is likewise macOS-only (register_tools_with_compat),
// so the flag is accepted-and-ignored here for CLI uniformity.
let _ = no_permissions_gate;
let _ = claude_code_compat;
// Serve mode needs the cursor overlay just like MCP mode.
let cursor_cfg = cursor_overlay::CursorConfig::from_args();
let reg = Arc::new(build_registry(cursor_cfg));
Expand Down Expand Up @@ -641,7 +649,7 @@ fn main() -> anyhow::Result<()> {
// Code over SSH lands in Session 0 and every desktop
// tool returns empty. See `cli::should_use_daemon_proxy`.
if cli::should_use_daemon_proxy(no_daemon_relaunch) {
if let Err(e) = cli::run_mcp_via_daemon_proxy(socket) {
if let Err(e) = cli::run_mcp_via_daemon_proxy(socket, claude_code_compat) {
eprintln!("cua-driver-rs: {e}");
std::process::exit(1);
}
Expand Down
Loading
Loading