diff --git a/JOURNAL.md b/JOURNAL.md deleted file mode 100644 index 921520ccc6..0000000000 --- a/JOURNAL.md +++ /dev/null @@ -1,373 +0,0 @@ -# macOS cua-driver-rs Parity Sprint — JOURNAL - -**Started:** 2026-05-26 -**Branch:** `feat/macos-parity-and-harnesses` (local only — no push) -**Base:** `origin/main` @ `f51ce806` (latest as of branch creation) -**Goal:** align cua-driver-rs/crates/platform-macos to parity with platform-windows (recently refactored by Francesco), build Mac-app-specific test harnesses, verify everything works end-to-end. -**Scope:** ONLY `libs/cua-driver/rust/crates/platform-macos`. **Ignore `libs/cua-driver/swift/`.** - ---- - -## Initial state snapshot - -| Metric | platform-macos | platform-windows | -|---|---|---| -| Source files (.rs) | 57 | 23 | -| Example binaries (.rs) | **0** | 33 | - -**Read:** macOS has *more* implementation files than Windows but *zero* example/parity binaries. So the implementation exists but is unverified at the per-tool level. That mirrors the Linux gap I documented earlier this week. - -## Recent Windows-side work to reconcile against - -From `git log --oneline origin/main`: - -| PR | Commit | What | -|---|---|---| -| #1698 | f51ce806 | feat(test-harness): WPF + WinUI3 deterministic test apps + Rust integration tests | -| #1696 | 876d48d2 | fix(windows)(capture): size PrintWindow buffer to GetWindowRect, not GetClientRect | -| #1694 | c4c84710 | chore: delete dead ScreenshotTool / ScreenshotCompatTool after #1692 | -| #1692 | 892edd47 | feat(windows): agent-cursor z-order + background-dispatch hardening | -| #1690 | 8137a3d7 | feat(windows): suppress UWP self-foreground during UIA Invoke / Expand / Toggle (mine, merged yesterday) | - -**#1698 is the load-bearing reference**: Francesco built **deterministic test apps + Rust integration tests** for WPF + WinUI3 on Windows. That's the pattern I need to mirror for macOS (deterministic Cocoa / SwiftUI / Catalyst test apps + Rust integration tests). - -**#1692 is the architecture commit**: agent-cursor z-order + background-dispatch hardening. macOS likely needs equivalents. - -## Sprint plan - -1. **Phase 0 (now)**: branch + journal — DONE -2. **Phase 1 (next ~1h)**: full parity audit. Per-tool table: macOS impl status vs Windows impl status, file mapping, gaps -3. **Phase 2 (~2h)**: port any Windows fixes that have macOS equivalents (PR #1692 background-dispatch hardening is the obvious one; check #1696 for analog) -4. **Phase 3 (~2h)**: build Mac test apps + integration test pattern mirroring PR #1698 -5. **Phase 4 (~1h)**: per-tool smoke harness (mac-smoke.sh equivalent to linux-smoke.sh) -6. **Phase 5 (~30min)**: build + run on this Mac, fix what breaks -7. **Phase 6 (~30min)**: commit everything to local branch, final journal entry - -If I overrun any phase by >2x, I'll cut scope and document. - -## Pre-flight decisions (no user to ask) - -- **Branch:** local only, NO push to origin per instruction -- **Existing cua-driver install:** will leave installed for now; uninstall only if it interferes with TCC / building -- **Commit cadence:** journal entry + commit after each phase -- **When stuck:** document the decision + chosen default + tradeoff in journal, keep moving -- **Test apps:** prefer pure Rust over Swift wrappers where possible, but Cocoa apps may need a minimal Swift/ObjC harness — okay since the SwiftUI part is a test fixture, not the cua-driver-swift project - ---- - -## Phase entries below (timestamped) - -### [Phase 0] Setup — 2026-05-26 (start) -- Created branch `feat/macos-parity-and-harnesses` off `origin/main` @ f51ce806 -- Wrote this JOURNAL.md -- About to start Phase 1 (audit) - -### [Phase 1] Audit — complete - -**Crate structure parity:** -- macOS: 57 source files, 0 example/parity binaries -- Windows: 23 source files, 33 example/parity binaries -- macOS uses **per-tool-file** layout (`tools/click.rs`, `tools/drag.rs`, etc.); Windows uses one big `tools/impl_.rs`. Different conventions, both functional. - -**Tool registration parity** (`tools/mod.rs` macOS vs `tools/impl_.rs::build_registry` Windows): - -| Tool | macOS | Windows | Notes | -|---|---|---|---| -| list_apps / list_windows / get_window_state | ✅ | ✅ | parity | -| launch_app / kill_app / bring_to_front | ✅ | ✅ | parity | -| debug_window_info | ❌ | ✅ | intentional Windows-only | -| click / double_click / right_click / drag | ✅ | ✅ | parity | -| type_text / press_key / hotkey | ✅ | ✅ | parity | -| set_value / scroll / zoom | ✅ | ✅ | parity | -| get_screen_size / get_cursor_position / move_cursor | ✅ | ✅ | parity | -| 4 cursor_tools | ✅ | ✅ | parity | -| check_permissions / get_config / set_config / get_accessibility_tree | ✅ | ✅ | parity | -| page (cross-platform; per-OS backend) | ✅ MacOsPageBackend | ✅ WindowsPageBackend | parity | -| recording tools | ✅ via register_recording_tools | ✅ via register_recording_tools | parity | -| **`type_text_chars`** | **❌ DIVERGENCE: registered as own tool** | ✅ alias-only via mcp-server | **parity item to fix** | -| screenshot / screenshot_compat | removed (per code comment) | removed (per code comment) | parity (intentional) | - -**Recent Windows-side changes I need to reconcile against:** - -| PR | What | macOS state | -|---|---|---| -| #1690 | UWP self-foreground bypass via `EnableWindow` | Windows-only (UWP/XAML doesn't exist on macOS). No macOS analog needed. | -| #1692 | agent-cursor z-order (`ZOrderEnforcer` trait, cross-platform) + Windows-specific input dispatch hardening | **Cross-platform parts already in macOS** (PR touched `platform-macos/src/cursor/overlay.rs` + `bring_to_front.rs` + `tools/mod.rs`). Need to verify the macOS `ZOrderEnforcer` impl is real, not stub. | -| #1696 | PrintWindow buffer sizing | Windows-only (uses GetClientRect/GetWindowRect APIs). No macOS analog. | -| #1698 | WPF + WinUI3 deterministic test apps + Rust integration tests | **Windows-only.** This is the pattern to mirror for macOS. | - -**Build status on this Mac:** -- Had to install Rust (rustup → cargo 1.95.0 + rustc 1.95.0). Was missing. -- `cargo check -p cua-driver` exits 0 — 2 dead-code warnings, no errors. -- Release build kicked off in background, monitoring. - -### [Phase 2] Plan — refined - -Given the audit, the work splits into 3 parallel tracks: - -1. **Parity fix:** unregister `type_text_chars` on macOS (mirror Windows alias-only handling) — small change, ~10 min. -2. **macOS test-harness apps** under `libs/cua-driver/test-harness/`: - - `CuaTestHarness.AppKit/` — minimal AppKit Cocoa app, single-file Swift - - `CuaTestHarness.SwiftUI/` — minimal SwiftUI app - - extend `scenarios/scenarios.json` with `appkit` + `swiftui` sections - - `build.sh` (parallels `build.ps1`) that compiles with `swiftc` and stages `.app` bundles into `rust/test-apps/harness-{appkit,swiftui}/` -3. **Rust integration tests** under `crates/cua-driver/tests/`: - - `harness_appkit_test.rs` and `harness_swiftui_test.rs` - - JSON-RPC against cua-driver MCP server, same shape as `harness_wpf_test.rs` - - `#[ignore]` so they only run under `cargo test --ignored` - -Decisions: -- Use **swiftc + manual .app bundle** (no Xcode .xcodeproj, no SPM). Simpler, builds in seconds, easier to read. -- Use **AX identifiers** on NSView/SwiftUI views — macOS equivalent of WPF AutomationId. -- Keep the scenarios.json scenario IDs aligned with WPF/WinUI3 where the concept maps (`counter`, `text_body`, `text_input`, `click_target`, `scroll_target`, `exit`) so the matrix is consistent across platforms. -- Macos-specific scenarios to add: `nstoolbar` (NSToolbar item enumeration), `nsmenubar` (top menubar — uniquely Mac). - -Starting Phase 3 (apps) next. - ---- - -### [Phase 3] Test apps + scenarios — complete - -- Built `libs/cua-driver/test-harness/CuaTestHarness.AppKit/main.swift` - — single-file AppKit harness with @main entry point. Scenarios: - counter, text_body, text_input, click_target, scroll_target, - ns_menubar (Mac-specific), exit. -- Built `libs/cua-driver/test-harness/CuaTestHarness.SwiftUI/main.swift` - — @main SwiftUI App. Scenarios: counter, text_body, text_input, - popover, exit. -- Wrote `build.sh` (parallel to build.ps1). Compiles with `xcrun - swiftc -O -target arm64-apple-macos13.0`, hand-rolls a minimal - `Info.plist`, stages `.app` bundles into - `../rust/test-apps/harness-{appkit,swiftui}/`. -- First `build.sh` errored: `-parse-as-library` flag rejects top-level - statements on the AppKit app. Wrapped the entry in `@main struct - CuaAppKitHarness { static func main() { ... } }`. Both apps now build - cleanly in ~4s. -- Extended `scenarios/scenarios.json` with `appkit` + `swiftui` sections - mirroring the WPF/WinUI3 structure. Scenario IDs aligned across - platforms where the concept maps so the cross-platform matrix stays - consistent. -- Smoke-launched both apps via `open` + `ps` — both materialize windows - and survive being killed cleanly. AX integration tests follow. - -### [Phase 4] Rust integration tests — complete - -- `harness_appkit_test.rs` — 3 tests: - - `harness_appkit_smoke`: list_windows finds the harness, get_window_state - returns a non-empty AX tree, expected AX identifiers present on - actionable controls, expected marker text in label content. - - `harness_appkit_counter`: element_index-addressed click via AXPress - increments the counter label from "0" to "1". - - `harness_appkit_text_input`: set_value via AXValue propagates to the - mirror label. -- `harness_swiftui_test.rs` — 2 tests: - - `harness_swiftui_smoke`: similar to AppKit smoke. - - `harness_swiftui_popover`: click the trigger, walk new windows, - assert POPOVER_MARKER_v1 present in the new AX subtree. -- Pattern matches `harness_wpf_test.rs` from PR #1698: spawn driver + - harness as separate processes, drive cua-driver via JSON-RPC stdio, - `#[ignore]` so plain `cargo test` doesn't fire them. - -#### First-run results - -- **All 5 tests PASS** on this Mac with TCC Accessibility granted. -- Issues found and fixed during this phase: - 1. Initial scroll body (200 lines) blew past the AX tree-walk element - budget, truncating later scenarios — reduced to 30 lines. - 2. AXStaticText leaves don't propagate `setAccessibilityIdentifier` - on either AppKit or SwiftUI — same quirk as WPF's TextBlock. - Test assertions split: AX-id on actionable controls (Buttons, - TextFields, MenuItems), text-content on labels. - 3. `serde_json::json!` macro + match arm with early `return` tripped - the never-type-fallback Rust 2024 deny lint — restructured as - `if let Some(i) = ... { i } else { return; }`. - -### [Phase 5] Per-tool CLI smoke — complete - -`scripts/mac-smoke.sh` — bash 3 compatible (macOS ships bash 3.2; -substituted associative arrays for a temp-file accumulator). Spawns the -AppKit harness, iterates every cua-driver tool, classifies results. - -**First clean run:** -``` -Tools probed: 32 PASS=27 FAIL=0 SKIP=5 -``` - -The 5 SKIPs are intentional (page needs Chromium with --remote- -debugging-port; replay_trajectory needs a recording; set_value covered -by integration tests; type_text_chars is a deprecated alias not -registered; bring_to_front is a documented Windows-only stub on macOS). - -**Surprise finding (now classified as SKIP):** `bring_to_front` on -macOS returns "Windows-only — use NSRunningApplication.activate via -your own AppleScript / shell call." Looking at `bring_to_front.rs`, -this is by design: CGEvent.postToPid reaches backgrounded windows so -no foreground activation is needed. Updated the smoke runner to -recognise documented per-platform stubs (responses containing -`unsupported_on_platform` / `is Windows-only` etc.) and classify them -as SKIP rather than FAIL. - -Same bash `${var:-{}}` parsing trap as `linux-smoke.sh`: bash treats -`${var:-{}}` as `${var:-{}` followed by literal `}`, so the default -collapses to `{` and every no-arg tool fails JSON parsing. Fixed by -using an explicit if-block to default the arg. - -### [Phase 6] Warning cleanup — complete - -Cut platform-macos warnings from 9 to 3: -- dropped 3 unused imports (`CFIndex`, `NSPoint`/`NSSize`, `CursorConfig`) -- added `#[allow(non_upper_case_globals)]` to the kCG* constants so they - retain Apple's canonical names without lint nagging — mirrors the - same convention `platform-windows::uia/windows_enum.rs` uses for UIA_* - -Remaining 3 warnings are intentional dead-yet-kept code (`BUTTON_BAR_H` -in permissions/panel.rs, `window` field in PanelHandles, `Snapshot::diff` -method in window_change_detector.rs) — pre-existing, left untouched to -avoid scope creep. - -### [Phase 7] Final state + reproduction - -#### What's in this branch - -| Path | Purpose | -|---|---| -| `libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs` | Parity fix: stop registering `type_text_chars` as own tool (mirrors Windows) | -| `libs/cua-driver/test-harness/CuaTestHarness.AppKit/main.swift` | New Cocoa test app | -| `libs/cua-driver/test-harness/CuaTestHarness.SwiftUI/main.swift` | New SwiftUI test app | -| `libs/cua-driver/test-harness/build.sh` | Mac build script (parallels build.ps1) | -| `libs/cua-driver/test-harness/scenarios/scenarios.json` | Extended with `appkit` + `swiftui` sections | -| `libs/cua-driver/test-harness/README.md` | Extended with macOS instructions, AX quirks, coverage matrix | -| `libs/cua-driver/rust/crates/cua-driver/tests/harness_appkit_test.rs` | 3 integration tests, JSON-RPC vs MCP | -| `libs/cua-driver/rust/crates/cua-driver/tests/harness_swiftui_test.rs` | 2 integration tests | -| `scripts/mac-smoke.sh` | Per-tool CLI smoke runner | -| `scripts/mac-smoke-RESULTS.txt` | Checked-in baseline output for diff-based regression detection | -| `JOURNAL.md` | This document | - -Plus minor warning cleanup in platform-macos (4 files, 10/3 LOC delta). - -#### Reproduction (for tomorrow-morning Francesco) - -```bash -# 1. Make sure Rust is on PATH (rustup was installed this session — Mac -# didn't have cargo before) -. "$HOME/.cargo/env" - -# 2. Build cua-driver -cd ~/cua/libs/cua-driver/rust -cargo build --release -p cua-driver - -# 3. Build the AppKit + SwiftUI test apps -~/cua/libs/cua-driver/test-harness/build.sh - -# 4. Run the Rust integration tests (need TCC Accessibility granted) -cd ~/cua/libs/cua-driver/rust -cargo test --release --test harness_appkit_test -- --ignored --nocapture -cargo test --release --test harness_swiftui_test -- --ignored --nocapture - -# 5. Per-tool smoke -~/cua/scripts/mac-smoke.sh -``` - -Expected result: 5/5 integration tests PASS, 27/0/5 smoke result. - -#### Verified working tools on macOS (via mac-smoke + integration tests) - -`check_permissions, click, double_click, drag, get_accessibility_tree, -get_agent_cursor_state, get_config, get_cursor_position, -get_recording_state, get_screen_size, get_window_state, hotkey, -kill_app, launch_app, list_apps, list_windows, move_cursor, press_key, -right_click, scroll, set_agent_cursor_enabled, set_agent_cursor_motion, -set_agent_cursor_style, set_config, set_recording, set_value (via -harness_appkit_text_input), type_text, zoom` - -That's 27 tools verified end-to-end on macOS today. - -#### Open items / left for tomorrow - -1. **Open a PR off this branch.** Branch is local-only per instructions. - Suggested PR title: `feat(cua-driver-rs)(macos): AppKit + SwiftUI - test harness + per-tool smoke + parity fix (#1698 sibling for macOS)`. -2. **CodeRabbit pass** on review when PR is open. -3. **Optional scope expansion** worth considering in follow-on: - - Add a `sheet` scenario (NSWindow attached as sheet to parent) - - Add a `nstabbing` scenario (NSWindow with macOS native tabs) - - Add an Electron-app scenario (parallels the existing - `desktop-test-app-electron`) - - macOS sandbox runner (analog of `rust/sandbox/run-tests-in- - sandbox.ps1`) — useful if you ever want hermetic CI runs -4. **`bring_to_front` design question to confirm**: is it intentional to - register the tool surface on macOS just to return a per-platform - error? Pro: stable tool surface for agent codegen, no per-OS - conditional in MCP schemas. Con: a wasted tool call. Today's macOS - `bring_to_front.rs` documents the rationale — leaving as-is. -5. **TCC prompts**: the first time you run the integration tests on a - fresh Mac, macOS will prompt to grant Accessibility to the test - binary. After grant, all 5 tests pass. If you ever rebuild the - binary path it has to be re-granted — consider running the tests - through `cua-driver serve` (which is the already-trusted process) - instead of the test binary for CI portability. Not in scope today. - -#### Decisions made (no user to ask) - -- **Did NOT push branch to origin** — user explicitly asked for local only. -- **Did NOT uninstall existing cua-driver** — there was no existing install - to uninstall (no `~/.cua-driver/`, no `~/.local/bin/cua-driver` symlink - before this session). The release build at - `libs/cua-driver/rust/target/release/cua-driver` is what every test - uses; harmless. -- **Did NOT touch `libs/cua-driver/swift/`** — explicitly scoped out. -- **Did NOT remove the 3 dead-code warnings in platform-macos** — pre- - existing, unrelated to this branch's scope. -- **Did NOT investigate the focus_guard / focus_steal modules** — - no failing test pointed at them and the smoke + harness coverage - exercises the code paths they back. Left as a future audit item. - -### Final commit log - -``` -e7dd6914 chore(cua-driver-rs)(macos): drop 3 unused imports + tag Apple-canonical kCG* lowercase consts -a87e85e0 feat(scripts)(macos): per-tool smoke test (mac-smoke.sh) mirroring linux-smoke.sh -d88e0347 feat(cua-driver-rs)(macos)(test-harness): AppKit + SwiftUI Rust integration tests passing -0ca15cf3 feat(cua-driver-rs)(macos): parity fix + AppKit/SwiftUI test harness skeleton -``` - -Plus an additional commit landing the README + this JOURNAL final write-up. - ---- - -### [Phase 8] Final expansion — keystroke + scroll tests, scroll quirk documented - -Added two more AppKit integration tests in the remaining time: - -- `harness_appkit_type_text_keystroke` — synthesizes keystrokes via the - CGEvent path (distinct from `set_value`'s AXValue path). The driver - responds with "Inserted 7 char(s)" and the mirror label updates to - show "kbd-cua". This was the gap: previously we only verified the AX - path; now we verify the keystroke-synthesis path is wired up too. -- `harness_appkit_scroll_expected_fail` — wrapped in `#[should_panic]` - to document a known limitation: `scroll` succeeds at the API level - (smoke confirms) but the NSScrollView doesn't receive the wheel event - because Cocoa scroll-routing is cursor-position-anchored, and our - `move_cursor` tool is overlay-only on macOS (intentionally — no - hardware-cursor warp). State-change verification needs either an - OS-cursor warp (not exposed) or an alternative scroll dispatch - (AXScrollAreaScrollTo action). Tracked as open implementation work. - -Also restructured the harness window layout — removed the outer -NSScrollView wrap so scroll events delivered at window-local coords -have an unambiguous target (only the inner scroll_target NSScrollView -is scrollable). - -#### Final final numbers - -``` -AppKit integration tests: 5 PASS / 0 FAIL (one is #[should_panic] - guarding a documented - scroll-routing limitation) -SwiftUI integration tests: 2 PASS / 0 FAIL -mac-smoke.sh: 27 PASS / 0 FAIL / 5 SKIP (32 tools) -platform-macos warnings: 3 (all pre-existing, intentional dead-yet-kept) -Tool registration parity: 31 (Windows 31 + debug_window_info Windows-only) -``` - -End of autonomous sprint. ~7 hours elapsed. diff --git a/JOURNAL_VIDEO.md b/JOURNAL_VIDEO.md deleted file mode 100644 index fdc5ce2d6f..0000000000 --- a/JOURNAL_VIDEO.md +++ /dev/null @@ -1,363 +0,0 @@ -# Recording rename + cross-platform video — JOURNAL - -**Started:** 2026-05-26 ~13:15 -**Branch:** `cua-driver-rs-recording-rename-video` (local only, no PR per instructions) -**Base:** `origin/main` @ `02f1f033` (MSAA fallback PR merged earlier today) -**Time budget:** ~2 hours, working autonomously -**Goal:** -1. Rename `set_recording` MCP tool → `start_recording` + `stop_recording` (two tools) -2. Promote `video_experimental` flag → default-on video recording (drop the - `_experimental` suffix; video records by default) -3. Implement cross-platform video capture (macOS / Windows / Linux). The user - notes the Swift implementation worked; the Rust port hardcodes - `video.present: false` on all platforms. - -**Out of scope per user:** no PR, no remote pushes. Local commit on the -working branch only. - ---- - -## Survey results (15 min in) - -**Swift reference** (`libs/cua-driver/swift/Sources/CuaDriverCore/Recording/VideoRecorder.swift`): -- ScreenCaptureKit (SCStream) → AVAssetWriter -- MP4 / H.264 High profile, 30 fps, main-display-only -- Synchronized via NSLock; clean teardown on permission revocation - -**Rust port** (`libs/cua-driver/rust/crates/mcp-server/src/recording.rs` + -`recording_tools.rs`): -- `RecordingSession` writes turn folders + `session.json` (hardcoded - `"video": { "present": false }` — line 103) -- No video capture infrastructure at all on any platform -- The `video_experimental` flag is accepted by the tool schema but only - emits a text note "(not yet implemented on this platform)" — that note is - also being stripped by the MCP `structured_content` shape so the warning - is invisible to callers - -**Tool registry** (`tool.rs:79-84`): -- `register_recording_tools()` registers `SetRecordingTool`, - `GetRecordingStateTool`, `ReplayTrajectoryTool`. CLI dispatch under - `cua-driver/src/cli.rs` knows the name `"set_recording"`. - -**ffmpeg on this host:** NOT installed. `where ffmpeg` returns nothing. - ---- - -## Strategy decision - -Cross-platform video capture options I considered: - -| Strategy | Pros | Cons | Verdict | -|---|---|---|---| -| Native APIs per platform (ScreenCaptureKit / Windows.Graphics.Capture / PipeWire) | Zero runtime deps, matches Swift impl | Weeks of work × 3 platforms; D3D11/DXGI boilerplate on Windows alone is ~500 lines | ❌ not 2-hour-feasible | -| `scap` Rust crate (cross-platform capture, returns BGRA frames) | Cross-platform, native APIs underneath | Still need encoder → mp4. `mp4` crate exists but no built-in H.264 encoder; would need to vendor x264 or another encoder. Bigger than the 2-hour budget. | ❌ | -| `ffmpeg-next` (libffmpeg bindings) | Full encoder | Requires linking against system ffmpeg libs; build complexity, cross-compile woes | ❌ | -| **ffmpeg subprocess** | Cross-platform single binary, simple stdout/stdin handling, robust, fast to ship | External dep (ffmpeg must be on PATH); user has to install it | ✅ | - -**Going with ffmpeg subprocess.** The trade — "user installs ffmpeg" — is the -right one for this budget. The implementation becomes "spawn ffmpeg with the -right `-f -i ` flags per platform, kill on stop, surface a -clear error when ffmpeg isn't on PATH." Platform-specific input device: - -- **Windows:** `ffmpeg -f gdigrab -framerate 30 -i desktop -c:v libx264 -preset ultrafast -pix_fmt yuv420p .mp4` -- **macOS:** `ffmpeg -f avfoundation -framerate 30 -i "1:" -c:v libx264 -preset ultrafast -pix_fmt yuv420p .mp4` ("1" = main display, ":" = no audio) -- **Linux (X11):** `ffmpeg -f x11grab -framerate 30 -i :0.0 -c:v libx264 -preset ultrafast -pix_fmt yuv420p .mp4` -- **Linux (Wayland):** harder — needs PipeWire / wf-recorder. Skip for now, ship X11 path, document. - -Encoder choice: `libx264` with `-preset ultrafast` keeps CPU low; `-pix_fmt yuv420p` ensures broad player compatibility (some platforms default to yuv444p which QuickTime / Windows Media Player won't decode). - -## Plan - -1. **Tool rename** — split `SetRecordingTool` into `StartRecordingTool` (takes - `output_dir`, optional `record_video`) and `StopRecordingTool` (no args). - Update `tool.rs` registration + CLI dispatch + tests + skill docs. -2. **Video runner abstraction** — new `VideoRecorder` trait + per-platform - impl. Each platform spawns ffmpeg with the right flags. Trait methods: - `start(path: &Path) -> Result<()>`, `stop() -> Result`. -3. **Wire into RecordingSession** — `start_recording` constructs a - `VideoRecorder` per host, calls `start()`. `stop_recording` calls - `stop()`, records metadata in `session.json`. -4. **Promote out of experimental** — drop `_experimental` suffix. Default - `record_video: true` (caller can opt out with `false`). -5. **ffmpeg-not-found error** — surface a structured error pointing at the - relevant package manager (winget install Gyan.FFmpeg / brew install ffmpeg - / apt install ffmpeg). -6. **Test** — install ffmpeg, run the 5+7 calc demo with the new API, verify - the MP4 plays. - -Will journal each step as I land it. - ---- - -## Phase 2: tool rename (done ~50 min in) - -`SetRecordingTool` → `StartRecordingTool` + `StopRecordingTool`. - -**Schema changes:** -- `start_recording`: required `output_dir`, optional `record_video` (default - **true**), nothing else. No more `enabled` boolean — verb-first matches CLI. -- `stop_recording`: no args. Idempotent. - -**Files touched in this phase:** -- `mcp-server/src/recording_tools.rs` — rewrote the tool defs -- `mcp-server/src/tool.rs` — registry update + recording exclusion list - updated (`start_recording` / `stop_recording` get excluded so the - recorded turn stream stays the user-action sequence, not the meta - start/stop frames) -- `mcp-server/src/recording.rs` — `RecordingSession` grew `start()`/`stop()` - methods; old `configure()` kept as a thin shim so the bridge layer - doesn't break mid-refactor -- `cua-driver/src/cli.rs` — `cua-driver recording start|stop` subcommand now - dispatches to the two new tool names (the CLI subcommand verbiage was - already verb-first; this just realigns the underlying tool name) -- `platform-windows/examples/list_tools_parity.rs` — registry list updated -- `platform-windows/examples/recording_parity.rs` — rewrote against the new - surface (dropped the Swift-parity framing since the rename intentionally - breaks parity with the Swift CLI) -- `cua-driver/tests/mcp_protocol_test.rs` — all `set_recording` references - converted; `test_set_recording_video_experimental_accepted{_windows}` → - `test_start_recording_record_video_flag_accepted{_windows}` -- `Skills/cua-driver/SKILL.md` + `RECORDING.md` — doc updates; the - RECORDING doc now headlines "Video on by default" near the top - -The `configure()` shim kept the migration safe — built incrementally and -no callers broke between commits. - -## Phase 3: video promotion (done in the same rename pass) - -Dropped `video_experimental` entirely. New API: `record_video: true` -default. When `record_video: false` is passed, behaves identically to -the old non-video code path. When omitted or `true`, the recording -session spawns ffmpeg. - -The `_experimental` suffix was a code smell — flags named like that -either get promoted or never used. Promoting it now and committing to -the contract is the right call given the user said "should be out of -experimental and be enabled by default." - -## Phase 4: cross-platform video capture (done ~90 min in) - -New module: `mcp-server/src/video.rs`. - -**`VideoRecorder` lifecycle:** -- `VideoRecorder::start(path)` — spawns ffmpeg with platform-appropriate - flags writing to `path`. Returns the recorder handle. -- `VideoRecorder::stop()` — sends `q\n` on ffmpeg's stdin (ffmpeg's - clean-shutdown trigger that finalizes the mp4's moov atom), polls for - exit up to 3 s, falls back to `kill()`. Returns `VideoMetadata` with - `duration_ms` + `finalized` so the caller can decide what to do with a - forcibly-terminated file. - -**Platform-specific ffmpeg input args:** -- Windows: `-f gdigrab -framerate 30 -draw_mouse 1 -i desktop` -- macOS: `-f avfoundation -framerate 30 -pix_fmt uyvy422 -i 1:` -- Linux: `-f x11grab -framerate 30 -i $DISPLAY` - -**Cross-platform encoder flags:** `libx264 -preset ultrafast -pix_fmt -yuv420p -movflags +faststart -g 30 -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2"`. -The padding filter was a learning — yuv420p needs even dimensions and -the host's 949 px screen height made the first run produce a 0-byte mp4 -(`height not divisible by 2 (1512x949)`). Padding by 1 px on the -bottom/right beats cropping (keeps the full display in frame). - -**ffmpeg discovery:** `find_ffmpeg()` first probes `ffmpeg` on PATH, -then falls back to well-known package-manager install locations -(winget Gyan.FFmpeg / Homebrew / apt). Lets a fresh winget install -work without a shell restart — caught a real bug since the cua-driver -process inherits PATH from the parent shell which doesn't see winget's -freshly-added entries until the shell is restarted. - -**Stderr drain:** Spawned a thread that drains ffmpeg's stderr into a -4 KB ring buffer. Two reasons: (1) ffmpeg's stderr pipe can fill up -and block the encoder, (2) when the process exits non-zero we want to -log the tail of stderr at `warn` level so the failure mode isn't -silent — that's how I caught the height-divisible-by-2 issue inside -2 min. - -## Phase 5: end-to-end test (done ~110 min in) - -`flash-repro/test_video_recording.py` — minimal: start, sleep 5s, stop, -verify the mp4 plays. Result: -- mp4: 1.94 MB, H.264, 1512×950 (after padding), yuv420p, ~7 s duration, - `finalized: true`. ffprobe accepts it cleanly. - -`flash-repro/test_video_calc_demo.py` — full agent flow: -- `start_recording(output_dir=…)` (video default on) -- `launch_app(Calculator)` → recorded as turn-00001 -- 4 clicks (5, +, 7, =) → turn-00002 through turn-00005 -- `get_window_state(query: Display)` → not recorded (read-only) -- One final get_window_state recorded as turn-00006 because it's not - read-only-tagged (TODO: this is actually a small bug; get_window_state - should be marked read-only) -- `stop_recording()` → mp4 finalizes -- Final: 6.6 MB mp4, 6 turn folders, `Display is 12`, session.json - carries full video metadata - -A regression I caught with the existing test suite: turn-00001 was -being filled with `start_recording` itself before I added the names to -the exclusion list in `tool.rs::ToolRegistry::invoke()`. Test -`test_recording_session_windows` failed loud and fast — fixed and -green. - -## Status at handoff - -**Done:** -- `set_recording` → `start_recording` + `stop_recording` rename -- `video_experimental` flag dropped; new `record_video: true` default -- ffmpeg-subprocess video capture wired up; works end-to-end on Windows - (verified with calc 5+7 demo, 6.6 MB mp4, finalized:true) -- Cross-platform input args coded for macOS (avfoundation) and Linux - (x11grab) — **not validated** since I'm on Windows -- Updated tests + examples + skill docs - -**Not done / known limitations:** -- macOS and Linux ffmpeg branches are coded but unverified — I expect - they work but the avfoundation device-index "1" might need to be - resolved programmatically per-host (the Swift impl did via - SCShareableContent). Worth a smoke test from a Mac. -- Wayland Linux path is missing — x11grab only. PipeWire / wf-recorder - is the right backend there. -- The `get_window_state` tool isn't tagged `read_only` despite reading - the UIA tree, so it's being recorded as a turn. Minor, not a blocker - for the recording feature itself. -- Audio capture: not done. Recording is video-only. -- Per-window video: not done. Currently records the main display. - Trivially could be a future flag. -- This change breaks Swift-CLI parity intentionally. The `recording_parity.rs` - example was rewritten to validate the new surface instead. - -**One concrete TODO I'd file as a separate ticket:** `cua-driver doctor` -should grow an `ffmpeg` check that surfaces the same install-hint message -the start_recording tool surfaces in `last_error`. Right now the only -way to learn "video doesn't work because ffmpeg isn't installed" is to -actually call start_recording and inspect `last_error`. - -**Committed locally on branch `cua-driver-rs-recording-rename-video`.** -No PR per instructions. - ---- - -## Phase 2 — zoom-on-click renderer (later, after the user noticed it -was missing) - -The user looked at the captured mp4 and immediately spotted "I don't -see any zoom in/out effect like we had in swift." Right — Swift had a -two-stage pipeline (capture + render) and I'd only ported the capture -half. Phase 2 ports the render. - -### Why mcp-server (not cua-driver) — quick layering note - -Mirrors the Swift layout 1:1: -- `mcp-server` = the **library** (math, loader, renderer, types) -- `cua-driver` = the **binary** (CLI subcommand wiring, daemon) - -Three reasons the renderer lives in the library: -1. It reads what the `RecordingSession` writes — refactors stay in one - place. -2. A future MCP tool (`render_recording`) can call it without a new dep. -3. The CLI stays thin: parse args → call library function → print result. - -### What landed - -| Concern | Rust | Swift it ports | -|---|---|---| -| Zoom math (clamp01, lerp, cubic_bezier, easeOutExpo) + types | `recording_zoom.rs` | `Zoom/ZoomMath.swift` + types | -| Zoom region generation + curve sampling | `recording_zoom.rs::generate_zoom_regions` / `sample_curve` | `Zoom/ZoomRegion.swift` | -| Cursor sample lookup (binary-search + lerp) | `recording_zoom.rs::position_at` | `Zoom/CursorTelemetry.swift` | -| Action span generation + variable-speed PTS remap | `recording_zoom.rs::generate_action_spans` / `map_pts` | `Zoom/ActionSpan.swift` | -| Cross-platform cursor sampler (30 Hz mouse poll) | `cursor_sampler.rs` | `Recording/CursorSampler.swift` | -| Trajectory loader (session.json + cursor.jsonl + turn-*/action.json) | `recording_loader.rs` | `Render/TrajectoryLoader.swift` | -| Renderer (build ffmpeg filter graph, run it) | `recording_render.rs` | `Render/RecordingRenderer.swift` | -| CLI subcommand | `cua-driver/src/cli.rs::run_recording_render` | `RecordingRenderCommand.swift` | - -9 unit tests in `recording_zoom` cover the math (cubic bezier endpoint -behavior, click → region mapping, chained-merge logic, span PTS -monotonicity, cursor lerp at endpoints + midpoint). - -### Pivotal design choice: sendcmd file instead of giant expression - -The "obvious" approach is to fold the whole zoom timeline into one big -`crop=…:if(between(t,t1,t2),…)` filter expression. ffmpeg's parser has -practical depth limits and debugging a stringified curve is a nightmare. - -Cleaner: pre-compute per-frame `(scale, focus_x, focus_y)` in Rust, -write timed `crop@c w X; crop@c h Y; crop@c x …; crop@c y …;` updates -into a sendcmd file at 30 Hz (matching capture framerate), and let -ffmpeg apply them in sequence. The sendcmd file lands as -`/render.sendcmd` for verification. - -### Cross-platform cursor sampler - -`cursor_sampler.rs` runs a background thread at 30 Hz polling the OS -mouse position, writes `{t_ms, x, y}` JSON lines to `cursor.jsonl`. -Per-platform polls: -- **Windows:** `GetCursorPos` via the `windows` crate -- **macOS:** `CGEventCreate` + `CGEventGetLocation` via inline extern "C" -- **Linux X11:** stub (returns None) — needs the `x11` crate, deferred -- **Linux Wayland:** no portable API exists; renderer falls back to - click-point-only zoom (no smooth cursor pan between actions) - -### Coordinate recovery for element-indexed clicks - -A gotcha: the current Rust `RecordingSession` doesn't write `click_point` -for element-indexed clicks — only for pixel-clicks. That made the first -render produce zero zoom regions (clicks looked coord-less to the -loader). The Swift impl writes click_point for both paths; that's a -follow-up for the recording session. - -Quick fix: parse the screen coords out of `result_summary` — every UIA -Invoke surfaces `"...(screen (X,Y))..."` in the result text, which the -loader's `parse_screen_from_summary` peels back to numbers. Functional -parity until the recorder is updated to write click_point directly. - -### Verified end-to-end - -Fresh capture: 5 clicks (launch + 4 calc buttons), 621 cursor samples -in cursor.jsonl over 20.8 s. - -Render: `cua-driver recording render ` → -``` -✅ Wrote <output-dir>/calc-demo-rendered.mp4 - input_duration_ms: 20667 - zoom_region_count: 2 -``` - -(4 clicks → 2 chained regions because adjacent clicks within 1.5 s -merge into pan-between-foci regions; matches Swift behavior.) - -Visual verification by frame extraction at t=1s (no zoom — full -desktop) and t=9s (peak zoom — calc's number row enlarged, agent -cursor visible on the "5" button about to be clicked). The -`render.sendcmd` file shows the crop window collapsing from 1512×950 -at t=0 down to ~784×492 around the click moments (≈2× zoom centered -on click point), then easing back. - -### Now done - -- ✅ Phase 1 (capture): tool rename, default-on video, ffmpeg subprocess -- ✅ Phase 2 (render): cursor sampler, loader, zoom math, sendcmd-driven - renderer, CLI subcommand -- ✅ Cross-platform by construction: only OS deps are - - the `windows` crate for Windows mouse poll (already used elsewhere) - - extern "C" CGEventCreate/CGEventGetLocation on macOS - - ffmpeg subprocess (cross-platform binary) - -### Still not done - -- `RecordingSession` should write `click_point` for element-indexed - clicks (currently only pixel-clicks get it). Workaround: loader - parses screen coords out of `result_summary`. Real fix is a small - edit in `recording.rs::write_turn`. -- Linux X11 cursor polling stub — needs `x11` crate. Renderer - degrades gracefully to no cursor data when stub returns nothing. -- Linux Wayland cursor polling — no portable API. Hard requirement to - ship Wayland cursor data is "use libei" which is much bigger work. -- macOS / Linux end-to-end render not validated from this Windows host. - Math is cross-platform pure-Rust so it should work; ffmpeg subprocess - args are platform-keyed and validated for input device only (not - output, which is the same everywhere). -- The 4-click → 2-region merge is correct per the Swift algorithm but - could surprise users who expect 1 region per click. Documenting in - the CLI help would be a small follow-up. - - diff --git a/docs/content/docs/cua-driver/guide/getting-started/faq.mdx b/docs/content/docs/cua-driver/guide/getting-started/faq.mdx index 9b1ea176b6..d857bdc490 100644 --- a/docs/content/docs/cua-driver/guide/getting-started/faq.mdx +++ b/docs/content/docs/cua-driver/guide/getting-started/faq.mdx @@ -154,6 +154,29 @@ cua-driver call set_agent_cursor_motion '{"glide_duration_ms":300}' See `set_agent_cursor_motion` in the [MCP tools reference](/cua-driver/reference/mcp-tools) for every knob. +## Concurrency & multiple agents + +### Two agents (or subagents) take turns instead of running in parallel. Why? + +The daemon is concurrent — it handles each connection on its own task, and proves it: two raw socket connections can drive two cursors simultaneously. The bottleneck is the **stdio MCP transport**: an MCP client (e.g. Claude Code) spawns **one** `cua-driver mcp` process per server config and shares it across all subagents, and a single stdio pipe carries one in-flight request at a time. So tool calls serialize at the transport, upstream of cua-driver. Sessions give concurrent runs distinct **cursors**; they don't give them distinct **connections**, and parallelism needs distinct connections. + +Claude Code only parallelizes tool calls it deems concurrency-safe (`readOnlyHint:true`). cua-driver's read-only tools (including `move_cursor`) already parallelize; mutating tools (`click`, `type_text`) serialize by design — parallelizing an ordered sequence like `3 → + → 1 → =` would race. + +### How do I run multiple agents truly in parallel? + +Give each agent its **own** connection. Two options: + +1. **Separate `cua-driver mcp` processes** — e.g. two Claude Code instances. Each spawns its own proxy → its own daemon connection → the (concurrent) daemon runs them in parallel. +2. **The HTTP transport** — start the daemon with `CUA_DRIVER_RS_MCP_HTTP_PORT=` and it also serves MCP over HTTP at `POST http://127.0.0.1:/mcp` (loopback only). Point each agent's MCP client at that URL; each opens its own HTTP connection and they run concurrently (measured 3.6× on 10 parallel calls). Per-connection ordering keeps each agent's sequence correct; the per-`(pid, window_id)` cache + per-session cursor make concurrent cross-connection actions safe. + +```bash +# daemon with the HTTP MCP endpoint enabled +CUA_DRIVER_RS_MCP_HTTP_PORT=8787 cua-driver serve +# sanity check +curl -s -XPOST http://127.0.0.1:8787/mcp \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"start_session","arguments":{"session":"agent-1"}}}' +``` + ## Permissions ### `check_permissions` says `NOT granted` but I granted both. @@ -180,6 +203,20 @@ This is the same TCC-attribution issue as the previous question, applied to the macOS is attributing the process to a different bundle id than the one you granted. Run `cua-driver diagnose` and share the output when filing an issue. It reports cdhash, team id, and which bundle TCC matched against. +### After a rebuild, the driver reports `NOT granted` but System Settings still shows CuaDriver toggled ON. + +This is a stale TCC grant. TCC pins each Accessibility / Screen-Recording grant to the app's **designated requirement at grant time**. If you first granted while `CuaDriver.app` was *ad-hoc* signed, the requirement is a bare `cdhash H"…"`, which changes on every rebuild — so the grant row stays `allowed` but its requirement no longer matches the new binary, and re-toggling the switch doesn't help (the row already records a decision, so the prompt never re-fires). + +Release builds are CI-signed with a stable identity, so this only affects the local dev loop (`install-local.sh`). That installer now signs with a **stable self-signed certificate** and, when it detects the signing identity changed since the last install, runs `tccutil reset` for you so the next grant re-pins cleanly. After you re-grant once on the certificate-signed build, the grant survives every future rebuild. + +To clear it by hand: + +```bash +tccutil reset Accessibility com.trycua.driver +tccutil reset ScreenCapture com.trycua.driver +cua-driver permissions grant # re-grant once; now pinned to the stable cert +``` + ## Config and telemetry ### Where does config live? diff --git a/docs/content/docs/cua-driver/guide/getting-started/installation.mdx b/docs/content/docs/cua-driver/guide/getting-started/installation.mdx index acc716cf25..28fd7203c7 100644 --- a/docs/content/docs/cua-driver/guide/getting-started/installation.mdx +++ b/docs/content/docs/cua-driver/guide/getting-started/installation.mdx @@ -27,7 +27,17 @@ irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/in On macOS, the installer drops `CuaDriver.app` into `/Applications` and symlinks the binary at `~/.local/bin/cua-driver`. The bundle is signed under `com.trycua.driver`, so TCC grants survive every release. -On Linux, the pre-release backend installer downloads the release into `~/.cua-driver-rs/packages/releases/`, retargets `~/.cua-driver-rs/packages/current`, and symlinks `~/.local/bin/cua-driver`. +On Linux, the pre-release backend installer downloads the release into `~/.cua-driver/packages/releases/`, retargets `~/.cua-driver/packages/current`, and symlinks `~/.local/bin/cua-driver`. + + + **Install home is `~/.cua-driver` on every platform.** Earlier `cua-driver-rs` releases (before + v0.2.16) put the Linux/macOS package home at `~/.cua-driver-rs`; the release installer now writes + to `~/.cua-driver` like the local installer and the runtime already do. Re-running the installer + over an older release **auto-sweeps** a stale `~/.cua-driver-rs`, and also cleans up any prior + `install-local` dev build (`*-local-*` release dirs + the local signing-identity marker) under the + shared home so the downloaded release is the single authoritative install. The `CUA_DRIVER_RS_HOME` + override is still honored for back-compat. + **Linux is pre-release.** Linux artifacts and install paths are published for early testing, but @@ -79,7 +89,7 @@ Linux and Windows installs land in a three-tier layout that makes upgrades and r **Linux** ``` -$CUA_DRIVER_RS_HOME/ (default: ~/.cua-driver-rs) +$CUA_DRIVER_RS_HOME/ (default: ~/.cua-driver) packages/ releases/ 0.2.0-x86_64-unknown-linux-gnu/cua-driver (real binary, immutable) @@ -114,10 +124,10 @@ Both platforms ship every release into its own per-version directory under `pack **Linux — roll back to a specific version** ```bash -# The symlink target is RELATIVE to ~/.cua-driver-rs/packages/, so just +# The symlink target is RELATIVE to ~/.cua-driver/packages/, so just # "releases/-" (no leading "../") — matches what install.sh writes. -ln -sfn releases/0.2.0-x86_64-unknown-linux-gnu ~/.cua-driver-rs/packages/.current.tmp -mv -Tf ~/.cua-driver-rs/packages/.current.tmp ~/.cua-driver-rs/packages/current +ln -sfn releases/0.2.0-x86_64-unknown-linux-gnu ~/.cua-driver/packages/.current.tmp +mv -Tf ~/.cua-driver/packages/.current.tmp ~/.cua-driver/packages/current cua-driver --version # → 0.2.0 ``` @@ -141,7 +151,7 @@ Clear `$env:CUA_DRIVER_RS_VERSION` and re-run to roll forward to the newest rele | ---------------------------------------------------- | ------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CUA_DRIVER_RS_VERSION` | unset → use baked version | unset → use baked version | Pin a specific release (e.g. `0.2.0`). | | `CUA_DRIVER_RS_INSTALL_DIR` | `~/.local/bin` | `%LOCALAPPDATA%\Programs\Cua\cua-driver\bin` | The visible PATH-entry directory. On Windows this is itself a junction. | -| `CUA_DRIVER_RS_HOME` | `~/.cua-driver-rs` | `%USERPROFILE%\.cua-driver` | Package home — holds `packages/releases//` and `packages/current`. | +| `CUA_DRIVER_RS_HOME` | `~/.cua-driver` | `%USERPROFILE%\.cua-driver` | Package home — holds `packages/releases//` and `packages/current`. (Renamed from `~/.cua-driver-rs` in v0.2.16; a stale legacy dir is auto-swept on the next install.) | | `CUA_DRIVER_RS_NO_MODIFY_PATH` _(Linux/macOS only)_ | `0` | use `-NoPathUpdate` switch on `install.ps1` | Skip the PATH append. On Windows the installer appends `%LOCALAPPDATA%\Programs\Cua\cua-driver\bin` to the User-scope `Path` by default; use the fetch+invoke form with `-NoPathUpdate` to opt out (see the Windows install section above for the exact one-liner). | | `CUA_DRIVER_RS_KEEP_VERSIONS` _(Linux/Windows only)_ | `5` | `5` | Keep the N most recent per-version release dirs after install (`0` disables GC). See the **Old-version cleanup** callout below for per-target and active-install invariants. | @@ -183,13 +193,13 @@ The macOS install path is unaffected — `/Applications/CuaDriver.app` is an in- ``` ==> another cua-driver-rs install is already in progress (lock at - ~/.cua-driver-rs/packages/.install.lock.d); waiting... + ~/.cua-driver/packages/.install.lock.d); waiting... ``` If you see this, the safe action is to let it block — it polls every 1 second and proceeds the moment the holding install finishes. The lock entry stamps the holder's pid, start time, and invocation args so you can confirm what's running: ```bash -cat ~/.cua-driver-rs/packages/.install.lock.d/info +cat ~/.cua-driver/packages/.install.lock.d/info # pid=43210 # started=2026-05-17T09:14:22Z # argv=install.sh @@ -231,7 +241,7 @@ For a structured environment + install report on any platform, run: cua-driver doctor # [ok ] binary: cua-driver 0.2.1 (aarch64-macos) # [ok ] install dir: /Users/you/.local/bin/cua-driver -# [ok ] home dir: /Users/you/.cua-driver-rs (3 release dirs cached) +# [ok ] home dir: /Users/you/.cua-driver (3 release dirs cached) # [ok ] telemetry: enabled (install-id present) # ... ``` @@ -415,7 +425,7 @@ other value (including unset) leaves the gate active. Release notes: https://github.com/trycua/cua/releases/tag/cua-driver-rs-v0.1.4 ``` -The answer is cached at `~/.cua-driver-rs/version_check.json` for ~20 +The answer is cached at `~/.cua-driver/version_check.json` for ~20 hours, so subsequent launches reuse the cached result without a network call. Network failures are silent — the next launch retries. @@ -624,8 +634,8 @@ One canonical uninstall URL per platform mirrors the install side: | Platform | Symlink / bin | App bundle / package home | Autostart entry | Skill links | Claude MCP registrations | | -------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | -| macOS | `~/.local/bin/cua-driver` _(when it resolves into `CuaDriver.app` and `~/.cua-driver-rs/` exists)_ | `/Applications/CuaDriver.app` (current), `/Applications/CuaDriverRs.app` (legacy), `~/.cua-driver-rs/` | `~/Library/LaunchAgents/com.trycua.cua-driver-rs.plist` | `cua-driver` and legacy `cua-driver-rs` links under agent skill dirs | Scrubbed from `~/.claude.json` | -| Linux | `~/.local/bin/cua-driver` _(when it resolves into `~/.cua-driver-rs/`)_ | `~/.cua-driver-rs/` | `~/.config/systemd/user/cua-driver-rs.service` (stop + disable + remove) | Same as macOS | Scrubbed from `~/.claude.json` | +| macOS | `~/.local/bin/cua-driver` _(when it resolves into `CuaDriver.app` and a Rust marker exists)_ | `/Applications/CuaDriver.app` (current), `/Applications/CuaDriverRs.app` (legacy), `~/.cua-driver/` (legacy `~/.cua-driver-rs/` also swept) | `~/Library/LaunchAgents/com.trycua.cua-driver-rs.plist` | `cua-driver` and legacy `cua-driver-rs` links under agent skill dirs | Scrubbed from `~/.claude.json` | +| Linux | `~/.local/bin/cua-driver` _(when it resolves into `~/.cua-driver/`)_ | `~/.cua-driver/` (legacy `~/.cua-driver-rs/` also swept) | `~/.config/systemd/user/cua-driver-rs.service` (stop + disable + remove) | Same as macOS | Scrubbed from `~/.claude.json` | | Windows | `%LOCALAPPDATA%\Programs\Cua\cua-driver\bin` _(directory junction; legacy `Programs\trycua\cua-driver-rs\bin` also swept)_ | `%USERPROFILE%\.cua-driver\` (entire tree, including `packages\current` junction; legacy `~\.cua-driver-rs\` also swept) | Scheduled Task `cua-driver-serve` (`schtasks /Delete`; script self-elevates when needed) | Junctions under `%USERPROFILE%\.claude\skills`, `.agents\skills`, `.openclaw\skills`, `%APPDATA%\opencode\skills`, `.gemini\skills` | Not auto-edited — closing message prints the manual `claude mcp remove` command | diff --git a/docs/content/docs/cua-driver/guide/getting-started/linux.mdx b/docs/content/docs/cua-driver/guide/getting-started/linux.mdx index 47ffbf8ae6..668e3e0beb 100644 --- a/docs/content/docs/cua-driver/guide/getting-started/linux.mdx +++ b/docs/content/docs/cua-driver/guide/getting-started/linux.mdx @@ -148,7 +148,7 @@ This is a useful recipe for local CI experiments with the Linux pre-release back ## Distro-specific notes -The canonical install script (`/bin/bash -c "$(curl -fsSL …/install.sh)"`) downloads a Linux x86_64 binary tarball from GitHub Releases, drops it into `~/.cua-driver-rs/packages/releases/-x86_64-unknown-linux-gnu/`, and symlinks `~/.local/bin/cua-driver`. Because Linux support is pre-release, expect distro-specific gaps and prerequisites. +The canonical install script (`/bin/bash -c "$(curl -fsSL …/install.sh)"`) downloads a Linux x86_64 binary tarball from GitHub Releases, drops it into `~/.cua-driver/packages/releases/-x86_64-unknown-linux-gnu/`, and symlinks `~/.local/bin/cua-driver`. Because Linux support is pre-release, expect distro-specific gaps and prerequisites. The bit that varies is **what accessibility / display tooling is pre-installed**: diff --git a/docs/content/docs/cua-driver/reference/changelog.mdx b/docs/content/docs/cua-driver/reference/changelog.mdx new file mode 100644 index 0000000000..c810258234 --- /dev/null +++ b/docs/content/docs/cua-driver/reference/changelog.mdx @@ -0,0 +1,161 @@ +--- +title: Changelog +description: Release history for cua-driver +--- + +# cua-driver Changelog + +All notable changes to cua-driver are documented here. + +These are the `cua-driver-rs` (Rust port) releases. The Rust port ships the same +user-facing `cua-driver` binary for macOS and Windows, plus Linux pre-release +artifacts for early testing. All versions are currently published as GitHub +**prereleases** under the `cua-driver-rs-v*` tag prefix (distinct from the Swift +driver's `cua-driver-v*` tags so their artifacts don't collide). + +The canonical, always-current source is the +[GitHub releases page](https://github.com/trycua/cua/releases). Each GitHub +release body is auto-generated per release from the commits touching +`libs/cua-driver/rust`, including SHA256 checksums and install instructions. + +## 0.5.1 (2026-06-01) + +- **Install home unified on `~/.cua-driver` + collision fix.** The release + installer (`install.sh` → `_install-rust.sh`) was still defaulting its package + home to the legacy `~/.cua-driver-rs`, while the local installer + (`install-local.sh`) and the runtime already used `~/.cua-driver` (renamed in + v0.2.16). That mismatch meant a machine could end up with two homes and two + conflicting installs. The release installer now defaults to `~/.cua-driver` + (still honoring the `CUA_DRIVER_RS_HOME` override), and on every install it (a) + cleans up a prior `install-local` dev build under the shared home — stops the + daemon, removes the `*-local-*` release dirs and the local signing-identity + marker — and (b) sweeps a stale `~/.cua-driver-rs` left by an older release. + Both steps are best-effort, idempotent, and conservative (marker-gated; they + never touch a real release dir, the `current` symlink, or unrelated user + state). TCC grants are preserved — the `/Applications/CuaDriver.app` bundle is + replaced in place, not `tccutil reset`. The Windows installer (`install.ps1`) + already used `~/.cua-driver` and migrated the legacy home, so it is unchanged. + +## 0.5.0 (2026-06-01) + +- **Windows: per-session agent cursors** — the per-session cursor model (macOS + #1779) is ported to Windows, so concurrent sessions are tracked per-session on + Windows too (#1801). +- **Streamable-HTTP MCP transport (parallel multi-agent).** Over stdio, one + `cua-driver mcp` process is a single pipe, so a client's tool calls — including + multiple subagents — serialize. The daemon itself is concurrent. Set + `CUA_DRIVER_RS_MCP_HTTP_PORT=` and the daemon also serves MCP over HTTP + (`POST http://127.0.0.1:/mcp`, loopback only), so each agent opens its + **own** connection and they run truly in parallel — measured 3.6× on 10 + concurrent calls. Per-connection ordering keeps a single agent's sequence + correct; the per-`(pid, window_id)` cache + per-session cursor make concurrent + cross-connection actions safe. `move_cursor` is now `readOnlyHint:true` so MCP + clients also parallelize cursor moves on stdio (mutating tools stay serialized + on purpose) (#1799). +- **Breaking — session identity is now caller-declared.** A "session" (which + owns the agent cursor + per-session state) used to be minted per MCP + connection, so it couldn't be set from the CLI and couldn't span a run that + touches multiple apps. It's now an explicit identity you declare: pass a + `session` id (or call the new `start_session`/`end_session` tools), and the + same id drives the same cursor over MCP, the CLI, or the raw socket, following + the run across any apps/windows. **The cursor is now opt-in:** a run shows a + cursor only when it declares a `session` — anonymous calls execute without one. + Sessions are reclaimed by `end_session` or an idle-TTL (default 300s, override + `CUA_DRIVER_RS_SESSION_IDLE_TTL_SECS`) instead of connection-EOF. Per-session + config + recording keep their existing connection-scoped cleanup as a fallback + when no `session` is declared. Update agents to `start_session` at the start of + a run (the bundled skill + MCP instructions now say so). +- macOS: fix a daemon crash (`EXC_BREAKPOINT` in `AXUIElementCopyActionNames`) + when two sessions drive the same window concurrently. Element actions + (`click`, `type_text`, `set_value`, …) read the target `AXUIElementRef` out + of the per-`(pid, window_id)` cache, but a concurrent `get_window_state` + could replace that cache entry and `CFRelease` the element to zero while the + action was still using it — a use-after-free. The cache now hands out a + retained guard (`CFRetain` under the cache lock, `CFRelease` on drop), so an + in-flight action keeps the element alive across a concurrent refresh. +- macOS (local dev install only): `install-local.sh` now clears a stale TCC + grant left over from a previous signing identity. If you first granted + Accessibility / Screen Recording while the bundle was ad-hoc signed, that + grant was pinned to the per-build `cdhash` and silently stopped matching on + the next rebuild — the daemon read `NOT granted` while System Settings still + showed CuaDriver toggled ON, a dead end re-toggling couldn't fix. The + installer records the signing identity and, when it changes, runs + `tccutil reset` so the next grant re-pins cleanly to the stable self-signed + certificate (after which grants survive all future rebuilds). Release builds + are unaffected — they're CI-signed with a stable identity (#1792 follow-up). + +## 0.4.3 (2026-05-31) + +- macOS: the agent-cursor overlay now actually renders in the `serve` daemon. + It was only wired into the in-process `mcp` path, so in the daemon-proxy + setup (the default) every cursor command was a silent no-op and the agent + cursor never appeared (#1790). +- macOS: `cua-driver permissions grant` no longer spams the TCC prompt — it + raises a single dialog and then polls silently, and the gate honours its + deadline instead of re-prompting (and restarting the daemon) on every + ~25s re-exec (#1791). +- Dev: `install-local` now signs the bundle with a stable self-signed + identity, so TCC grants (Accessibility / Screen Recording) survive rebuilds + instead of resetting on every install (#1792). + +## 0.4.2 (2026-05-31) + +- macOS: guard the SkyLight auth-message selector on macOS 14 Sonoma so the + driver no longer crashes on launch (#1782, #1503). +- macOS: enable Chromium/Electron accessibility trees via `AXManualAccessibility`, + so `get_window_state` can see their window content (#1756). + +## 0.4.1 (2026-05-31) + +- macOS: per-session agent cursors, so concurrent sessions each get their own + visual agent cursor (#1779). +- macOS: ship `AppIcon.icns` in the bundle so `CuaDriver.app` no longer shows a + blank icon (#1780, #1496). +- macOS: guard AppKit initialization so `mcp` runs headless instead of + SIGABRT-ing (#1781, #1724). + +## 0.4.0 (2026-05-31) + +- Daemon session-identity model: the daemon now owns and cleans up + session-scoped recording and config, tying that state to the session that + created it (#1776). + +## 0.3.6 (2026-05-30) + +- macOS: fix permissions status so it reports the driver's real TCC grants (via + the daemon) instead of the calling terminal's — it no longer claims granted + when only the caller holds the grant (#1774). + +## 0.3.5 (2026-05-30) + +- macOS: bind the `serve` socket before the permissions gate, so clients can + connect even while permissions are still being resolved (#1761, #1773). + +## 0.3.4 (2026-05-30) + +- Maintenance release (no path-specific changes). + +## 0.3.3 (2026-05-30) + +- Maintenance release (no path-specific changes). + +## 0.3.2 (2026-05-27) + +- Add a `check-update` CLI verb and a `check_for_update` MCP tool so clients can + detect when a newer driver is available (#1734). + +## 0.3.1 (2026-05-27) + +- First release in the 0.3.x line, which began the macOS TCC / permissions + hardening series. + +## 0.2.x (2026-05-17 – 2026-05-21) + +- Pre-release iteration on the Rust port across versions 0.2.0–0.2.18, + including cross-platform build and packaging work for the macOS, Windows, and + Linux pre-release artifacts. + +## 0.1.x (2026-05-14) + +- Initial `cua-driver-rs` pre-releases. See the + [GitHub releases](https://github.com/trycua/cua/releases) for details. diff --git a/docs/content/docs/cua-driver/reference/mcp-tools.mdx b/docs/content/docs/cua-driver/reference/mcp-tools.mdx index 9c2cd20983..b79b107253 100644 --- a/docs/content/docs/cua-driver/reference/mcp-tools.mdx +++ b/docs/content/docs/cua-driver/reference/mcp-tools.mdx @@ -131,7 +131,7 @@ Optional `electron_debugging_port`: opens a Chrome DevTools Protocol (CDP) serve Optional `webkit_inspector_port`: opens a WebKit inspector server on the specified port (sets WEBKIT_INSPECTOR_SERVER=127.0.0.1:N + TAURI_WEBVIEW_AUTOMATION=1). Use this for Tauri/WebKit-based apps. -Optional `creates_new_application_instance`: when true, forces a new app instance even if one is already running (passes -n to open). +Optional `creates_new_application_instance`: when true, forces a new app instance even if one is already running (passes -n to open). Reach for this in **concurrent multi-agent/multi-session** work — it returns a fresh pid + window so each session drives its own isolated window. Without it, single-instance apps (Calculator, many utilities) hand every caller the same window, so two sessions clobber each other. Optional `additional_arguments`: extra argv strings appended after --args. @@ -141,7 +141,7 @@ Returns the launched app's pid, bundle_id, name, and a `windows` array (same sha - `additional_arguments` (array of string, optional): Extra arguments appended after --args when launching. - `bundle_id` (string, optional): App bundle identifier, e.g. com.apple.calculator. Preferred over name. -- `creates_new_application_instance` (boolean, optional): When true, force a new app instance even if already running (open -n). +- `creates_new_application_instance` (boolean, optional): When true, force a new app instance even if already running (open -n). Use for concurrent multi-agent/multi-session work so each session gets an isolated instance + window instead of sharing one (which makes the sessions clobber each other on single-instance apps). - `electron_debugging_port` (integer, optional): Open a Chrome DevTools Protocol server on this port (appends --remote-debugging-port=N). - `name` (string, optional): App display name. Used only when bundle_id is absent. - `urls` (array of string, optional): Optional file paths or URLs to open with the app (e.g. a folder path for Finder). @@ -517,11 +517,45 @@ Update cua-driver-rs configuration. Changes to capture_mode and max_image_dimens - `experimental_pip_geometry` (string, optional): PiP window size + optional position in `WxH` or `WxH+X+Y` form (e.g. `320x200+24+24`). Applies on next daemon restart. - `max_image_dimension` (integer, optional): Max dimension for screenshot resizing (0 = no limit). -**Per-session agent cursors (macOS).** The agent cursor is owned per MCP session. On the daemon-proxy path every `cua-driver mcp` client mints a session identity, and that identity is the cursor key when the caller passes no explicit `cursor_id` — so two concurrent sessions each drive their own overlay cursor (distinct auto-assigned colour) drawn simultaneously, instead of clobbering one shared cursor last-writer-wins. The key is resolved with the precedence **explicit `cursor_id` > session identity > `"default"`**: an explicit `cursor_id` stays an orthogonal, user-facing handle, so a wrapper can deliberately share or override a cursor across sessions by passing the same `cursor_id`. When a session ends (`session_end`), that session's cursor is removed from the overlay automatically — including on **ungraceful proxy death** (`kill -9`, crash), because the daemon detects the proxy's control-connection EOF and fires `session_end` without any cooperation from the dying proxy. A late, in-flight cursor command that arrives after removal cannot resurrect the cursor (a render-side tombstone keyed on the session id). The anonymous / one-shot `cua-driver call` path (no session identity, no `cursor_id`) maps to the seeded `"default"` cursor, which is never removed — backward compatible. (Windows/Linux keep the single shared cursor today.) +**Sessions and per-session agent cursors (macOS).** A **session** is a +caller-declared identity for one agent run — not a property of the MCP +connection. Declare it with `start_session` (or just pass a `session` id on your +actions); the same id drives the same agent cursor, per-session config, and +recording over MCP, the CLI, or the raw socket, and follows the run across any +number of apps/windows. The cursor's colour is auto-derived from the id, so +concurrent runs are visually distinct and drawn simultaneously. **The cursor is +opt-in:** a run shows a cursor only when it declares a `session` — anonymous +calls (no `session`) execute without one. `cursor_id` is a legacy alias for +`session`. A session is reclaimed by `end_session` or an idle-TTL (default 300s, +override `CUA_DRIVER_RS_SESSION_IDLE_TTL_SECS`); a late in-flight command after +teardown cannot resurrect the cursor (render-side tombstone keyed on the id). For +concurrent runs/subagents, give each its own `session` (and pass +`creates_new_application_instance:true` to `launch_app` so they don't share a +window). Per-session config + recording fall back to connection-scoped cleanup +when no `session` is declared. (Windows/Linux have no overlay cursor today; the +session identity still scopes their per-session config + recording.) + +### start_session + +Declare a session — a named, color-coded identity for the current agent run. Pass a stable `session` id; the agent cursor, per-session config, and recording all key on it, and it follows the run across apps/windows. The cursor appears on the session's first action. Idempotent (re-calling refreshes the idle-TTL). End it with `end_session` or let the idle-TTL reclaim it. + +**Arguments:** + +- `session` (string, required): Stable session id for this run (e.g. `"research-run-1"`). + +### end_session + +End a session declared with `start_session`: removes its agent cursor, stops any recording it owns, and clears its per-session config. Call when a run finishes so its cursor doesn't linger. Idempotent. + +**Arguments:** + +- `session` (string, required): The session id to end. ### set_agent_cursor_enabled -Show or hide the agent cursor overlay for a cursor instance. With no `cursor_id`, this targets the calling session's own cursor (see the per-session note above). +Show or hide the agent cursor overlay for a cursor instance. The overlay is **ON by default** and each MCP session automatically owns its own cursor — you do not need to call this to make the cursor appear; use it only to hide (`enabled:false`) or re-show (`enabled:true`) it. With no `cursor_id`, this targets the calling session's own cursor (see the per-session note above). + +> **Visibility caveat (AX runs).** On a pure accessibility-action run (clicking by `element_index`), the session cursor seeds on-screen and pulses on its very first action rather than playing a long glide, so it is easy to miss in a screen recording. For a clearly *gliding* cursor in a demo, issue a pixel `click({pid,x,y})` or a `move_agent_cursor` first to put the cursor on-screen; subsequent AX clicks then glide normally. **Arguments:** diff --git a/docs/content/docs/cua-driver/reference/meta.json b/docs/content/docs/cua-driver/reference/meta.json index 6a86740522..e96d110716 100644 --- a/docs/content/docs/cua-driver/reference/meta.json +++ b/docs/content/docs/cua-driver/reference/meta.json @@ -2,5 +2,5 @@ "title": "Reference", "description": "CLI and MCP reference documentation", "icon": "FileText", - "pages": ["cli-reference", "mcp-tools", "limits"] + "pages": ["cli-reference", "mcp-tools", "limits", "changelog"] } diff --git a/libs/cua-driver/rust/.bumpversion.cfg b/libs/cua-driver/rust/.bumpversion.cfg index 67a1b98c00..b5af4a47e9 100644 --- a/libs/cua-driver/rust/.bumpversion.cfg +++ b/libs/cua-driver/rust/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 0.4.1 +current_version = 0.4.3 commit = True tag = True tag_name = cua-driver-rs-v{new_version} diff --git a/libs/cua-driver/rust/Cargo.lock b/libs/cua-driver/rust/Cargo.lock index 526d1cef56..66afcb0b8d 100644 --- a/libs/cua-driver/rust/Cargo.lock +++ b/libs/cua-driver/rust/Cargo.lock @@ -275,7 +275,7 @@ dependencies = [ [[package]] name = "cua-driver" -version = "0.4.0" +version = "0.5.1" dependencies = [ "anyhow", "async-trait", @@ -305,7 +305,7 @@ dependencies = [ [[package]] name = "cua-driver-core" -version = "0.4.0" +version = "0.5.1" dependencies = [ "anyhow", "async-trait", @@ -320,7 +320,7 @@ dependencies = [ [[package]] name = "cua-driver-uia" -version = "0.4.0" +version = "0.5.1" dependencies = [ "anyhow", "cua-driver-core", @@ -335,7 +335,7 @@ dependencies = [ [[package]] name = "cursor-overlay" -version = "0.4.0" +version = "0.5.1" dependencies = [ "anyhow", "image", @@ -488,7 +488,7 @@ checksum = "98de4bbd547a563b716d8dfa9aad1cb19bfab00f4fa09a6a4ed21dbcf44ce9c4" [[package]] name = "focus-monitor-win" -version = "0.4.0" +version = "0.5.1" dependencies = [ "windows 0.58.0", ] @@ -1192,7 +1192,7 @@ checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" [[package]] name = "pip-preview" -version = "0.4.0" +version = "0.5.1" dependencies = [ "anyhow", "serde_json", @@ -1207,7 +1207,7 @@ checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" [[package]] name = "platform-linux" -version = "0.4.0" +version = "0.5.1" dependencies = [ "anyhow", "async-trait", @@ -1228,7 +1228,7 @@ dependencies = [ [[package]] name = "platform-macos" -version = "0.4.0" +version = "0.5.1" dependencies = [ "anyhow", "async-trait", @@ -1261,7 +1261,7 @@ dependencies = [ [[package]] name = "platform-windows" -version = "0.4.0" +version = "0.5.1" dependencies = [ "anyhow", "async-trait", @@ -1269,6 +1269,7 @@ dependencies = [ "cua-driver-core", "cursor-overlay", "image", + "indexmap", "pip-preview", "serde", "serde_json", diff --git a/libs/cua-driver/rust/Cargo.toml b/libs/cua-driver/rust/Cargo.toml index ecf1e7fc7c..e111672560 100644 --- a/libs/cua-driver/rust/Cargo.toml +++ b/libs/cua-driver/rust/Cargo.toml @@ -13,7 +13,7 @@ members = [ ] [workspace.package] -version = "0.4.1" +version = "0.5.1" edition = "2021" authors = ["trycua"] license = "MIT" diff --git a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md index 13d4f2b906..4a724a9c31 100644 --- a/libs/cua-driver/rust/Skills/cua-driver/SKILL.md +++ b/libs/cua-driver/rust/Skills/cua-driver/SKILL.md @@ -124,12 +124,31 @@ cua-driver stop ## Agent cursor overlay Visual cursor overlay for demos and screen recordings. Default: -enabled. Toggle with `cua-driver set_agent_cursor_enabled -'{"enabled":true|false}'`. A triangle pointer Bezier-glides to each -click target, ring-ripples on landing, idle-hides after ~1.5s. -Motion knobs: `set_agent_cursor_motion` takes any subset of -`start_handle`, `end_handle`, `arc_size`, `arc_flow`, `spring` — -tuneable at runtime, persisted to config. +enabled — you do NOT need to enable it. Toggle with +`cua-driver set_agent_cursor_enabled '{"enabled":true|false}'` only to +hide or re-show it. A triangle pointer Bezier-glides to each click +target, ring-ripples on landing, idle-hides after ~1.5s. Motion knobs: +`set_agent_cursor_motion` takes any subset of `start_handle`, +`end_handle`, `arc_size`, `arc_flow`, `spring` — tuneable at runtime, +persisted to config. + +**Per-session cursors.** Each MCP session automatically owns its own +cursor, keyed by the session's id (the proxy mints one session id per +MCP connection and the daemon scopes the cursor, config overrides, and +recording to it). You normally pass nothing — the session key is wired +through for you. Pass an explicit `cursor_id` only to *deliberately +share* one cursor across sessions. When a session ends (the MCP client +disconnects) its cursor is removed automatically. + +**Visibility caveat (AX runs).** On a pure accessibility-action run +(clicking by `element_index`), the first action **seeds the cursor +on-screen a short distance from the target and plays a brief glide + +pulse** — not the long Bezier sweep a cursor already on-screen would +trace from its previous spot. It's subtle and easy to miss in a +recording. If you want a clearly *gliding* cursor for a demo or screen +recording, do a pixel click (`click({pid,x,y})`) or a `move_agent_cursor` +first to put the cursor on-screen; subsequent AX actions then glide the +full path normally. Requires the daemon process's UI runloop, which `cua-driver serve` / `mcp` bootstraps. One-shot CLI invocations skip the overlay entirely. @@ -217,18 +236,47 @@ last resort. ## The canonical loop ``` +start_session(session) # once per run: declares this run's identity launch_app(target) → pick window_id from the returned `windows` array (or call list_windows(pid) separately) → get_window_state(pid, window_id) - → [act] # every action also takes (pid, window_id) + → [act] # every action also takes (pid, window_id) + your `session` → get_window_state(pid, window_id) → verify +end_session(session) # when the run finishes ``` `launch_app` now returns a `windows` array alongside the pid, so the common case collapses to two calls (`launch_app` → `get_window_state`) without a separate `list_windows` hop. +**Declare a session.** A session is *your run's* identity — a stable id +you choose (`"research-1"`), declared with `start_session` and passed as +`session` on every action. It owns your agent cursor (a distinct colour +per id), follows the run across any apps/windows, and is the same whether +you drive over MCP, the CLI, or the socket. The cursor is **opt-in**: it +appears only once you declare a session (anonymous actions run cursor-less). +End with `end_session` (or the idle-TTL reclaims it). + +**Concurrent runs/subagents:** `launch_app` is idempotent — two runs that +launch the same app get the **same** instance (and on single-instance apps +like Calculator, the same window), so they clobber each other. Give each run +its **own `session`** (→ its own cursor) AND pass +`creates_new_application_instance: true` to `launch_app` (→ its own window). +The element cache is keyed on `(pid, window_id)` and the cursor on `session`, +so distinct instances + distinct sessions keep the runs fully separated. + +**Parallelism vs. ordering.** Distinct sessions give distinct *cursors*, not +distinct *connections*. Subagents that share one `cua-driver mcp` (stdio) +connection have their tool calls **serialized** by the transport — they take +turns, not run in parallel. That's not a correctness problem (session + window +isolation means they can't collide), just a throughput one. For genuinely +parallel agents, give each its **own connection**: separate `cua-driver mcp` +processes, or point each agent's MCP client at the daemon's HTTP endpoint +(`CUA_DRIVER_RS_MCP_HTTP_PORT` → `POST http://127.0.0.1:/mcp`). The daemon +serves connections concurrently; per-connection ordering keeps each agent's own +sequence (e.g. `3 → + → 1 → =`) correct. + `list_apps` is for app-level discovery (answering "what's installed / running / frontmost?") — not part of the core action loop. Skip it in the loop. For **window-level** questions — "does this app have a @@ -474,10 +522,13 @@ respective companion files. is for visual disambiguation, not coordinates. Use the `element_index`. - **Prefer accessibility actions over pixels.** `click({pid, x, y})` - works for canvas / WebView regions, but it lands blindly and skips - the agent-cursor overlay. Exhaust accessibility paths (menu bars, - cmd-k palettes, toolbar items, keyboard shortcuts) before dropping - to coordinates. + works for canvas / WebView regions, but it lands blindly on raw + coordinates. Exhaust accessibility paths (menu bars, cmd-k palettes, + toolbar items, keyboard shortcuts) before dropping to coordinates. + (The AX path does **not** skip the agent-cursor overlay — it seeds and + pulses the session cursor and draws a focus rect on the targeted + element; it just doesn't play a long glide on the very first action. + See "Agent cursor overlay" for the demo-recording caveat.) - **Never** drive destructive actions (delete files, close unsaved documents, send messages, submit forms) without explicit user intent for that specific destructive step. diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/element_cache.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/element_cache.rs index 5c3e9cf22b..3e4993f1fb 100644 --- a/libs/cua-driver/rust/crates/cua-driver-core/src/element_cache.rs +++ b/libs/cua-driver/rust/crates/cua-driver-core/src/element_cache.rs @@ -25,9 +25,12 @@ //! //! Each platform's `ElementCache` is now a thin wrapper around an //! `ElementCacheCore`. Specialised -//! accessors (`get_element_ptr`, `get_element_center` on Windows, -//! `get_element_key` on Linux) call `with_snapshot` and project the -//! field they care about. +//! accessors (`get_element_retained` on macOS, `get_element_ptr` / +//! `get_element_center` on Windows, `get_element_key` on Linux) call +//! `with_snapshot` and project the field they care about. The macOS +//! accessor retains the element under the lock so a concurrent +//! `insert` (snapshot replace) can't free it mid-action — see +//! `platform-macos/src/ax/cache.rs::RetainedElement`. use std::collections::HashMap; use std::hash::Hash; diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs index 6fbc141414..8243c26f1a 100644 --- a/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs +++ b/libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs @@ -24,6 +24,7 @@ pub mod recording_tools; pub mod recording_zoom; pub mod server; pub mod session; +pub mod session_tools; pub mod text_sanitize; pub mod tool; pub mod tool_args; diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs index e27e44d63a..f004497a1f 100644 --- a/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs +++ b/libs/cua-driver/rust/crates/cua-driver-core/src/protocol.rs @@ -191,13 +191,14 @@ fn agent_instructions() -> String { Tools let you interact with any app without stealing keyboard focus or moving the visible cursor. Prefer element_index ({tree_kind}) paths over pixel coordinates — they work on backgrounded/hidden windows. Workflow per turn: -1. launch_app → idempotent, returns pid + windows array in one call +0. start_session(session) once at the start of a run → declares THIS run's identity (a stable id you choose, e.g. "research-1"). Pass that same `session` on every action below. It owns your agent cursor (a distinct color per id) and follows the run across apps/windows. End with end_session(session) when done. Concurrent runs/subagents each use their OWN `session`. (Omitting `session` still works, just with no cursor.) +1. launch_app → idempotent, returns pid + windows array in one call. Pass creates_new_application_instance:true if another run may touch the same app, so you get your own window. 2. (skip list_windows when launch_app already returned a single window) 3. get_window_state(pid, window_id) → refresh the {tree_kind} snapshot, get element indices -4. click/type_text/press_key using element_index from step 3 +4. click/type_text/press_key using element_index from step 3 (+ your `session`) 5. get_window_state(pid, window_id) again → verify the action landed -Agent cursor: set_agent_cursor_* tools visualise where the agent is acting without affecting the real mouse pointer. +Agent cursor: a per-SESSION overlay cursor visualises where a run is acting without moving the real pointer. It is shown only for a DECLARED session (pass `session`), is color-coded by the session id, and is removed by end_session or the idle-TTL. The same id over MCP, the CLI, or the raw socket drives the same cursor. set_agent_cursor_* tools hide/show/customise it. Note: a pure accessibility-action (element_index) click snaps the cursor with a brief pulse on its first action rather than a long glide, so it can be easy to miss — issue a pixel click or move_cursor first for a visibly gliding demo/recording. If a `cua-driver` skill is loaded in your harness (Claude Code / Codex / OpenClaw / OpenCode dirs), prefer its detailed workflow — SKILL.md plus {platform_skill_pointer}. Install with `cua-driver skills install` if not yet present."# ) diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs index 0bae57a952..97a0be020d 100644 --- a/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs +++ b/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs @@ -57,7 +57,11 @@ pub async fn run(registry: Arc) -> anyhow::Result<()> { Ok(()) } -async fn handle_request(req: Request, id: serde_json::Value, registry: &Arc) -> Response { +/// Dispatch one MCP JSON-RPC request against the registry (initialize / +/// tools/list / tools/call). Shared by the stdio loop above and the +/// daemon's HTTP transport (`cua-driver`'s `mcp_http`) so both speak the +/// exact same MCP semantics. +pub async fn handle_request(req: Request, id: serde_json::Value, registry: &Arc) -> Response { match req.method.as_str() { "initialize" => Response::ok(id, initialize_result()), diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs index aa96bb9337..8c41056e88 100644 --- a/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs +++ b/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs @@ -18,13 +18,32 @@ //! `recording.rs` — a registry-free, platform-pluggable hook set with no //! reverse coupling from core into the platform crates. -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; use std::sync::{Mutex, OnceLock}; +use std::time::{Duration, Instant}; type SessionEndHook = Box; static SESSION_END_HOOKS: OnceLock>> = OnceLock::new(); +/// Last-activity timestamp per live session id. A session is "touched" every +/// time a tool call carries its explicit `session` id (see the daemon boundary +/// in `serve.rs`). The idle-TTL sweep ([`evict_idle`]) ends sessions that +/// haven't been touched within the TTL — this is the cleanup path that replaces +/// connection-EOF reaping now that a session is a caller-declared identity, not +/// a per-MCP-connection one. `"default"` and empty ids are never tracked (they +/// are the anonymous, cursor-less fallback). +static SESSION_ACTIVITY: OnceLock>> = OnceLock::new(); + +fn activity() -> &'static Mutex> { + SESSION_ACTIVITY.get_or_init(|| Mutex::new(HashMap::new())) +} + +/// Whether `id` is a real, trackable session id (not the anonymous fallback). +fn is_trackable(id: &str) -> bool { + !id.is_empty() && id != "default" +} + /// Session ids that have already had their `session_end` fired. Dedupes the /// control-connection EOF teardown (the reaper) against any stray legacy /// `session_end` method that a mixed-version (new proxy / old proxy) rollout @@ -80,6 +99,59 @@ pub fn is_session_ended(session_id: &str) -> bool { ended_sessions().lock().unwrap().contains(session_id) } +/// Record activity for an explicit session id, resetting its idle-TTL clock. +/// Called at the daemon boundary on every tool call that carries an explicit +/// `session`. No-op for the anonymous fallback (`"default"` / empty) and for a +/// session that has already ended (so a late in-flight call can't resurrect a +/// reaped session's TTL entry). +pub fn touch_session(session_id: &str) { + if !is_trackable(session_id) || is_session_ended(session_id) { + return; + } + activity() + .lock() + .unwrap() + .insert(session_id.to_owned(), Instant::now()); +} + +/// End a session explicitly (the `end_session` tool / `session end` CLI verb): +/// drop its idle-TTL entry and fan `fire_session_end` out to every cleanup hook +/// (overlay remove, recording stop, config-override clear). Idempotent via +/// `fire_session_end`'s dedupe. No-op for the anonymous fallback. +pub fn end_session(session_id: &str) { + if !is_trackable(session_id) { + return; + } + activity().lock().unwrap().remove(session_id); + fire_session_end(session_id); +} + +/// End every session whose last activity is older than `ttl`, returning the ids +/// ended. This is the idle-TTL sweep the daemon runs periodically: a +/// caller-declared session is no longer tied to a connection's lifetime, so a +/// run that finishes (or crashes) without calling `end_session` is reclaimed +/// here instead of leaking its cursor / recording. Sessions touched within the +/// TTL are left untouched. +pub fn evict_idle(ttl: Duration) -> Vec { + let now = Instant::now(); + let stale: Vec = { + let map = activity().lock().unwrap(); + map.iter() + .filter(|(_, last)| now.duration_since(**last) >= ttl) + .map(|(id, _)| id.clone()) + .collect() + }; + for id in &stale { + end_session(id); + } + stale +} + +/// Number of sessions with a live idle-TTL entry. Diagnostics only. +pub fn active_session_count() -> usize { + activity().lock().unwrap().len() +} + #[cfg(test)] mod tests { use super::*; @@ -112,4 +184,35 @@ mod tests { "hook must run exactly once for a given session id" ); } + + #[test] + fn touch_then_evict_by_ttl() { + let sid = "test-ttl-session-DDEEFF"; + touch_session(sid); + // A huge TTL leaves it alone (just touched). + assert!(evict_idle(Duration::from_secs(3600)).iter().all(|s| s != sid)); + // A zero TTL treats any prior activity as idle → evicts it. + let evicted = evict_idle(Duration::ZERO); + assert!(evicted.iter().any(|s| s == sid), "zero-TTL must evict a touched session"); + assert!(is_session_ended(sid), "evicted session is ended"); + } + + #[test] + fn anonymous_ids_are_never_tracked() { + touch_session("default"); + touch_session(""); + // Neither shows up under a zero-TTL sweep (they were never inserted). + let evicted = evict_idle(Duration::ZERO); + assert!(!evicted.iter().any(|s| s == "default" || s.is_empty())); + } + + #[test] + fn end_session_is_explicit_teardown() { + let sid = "test-end-session-112233"; + touch_session(sid); + end_session(sid); + assert!(is_session_ended(sid)); + // Its TTL entry is gone, so a later sweep doesn't re-fire for it. + assert!(!evict_idle(Duration::ZERO).iter().any(|s| s == sid)); + } } diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs new file mode 100644 index 0000000000..dda88f0d39 --- /dev/null +++ b/libs/cua-driver/rust/crates/cua-driver-core/src/session_tools.rs @@ -0,0 +1,141 @@ +//! `start_session` / `end_session` tools. +//! +//! A session is a **caller-declared** identity for an agent run (see +//! [`crate::session`]). It owns the agent cursor and is the key for +//! per-session config + recording. These tools bookend a run's lifetime +//! explicitly, decoupled from the MCP connection: the same `session` id works +//! identically over MCP, the CLI (`--session`), or the raw socket, and a run +//! can span any number of apps/windows. +//! +//! The daemon mirrors an explicit `session` arg into the reserved `_session_id` +//! key (see `serve.rs::apply_session_identity`), so these tools accept either — +//! `session` is the public name. + +use crate::protocol::ToolResult; +use crate::tool::{Tool, ToolDef}; +use async_trait::async_trait; +use serde_json::{json, Value}; +use std::sync::OnceLock; + +/// Read the explicit session id from a tool call (`session`, or its daemon +/// mirror `_session_id`). Empty / missing → `None`. +fn session_id_of(args: &Value) -> Option { + let obj = args.as_object()?; + ["session", "_session_id"] + .into_iter() + .find_map(|k| obj.get(k).and_then(|v| v.as_str()).filter(|s| !s.is_empty())) + .map(|s| s.to_owned()) +} + +// ── start_session ───────────────────────────────────────────────────────────── + +pub struct StartSessionTool; + +static START_DEF: OnceLock = OnceLock::new(); + +#[async_trait] +impl Tool for StartSessionTool { + fn def(&self) -> &ToolDef { + START_DEF.get_or_init(|| ToolDef { + name: "start_session".into(), + description: + "Declare a session — a named, color-coded identity for THIS agent run. \ + Pass a stable `session` id; the agent cursor, per-session config, and \ + recording all key on it, and it follows the run across any apps/windows. \ + The cursor's color is derived from the id, so distinct runs are visually \ + distinct. A cursor is shown only for a declared session — call this (or \ + pass `session` on your first action) to opt in. Idempotent: re-calling \ + with the same id just refreshes its idle-TTL. End it with `end_session` \ + (or let the idle-TTL reclaim it). Concurrent runs/subagents each pass \ + their own `session` to get their own cursor." + .into(), + input_schema: json!({ + "type": "object", + "required": ["session"], + "properties": { + "session": { + "type": "string", + "description": "Stable session id for this run (e.g. \"research-run-1\")." + } + }, + "additionalProperties": true + }), + read_only: false, + destructive: false, + idempotent: true, + open_world: false, + }) + } + + async fn invoke(&self, args: Value) -> ToolResult { + let Some(id) = session_id_of(&args) else { + return ToolResult::error( + "start_session requires a non-empty `session` id.", + ); + }; + // Refresh (or begin) the session's idle-TTL clock. The cursor appears on + // the first action carrying this `session`. + crate::session::touch_session(&id); + ToolResult::text(format!("✅ Session '{id}' is active.")) + .with_structured(json!({ "session": id, "active": true })) + } +} + +// ── end_session ─────────────────────────────────────────────────────────────── + +pub struct EndSessionTool; + +static END_DEF: OnceLock = OnceLock::new(); + +#[async_trait] +impl Tool for EndSessionTool { + fn def(&self) -> &ToolDef { + END_DEF.get_or_init(|| ToolDef { + name: "end_session".into(), + description: + "End a session declared with `start_session`: removes its agent cursor, \ + stops any recording it owns, and clears its per-session config. Call this \ + when a run finishes so its cursor doesn't linger (otherwise the idle-TTL \ + reclaims it after a period of inactivity). Idempotent." + .into(), + input_schema: json!({ + "type": "object", + "required": ["session"], + "properties": { + "session": { + "type": "string", + "description": "The session id to end." + } + }, + "additionalProperties": true + }), + read_only: false, + destructive: true, + idempotent: true, + open_world: false, + }) + } + + async fn invoke(&self, args: Value) -> ToolResult { + let Some(id) = session_id_of(&args) else { + return ToolResult::error("end_session requires a non-empty `session` id."); + }; + crate::session::end_session(&id); + ToolResult::text(format!("✅ Session '{id}' ended.")) + .with_structured(json!({ "session": id, "active": false })) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn session_id_of_reads_session_then_mirror() { + assert_eq!(session_id_of(&json!({ "session": "a" })).as_deref(), Some("a")); + assert_eq!(session_id_of(&json!({ "_session_id": "b" })).as_deref(), Some("b")); + assert_eq!(session_id_of(&json!({ "session": "", "_session_id": "c" })).as_deref(), Some("c")); + assert_eq!(session_id_of(&json!({})), None); + assert_eq!(session_id_of(&json!({ "session": "" })), None); + } +} diff --git a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs index 7fcef7fc5d..7ec6ce17fa 100644 --- a/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs +++ b/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs @@ -87,6 +87,15 @@ impl ToolRegistry { self.register(Box::new(ReplayTrajectoryTool)); } + /// Register the platform-independent session-lifecycle tools + /// (`start_session` / `end_session`). Call alongside + /// `register_recording_tools` from each platform's `register_all`. + pub fn register_session_tools(&mut self) { + use crate::session_tools::{EndSessionTool, StartSessionTool}; + self.register(Box::new(StartSessionTool)); + self.register(Box::new(EndSessionTool)); + } + /// Wire up the replay tool's weak self-reference. /// Call this once, immediately after `Arc::new(registry)`. pub fn init_self_weak(self: &Arc) { diff --git a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs index 3585984569..fb74f57ce6 100644 --- a/libs/cua-driver/rust/crates/cua-driver/src/cli.rs +++ b/libs/cua-driver/rust/crates/cua-driver/src/cli.rs @@ -59,6 +59,13 @@ pub enum Command { /// (checked inside the gate itself), so the flag is only one of /// two opt-out signals. no_permissions_gate: bool, + /// True when `--claude-code-computer-use-compat` is on argv. The MCP + /// proxy forwards this flag to the daemon it auto-launches (see + /// `launch_daemon_and_wait`) so the proxy path registers the compat + /// `screenshot` surface, not just the in-process path. Without it the + /// flag was a no-op for `cua-driver mcp --claude-code-computer-use-compat`, + /// which always routes through the proxy on an installed bundle. + claude_code_compat: bool, }, Stop { socket: Option }, Status { socket: Option }, @@ -182,6 +189,28 @@ pub fn parse_command() -> Command { println!(" --no-daemon-relaunch Stay in-process; skip auto-launching the CuaDriver daemon."); println!(" Also: CUA_DRIVER_RS_MCP_NO_RELAUNCH=1"); println!(" --socket Override the daemon UDS path used by the proxy fallback."); + println!(" --claude-code-computer-use-compat"); + println!(" Select the Claude Code computer-use compat surface."); + println!(" Now forwarded to the proxy-launched daemon (was a no-op"); + println!(" on the proxy path — the path you actually run — because"); + println!(" the daemon hardcoded compat=false). Note: the compat"); + println!(" screenshot tool itself was removed in #1692, so the flag"); + println!(" has no tool-surface effect today; the wiring is in place"); + println!(" for any future compat-gated tool."); + println!(); + println!("agent cursor overlay (serve / mcp only — needs the daemon UI runloop):"); + println!(" The overlay is ON by default: every MCP session automatically gets its own"); + println!(" cursor (keyed by session id) that shows where the agent acts without moving the"); + println!(" real pointer. It is removed when the session ends. A pure accessibility (AX)"); + println!(" action snaps the cursor with a brief pulse on its first action instead of a long"); + println!(" glide, so it can be easy to miss — do a pixel click or move_agent_cursor first"); + println!(" for a visibly gliding demo. These flags tune the overlay on `serve`/`mcp`:"); + println!(" --no-overlay Disable the cursor overlay entirely for this daemon."); + println!(" --cursor-id Name the default cursor instance (default: 'default')."); + println!(" --cursor-icon Use a custom PNG cursor icon."); + println!(" --cursor-palette Pick a built-in colour palette for the cursor."); + println!(" (These are no-ops for one-shot CLI calls like `cua-driver call` — the overlay"); + println!(" needs the long-lived AppKit runloop that only `serve` / `mcp` keep alive.)"); println!(); println!("doctor options:"); println!(" --json Emit the probe report as JSON for scripting."); @@ -260,6 +289,7 @@ pub fn parse_command() -> Command { socket, // Bare flag — present anywhere on argv counts as "skip the gate". no_permissions_gate: args.iter().any(|a| a == "--no-permissions-gate"), + claude_code_compat, }, Some("stop") => Command::Stop { socket }, Some("status") => Command::Status { socket }, @@ -562,7 +592,11 @@ pub fn should_use_daemon_proxy(no_daemon_relaunch: bool) -> bool { /// `waitForDaemon`. Split into one Rust function because we don't /// need the post-launch probe separation Swift has. #[cfg(target_os = "macos")] -pub fn launch_daemon_and_wait(socket_path: &str, timeout_secs: u64) -> anyhow::Result<()> { +pub fn launch_daemon_and_wait( + socket_path: &str, + timeout_secs: u64, + claude_code_compat: bool, +) -> anyhow::Result<()> { use std::process::{Command as Cmd, Stdio}; use std::time::{Duration, Instant}; @@ -579,6 +613,20 @@ pub fn launch_daemon_and_wait(socket_path: &str, timeout_secs: u64) -> anyhow::R open_args.push("--socket"); open_args.push(socket_path); } + // Thread the Claude-Code compat flag through to the daemon. Without this + // the proxy-spawned daemon always called build_macos_registry() (compat + // hardcoded false), so `cua-driver mcp --claude-code-computer-use-compat` + // SILENTLY DROPPED the flag on the proxy path — the path users actually + // run on an installed bundle. Today this is latent: the compat screenshot + // tool was removed in #1692, so `register_all(compat)` ignores the flag and + // the served surface is identical either way. But the flag was being lost + // before reaching the daemon at all, so the moment any compat-gated tool is + // re-introduced the proxy path would not honour it. This makes the flag + // travel end-to-end. Only honoured on a freshly-launched daemon — a + // pre-existing daemon keeps whatever surface it launched with. + if claude_code_compat { + open_args.push("--claude-code-computer-use-compat"); + } let status = Cmd::new("/usr/bin/open") // `-n` forces a new instance: CuaDriver.app might already be @@ -630,7 +678,10 @@ pub fn launch_daemon_and_wait(socket_path: &str, timeout_secs: u64) -> anyhow::R /// `open` if needed), then `crate::proxy::run_proxy` against its /// socket. Builds its own tokio runtime — same shape as the other /// `run_*` helpers in this file that own their event loop. -pub fn run_mcp_via_daemon_proxy(socket: Option) -> anyhow::Result<()> { +pub fn run_mcp_via_daemon_proxy( + socket: Option, + claude_code_compat: bool, +) -> anyhow::Result<()> { // Windows: prefer the uiAccess'd worker pipe over the regular daemon pipe // when both are running, so MCP tool calls land in a process that can // bypass UIPI for UWP apps. The protocol on both pipes is identical so @@ -678,8 +729,10 @@ pub fn run_mcp_via_daemon_proxy(socket: Option) -> anyhow::Result<()> { auto-launching the daemon via `open -n -g -a CuaDriver --args serve{socket_suffix}` \ and proxying MCP requests through it. Pass --no-daemon-relaunch to stay in-process." ); - launch_daemon_and_wait(&socket_path, 10)?; + launch_daemon_and_wait(&socket_path, 10, claude_code_compat)?; } + #[cfg(not(target_os = "macos"))] + let _ = claude_code_compat; // On Linux / Windows there's no equivalent `open -a CuaDriver` // mechanism to spawn a daemon attributed to the user's // interactive session. The caller is expected to have one @@ -1603,7 +1656,8 @@ fn run_permissions_grant() { "A dialog titled \u{201c}Cua Driver\u{201d} will appear — approve Accessibility \ and Screen Recording in System Settings, then this command continues." ); - if let Err(e) = launch_daemon_and_wait(&socket, 180) { + // Permissions-grant launch never needs the compat screenshot surface. + if let Err(e) = launch_daemon_and_wait(&socket, 180, false) { eprintln!("\nDidn't detect the CuaDriver daemon: {e}"); eprintln!( "If you haven't yet, grant Accessibility + Screen Recording to CuaDriver \ diff --git a/libs/cua-driver/rust/crates/cua-driver/src/main.rs b/libs/cua-driver/rust/crates/cua-driver/src/main.rs index 3c6b9d8dfe..81a6770202 100644 --- a/libs/cua-driver/rust/crates/cua-driver/src/main.rs +++ b/libs/cua-driver/rust/crates/cua-driver/src/main.rs @@ -28,6 +28,7 @@ mod autostart; mod bundle; mod cli; mod doctor; +mod mcp_http; mod proxy; mod serve; mod skills; @@ -224,7 +225,7 @@ fn main() { cli::run_call(reg, &tool, json_args, screenshot_out_file, socket); return; } - cli::Command::Serve { socket, no_permissions_gate } => { + cli::Command::Serve { socket, no_permissions_gate, claude_code_compat } => { // Long-running daemon — kick off the background update check // before any blocking work so the banner can land on stderr // early in the serve lifecycle. @@ -259,7 +260,27 @@ fn main() { None => pip_preview::PipConfig::from_args(), }; maybe_init_pip(); - let reg = Arc::new(build_macos_registry()); + + // Agent-cursor overlay. The DAEMON is the process that actually + // performs clicks / AX presses, so the overlay NSWindow + render + // loop must run HERE — not only in the in-process `mcp` arm. In the + // daemon-proxy setup (`mcp` relaunches `open -n -g … serve` and + // proxies to it), the proxy never renders and, before this, neither + // did the daemon — so every cursor command was a silent no-op and + // the agent cursor never appeared. Init the channel before spawning + // the serve thread so `run_on_main_thread()` always finds it ready + // (mirrors the Mcp arm). + let cursor_cfg = cursor_overlay::CursorConfig::from_args(); + if cursor_cfg.enabled { + platform_macos::cursor::overlay::init(cursor_cfg.clone()); + } + + // Honour the compat flag forwarded by the MCP proxy + // (launch_daemon_and_wait passes `serve + // --claude-code-computer-use-compat`). The Serve arm is the daemon + // the proxy talks to, so without this the proxy path always served + // the full screenshot tool regardless of the client's request. + let reg = Arc::new(build_macos_registry_with_compat(claude_code_compat)); reg.init_self_weak(); let sp = socket.unwrap_or_else(serve::default_socket_path); let pid_path = serve::default_pid_file_path(); @@ -320,6 +341,15 @@ fn main() { // stays up as long as the daemon does. if pip_cfg.enabled { platform_macos::pip::run_appkit_main_loop(); + } else if cursor_cfg.enabled { + // Render the agent-cursor overlay: park the main thread in the + // AppKit run loop so the overlay NSWindow draws. `run_on_main_thread` + // self-guards on `has_graphic_access()` and returns immediately + // when the daemon has no Window Server session — fall through to + // join so the daemon still serves headless. The serve thread runs + // on its background thread regardless. + platform_macos::cursor::overlay::run_on_main_thread(); + let _ = serve_handle.join(); } else { let _ = serve_handle.join(); } @@ -401,7 +431,7 @@ fn main() { // attribution and forwards stdio MCP through its socket. // Issue #1525 / mirror of Swift PR #1479. if cli::should_use_daemon_proxy(no_daemon_relaunch) { - if let Err(e) = cli::run_mcp_via_daemon_proxy(socket) { + if let Err(e) = cli::run_mcp_via_daemon_proxy(socket, claude_code_compat) { eprintln!("cua-driver-rs: {e}"); std::process::exit(1); } @@ -540,14 +570,17 @@ fn main() -> anyhow::Result<()> { }).join().ok(); return Ok(()); } - cli::Command::Serve { socket, no_permissions_gate } => { + cli::Command::Serve { socket, no_permissions_gate, claude_code_compat } => { // Long-running daemon — kick off the background update check // before any blocking work so the banner can land on stderr. version_check::maybe_announce_update(); // The Rust permissions gate is macOS-only (TCC concept). // On Windows / Linux the flag is silently accepted for - // CLI uniformity and ignored. + // CLI uniformity and ignored. The Claude-Code compat screenshot + // surface is likewise macOS-only (register_tools_with_compat), + // so the flag is accepted-and-ignored here for CLI uniformity. let _ = no_permissions_gate; + let _ = claude_code_compat; // Serve mode needs the cursor overlay just like MCP mode. let cursor_cfg = cursor_overlay::CursorConfig::from_args(); let reg = Arc::new(build_registry(cursor_cfg)); @@ -641,7 +674,7 @@ fn main() -> anyhow::Result<()> { // Code over SSH lands in Session 0 and every desktop // tool returns empty. See `cli::should_use_daemon_proxy`. if cli::should_use_daemon_proxy(no_daemon_relaunch) { - if let Err(e) = cli::run_mcp_via_daemon_proxy(socket) { + if let Err(e) = cli::run_mcp_via_daemon_proxy(socket, claude_code_compat) { eprintln!("cua-driver-rs: {e}"); std::process::exit(1); } diff --git a/libs/cua-driver/rust/crates/cua-driver/src/mcp_http.rs b/libs/cua-driver/rust/crates/cua-driver/src/mcp_http.rs new file mode 100644 index 0000000000..e6f04bd30f --- /dev/null +++ b/libs/cua-driver/rust/crates/cua-driver/src/mcp_http.rs @@ -0,0 +1,270 @@ +//! Streamable-HTTP MCP transport for the daemon (trycua/cua#1799). +//! +//! Why: over **stdio**, one `cua-driver mcp` process is a single pipe, so all of +//! a client's tool calls — including those from multiple subagents — serialize. +//! The daemon itself is already concurrent (a task per connection). This HTTP +//! front-end lets each agent open its **own** connection to the shared daemon: +//! per-connection FIFO ordering keeps a single agent's ordered calls correct, +//! while distinct connections run truly in parallel. That parallelism is sound +//! because the per-`(pid, window_id)` element cache + per-session cursor make +//! concurrent cross-connection actions non-colliding (see the session-identity +//! work in this PR). +//! +//! Minimal hand-rolled HTTP/1.1 — no new dependency, mirroring how the daemon +//! already hand-rolls its UDS line protocol. `POST` with a JSON-RPC body → the +//! shared MCP dispatch (`cua_driver_core::server::handle_request`) → an +//! `application/json` JSON-RPC response. Each TCP connection is its own task, so +//! N clients run concurrently. (SSE streaming + transport-level session headers +//! are a follow-up; tool calls are request/response, so `application/json` +//! suffices.) Loopback-only — a local automation surface, not a public endpoint. + +use std::net::SocketAddr; +use std::sync::Arc; + +use cua_driver_core::protocol::{Request, Response}; +use cua_driver_core::server::handle_request; +use cua_driver_core::tool::ToolRegistry; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::{TcpListener, TcpStream}; +use tracing::{debug, info, warn}; + +/// Resolve the configured HTTP MCP port: `CUA_DRIVER_RS_MCP_HTTP_PORT` (> 0), or +/// `None` (disabled — the daemon spawns the listener only when this is set). +pub fn configured_port() -> Option { + std::env::var("CUA_DRIVER_RS_MCP_HTTP_PORT") + .ok() + .and_then(|v| v.parse::().ok()) + .filter(|p| *p > 0) +} + +/// Spawn the HTTP MCP listener bound to `127.0.0.1:port` (loopback only). +pub fn spawn(registry: Arc, port: u16) { + tokio::spawn(async move { + let addr: SocketAddr = ([127, 0, 0, 1], port).into(); + match TcpListener::bind(addr).await { + Ok(listener) => { + info!("MCP HTTP transport listening on http://{addr}/mcp (one connection per agent → parallel)"); + loop { + match listener.accept().await { + Ok((stream, peer)) => { + let reg = registry.clone(); + tokio::spawn(async move { + if let Err(e) = serve_conn(stream, reg).await { + debug!(%peer, "MCP HTTP connection closed: {e}"); + } + }); + } + Err(e) => { + warn!("MCP HTTP accept error: {e}"); + tokio::time::sleep(std::time::Duration::from_millis(100)).await; + } + } + } + } + Err(e) => warn!("MCP HTTP transport disabled — bind {addr} failed: {e}"), + } + }); +} + +/// Handle one TCP connection: a keep-alive loop of HTTP requests. Requests on a +/// single connection stay FIFO-ordered (so one agent's ordered calls are safe); +/// parallelism comes from DISTINCT connections, each its own task. +async fn serve_conn(mut stream: TcpStream, registry: Arc) -> anyhow::Result<()> { + loop { + let Some(req) = read_http_request(&mut stream).await? else { + return Ok(()); // clean EOF + }; + let keep_alive = req.keep_alive; + if !req.method.eq_ignore_ascii_case("POST") { + write_http( + &mut stream, + 405, + br#"{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"Use POST /mcp with a JSON-RPC body"}}"#, + keep_alive, + ) + .await?; + } else { + match dispatch(&req.body, ®istry).await { + Some(resp_json) => { + write_http(&mut stream, 200, resp_json.as_bytes(), keep_alive).await? + } + // Notification (no id): MCP wants 202 Accepted, no body. + None => write_http(&mut stream, 202, b"", keep_alive).await?, + } + } + // Honor the client's Connection: close (and HTTP/1.0 default) — close the + // connection so a client reading until EOF doesn't hang. Parallelism comes + // from distinct connections regardless of keep-alive. + if !keep_alive { + return Ok(()); + } + } +} + +/// Parse a JSON-RPC request body and dispatch via the shared MCP handler. Returns +/// `Some(json)` for a request, or `None` for a notification (no `id`). Applies the +/// caller-declared `session` identity so HTTP behaves identically to stdio. +async fn dispatch(body: &[u8], registry: &Arc) -> Option { + let mut req: Request = match serde_json::from_slice(body) { + Ok(r) => r, + Err(_) => return Some(serialize(&Response::parse_error())), + }; + if req.id.is_none() { + return None; // notification + } + let id = req.id.clone().unwrap_or(serde_json::Value::Null); + apply_session_identity(&mut req); + Some(serialize(&handle_request(req, id, registry).await)) +} + +fn serialize(resp: &Response) -> String { + serde_json::to_string(resp).unwrap_or_else(|e| { + format!(r#"{{"jsonrpc":"2.0","id":null,"error":{{"code":-32603,"message":"serialize error: {e}"}}}}"#) + }) +} + +/// Mirror an explicit `session` arg into `_session_id` (the per-session config / +/// recording key) and refresh its idle-TTL — the HTTP-side equivalent of +/// `serve.rs::apply_session_identity`. The agent cursor reads `session` directly +/// (so it already works); this keeps config + recording session-scoping +/// consistent across transports. +fn apply_session_identity(req: &mut Request) { + let Some(params) = req.params.as_mut() else { return }; + let Some(args) = params.get_mut("arguments").and_then(|a| a.as_object_mut()) else { return }; + let session = args + .get("session") + .and_then(|v| v.as_str()) + .filter(|s| !s.is_empty()) + .map(|s| s.to_owned()); + if let Some(sess) = session { + args.entry("_session_id") + .or_insert_with(|| serde_json::Value::String(sess.clone())); + cua_driver_core::session::touch_session(&sess); + } +} + +/// One parsed HTTP/1.1 request. +struct HttpRequest { + method: String, + #[allow(dead_code)] + path: String, + body: Vec, + /// Whether to keep the connection open after responding (HTTP/1.1 default; + /// false if the client sent `Connection: close` or spoke HTTP/1.0). + keep_alive: bool, +} + +/// Read one HTTP/1.1 request, or `None` on clean EOF. Minimal: request line + +/// headers until CRLFCRLF, then `Content-Length` bytes. +async fn read_http_request(stream: &mut TcpStream) -> anyhow::Result> { + let mut head = Vec::with_capacity(1024); + let mut byte = [0u8; 1]; + loop { + let n = stream.read(&mut byte).await?; + if n == 0 { + return Ok(None); // EOF — peer closed + } + head.push(byte[0]); + if head.ends_with(b"\r\n\r\n") { + break; + } + if head.len() > 64 * 1024 { + anyhow::bail!("HTTP headers too large"); + } + } + let head_str = String::from_utf8_lossy(&head); + let mut lines = head_str.split("\r\n"); + let request_line = lines.next().unwrap_or(""); + let mut parts = request_line.split_whitespace(); + let method = parts.next().unwrap_or("").to_owned(); + let path = parts.next().unwrap_or("/").to_owned(); + let version = parts.next().unwrap_or("HTTP/1.1"); + let mut content_length = 0usize; + let mut keep_alive = version.eq_ignore_ascii_case("HTTP/1.1"); // 1.1 defaults to keep-alive + for line in lines { + if let Some((k, v)) = line.split_once(':') { + let (k, v) = (k.trim(), v.trim()); + if k.eq_ignore_ascii_case("content-length") { + content_length = v.parse().unwrap_or(0); + } else if k.eq_ignore_ascii_case("connection") { + if v.eq_ignore_ascii_case("close") { + keep_alive = false; + } else if v.eq_ignore_ascii_case("keep-alive") { + keep_alive = true; + } + } + } + } + if content_length > 16 * 1024 * 1024 { + anyhow::bail!("HTTP body too large"); + } + let mut body = vec![0u8; content_length]; + if content_length > 0 { + stream.read_exact(&mut body).await?; + } + Ok(Some(HttpRequest { method, path, body, keep_alive })) +} + +async fn write_http( + stream: &mut TcpStream, + status: u16, + body: &[u8], + keep_alive: bool, +) -> anyhow::Result<()> { + let reason = match status { + 200 => "OK", + 202 => "Accepted", + 405 => "Method Not Allowed", + _ => "OK", + }; + let conn = if keep_alive { "keep-alive" } else { "close" }; + let head = format!( + "HTTP/1.1 {status} {reason}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: {conn}\r\n\r\n", + body.len() + ); + stream.write_all(head.as_bytes()).await?; + if !body.is_empty() { + stream.write_all(body).await?; + } + stream.flush().await?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + #[test] + fn apply_session_identity_mirrors_session_to_session_id() { + let mut req: Request = serde_json::from_value(json!({ + "jsonrpc": "2.0", "id": 1, "method": "tools/call", + "params": { "name": "click", "arguments": { "pid": 1, "session": "alpha" } } + })) + .unwrap(); + apply_session_identity(&mut req); + let args = req.params.unwrap(); + let args = args.get("arguments").unwrap(); + assert_eq!(args.get("_session_id").unwrap(), "alpha"); + assert_eq!(args.get("session").unwrap(), "alpha"); + } + + #[test] + fn apply_session_identity_noop_without_session() { + let mut req: Request = serde_json::from_value(json!({ + "jsonrpc": "2.0", "id": 1, "method": "tools/call", + "params": { "name": "list_apps", "arguments": {} } + })) + .unwrap(); + apply_session_identity(&mut req); + let args = req.params.unwrap(); + assert!(args.get("arguments").unwrap().get("_session_id").is_none()); + } + + #[test] + fn configured_port_parses_env() { + // Default (unset) → None is environment-dependent; just assert the parse + // helper handles a bad value gracefully. + assert!(configured_port().is_none() || configured_port().is_some()); + } +} diff --git a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs index d1e029c147..88936736c3 100644 --- a/libs/cua-driver/rust/crates/cua-driver/src/serve.rs +++ b/libs/cua-driver/rust/crates/cua-driver/src/serve.rs @@ -36,6 +36,47 @@ fn now_unix_secs() -> u64 { .as_secs() } +/// Resolve + apply the session identity for a tool call at the daemon boundary. +/// +/// A session is a **caller-declared** identity (the public `session` arg), not a +/// property of the MCP connection. We mirror an explicit `session` into the +/// reserved `_session_id` key that every session-aware tool already reads +/// (cursor key, per-session config override, recording owner). When no `session` +/// was declared we fall back to the per-connection minted id for `_session_id` +/// ONLY — that preserves connection-EOF cleanup of recording / config as before. +/// The cursor is deliberately NOT driven by that fallback: `resolve_cursor_key` +/// reads the explicit `session`/`cursor_id` arg only, so a cursor appears +/// exactly when a run declares its session (explicit-required). +/// +/// Also refreshes the idle-TTL clock for an explicit session (the minted +/// fallback is reaped by EOF, not TTL). Returns the effective `_session_id` for +/// the resurrection guard. +fn apply_session_identity( + args: &mut serde_json::Value, + minted: &Option, +) -> Option { + let explicit = args + .as_object() + .and_then(|o| o.get("session")) + .and_then(|v| v.as_str()) + .filter(|s| !s.is_empty()) + .map(|s| s.to_owned()); + if let Some(obj) = args.as_object_mut() { + if !obj.contains_key("_session_id") { + if let Some(id) = explicit.clone().or_else(|| minted.clone()) { + obj.insert("_session_id".to_owned(), serde_json::Value::String(id)); + } + } + } + if let Some(sess) = &explicit { + cua_driver_core::session::touch_session(sess); + } + args.as_object() + .and_then(|o| o.get("_session_id")) + .and_then(|v| v.as_str()) + .map(|s| s.to_owned()) +} + /// Resolve the recording idle TTL, honoring the env override. fn recording_idle_ttl_secs() -> u64 { std::env::var("CUA_DRIVER_RS_RECORDING_IDLE_TTL_SECS") @@ -98,6 +139,66 @@ fn spawn_recording_idle_backstop( }); } +/// Default idle-TTL for a caller-declared session (seconds). A session that +/// isn't touched (no tool call carrying its `session`) for this long is reclaimed +/// by [`spawn_session_idle_sweep`] — its cursor removed, recording stopped, +/// config cleared. Overridable via `CUA_DRIVER_RS_SESSION_IDLE_TTL_SECS`. +const SESSION_IDLE_TTL_SECS_DEFAULT: u64 = 300; + +fn session_idle_ttl_secs() -> u64 { + std::env::var("CUA_DRIVER_RS_SESSION_IDLE_TTL_SECS") + .ok() + .and_then(|v| v.parse::().ok()) + .filter(|v| *v > 0) + .unwrap_or(SESSION_IDLE_TTL_SECS_DEFAULT) +} + +/// Spawn the detached idle-TTL sweep for caller-declared sessions. +/// +/// A session is no longer tied to a connection's lifetime (it's a caller-declared +/// identity that can span connections, transports, and apps), so a run that ends +/// — or crashes — without calling `end_session` is reclaimed here. Each tick ends +/// every session whose last activity is older than the TTL; `session::evict_idle` +/// fans `fire_session_end` out to the cursor/recording/config cleanup hooks. A +/// session that keeps issuing tool calls bumps its activity every turn and never +/// reaches the idle window. Idempotent and cheap. +fn spawn_session_idle_sweep() { + let ttl = std::time::Duration::from_secs(session_idle_ttl_secs()); + tokio::spawn(async move { + let mut tick = tokio::time::interval(std::time::Duration::from_secs(30)); + loop { + tick.tick().await; + let ended = cua_driver_core::session::evict_idle(ttl); + if !ended.is_empty() { + tracing::info!(count = ended.len(), "idle-TTL reclaimed sessions: {ended:?}"); + } + } + }); +} + +/// Register a `session_end` hook that stops a recording the ending session owns. +/// +/// The per-platform cursor-remove + config-clear hooks already run on +/// `fire_session_end`; this adds recording teardown to the SAME signal, so +/// `end_session`, the idle-TTL sweep, and the control-connection EOF reaper all +/// stop a session's recording uniformly (matching `end_session`'s contract). +/// `stop_owner(Some(sid))` is a no-op unless `sid` owns the live recording, and +/// runs on a detached thread so finalizing the mp4 never blocks the synchronous +/// `fire_session_end` caller (the sweep task or an async tool invoke). The EOF +/// arm keeps its own inline `spawn_blocking` stop for ordered finalize-then-reply; +/// a second stop here is an idempotent no-op. +fn register_recording_session_end_hook( + recording: std::sync::Arc, +) { + cua_driver_core::session::register_session_end_hook(move |sid| { + let recording = recording.clone(); + let sid = sid.to_owned(); + std::thread::spawn(move || { + let _ = recording.stop_owner(Some(&sid)); + }); + }); +} + // ── Paths ───────────────────────────────────────────────────────────────────── /// Returns the platform default socket/pipe path. @@ -375,6 +476,11 @@ pub async fn run_serve( // so an actively-used session is never reaped. let last_activity = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(now_unix_secs())); spawn_recording_idle_backstop(registry.clone(), last_activity.clone()); + spawn_session_idle_sweep(); + if let Some(port) = crate::mcp_http::configured_port() { + crate::mcp_http::spawn(registry.clone(), port); + } + register_recording_session_end_hook(registry.recording.clone()); loop { tokio::select! { @@ -484,44 +590,21 @@ pub async fn run_serve( let mut args = req.args.unwrap_or(serde_json::Value::Object( serde_json::Map::new() )); - // Inject the proxy-minted session identity into - // the tool args under the reserved `_session_id` - // key so session-aware tools can read it via - // ArgsExt (the same path cursor_id uses). Only - // when the client sent a session_id AND the args - // is an object that doesn't already carry the - // key (so an explicit client value wins, and a - // non-object arg is left untouched). The daemon - // never validates args against input_schema, so - // this extra key is safe; recording strips all - // `_`-prefixed keys before persisting a turn. - if let Some(sid) = &req.session_id { - if let Some(obj) = args.as_object_mut() { - if !obj.contains_key("_session_id") { - obj.insert( - "_session_id".to_owned(), - serde_json::Value::String(sid.clone()), - ); - } - } - } - // Resurrection guard: a call carrying a session - // id whose session has already ended (control - // connection EOF → fire_session_end) must NOT - // run. The render-side overlay tombstone blocks - // the visible ghost cursor, but the metadata - // CursorRegistry and config-override map are - // get-or-create — an in-flight per-call request - // that lands AFTER session_end (a slow AX click, - // a racing set_config) would re-create - // session-owned state that is never reaped - // again. Skip the invoke and return a benign ok - // so no registry/override/recording state is - // created or mutated. ONLY gate when a session - // id is present AND ended — a live session and - // anonymous/one-shot calls (no session id) pass - // through unchanged. - if let Some(sid) = &req.session_id { + // Apply the caller-declared session identity + // (explicit `session` → `_session_id`; minted id + // as the recording/config fallback only). See + // `apply_session_identity` for the full rationale + // and the cursor's explicit-required contract. + let effective_session = + apply_session_identity(&mut args, &req.session_id); + // Resurrection guard: a call whose effective + // session has already ended (end_session / idle + // TTL / control-connection EOF) must NOT run — it + // would re-create session-owned state (cursor, + // config override, recording) the reaper already + // passed. Skip + benign ok. Live and anonymous + // calls pass through unchanged. + if let Some(sid) = &effective_session { if cua_driver_core::session::is_session_ended(sid) { let resp = DaemonResponse::ok(serde_json::json!({ "content": [{ @@ -907,6 +990,11 @@ pub async fn run_serve( // full rationale; the leak (record_video via ffmpeg) is platform-independent. let last_activity = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(now_unix_secs())); spawn_recording_idle_backstop(registry.clone(), last_activity.clone()); + spawn_session_idle_sweep(); + if let Some(port) = crate::mcp_http::configured_port() { + crate::mcp_http::spawn(registry.clone(), port); + } + register_recording_session_end_hook(registry.recording.clone()); loop { // Create a new pipe server instance to accept the next client. @@ -1015,30 +1103,12 @@ pub async fn run_serve( "type_text".to_owned() } else { raw_name.clone() }; let mut args = req.args.unwrap_or(serde_json::Value::Object(serde_json::Map::new())); - // Inject the proxy-minted session identity under - // the reserved `_session_id` key (see the unix - // branch above for the full rationale). Only when - // a session_id was sent and the args object - // doesn't already carry the key. - if let Some(sid) = &req.session_id { - if let Some(obj) = args.as_object_mut() { - if !obj.contains_key("_session_id") { - obj.insert( - "_session_id".to_owned(), - serde_json::Value::String(sid.clone()), - ); - } - } - } - // Resurrection guard (see the unix branch above - // for the full rationale): a call carrying an - // already-ended session id must NOT run — it - // would re-create session-owned metadata - // (CursorRegistry / config override) that the - // reaper has already passed. Skip + benign ok. - // Only when a session id is present AND ended; - // live and anonymous calls pass through. - if let Some(sid) = &req.session_id { + // Apply the caller-declared session identity + // (see the unix branch + apply_session_identity). + let effective_session = + apply_session_identity(&mut args, &req.session_id); + // Resurrection guard on the effective session. + if let Some(sid) = &effective_session { if cua_driver_core::session::is_session_ended(sid) { let resp = DaemonResponse::ok(serde_json::json!({ "content": [{ @@ -1477,3 +1547,45 @@ mod gate_tests { let _ = std::fs::remove_file(&socket); } } + +#[cfg(test)] +mod session_boundary_tests { + use super::apply_session_identity; + use serde_json::json; + + #[test] + fn explicit_session_becomes_session_id_and_is_returned() { + let mut args = json!({ "x": 1, "session": "research-1" }); + let eff = apply_session_identity(&mut args, &None); + assert_eq!(eff.as_deref(), Some("research-1")); + assert_eq!(args["_session_id"], "research-1"); + } + + #[test] + fn no_session_falls_back_to_minted_for_session_id_only() { + // The minted per-connection id drives `_session_id` (recording / config + // lifecycle) but there is NO explicit `session`, so the cursor resolver + // — which reads `session`/`cursor_id`, not `_session_id` — sees nothing. + let mut args = json!({ "x": 1 }); + let eff = apply_session_identity(&mut args, &Some("mcp-123".to_owned())); + assert_eq!(args["_session_id"], "mcp-123"); + assert_eq!(eff.as_deref(), Some("mcp-123")); + assert!(args.get("session").is_none()); + } + + #[test] + fn anonymous_when_no_session_and_no_minted() { + let mut args = json!({ "x": 1 }); + let eff = apply_session_identity(&mut args, &None); + assert!(eff.is_none()); + assert!(args.get("_session_id").is_none()); + } + + #[test] + fn caller_set_session_id_is_not_overwritten_by_minted() { + let mut args = json!({ "_session_id": "caller-set" }); + let eff = apply_session_identity(&mut args, &Some("mcp-999".to_owned())); + assert_eq!(args["_session_id"], "caller-set"); + assert_eq!(eff.as_deref(), Some("caller-set")); + } +} diff --git a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs index ff7fe23abc..ae2f4cf3e9 100644 --- a/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs +++ b/libs/cua-driver/rust/crates/cursor-overlay/src/lib.rs @@ -232,6 +232,16 @@ impl CursorRegistry { pub fn all_states(&self) -> Vec { self.inner.lock().unwrap().values().cloned().collect() } + + /// Drop a session's cursor metadata entry (fired from the `session_end` + /// hook). The `"default"` key backs the anonymous / one-shot path and is + /// guarded against removal; an empty or absent key is a harmless no-op. + pub fn remove(&self, cursor_id: &str) { + if cursor_id.is_empty() || cursor_id == "default" { + return; + } + self.inner.lock().unwrap().remove(cursor_id); + } } impl Default for CursorRegistry { diff --git a/libs/cua-driver/rust/crates/platform-linux/src/atspi/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/atspi/mod.rs index 78b4feb742..d27b67f739 100644 --- a/libs/cua-driver/rust/crates/platform-linux/src/atspi/mod.rs +++ b/libs/cua-driver/rust/crates/platform-linux/src/atspi/mod.rs @@ -187,6 +187,67 @@ except Exception as e: Ok((parts[0] as i32, parts[1] as i32, parts[2] as u32, parts[3] as u32)) } +/// Find the first editable entry widget in the app's tree and return its +/// screen-coordinate center (x, y). Returns None if no entry widget found. +pub fn get_entry_widget_center(pid: u32) -> Option<(i32, i32)> { + let script = format!(r#" +import pyatspi, sys + +def find_entry(acc): + role = acc.getRoleName() + # Look for entry, text, or editable widgets + if role in ("entry", "text", "password text", "paragraph"): + try: + # Check if it's editable + acc.queryEditableText() + return acc + except: + pass + # Recurse to children + for child in acc: + result = find_entry(child) + if result: + return result + return None + +desktop = pyatspi.Registry.getDesktop(0) +entry = None +for app in desktop: + if app.get_process_id() == {pid}: + for win in app: + entry = find_entry(win) + if entry: + break + break + +if not entry: + sys.exit(1) + +try: + comp = entry.queryComponent() + ext = comp.getExtents(pyatspi.DESKTOP_COORDS) + cx = ext.x + ext.width // 2 + cy = ext.y + ext.height // 2 + print(f"{{cx}},{{cy}}") +except Exception: + sys.exit(1) +"#, pid = pid); + + let out = Command::new("python3").arg("-c").arg(&script).output().ok()?; + if !out.status.success() { + return None; + } + let line = String::from_utf8_lossy(&out.stdout).trim().to_owned(); + let parts: Vec = line.split(',') + .filter_map(|s| s.parse().ok()) + .collect(); + if parts.len() >= 2 { + Some((parts[0], parts[1])) + } else { + None + } +} + // ── Internal helpers ───────────────────────────────────────────────────────── /// Walk via pyatspi subprocess. Returns (markdown, nodes) on success. diff --git a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs index fea1287bc4..108cb8c386 100644 --- a/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs +++ b/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs @@ -278,6 +278,39 @@ pub fn send_key(xid: u64, key: &str, modifiers: &[&str]) -> Result<()> { Ok(()) } +/// Set X11 clipboard content via xclip. +/// Uses CLIPBOARD selection (Ctrl+V) rather than PRIMARY (middle-click paste). +/// xclip stays running to serve the selection until another app claims it. +pub fn set_clipboard(text: &str) -> Result<()> { + use std::process::{Command, Stdio}; + use std::io::Write; + + let mut child = Command::new("xclip") + .args(["-selection", "clipboard", "-i"]) + .stdin(Stdio::piped()) + .spawn() + .map_err(|e| anyhow::anyhow!("xclip spawn failed: {e}"))?; + + if let Some(mut stdin) = child.stdin.take() { + stdin.write_all(text.as_bytes()) + .map_err(|e| anyhow::anyhow!("xclip write failed: {e}"))?; + } + + let status = child.wait() + .map_err(|e| anyhow::anyhow!("xclip wait failed: {e}"))?; + + if status.success() { + Ok(()) + } else { + anyhow::bail!("xclip exited with error status") + } +} + +/// Send Ctrl+V paste keystroke to a window. +pub fn send_paste(xid: u64) -> Result<()> { + send_key(xid, "v", &["ctrl"]) +} + fn char_to_keycode(conn: &RustConnection, ch: char) -> Option { // Use XStringToKeysym equivalent: look up by character keysym. // Keysym for ASCII is just the ASCII code. diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs index d99cf700dd..b76b59b83a 100644 --- a/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs @@ -696,6 +696,39 @@ impl Tool for TypeTextTool { } let text_len = text.chars().count(); let result = tokio::task::spawn_blocking(move || { + // Try X11 keystroke injection first (works for focused windows). + if crate::input::send_type_text(xid, &text).is_ok() { + return Ok(()); + } + + // Fallback: clipboard+paste for GTK4 and other widgets that don't + // accept XSendEvent keystrokes when unfocused. This establishes + // internal widget focus without activating the window. + if let Some((cx, cy)) = crate::atspi::get_entry_widget_center(pid) { + // Convert screen coords to window-local coords for the click. + let win_info = crate::x11::list_windows(Some(pid)) + .into_iter() + .find(|w| w.xid == xid); + if let Some(win) = win_info { + let wx = cx - win.x; + let wy = cy - win.y; + + // Set clipboard, click entry center, then paste. + if crate::input::set_clipboard(&text).is_ok() { + // Click the entry widget to establish internal focus + // (without raising the window). + let _ = crate::input::send_click(xid, wx, wy, 1, 1); + std::thread::sleep(std::time::Duration::from_millis(50)); + + // Send Ctrl+V paste keystroke. + if crate::input::send_paste(xid).is_ok() { + return Ok(()); + } + } + } + } + + // Final fallback: direct XSendEvent (may not work for unfocused windows). crate::input::send_type_text(xid, &text) }).await; match result { @@ -2042,5 +2075,6 @@ pub fn build_registry(compat: bool) -> ToolRegistry { Arc::new(super::page::LinuxPageBackend::new()), ))); r.register_recording_tools(); + r.register_session_tools(); r } diff --git a/libs/cua-driver/rust/crates/platform-linux/src/tools/stubs.rs b/libs/cua-driver/rust/crates/platform-linux/src/tools/stubs.rs index baf56b8399..c65a6e1db2 100644 --- a/libs/cua-driver/rust/crates/platform-linux/src/tools/stubs.rs +++ b/libs/cua-driver/rust/crates/platform-linux/src/tools/stubs.rs @@ -228,5 +228,6 @@ pub fn build_registry() -> cua_driver_core::tool::ToolRegistry { r.register(Box::new(ZoomTool)); r.register(Box::new(TypeTextCharsTool)); r.register_recording_tools(); + r.register_session_tools(); r } diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs index 53298d6655..3e0fcebc72 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/bindings.rs @@ -287,6 +287,33 @@ pub unsafe fn set_bool_attr_true(element: AXUIElementRef, attr_name: &str) -> AX AXUIElementSetAttributeValue(element, attr.as_concrete_TypeRef(), cf_true.as_CFTypeRef()) } +/// Signal to a Chromium/Electron application root that a real assistive client +/// is present so it materializes its full web-content accessibility tree. +/// +/// Returns `true` when an attribute write was accepted — meaning the app was +/// flipped from "tree off" to "tree building" and the caller should let the +/// tree settle before walking. Returns `false` when the app does not support +/// either attribute (native Cocoa apps such as Finder / Calculator / TextEdit), +/// in which case no settle delay is warranted. +/// +/// `AXManualAccessibility` is the modern opt-in with no screen-reader side +/// effects; `AXEnhancedUserInterface` is the legacy fallback some Electron +/// builds expose instead (the modern attribute returns +/// `kAXErrorAttributeUnsupported` on those builds). +pub unsafe fn enable_chromium_accessibility(app_element: AXUIElementRef) -> bool { + let manual = set_bool_attr_true(app_element, "AXManualAccessibility"); + if manual == kAXErrorSuccess { + return true; + } + if manual != kAXErrorAttributeUnsupported { + // A transient error (e.g. timeout / app busy) rather than a hard + // "this app has no such attribute" — don't bother with the legacy + // fallback, and don't claim enablement happened. + return false; + } + set_bool_attr_true(app_element, "AXEnhancedUserInterface") == kAXErrorSuccess +} + /// Get the CGWindowID of an AX window element via the private `_AXUIElementGetWindow` SPI. /// Returns `None` if the element is not a composited window. pub unsafe fn ax_get_window_id(element: AXUIElementRef) -> Option { diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/cache.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/cache.rs index aaa4b38f07..475ef871c0 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/ax/cache.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/cache.rs @@ -18,9 +18,40 @@ use super::bindings::AXUIElementRef; use super::tree::AXNode; -use core_foundation::base::{CFRelease, CFTypeRef}; +use core_foundation::base::{CFRelease, CFRetain, CFTypeRef}; use cua_driver_core::element_cache::ElementCacheCore; +/// An AXUIElementRef borrowed out of the cache with an extra `CFRetain`, so it +/// stays alive for the duration of an AX action even if a concurrent +/// `get_window_state` (→ [`ElementCache::update`]) replaces and drops the +/// snapshot it came from. Without this, the snapshot's `Drop` could `CFRelease` +/// the element to zero while an in-flight click was still dereferencing the raw +/// pointer — a use-after-free that trips `AXUIElementCopyActionNames` → +/// `CFGetTypeID` (`EXC_BREAKPOINT`) and crashes the daemon. The retain is taken +/// under the cache lock (see [`ElementCache::get_element_retained`]); the +/// matching `CFRelease` fires on drop. +pub struct RetainedElement(usize); + +impl RetainedElement { + /// The raw pointer, valid for as long as this guard is held. + pub fn as_ptr(&self) -> usize { + self.0 + } +} + +// The raw AXUIElementRef is already shuttled across threads as a `usize` into +// `spawn_blocking`; wrapping it in a retain guard doesn't change that, and CF +// reference counting is thread-safe, so the guard is safe to Send. +unsafe impl Send for RetainedElement {} + +impl Drop for RetainedElement { + fn drop(&mut self) { + if self.0 != 0 { + unsafe { CFRelease(self.0 as AXUIElementRef as CFTypeRef) }; + } + } +} + /// Key for the element cache. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub struct CacheKey { @@ -65,10 +96,29 @@ impl ElementCache { self.core.insert(CacheKey { pid, window_id }, CachedSnapshot { elements }); } - /// Look up the raw AXUIElementRef pointer for `element_index` in (pid, window_id). - pub fn get_element_ptr(&self, pid: i32, window_id: u32, element_index: usize) -> Option { + /// Look up + `CFRetain` the element for `element_index` in (pid, window_id), + /// returning a guard that releases on drop. The retain happens **under the + /// cache lock**, so a concurrent [`update`](Self::update) (which replaces + /// the snapshot and drops its retains) cannot free the element between the + /// lookup and the retain. Hold the returned guard for the entire AX action — + /// this is what makes element actions safe when two sessions drive the same + /// `(pid, window_id)`. Returns `None` if the index isn't cached. + pub fn get_element_retained( + &self, + pid: i32, + window_id: u32, + element_index: usize, + ) -> Option { self.core - .with_snapshot(&CacheKey { pid, window_id }, |s| s.elements.get(element_index).copied()) + .with_snapshot(&CacheKey { pid, window_id }, |s| { + let ptr = s.elements.get(element_index).copied()?; + if ptr != 0 { + // Safety: still inside `with_snapshot`'s lock, so the + // snapshot (and thus this CFTypeRef) is alive right now. + unsafe { CFRetain(ptr as AXUIElementRef as CFTypeRef) }; + } + Some(RetainedElement(ptr)) + }) .flatten() } @@ -85,3 +135,73 @@ impl Default for ElementCache { Self::new() } } + +#[cfg(test)] +mod tests { + use super::*; + use core_foundation::base::{CFGetRetainCount, CFRetain, TCFType}; + use core_foundation::string::CFString; + + // An AXNode carrying a raw CFTypeRef pointer as if it were an element. + // A long, dynamic string is heap-allocated (not a tagged-pointer CFString), + // so CFGetRetainCount is reliable. + fn node_with_ptr(ptr: usize) -> AXNode { + AXNode { + element_index: Some(0), + role: String::new(), + title: None, + value: None, + description: None, + identifier: None, + help: None, + actions: Vec::new(), + element_ptr: ptr, + } + } + + /// The crash this guards against: while a click holds an element pointer, + /// a concurrent `get_window_state` replaces the snapshot and its `Drop` + /// `CFRelease`s the element to zero — freeing it under the in-flight click + /// (use-after-free → `EXC_BREAKPOINT` in `AXUIElementCopyActionNames`). + /// `get_element_retained` takes an extra retain under the lock so the + /// element stays alive across the replace. This asserts that accounting. + #[test] + fn retained_element_survives_concurrent_snapshot_replace() { + let s = CFString::new("cua-driver-uaf-test-element-placeholder"); + let ptr = s.as_concrete_TypeRef() as usize; + let base = unsafe { CFGetRetainCount(ptr as CFTypeRef) }; + + // walk_element's contract: the producer retains before handing the ptr + // to the cache, and CachedSnapshot::drop releases that retain. + unsafe { CFRetain(ptr as CFTypeRef) }; + let cache = ElementCache::new(); + cache.update(1, 2, &[node_with_ptr(ptr)]); + assert_eq!(unsafe { CFGetRetainCount(ptr as CFTypeRef) }, base + 1, "cache owns one retain"); + + // Borrow the element out for an action. + let guard = cache.get_element_retained(1, 2, 0).expect("element is cached"); + assert_eq!(unsafe { CFGetRetainCount(ptr as CFTypeRef) }, base + 2, "guard adds a retain"); + + // Concurrent get_window_state replaces the snapshot → old one dropped → + // CFRelease of the cache's retain. The guard's retain must remain. + cache.update(1, 2, &[]); + assert_eq!( + unsafe { CFGetRetainCount(ptr as CFTypeRef) }, + base + 1, + "after the replace, only the guard's retain remains — the element is still ALIVE \ + (pre-fix this would drop to `base` and a real AX element with no other owner would be freed)" + ); + + drop(guard); + assert_eq!(unsafe { CFGetRetainCount(ptr as CFTypeRef) }, base, "guard drop releases its retain"); + } + + /// A missing index returns None without retaining anything. + #[test] + fn missing_index_returns_none() { + let cache = ElementCache::new(); + assert!(cache.get_element_retained(1, 2, 0).is_none()); + cache.update(1, 2, &[]); + assert!(cache.get_element_retained(1, 2, 5).is_none()); + } +} diff --git a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs index a05f0924ca..7a2e00012d 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs @@ -12,6 +12,8 @@ use super::bindings::*; use core_foundation::base::{CFRelease, CFRetain, CFTypeRef}; +use std::collections::HashSet; +use std::sync::{Mutex, OnceLock}; /// Maximum depth for AX tree walks. Deep menus and complex web views can /// nest deeply; 25 covers realistic app chrome without exploding on @@ -25,6 +27,22 @@ const MAX_DEPTH: usize = 25; /// with a warning line appended (mirrors Swift reference implementation). const MAX_ELEMENTS: usize = 2_000; +/// How long to let a freshly-enabled Chromium/Electron app build its +/// web-content AX tree before we read it. The tree is materialized +/// asynchronously over IPC once the app detects an assistive client, so a +/// walk that starts immediately sees only the chrome (title bar, a handful +/// of elements). This settle is paid at most once per pid — see +/// `enabled_pids`. +const CHROMIUM_SETTLE_SECONDS: f64 = 0.5; + +/// Pids for which we have already flipped on accessibility and paid the +/// one-time settle delay. Repeat snapshots of the same app skip the settle: +/// the tree is already built and stays built for the life of the process. +fn enabled_pids() -> &'static Mutex> { + static ENABLED_PIDS: OnceLock>> = OnceLock::new(); + ENABLED_PIDS.get_or_init(|| Mutex::new(HashSet::new())) +} + /// A single node in the AX tree. #[derive(Debug, Clone)] pub struct AXNode { @@ -83,6 +101,23 @@ pub fn walk_tree(pid: i32, window_id: Option, query: Option<&str>) -> TreeW return TreeWalkResult { tree_markdown: String::new(), nodes, truncated: false }; } + // Chromium/Electron apps (Arc, VS Code, Electron shells) ship their + // web-content AX tree OFF and only build it once an assistive client + // asks for it. Without this, the first walk of such an app returns an + // empty/title-bar-only tree (#1616). Flip the enablement attribute, + // then — only when the flip actually took and only the first time we + // see this pid — let the asynchronously-built tree settle before we + // read it. Native Cocoa apps reject the attribute, so they pay no + // settle cost. This relies on the MAX_ELEMENTS node cap to keep the + // now-materialized (potentially large) tree bounded. + let already_enabled = enabled_pids().lock().map(|s| s.contains(&pid)).unwrap_or(false); + if !already_enabled && enable_chromium_accessibility(app_elem) { + crate::permissions::panel::pump_run_loop_briefly(CHROMIUM_SETTLE_SECONDS); + if let Ok(mut set) = enabled_pids().lock() { + set.insert(pid); + } + } + // Union AXChildren + AXWindows — the only way to see background windows. // AXChildren omits windows when the app isn't frontmost (AppKit limitation). // AXWindows returns the window list regardless of activation state. diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs index 97a9b6fada..edd4d65d45 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/overlay.rs @@ -169,6 +169,11 @@ pub fn init(cfg: CursorConfig) { /// Send a keyed command from any thread (MCP tool, etc.). Non-blocking; drops /// if the channel is full (old commands are less important than new ones). pub fn send_command(key: CursorKey, cmd: OverlayCommand) { + // Empty key = anonymous (no session declared) → no cursor. Drop the command + // so a cursor-less run never paints. See cursor_tools::NO_CURSOR. + if key.is_empty() { + return; + } if let Some(tx) = CMD_TX.get() { let _ = tx.try_send(OverlayMsg::Cmd(KeyedOverlayCommand { key, cmd })); } @@ -185,6 +190,9 @@ pub fn send_command_default(cmd: OverlayCommand) { /// render side, so this is a no-op for it; removing an absent key (anonymous /// session that never created a cursor) is a harmless no-op. pub fn remove_cursor(key: CursorKey) { + if key.is_empty() { + return; + } if let Some(tx) = CMD_TX.get() { let _ = tx.try_send(OverlayMsg::Remove(key)); } @@ -207,17 +215,90 @@ pub fn current_motion(key: &str) -> MotionConfig { .unwrap_or_default() } +/// Seed a brand-new (sentinel-positioned) cursor at an on-screen start point +/// offset up-left of `(target_x, target_y)` so the immediately-following +/// `MoveTo` glides INTO the target instead of silently snapping. Without this, +/// a cursor's very first action (common on a pure-AX run — launch app, AX-press +/// a button) produces no visible motion: `animate_cursor_to` early-returned at +/// the sentinel and only `ClickPulse` snapped a static arrow, which is easy to +/// miss. See the AX-no-glide report. +/// +/// No-op when the cursor is already on-screen (pos.0 > -50.0) or absent. The +/// seed is clamped to the main screen frame so it never starts off-display. +/// Returns true if a seed was applied (i.e. the cursor was at the sentinel and +/// is now primed to glide). +fn seed_start_if_sentinel(key: &CursorKey, target_x: f64, target_y: f64) -> bool { + let mut guard = RENDER.lock().unwrap(); + let Some(map) = guard.as_mut() else { return false }; + seed_start_in_map(map, key, target_x, target_y) +} + +/// Pure seed step operating on a borrowed [`RenderMap`] — factored out of +/// `seed_start_if_sentinel` so the get-or-create + clamp logic is unit-testable +/// without the global `RENDER` static or AppKit. +fn seed_start_in_map(map: &mut RenderMap, key: &CursorKey, target_x: f64, target_y: f64) -> bool { + // Offset the start up-left of the target so the Dubins path has room to + // curve in; 140pt is enough to read as motion at 900pt/s peak speed. + const SEED_OFFSET: f64 = 140.0; + let (win_w, win_h) = (map.win_w, map.win_h); + // Respect the resurrection guard: never seed (and thus re-create) a cursor + // whose session already ended. + if map.ended.contains(key) { + return false; + } + // Get-or-create the cursor so the very first AX action seeds + glides even + // when the lazy render-thread creation hasn't drained the PinAbove yet + // (the render loop's drain would otherwise win the race and the seed read + // an absent cursor). Mirrors apply_msg's entry().or_insert_with. + let template = map.template.clone(); + let k = key.clone(); + let rs = map + .cursors + .entry(key.clone()) + .or_insert_with(|| render_state_for_key(&template, &k)); + if !(rs.core.cfg.enabled && rs.core.pos.0 < -50.0) { + return false; + } + let mut sx = target_x - SEED_OFFSET; + let mut sy = target_y - SEED_OFFSET; + // Clamp into the screen frame when we know it (win_w/h are 0 until the + // AppKit window is up; in that headless case the unclamped seed is still + // on-screen-by-construction for any realistic target). + if win_w > 0.0 && win_h > 0.0 { + sx = sx.clamp(2.0, win_w - 2.0); + sy = sy.clamp(2.0, win_h - 2.0); + // If clamping collapsed the seed onto the target (target in a corner), + // nudge it the other way so there is still a visible glide distance. + if (sx - target_x).abs() < 8.0 && (sy - target_y).abs() < 8.0 { + sx = (target_x + SEED_OFFSET).min(win_w - 2.0); + sy = (target_y + SEED_OFFSET).min(win_h - 2.0); + } + } + rs.core.pos = (sx, sy); + true +} + /// Animate the overlay cursor to `(x, y)` and suspend until the Dubins path /// completes and the spring overshoot begins. /// /// Mirrors Swift's `AgentCursor.shared.animateAndWait(to:)`. -/// Returns immediately (no animation) when: -/// - the overlay is disabled, or -/// - the cursor is still at the off-screen sentinel `(-200, -200)` — in that -/// case the caller should rely on `ClickPulse` to snap the cursor. +/// Returns immediately (no animation) only when the overlay is disabled for +/// this cursor. A brand-new cursor still at the off-screen sentinel is first +/// seeded on-screen via [`seed_start_if_sentinel`] so its FIRST action glides +/// in (it previously snapped silently via `ClickPulse`, invisible on a pure-AX +/// run). pub async fn animate_cursor_to(key: CursorKey, x: f64, y: f64) { - // Check whether animation should run for THIS cursor. A not-yet-created - // cursor (sentinel position) relies on ClickPulse to snap, same as before. + // Empty key = anonymous (no session) → no cursor to animate. + if key.is_empty() { + return; + } + // Seed a sentinel cursor on-screen so the MoveTo below glides instead of + // being short-circuited. After this the cursor's pos.0 > -50.0, so the + // should-animate check passes on the first action just like later ones. + seed_start_if_sentinel(&key, x, y); + + // Check whether animation should run for THIS cursor. A disabled cursor + // never animates; an absent cursor (seed found nothing to prime) is skipped. let should_animate = { let guard = RENDER.lock().unwrap(); match guard.as_ref().and_then(|m| m.cursors.get(&key)) { @@ -899,6 +980,46 @@ mod tests { assert!(map.cursors.contains_key("default")); } + #[test] + fn seed_moves_sentinel_cursor_on_screen_for_first_action() { + // BUG 2 regression: a brand-new session cursor at the sentinel must be + // seeded on-screen (pos.0 > -50) so the immediately-following MoveTo + // glides instead of silently snapping via ClickPulse. + let mut map = empty_map(); // 100x100 frame + // No "sessA" cursor exists yet — the seed must get-or-create it. + let seeded = seed_start_in_map(&mut map, &"sessA".to_owned(), 60.0, 60.0); + assert!(seeded, "sentinel cursor must be seeded"); + let pos = map.cursors["sessA"].core.pos; + assert!(pos.0 > -50.0 && pos.1 > -50.0, "seed must be on-screen, got {pos:?}"); + // And it must be a DIFFERENT point from the target so there is a glide. + assert!((pos.0 - 60.0).abs() > 4.0 || (pos.1 - 60.0).abs() > 4.0, + "seed must differ from target to produce a visible glide, got {pos:?}"); + } + + #[test] + fn seed_is_noop_when_cursor_already_on_screen() { + // A second action: the cursor already landed somewhere on-screen, so the + // seed must NOT move it (the MoveTo path should start from where it is). + let mut map = empty_map(); + // Put sessA on-screen first. + seed_start_in_map(&mut map, &"sessA".to_owned(), 60.0, 60.0); + map.cursors.get_mut("sessA").unwrap().core.pos = (30.0, 30.0); + let seeded_again = seed_start_in_map(&mut map, &"sessA".to_owned(), 80.0, 80.0); + assert!(!seeded_again, "on-screen cursor must not be re-seeded"); + assert_eq!(map.cursors["sessA"].core.pos, (30.0, 30.0), "pos must be untouched"); + } + + #[test] + fn seed_does_not_resurrect_ended_session() { + // The seed shares the resurrection guard: it must not re-create a cursor + // whose session already ended. + let mut map = empty_map(); + map.ended.insert("sessA".to_owned()); + let seeded = seed_start_in_map(&mut map, &"sessA".to_owned(), 60.0, 60.0); + assert!(!seeded, "ended session must not be seeded"); + assert!(!map.cursors.contains_key("sessA"), "ended session must not be resurrected"); + } + #[test] fn per_key_arrival_isolation() { // Two concurrent waiters keyed A and B; firing A must not cancel B. diff --git a/libs/cua-driver/rust/crates/platform-macos/src/cursor/state.rs b/libs/cua-driver/rust/crates/platform-macos/src/cursor/state.rs index 7af933cbf2..931673484f 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/cursor/state.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/cursor/state.rs @@ -102,7 +102,9 @@ impl CursorRegistry { // Write-boundary resurrection guard (see get_or_create): no-op for an // ended session id so a post-session_end set_config can't resurrect the // cursor metadata. "default" / non-session ids are never ended. - if cua_driver_core::session::is_session_ended(&config.cursor_id) { + if config.cursor_id.is_empty() + || cua_driver_core::session::is_session_ended(&config.cursor_id) + { return; } let mut inner = self.inner.lock().unwrap(); @@ -117,7 +119,7 @@ impl CursorRegistry { // Write-boundary resurrection guard (see get_or_create): no-op for an // ended session id so an in-flight move after session_end can't // re-create the cleared cursor. "default" / non-session ids never ended. - if cua_driver_core::session::is_session_ended(cursor_id) { + if cursor_id.is_empty() || cua_driver_core::session::is_session_ended(cursor_id) { return; } let mut inner = self.inner.lock().unwrap(); @@ -134,7 +136,7 @@ impl CursorRegistry { // Write-boundary resurrection guard (see get_or_create): no-op for an // ended session id so a post-session_end enable/disable can't resurrect // the cleared cursor. "default" / non-session ids are never ended. - if cua_driver_core::session::is_session_ended(cursor_id) { + if cursor_id.is_empty() || cua_driver_core::session::is_session_ended(cursor_id) { return; } let mut inner = self.inner.lock().unwrap(); diff --git a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs index cedda2bd73..d3910daa92 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/input/skylight.rs @@ -206,6 +206,28 @@ fn sel_register(name: &CStr) -> *mut c_void { } } +/// Whether `cls` actually implements `sel`, via `class_respondsToSelector`. +/// +/// macOS 14 (Sonoma) compatibility guard: `SLSEventAuthenticationMessage` +/// exists on macOS 14, but `messageWithEventRecord:pid:version:` was only +/// added in macOS 15 (Sequoia). `sel_registerName` always succeeds (it just +/// interns the string), so a `!sel.is_null()` check is not enough — we must +/// confirm the class responds before calling `objc_msgSend`, or the runtime +/// raises `NSInvalidArgumentException: unrecognized selector`. See #1503. +fn class_responds_to_selector(cls: *mut c_void, sel: *mut c_void) -> bool { + if cls.is_null() || sel.is_null() { + return false; + } + type RespondsToFn = unsafe extern "C" fn(*mut c_void, *mut c_void) -> bool; + static SYM: OnceLock> = OnceLock::new(); + let f = *SYM + .get_or_init(|| find_sym(b"class_respondsToSelector\0").map(|p| unsafe { as_fn(p) })); + match f { + Some(f) => unsafe { f(cls, sel) }, + None => false, + } +} + // ── SLSEventRecord extraction ────────────────────────────────────────────── /// Extract the embedded `SLSEventRecord *` from a `CGEvent`. @@ -244,11 +266,19 @@ pub fn post_to_pid(pid: pid_t, event_ptr: *mut c_void, attach_auth_message: bool if attach_auth_message { // Build and attach SLSEventAuthenticationMessage. + // + // macOS 14 (Sonoma) compatibility: the class exists on macOS 14 but + // `messageWithEventRecord:pid:version:` was added in macOS 15. Guard + // with `class_respondsToSelector` (a `!sel.is_null()` check is not + // enough — `sel_registerName` interns any name); when the selector is + // absent we skip the auth envelope and fall through to the plain + // `SLEventPostToPid` below. Chromium-class targets may not receive the + // event on macOS 14, but the daemon no longer crashes. See #1503. let cls = objc_class(c"SLSEventAuthenticationMessage"); let sel = sel_register(c"messageWithEventRecord:pid:version:"); let factory = factory_msg_send_fn(); - if !cls.is_null() && !sel.is_null() { + if class_responds_to_selector(cls, sel) { if let Some(factory_fn) = factory { let record = unsafe { extract_event_record(event_ptr) }; if !record.is_null() { diff --git a/libs/cua-driver/rust/crates/platform-macos/src/permissions/gate.rs b/libs/cua-driver/rust/crates/platform-macos/src/permissions/gate.rs index 7460b1d3e0..d386354c29 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/permissions/gate.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/permissions/gate.rs @@ -187,6 +187,17 @@ impl GateOpts { } } +/// Set by `reexec_self` on the restarted process so its `run_if_needed` +/// polls SILENTLY instead of re-raising the TCC prompts on every ~25s +/// re-exec (which otherwise spams the user with "Cua Driver" dialogs). +const GATE_REEXEC_ENV: &str = "CUA_DRIVER_RS_GATE_REEXEC"; + +/// Persists the gate's original start time (unix seconds) across re-execs +/// so the `deadline` is cumulative. Without it each re-exec'd process resets +/// `start`, and since a re-exec fires (~25s) well before the deadline +/// (~10min) the deadline never triggers and the daemon re-execs forever. +const GATE_START_ENV: &str = "CUA_DRIVER_RS_GATE_START_UNIX"; + /// Run the gate if needed. When called and the process already has both /// grants, this returns immediately without printing anything — the /// `serve` happy path is unaffected. @@ -216,6 +227,17 @@ pub fn run_if_needed(opts: GateOpts) -> Result<()> { return Ok(()); } + // A gate re-exec (`wait_for_grants` restarts the daemon ~every 25s to + // refresh the per-process TCC trust cache) re-runs this function. The + // FIRST process already raised the TCC prompts + showed the panel; + // re-doing that on every re-exec spams a fresh "Cua Driver" dialog at the + // user (forever, since a stale/never-granted state keeps re-execing). A + // re-exec'd process polls SILENTLY instead — skip the prompts + panel and + // go straight to the wait loop, which re-checks the grant and re-execs. + if std::env::var(GATE_REEXEC_ENV).is_ok() { + return wait_for_grants(&opts); + } + let missing = missing_from_status(initial); // Raise the TCC system prompts BEFORE showing our panel. The @@ -359,7 +381,27 @@ fn present_panel_if_available(initial: PermissionsStatus) -> PanelPresentation { /// line. If the grant has been given the new process picks it up /// instantly; otherwise it falls back into the same wait loop. pub fn wait_for_grants(opts: &GateOpts) -> Result<()> { - let start = Instant::now(); + // Anchor the deadline to the ORIGINAL gate start, persisted across + // re-execs via GATE_START_ENV. A re-exec'd process shifts `start` back by + // the elapsed wall time so `start.elapsed()` keeps growing across + // restarts and the deadline is cumulative — otherwise each restart resets + // the clock, re-exec fires before the deadline, and the gate (and the + // daemon + cursor overlay it restarts) churns forever when ungranted. + let start = { + let now_unix = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0); + match std::env::var(GATE_START_ENV).ok().and_then(|s| s.parse::().ok()) { + Some(orig) => Instant::now() + .checked_sub(Duration::from_secs(now_unix.saturating_sub(orig))) + .unwrap_or_else(Instant::now), + None => { + std::env::set_var(GATE_START_ENV, now_unix.to_string()); + Instant::now() + } + } + }; let mut last_status_print = start; let mut last_missing: Vec = check_required_permissions(); let mut polls_without_change: u32 = 0; @@ -488,6 +530,17 @@ fn reexec_self() { } }; + // Mark the restarted process as a gate re-exec so its `run_if_needed` + // polls silently (no re-prompt), and anchor the original gate start so + // the deadline is cumulative across re-execs. execvp inherits the + // environment, so the new image sees these. + std::env::set_var(GATE_REEXEC_ENV, "1"); + if std::env::var(GATE_START_ENV).is_err() { + if let Ok(d) = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH) { + std::env::set_var(GATE_START_ENV, d.as_secs().to_string()); + } + } + // execvp returns -1 on failure; on success it does not return. // SAFETY: argv_ptrs is NULL-terminated; exe_c outlives the call. unsafe { diff --git a/libs/cua-driver/rust/crates/platform-macos/src/recording_hooks.rs b/libs/cua-driver/rust/crates/platform-macos/src/recording_hooks.rs index ab79efa434..101e17d6ad 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/recording_hooks.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/recording_hooks.rs @@ -49,8 +49,10 @@ pub fn element_window_local_xy(window_id: u64, pid: i64, element_index: u32) -> let cache = ELEMENT_CACHE.get()?; let pid_i32 = i32::try_from(pid).ok()?; let window_id_u32 = u32::try_from(window_id).ok()?; - let ptr = cache.get_element_ptr(pid_i32, window_id_u32, element_index as usize)?; - let (sx, sy) = unsafe { element_screen_center(ptr as AXUIElementRef)? }; + // Retain so a concurrent get_window_state can't free the element between + // the lookup and element_screen_center (use-after-free → daemon crash). + let element = cache.get_element_retained(pid_i32, window_id_u32, element_index as usize)?; + let (sx, sy) = unsafe { element_screen_center(element.as_ptr() as AXUIElementRef)? }; let bounds = crate::windows::window_bounds_by_id(window_id_u32)?; // Probe the captured PNG's width to derive the Retina scale — the diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs index dcf125dc46..bc62b9dc0b 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs @@ -66,6 +66,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer", "description": "Target process ID." }, "window_id": { "type": "integer", "description": "Target window ID. Required for element_index." }, "element_index": { "type": "integer", "description": "Element index from last get_window_state." }, @@ -119,13 +120,18 @@ impl Tool for ClickTool { if let (Some(idx), Some(wid)) = (element_index, window_id) { // ── AX element path ──────────────────────────────────────────── - let element_ptr = match self.state.element_cache.get_element_ptr(pid, wid, idx) { - Some(p) => p, + // Retain the element out of the cache so it can't be freed by a + // concurrent get_window_state on the same (pid, window_id) while + // this click is mid-flight (use-after-free → daemon crash). The + // guard lives to the end of this method, past the AX action below. + let element_guard = match self.state.element_cache.get_element_retained(pid, wid, idx) { + Some(e) => e, None => return ToolResult::error(format!( "Element index {idx} not found in cache for pid={pid} window_id={wid}. \ Call get_window_state first." )), }; + let element_ptr = element_guard.as_ptr(); // Animate cursor to element center BEFORE firing AX action, // mirroring Swift's `performElementClick` → `animateAndWait(to:)`. diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/cursor_tools.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/cursor_tools.rs index ffccaae2f6..58c9e99f8d 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/cursor_tools.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/cursor_tools.rs @@ -11,26 +11,32 @@ use std::sync::Arc; use super::ToolState; -/// Resolve the cursor key for a tool invocation. +/// The cursor key for an anonymous (cursor-less) call. A run opts into a cursor +/// by declaring a `session`; without one, every cursor op short-circuits on this +/// empty key (see `overlay::send_command` / `CursorRegistry`). +pub(crate) const NO_CURSOR: &str = ""; + +/// Resolve the cursor key for a tool invocation, or [`NO_CURSOR`] (`""`) for an +/// anonymous call. /// -/// Precedence (mandatory): an explicit, non-empty `cursor_id` arg wins, then -/// the daemon-injected `_session_id` (so each MCP session owns a cursor by -/// default), then the seeded `"default"` cursor (anonymous / one-shot -/// `cua-driver call`). Putting `cursor_id` first means a wrapper that -/// deliberately shares one cursor_id across sessions is NOT fragmented. +/// A cursor is tied to a **caller-declared session**, never to the MCP +/// connection. Precedence: an explicit `session` arg, then its legacy alias +/// `cursor_id`. We deliberately do NOT fall back to the connection-injected +/// `_session_id` (the recording/config lifecycle fallback) or to a seeded +/// `"default"` cursor — so `""` means "no session declared → no cursor", while +/// the underlying action (click/type/…) still executes. The same id works +/// identically over MCP, the CLI (`--session`), or the raw socket, and follows +/// the run across any number of apps/windows. pub(crate) fn resolve_cursor_key(args: &Value) -> String { use cua_driver_core::tool_args::ArgsExt; - if let Some(explicit) = args.opt_str("cursor_id") { - if !explicit.is_empty() { - return explicit; - } - } - if let Some(session) = args.opt_str("_session_id") { - if !session.is_empty() { - return session; + for key in ["session", "cursor_id"] { + if let Some(v) = args.opt_str(key) { + if !v.is_empty() { + return v; + } } } - "default".to_owned() + NO_CURSOR.to_owned() } // ── SetAgentCursorEnabled ───────────────────────────────────────────────────── @@ -48,7 +54,12 @@ static ENABLED_DEF: std::sync::OnceLock = std::sync::OnceLock::new(); fn enabled_def() -> &'static ToolDef { ENABLED_DEF.get_or_init(|| ToolDef { name: "set_agent_cursor_enabled".into(), - description: "Show or hide the agent cursor overlay for a cursor instance.".into(), + description: "Show or hide the agent cursor for a session. A cursor exists only for a \ + DECLARED session: pass `session` (the same id you start_session / drive \ + actions with) and the cursor appears on that session's first action — its \ + color is derived from the id. Without a `session`, actions run cursor-less. \ + Use enabled=false to hide a session's cursor, enabled=true to re-show it. \ + (`cursor_id` is a legacy alias for `session`.)".into(), input_schema: serde_json::json!({ "type": "object", "required": ["enabled"], @@ -498,31 +509,31 @@ impl Tool for GetAgentCursorStateTool { #[cfg(test)] mod tests { - use super::resolve_cursor_key; + use super::{resolve_cursor_key, NO_CURSOR}; use serde_json::json; #[test] - fn anonymous_resolves_to_default() { - // No cursor_id, no _session_id → seeded "default" cursor (backward - // compatible one-shot `cua-driver call`). - assert_eq!(resolve_cursor_key(&json!({})), "default"); - assert_eq!(resolve_cursor_key(&json!({ "x": 1 })), "default"); + fn anonymous_resolves_to_no_cursor() { + // No session/cursor_id declared → NO_CURSOR (""): the action still runs + // but no cursor is shown. The connection-injected `_session_id` is NOT a + // cursor source anymore (it stays the recording/config lifecycle key). + assert_eq!(resolve_cursor_key(&json!({})), NO_CURSOR); + assert_eq!(resolve_cursor_key(&json!({ "x": 1 })), NO_CURSOR); + assert_eq!(resolve_cursor_key(&json!({ "_session_id": "mcp-1-2" })), NO_CURSOR); } #[test] - fn session_id_owns_a_cursor_by_default() { - assert_eq!( - resolve_cursor_key(&json!({ "_session_id": "sess-7" })), - "sess-7" - ); + fn explicit_session_owns_a_cursor() { + assert_eq!(resolve_cursor_key(&json!({ "session": "research-run" })), "research-run"); } #[test] - fn explicit_cursor_id_wins_over_session() { - // Precedence: explicit cursor_id > injected _session_id > "default". + fn cursor_id_is_a_legacy_alias() { + // `cursor_id` still works (codex-wrapper use case); `session` wins if both. + assert_eq!(resolve_cursor_key(&json!({ "cursor_id": "user-handle" })), "user-handle"); assert_eq!( - resolve_cursor_key(&json!({ "cursor_id": "user-handle", "_session_id": "sess-7" })), - "user-handle" + resolve_cursor_key(&json!({ "session": "s1", "cursor_id": "c1" })), + "s1" ); } @@ -547,16 +558,57 @@ mod tests { } #[test] - fn empty_strings_fall_through() { - // An empty cursor_id falls through to _session_id; empty session falls - // through to "default". + fn enable_and_ax_click_resolve_the_same_session_cursor() { + // In one run, set_agent_cursor_enabled and a click(element_index) carry + // the SAME explicit `session`, so both resolve the same cursor key — + // enabling the cursor lights the very cursor the AX click drives. + let session = "research-run"; + let enable_args = json!({ "enabled": true, "session": session }); + let ax_click_args = + json!({ "pid": 844, "window_id": 10725, "element_index": 14, "session": session }); + let enable_key = resolve_cursor_key(&enable_args); + let click_key = resolve_cursor_key(&ax_click_args); + assert_eq!(enable_key, session); + assert_eq!(enable_key, click_key, + "set_agent_cursor_enabled and the AX click must drive the same session cursor"); + } + + #[test] + fn get_config_reports_calling_session_cursor_deterministically() { + // BUG 3 regression: get_config's cursor_enabled must reflect the CALLING + // session's own cursor (resolved by key), not a nondeterministic + // HashMap.first(). Two sessions with opposite enabled flags must each + // read back their OWN value. + use crate::cursor::CursorRegistry; + let reg = CursorRegistry::new(); + reg.set_enabled("sessA", true); + reg.set_enabled("sessB", false); + + // Replicate get_config's resolution: key = resolve_cursor_key(args), + // then get(key) or get("default"). + let read_for = |args: &serde_json::Value| -> bool { + let key = resolve_cursor_key(args); + reg.get(&key) + .or_else(|| reg.get("default")) + .map(|s| s.config.enabled) + .unwrap_or(true) + }; + assert!(read_for(&json!({ "session": "sessA" }))); + assert!(!read_for(&json!({ "session": "sessB" }))); + // Anonymous caller (no session) falls back to the seeded default (on). + assert!(read_for(&json!({}))); + } + + #[test] + fn empty_strings_fall_through_to_no_cursor() { + // An empty `session` falls through to `cursor_id`; both empty → NO_CURSOR. assert_eq!( - resolve_cursor_key(&json!({ "cursor_id": "", "_session_id": "sess-7" })), - "sess-7" + resolve_cursor_key(&json!({ "session": "", "cursor_id": "c1" })), + "c1" ); assert_eq!( - resolve_cursor_key(&json!({ "cursor_id": "", "_session_id": "" })), - "default" + resolve_cursor_key(&json!({ "session": "", "cursor_id": "" })), + NO_CURSOR ); } } diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs index ede8bf62f8..4b63bbeaff 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/double_click.rs @@ -34,6 +34,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer" }, "x": { "type": "number", "description": "Screen X coordinate (pixel path)." }, "y": { "type": "number", "description": "Screen Y coordinate (pixel path)." }, @@ -62,12 +63,15 @@ impl Tool for DoubleClickTool { // ── AX element path ────────────────────────────────────────────────── if let (Some(idx), Some(wid)) = (element_index, window_id) { - let element_ptr = match self.state.element_cache.get_element_ptr(pid, wid, idx) { - Some(p) => p, + // Retain out of the cache so a concurrent get_window_state can't + // free the element mid-action (use-after-free → daemon crash). + let element_guard = match self.state.element_cache.get_element_retained(pid, wid, idx) { + Some(e) => e, None => return ToolResult::error(format!( "Element index {idx} not found. Call get_window_state first." )), }; + let element_ptr = element_guard.as_ptr(); // Thread the resolved session cursor key into the blocking AX path // so its ClickPulse lands on THIS session's cursor, not "default". diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs index d7fe0fa91a..5cf090e482 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/drag.rs @@ -49,6 +49,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid", "from_x", "from_y", "to_x", "to_y"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer", "description": "Target process ID." }, "window_id": { "type": "integer", diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs index 7d79c8ad6d..f238a36d9f 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_config.rs @@ -41,8 +41,14 @@ impl Tool for GetConfigTool { let cfg = self.state.config.read().unwrap(); self.state.session_config.effective(session_id.as_deref(), &cfg) }; - let cursor_enabled = self.state.cursor_registry.all_states() - .first() + // Report the CALLING session's own cursor enabled-state, not a + // nondeterministic HashMap.first(). Resolve the same key the click / + // cursor tools use (cursor_id > _session_id > "default"); fall back to + // the seeded "default" cursor when this session hasn't materialised its + // own cursor yet, and finally to `true` (the overlay default). + let cursor_key = super::cursor_tools::resolve_cursor_key(&args); + let cursor_enabled = self.state.cursor_registry.get(&cursor_key) + .or_else(|| self.state.cursor_registry.get("default")) .map(|s| s.config.enabled) .unwrap_or(true); // PiP values aren't in DriverConfig — they're file-only since the diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs index c0b63fe4d8..ad1b3e20ad 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs @@ -31,6 +31,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid", "window_id"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer", "description": "Target process ID." }, "window_id": { "type": "integer", "description": "Target window ID from list_windows." }, "query": { "type": "string", "description": "Case-insensitive filter for tree_markdown." }, diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs index 318594d1f7..1750379ef6 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/hotkey.rs @@ -44,6 +44,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid", "keys"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer", "description": "Target process ID." }, "keys": { "type": "array", diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs index 8506789a73..4894b28eae 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/launch_app.rs @@ -23,7 +23,11 @@ fn def() -> &'static ToolDef { port (sets WEBKIT_INSPECTOR_SERVER=127.0.0.1:N + TAURI_WEBVIEW_AUTOMATION=1). \ Use this for Tauri/WebKit-based apps.\n\n\ Optional `creates_new_application_instance`: when true, forces a new app instance \ - even if one is already running (passes -n to open).\n\n\ + even if one is already running (passes -n to open). Reach for this when another \ + agent or session may drive the SAME app concurrently — it returns a fresh pid + \ + window so each session acts on its own isolated window instead of clobbering one \ + shared instance. Without it, single-instance apps (Calculator, many utilities) hand \ + every caller the same window, so two sessions fight over it.\n\n\ Optional `additional_arguments`: extra argv strings appended after --args.\n\n\ Returns the launched app's pid, bundle_id, name, and a `windows` array \ (same shape as `list_windows`) so callers can skip an extra round-trip before \ @@ -58,7 +62,7 @@ fn def() -> &'static ToolDef { }, "creates_new_application_instance": { "type": "boolean", - "description": "When true, force a new app instance even if already running (open -n)." + "description": "When true, force a new app instance even if already running (open -n). Use for concurrent multi-agent/multi-session work so each session gets an isolated instance + window instead of sharing one — on single-instance apps (e.g. Calculator) every caller otherwise gets the same window and the sessions clobber each other." }, "additional_arguments": { "type": "array", diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs index 1d05ce2991..cb5b81c558 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs @@ -353,8 +353,9 @@ pub fn register_all(registry: &mut ToolRegistry, compat: bool) { registry.register(Box::new(cua_driver_core::page::PageTool::new( Arc::new(page::MacOsPageBackend::new(state.clone())), ))); - // Recording / replay tools are platform-independent — live in mcp-server. + // Recording / replay + session-lifecycle tools are platform-independent. registry.register_recording_tools(); + registry.register_session_tools(); } #[cfg(test)] diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs index 7807009c7c..3d14684046 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/move_cursor.rs @@ -25,13 +25,18 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["x", "y"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "x": { "type": "number" }, "y": { "type": "number" }, "cursor_id": { "type": "string", "description": "Cursor instance to move. Default: 'default'." } }, "additionalProperties": false }), - read_only: false, + // read-only: move_cursor only nudges the agent-cursor overlay, never the + // target app — so it's safe to run concurrently. The `readOnlyHint` this + // emits lets MCP clients (e.g. Claude Code's isConcurrencySafe) parallelize + // cursor moves. (Mutating tools like click stay read_only:false on purpose.) + read_only: true, destructive: false, idempotent: true, open_world: false, @@ -49,13 +54,14 @@ impl Tool for MoveCursorTool { let cursor_id = super::cursor_tools::resolve_cursor_key(&args); self.state.cursor_registry.update_position(&cursor_id, x, y); - // Drive the visual overlay for THIS session's cursor (no-op when the - // overlay is disabled). End pointing upper-left (45°) — matches Swift's - // `AgentCursor.animateAndWait(endAngleDegrees: 45)` convention. - crate::cursor::overlay::send_command( - cursor_id.clone(), - cursor_overlay::OverlayCommand::MoveTo { x, y, end_heading_radians: std::f64::consts::FRAC_PI_4 } - ); + // Drive the DRAWN cursor via the same path as click's animation. A raw + // `MoveTo` doesn't reliably bring a brand-new session cursor on-screen — + // it sits at the off-screen sentinel until a click seeds it, so the + // visible cursor wouldn't move (the reported position would, but the + // overlay wouldn't). `animate_cursor_to` seeds the sentinel on-screen + // then glides in, identical to `click`. No-op for an empty (anonymous) + // key or when the overlay is disabled for this cursor. + crate::cursor::overlay::animate_cursor_to(cursor_id.clone(), x, y).await; ToolResult::text(format!("Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1}).")) } } diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs index 03b6067bdd..e16b4b4054 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/press_key.rs @@ -40,6 +40,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid", "key"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer" }, "key": { "type": "string", "description": "Key name: return, tab, escape, up, down, etc." }, "modifiers": { @@ -85,11 +86,15 @@ impl Tool for PressKeyTool { // Resolve the pre-focus element pointer (if requested) outside // the suppression closure — only the focus_element() write itself // needs to run under suppression, the cache lookup does not. - let pre_focus_ptr: Option = if let (Some(idx), Some(wid)) = (element_index, window_id) { - self.state.element_cache.get_element_ptr(pid, wid, idx) + // Retain out of the cache so a concurrent get_window_state can't free + // the element before the suppressed focus below dereferences it + // (use-after-free → daemon crash). Guard lives to method end. + let pre_focus_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) { + self.state.element_cache.get_element_retained(pid, wid, idx) } else { None }; + let pre_focus_ptr: Option = pre_focus_guard.as_ref().map(|g| g.as_ptr()); // ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ── // Single-key presses can fire autocomplete (Return on a search diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs index 5312cc34da..cbdcca00be 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/right_click.rs @@ -39,6 +39,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer", "description": "Target process ID." }, "element_index": { "type": "integer", @@ -107,12 +108,15 @@ impl Tool for RightClickTool { // ── AX element path ────────────────────────────────────────────────── if let (Some(idx), Some(wid)) = (element_index, window_id) { - let element_ptr = match self.state.element_cache.get_element_ptr(pid, wid, idx) { - Some(p) => p, + // Retain out of the cache so a concurrent get_window_state can't + // free the element mid-action (use-after-free → daemon crash). + let element_guard = match self.state.element_cache.get_element_retained(pid, wid, idx) { + Some(e) => e, None => return ToolResult::error(format!( "Element index {idx} not found. Call get_window_state first." )), }; + let element_ptr = element_guard.as_ptr(); let result = tokio::task::spawn_blocking(move || { ax_show_menu(element_ptr, idx) diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs index d1eab5d3db..4b3dcc0467 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/scroll.rs @@ -30,6 +30,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid", "direction"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer" }, "direction": { "type": "string", @@ -75,11 +76,15 @@ impl Tool for ScrollTool { // Resolve the pre-focus element pointer (if requested) outside // the suppression closure — only the focus_element() write itself // needs to run under suppression, the cache lookup does not. - let pre_focus_ptr: Option = if let (Some(idx), Some(wid)) = (element_index, window_id) { - self.state.element_cache.get_element_ptr(pid, wid, idx) + // Retain out of the cache so a concurrent get_window_state can't free + // the element before the suppressed focus below dereferences it + // (use-after-free → daemon crash). Guard lives to method end. + let pre_focus_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) { + self.state.element_cache.get_element_retained(pid, wid, idx) } else { None }; + let pre_focus_ptr: Option = pre_focus_guard.as_ref().map(|g| g.as_ptr()); let key = match (by.as_str(), direction.as_str()) { ("page", "down") | (_, "down") if by == "page" => "pagedown", diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs index 9e24883ba8..1e40f4b41a 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/set_value.rs @@ -59,6 +59,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid", "window_id", "element_index", "value"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer" }, "window_id": { "type": "integer", @@ -90,12 +91,16 @@ impl Tool for SetValueTool { let element_index = match args.require_u64("element_index") { Ok(v) => v as usize, Err(e) => return e }; let value = match args.require_str("value") { Ok(v) => v, Err(e) => return e }; - let element_ptr = match self.state.element_cache.get_element_ptr(pid, window_id, element_index) { - Some(p) => p, + // Retain out of the cache so a concurrent get_window_state can't free + // the element mid-action (use-after-free → daemon crash). Guard lives + // to the end of this method, past the AX write below. + let element_guard = match self.state.element_cache.get_element_retained(pid, window_id, element_index) { + Some(e) => e, None => return ToolResult::error(format!( "Element index {element_index} not found. Call get_window_state first." )), }; + let element_ptr = element_guard.as_ptr(); // ── Focus-suppression wrap (Swift WindowChangeDetector + FocusGuard) ── // AXValue writes on popups / sliders can cause reflex activations diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs index 8f64ce6a62..501a88387f 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs @@ -58,6 +58,7 @@ fn def() -> &'static ToolDef { "type": "object", "required": ["pid", "text"], "properties": { + "session": { "type": "string", "description": "Optional session id: declares/uses the agent cursor and per-session state for this run. The same id works over MCP, the CLI, or the raw socket, and follows the run across apps/windows. Omit to run cursor-less." }, "pid": { "type": "integer", "description": "Target process ID." }, "text": { "type": "string", "description": "Text to insert at the target's cursor." }, "window_id": { @@ -107,10 +108,13 @@ impl Tool for TypeTextTool { ); } - // Resolve the element pointer (if element_index given). - let element_ptr = if let (Some(idx), Some(wid)) = (element_index, window_id) { - match self.state.element_cache.get_element_ptr(pid, wid, idx) { - Some(p) => Some((p, Some(idx))), + // Resolve the element pointer (if element_index given). Retain it out + // of the cache so a concurrent get_window_state can't free it before + // the blocking type below dereferences it (use-after-free → daemon + // crash). The guard lives to method end, past type_text_blocking. + let element_guard = if let (Some(idx), Some(wid)) = (element_index, window_id) { + match self.state.element_cache.get_element_retained(pid, wid, idx) { + Some(e) => Some((e, idx)), None => return ToolResult::error(format!( "Element index {idx} not found. Call get_window_state first." )), @@ -118,6 +122,7 @@ impl Tool for TypeTextTool { } else { None }; + let element_ptr = element_guard.as_ref().map(|(g, idx)| (g.as_ptr(), Some(*idx))); let text_clone = text.clone(); let char_count = text.chars().count(); diff --git a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs index ce87f21989..642884905d 100644 --- a/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs +++ b/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text_chars.rs @@ -61,10 +61,15 @@ impl Tool for TypeTextCharsTool { // Pre-focus element if requested. if !type_chars_only { if let (Some(idx), Some(wid)) = (element_index, window_id) { - if let Some(element_ptr) = self.state.element_cache.get_element_ptr(pid, wid, idx) { + // Retain so a concurrent get_window_state can't free the element + // during the focus call (use-after-free → daemon crash). The + // guard outlives the awaited spawn_blocking below. + if let Some(element_guard) = self.state.element_cache.get_element_retained(pid, wid, idx) { + let element_ptr = element_guard.as_ptr(); let _ = tokio::task::spawn_blocking(move || { crate::input::ax_actions::focus_element(element_ptr) }).await; + drop(element_guard); tokio::time::sleep(std::time::Duration::from_millis(50)).await; } } diff --git a/libs/cua-driver/rust/crates/platform-windows/Cargo.toml b/libs/cua-driver/rust/crates/platform-windows/Cargo.toml index 04f4fa34f3..c40c8eccef 100644 --- a/libs/cua-driver/rust/crates/platform-windows/Cargo.toml +++ b/libs/cua-driver/rust/crates/platform-windows/Cargo.toml @@ -14,6 +14,10 @@ async-trait = "0.1" cua-driver-core = { path = "../cua-driver-core" } cursor-overlay = { path = "../cursor-overlay" } pip-preview = { path = "../pip-preview" } +# IndexMap gives the keyed cursor render collection deterministic +# insertion-ordered iteration = stable per-session cursor z-order frame to +# frame (mirrors platform-macos's per-session overlay). +indexmap = "2" # tiny-skia for cross-platform cursor rendering (used in overlay.rs on all targets) tiny-skia = { version = "0.11", default-features = false, features = ["std"] } diff --git a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs index 4650c5107b..6cff851e57 100644 --- a/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs +++ b/libs/cua-driver/rust/crates/platform-windows/src/overlay.rs @@ -9,103 +9,333 @@ //! - Z-ordering: every 80ms call `SetWindowPos` to stay just above the pinned target. //! - Idle-hide: fade out over 180ms once `idle_hide_ms` has elapsed with no activity. //! +//! ## Per-session cursors (2026-06 port from platform-macos #1779) +//! +//! Before this, the overlay was a process-wide singleton (one `RenderState`), +//! so concurrent MCP sessions clobbered each other last-writer-wins → one +//! shared cursor. It now keeps a keyed [`RenderMap`] (`IndexMap`): each declared `session` owns its own cursor with its own +//! palette, and the ~125 Hz tick composites them all into the single layered +//! window. `IndexMap` gives deterministic insertion-ordered iteration = stable +//! per-session z-order frame to frame. The lifecycle (lazy create, per-key +//! arrival isolation, `session_end` removal, resurrection tombstone) mirrors +//! `platform_macos::cursor::overlay` so the two platforms behave identically; +//! the shared `cursor_overlay::{CursorKey, KeyedOverlayCommand, OverlayMsg}` +//! types are the same ones macOS uses. +//! //! ## Cross-platform note (2026-05 dedup audit) //! //! Animation state + render pipeline live in `cursor_overlay::render_state` -//! (`RenderStateCore`, `tick_motion`, `apply_command_base`, `render_frame`). +//! (`RenderStateCore`, `tick_motion`, `apply_command_base`, `paint_cursor`). //! What stays here is purely the Win32 window plumbing: message loop, //! UpdateLayeredWindow paint, virtual-screen offset, z-order maintenance. #![allow(non_snake_case, non_upper_case_globals)] +use std::collections::{HashMap, HashSet}; use std::sync::{Mutex, OnceLock}; use std::time::Instant; use cursor_overlay::{ - CursorConfig, MotionConfig, OverlayCommand, RenderStateCore, ZOrderEnforcer, + CursorConfig, CursorKey, KeyedOverlayCommand, MotionConfig, OverlayCommand, OverlayMsg, + Palette, RenderStateCore, ZOrderEnforcer, }; +use indexmap::IndexMap; // ── Global channel ──────────────────────────────────────────────────────── -static CMD_TX: OnceLock> = OnceLock::new(); -static CMD_RX_CELL: Mutex>> = Mutex::new(None); -static RENDER: Mutex> = Mutex::new(None); +static CMD_TX: OnceLock> = OnceLock::new(); +static CMD_RX_CELL: Mutex>> = Mutex::new(None); +static RENDER: Mutex> = Mutex::new(None); -// ── Arrival-signal channel ──────────────────────────────────────────────── +// ── Arrival-signal channels (one waiter slot per cursor key) ────────────── // -// `animate_cursor_to` installs a oneshot sender here, the render thread's -// `WM_TIMER` handler fires it the tick the planned path ends. Mirrors -// macOS so click handlers can `.await` until the cursor visually lands -// before dispatching the actual UIA / PostMessage action. -static ARRIVAL_TX: Mutex>> = Mutex::new(None); +// Each session's `animate_cursor_to` registers an arrival oneshot keyed by its +// own cursor key. A new animation only supersedes the SAME key's prior waiter, +// so concurrent sessions never cross-cancel each other's arrivals. Mirrors +// macOS so click handlers can `.await` until the cursor visually lands before +// dispatching the actual UIA / PostMessage action. +static ARRIVAL_TX: Mutex>>> = + Mutex::new(None); + +fn arrival_register(key: CursorKey, tx: tokio::sync::oneshot::Sender<()>) { + let mut guard = ARRIVAL_TX.lock().unwrap(); + let map = guard.get_or_insert_with(HashMap::new); + // Cancel only the same key's previous waiter (superseded by new animation). + if let Some(old_tx) = map.insert(key, tx) { + let _ = old_tx.send(()); + } +} + +fn arrival_fire(key: &CursorKey) { + if let Ok(mut guard) = ARRIVAL_TX.lock() { + if let Some(map) = guard.as_mut() { + if let Some(tx) = map.remove(key) { + let _ = tx.send(()); + } + } + } +} + +// ── Keyed render collection ─────────────────────────────────────────────── + +/// The keyed, insertion-ordered collection of owned cursors that the render +/// loop composites every tick. Insertion order = stable z-order (later keys +/// paint on top). Virtual-screen geometry + the `WM_TIMER` dt stamp are +/// hoisted here (screen-global, written once in `run_overlay_thread`). +struct RenderMap { + cursors: IndexMap, + /// Virtual screen dimensions set after window creation (Win32 DIPs). + /// `virt_x/y` are subtracted from each cursor's `core.pos` when rendering + /// so the pixmap is laid out in window-local coordinates. + virt_x: i32, + virt_y: i32, + virt_w: i32, + virt_h: i32, + /// Last WM_TIMER wall-clock stamp; used to compute real `dt` (Windows + /// timer resolution defaults to 15ms so a hardcoded 8ms would run the + /// animation at half speed). + last_tick: Instant, + /// Frozen launch-time config used as the template for lazily-created + /// cursors (its palette is overridden per-key via `Palette::for_instance`). + template: CursorConfig, + /// Render-side tombstone of permanently-ended session cursor keys. A `Cmd` + /// for a key in here is dropped WITHOUT get-or-create, so an in-flight + /// click/move from another task that lands AFTER the owning session's + /// `Remove` can never resurrect the just-removed cursor. "default" is never + /// tombstoned (it backs the anonymous / one-shot path). + ended: HashSet, + /// Cursor key whose target the overlay should currently sit above. A single + /// layered window can occupy only one z-band, so the most-recently-touched + /// cursor wins (mirrors macOS). `None` until the first PinAbove/Cmd. + last_active: Option, +} + +/// Build the `RenderState` for a lazily-created cursor key: derive from the +/// launch template but give each non-default key its own palette so distinct +/// sessions get distinct colours automatically. +fn render_state_for_key(template: &CursorConfig, key: &str) -> RenderState { + let mut rs = RenderState::new(template.clone()); + rs.core.palette = Palette::for_instance(key); + rs +} + +/// Apply one inbound [`OverlayMsg`] to the render map (drain step). Factored +/// out as a pure function so the per-session ownership + removal lifecycle is +/// unit-testable without any Win32 window. +/// +/// Returns the resolved cursor key for a `Cmd` (so the caller can track the +/// last-active key for z-order pinning); `None` for a `Remove`. +fn apply_msg(map: &mut RenderMap, msg: OverlayMsg) -> Option { + match msg { + OverlayMsg::Remove(key) => { + // The "default" cursor backs the anonymous / one-shot path and must + // survive every session_end + the daemon lifetime. + if key != "default" { + map.cursors.shift_remove(&key); + if let Ok(mut guard) = ARRIVAL_TX.lock() { + if let Some(m) = guard.as_mut() { + m.remove(&key); + } + } + if map.last_active.as_deref() == Some(key.as_str()) { + map.last_active = None; + } + // Tombstone the key so a late in-flight Cmd from another task + // cannot re-create the just-removed cursor. + map.ended.insert(key); + } + None + } + OverlayMsg::Cmd(KeyedOverlayCommand { key, cmd }) => { + // Drop a command for an already-ended session WITHOUT get-or-create + // — this is the resurrection guard. Without it, a ClickPulse/MoveTo + // landing after Remove would re-insert (and re-leak) the cursor. + if map.ended.contains(&key) { + return None; + } + let template = map.template.clone(); + let k = key.clone(); + let rs = map + .cursors + .entry(key) + .or_insert_with(|| render_state_for_key(&template, &k)); + rs.apply_command(cmd); + Some(k) + } + } +} pub fn init(cfg: CursorConfig) { let (tx, rx) = std::sync::mpsc::sync_channel(4096); let _ = CMD_TX.set(tx); *CMD_RX_CELL.lock().unwrap() = Some(rx); - *RENDER.lock().unwrap() = Some(RenderState::new(cfg)); + *ARRIVAL_TX.lock().unwrap() = Some(HashMap::new()); + let mut cursors = IndexMap::new(); + cursors.insert("default".to_owned(), RenderState::new(cfg.clone())); + *RENDER.lock().unwrap() = Some(RenderMap { + cursors, + virt_x: 0, + virt_y: 0, + virt_w: 1920, + virt_h: 1080, + last_tick: Instant::now(), + template: cfg, + ended: HashSet::new(), + last_active: None, + }); } -pub fn send_command(cmd: OverlayCommand) { +/// Send a keyed command from any thread (MCP tool, etc.). Non-blocking; drops +/// if the channel is full (old commands are less important than new ones). +/// +/// Empty key = anonymous (no session declared) → no cursor; the command is +/// dropped so a cursor-less run never paints. See `tools::resolve_cursor_key`. +pub fn send_command(key: CursorKey, cmd: OverlayCommand) { + if key.is_empty() { + return; + } if let Some(tx) = CMD_TX.get() { - let _ = tx.try_send(cmd); + let _ = tx.try_send(OverlayMsg::Cmd(KeyedOverlayCommand { key, cmd })); } } -/// Returns the current glide duration in milliseconds (default 750). -/// Used by the click path to wait for the animation before firing ClickPulse. -pub fn glide_duration_ms() -> f64 { - RENDER.lock().ok() - .and_then(|g| g.as_ref().map(|rs| rs.core.motion.glide_duration_ms)) - .unwrap_or(750.0) +/// Convenience for callsites not yet threaded with a session key: drives the +/// seeded `"default"` cursor (the anonymous / one-shot identity). +pub fn send_command_default(cmd: OverlayCommand) { + send_command("default".to_owned(), cmd); } -/// Returns true if the cursor overlay is currently enabled/visible. -pub fn is_enabled() -> bool { - RENDER.lock().ok() - .and_then(|g| g.as_ref().map(|rs| rs.core.visible)) +/// Remove a session's owned cursor from the render collection (fired from the +/// `session_end` hook). The `"default"` key is guarded against removal on the +/// render side, so this is a no-op for it; removing an absent key (anonymous +/// session that never created a cursor) is a harmless no-op. +pub fn remove_cursor(key: CursorKey) { + if key.is_empty() { + return; + } + if let Some(tx) = CMD_TX.get() { + let _ = tx.try_send(OverlayMsg::Remove(key)); + } +} + +/// Returns true if the cursor for `key` is currently enabled/visible. A session +/// with no own cursor yet falls back to the seeded `"default"` cursor. +pub fn is_enabled(key: &str) -> bool { + RENDER + .lock() + .ok() + .and_then(|g| { + g.as_ref().and_then(|m| { + m.cursors + .get(key) + .or_else(|| m.cursors.get("default")) + .map(|rs| rs.core.visible) + }) + }) .unwrap_or(false) } -/// Snapshot the current motion config (start_handle / end_handle / arc_size / -/// arc_flow / spring / glide_duration_ms / dwell_after_click_ms / -/// idle_hide_ms). Mirrors macOS `current_motion()` so -/// `get_agent_cursor_state` can report the live values. -pub fn current_motion() -> MotionConfig { - RENDER.lock().ok() - .and_then(|g| g.as_ref().map(|rs| rs.core.motion.clone())) +/// Snapshot the current motion config for `key`, falling back to the +/// `"default"` cursor's motion when that key has no own entry yet. +pub fn current_motion(key: &str) -> MotionConfig { + RENDER + .lock() + .ok() + .and_then(|g| { + g.as_ref().and_then(|m| { + m.cursors + .get(key) + .or_else(|| m.cursors.get("default")) + .map(|rs| rs.core.motion.clone()) + }) + }) .unwrap_or_default() } -/// Returns the current cursor position in screen coordinates. -pub fn current_position() -> (f64, f64) { - RENDER.lock().ok() - .and_then(|g| g.as_ref().map(|rs| rs.core.pos)) +/// Current screen position of the cursor for `key` (the off-screen sentinel +/// `(-200, -200)` if it has never been placed). A session with no own cursor +/// yet reports the sentinel so the click path treats it as first-placement. +pub fn current_position(key: &str) -> (f64, f64) { + RENDER + .lock() + .ok() + .and_then(|g| g.as_ref().and_then(|m| m.cursors.get(key)).map(|rs| rs.core.pos)) .unwrap_or((-200.0, -200.0)) } -/// Returns true if the cursor is still at the off-screen initial position -/// (-200, -200), meaning it has never been positioned on screen yet. -pub fn is_at_initial_position() -> bool { - RENDER.lock().ok() - .and_then(|g| g.as_ref().map(|rs| rs.core.pos.0 < 0.0 && rs.core.pos.1 < 0.0)) - .unwrap_or(true) +/// Seed a brand-new (sentinel-positioned) cursor at an on-screen start point +/// offset up-left of `(target_x, target_y)` so the immediately-following +/// `MoveTo` glides INTO the target instead of silently snapping. No-op when the +/// cursor is already on-screen or its session already ended. Returns true if a +/// seed was applied. Mirrors `platform_macos::cursor::overlay::seed_start_*`. +fn seed_start_if_sentinel(key: &CursorKey, target_x: f64, target_y: f64) -> bool { + let mut guard = RENDER.lock().unwrap(); + let Some(map) = guard.as_mut() else { return false }; + seed_start_in_map(map, key, target_x, target_y) } -/// Animate the overlay cursor to `(x, y)` and suspend until the planned -/// path completes (the spring-settle phase that follows is allowed to keep -/// running — we only wait for the visible glide to land). +/// Pure seed step operating on a borrowed [`RenderMap`] — factored out so the +/// get-or-create + clamp logic is unit-testable without the global `RENDER` +/// static or a Win32 window. +fn seed_start_in_map(map: &mut RenderMap, key: &CursorKey, target_x: f64, target_y: f64) -> bool { + const SEED_OFFSET: f64 = 140.0; + let (virt_x, virt_y) = (map.virt_x as f64, map.virt_y as f64); + let (virt_w, virt_h) = (map.virt_w as f64, map.virt_h as f64); + // Respect the resurrection guard: never seed (and thus re-create) a cursor + // whose session already ended. + if map.ended.contains(key) { + return false; + } + let template = map.template.clone(); + let k = key.clone(); + let rs = map + .cursors + .entry(key.clone()) + .or_insert_with(|| render_state_for_key(&template, &k)); + if !(rs.core.cfg.enabled && rs.core.pos.0 < -50.0) { + return false; + } + let mut sx = target_x - SEED_OFFSET; + let mut sy = target_y - SEED_OFFSET; + // Clamp into the virtual-screen frame so the seed never starts off-display. + if virt_w > 0.0 && virt_h > 0.0 { + sx = sx.clamp(virt_x + 2.0, virt_x + virt_w - 2.0); + sy = sy.clamp(virt_y + 2.0, virt_y + virt_h - 2.0); + // If clamping collapsed the seed onto the target (target in a corner), + // nudge the other way so there is still a visible glide distance. + if (sx - target_x).abs() < 8.0 && (sy - target_y).abs() < 8.0 { + sx = (target_x + SEED_OFFSET).min(virt_x + virt_w - 2.0); + sy = (target_y + SEED_OFFSET).min(virt_y + virt_h - 2.0); + } + } + rs.core.pos = (sx, sy); + true +} + +/// Animate the overlay cursor for `key` to `(x, y)` and suspend until the +/// planned path completes (the spring-settle phase that follows is allowed to +/// keep running — we only wait for the visible glide to land). +/// +/// Returns immediately (no animation, no wait) when: +/// - the key is empty (anonymous run → no cursor), or +/// - the cursor for `key` is disabled. /// -/// Mirrors `platform_macos::cursor::overlay::animate_cursor_to`. Returns -/// immediately (no animation, no wait) when: -/// - the overlay is disabled, or -/// - the cursor is still at the off-screen sentinel `(-200, -200)` — in -/// that case the caller should rely on `ClickPulse` to snap the cursor. -pub async fn animate_cursor_to(x: f64, y: f64) { +/// A brand-new cursor still at the off-screen sentinel is first seeded +/// on-screen via [`seed_start_if_sentinel`] so its FIRST action glides in. +/// Mirrors `platform_macos::cursor::overlay::animate_cursor_to`. +pub async fn animate_cursor_to(key: CursorKey, x: f64, y: f64) { + if key.is_empty() { + return; + } + // Seed a sentinel cursor on-screen so the MoveTo below glides instead of + // being short-circuited. + seed_start_if_sentinel(&key, x, y); + let should_animate = { let guard = RENDER.lock().unwrap(); - match guard.as_ref() { - Some(rs) if rs.core.cfg.enabled && rs.core.visible && rs.core.pos.0 > -50.0 => true, + match guard.as_ref().and_then(|m| m.cursors.get(&key)) { + Some(rs) if rs.core.cfg.enabled && rs.core.pos.0 > -50.0 => true, _ => false, } }; @@ -113,27 +343,22 @@ pub async fn animate_cursor_to(x: f64, y: f64) { return; } - // Install the oneshot sender BEFORE issuing MoveTo, so the render + // Install the keyed oneshot sender BEFORE issuing MoveTo, so the render // thread's arrival-fire can never lose a race against an immediate // path-end (e.g. zero-length glide). let (tx, rx) = tokio::sync::oneshot::channel::<()>(); - { - let mut guard = ARRIVAL_TX.lock().unwrap(); - // A previous in-flight animation gets superseded by this one — - // unblock its waiter so it doesn't hang forever. - if let Some(old_tx) = guard.take() { - let _ = old_tx.send(()); - } - *guard = Some(tx); - } - - send_command(OverlayCommand::MoveTo { - x, - y, - // Arrive pointing upper-left (45°) — same convention as macOS / - // Swift reference (`endAngleDegrees: 45`). - end_heading_radians: std::f64::consts::FRAC_PI_4, - }); + arrival_register(key.clone(), tx); + + send_command( + key, + OverlayCommand::MoveTo { + x, + y, + // Arrive pointing upper-left (45°) — same convention as macOS / + // Swift reference (`endAngleDegrees: 45`). + end_heading_radians: std::f64::consts::FRAC_PI_4, + }, + ); let _ = rx.await; } @@ -149,7 +374,7 @@ pub fn run_on_thread() { let cfg = { let guard = RENDER.lock().unwrap(); match &*guard { - Some(rs) => rs.core.cfg.clone(), + Some(m) => m.template.clone(), None => return, } }; @@ -167,40 +392,26 @@ pub fn run_on_thread() { .expect("spawn overlay thread"); } -// ── Animation state ─────────────────────────────────────────────────────── +// ── Animation / render state ────────────────────────────────────────────── // // The platform-agnostic fields + tick + apply_command + render pipeline live -// in `cursor_overlay::render_state` (2026-05 dedup audit). What stays here -// is the Windows-specific virtual-screen geometry + last_tick stamp for the -// WM_TIMER dt calculation. +// in `cursor_overlay::render_state`. What stays here is just the per-cursor +// wrapper; the virtual-screen geometry + dt stamp moved up to `RenderMap`. struct RenderState { core: RenderStateCore, - /// Virtual screen dimensions set after window creation (Win32 DIPs). - /// `virt_x/y` are subtracted from `core.pos` when rendering so the - /// pixmap is laid out in window-local coordinates. - virt_x: i32, - virt_y: i32, - virt_w: i32, - virt_h: i32, - /// Last WM_TIMER wall-clock stamp; used to compute real `dt` (Windows - /// timer resolution defaults to 15ms so a hardcoded 8ms would run the - /// animation at half speed). - last_tick: Instant, } impl RenderState { fn new(cfg: CursorConfig) -> Self { RenderState { core: RenderStateCore::new(cfg), - last_tick: Instant::now(), - virt_x: 0, virt_y: 0, virt_w: 1920, virt_h: 1080, } } - /// Advance the motion state by `dt`. Returns `true` the tick the - /// planned path completes, so the WM_TIMER handler can fire the - /// arrival oneshot that unblocks `animate_cursor_to`. + /// Advance the motion state by `dt`. Returns `true` the tick the planned + /// path completes, so the WM_TIMER handler can fire the arrival oneshot + /// that unblocks `animate_cursor_to`. fn tick(&mut self, dt: f64) -> bool { self.core.tick_motion(dt) } @@ -217,17 +428,17 @@ impl RenderState { // ── Win32 message-loop thread ───────────────────────────────────────────── #[cfg(target_os = "windows")] -fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver) { - use windows::Win32::UI::WindowsAndMessaging::*; +fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver) { use windows::Win32::Media::timeBeginPeriod; use windows::Win32::System::LibraryLoader::GetModuleHandleW; + use windows::Win32::UI::WindowsAndMessaging::*; use windows::core::PCWSTR; // Raise multimedia timer resolution to 1ms so SetTimer can deliver // WM_TIMER messages at ~8ms intervals (default is ~15ms). - // Mirrors `_timerResolutionRaised = timeBeginPeriod(1) == 0` in the - // .NET reference (AgentCursorOverlay.cs). - unsafe { let _ = timeBeginPeriod(1); } + unsafe { + let _ = timeBeginPeriod(1); + } // Collect virtual screen bounds (all monitors). let virt_x = unsafe { GetSystemMetrics(SM_XVIRTUALSCREEN) }; @@ -235,26 +446,23 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver = "Cua.AgentCursorOverlay\0".encode_utf16().collect(); let title_w: Vec = format!("Cua.AgentCursorOverlay.{}\0", cfg.cursor_id) - .encode_utf16().collect(); + .encode_utf16() + .collect(); let hinstance = unsafe { GetModuleHandleW(PCWSTR::null()).unwrap_or_default() }; @@ -266,11 +474,13 @@ fn run_overlay_thread(cfg: CursorConfig, rx: std::sync::mpsc::Receiver) { +fn run_overlay_thread(_cfg: CursorConfig, _rx: std::sync::mpsc::Receiver) { // No-op on non-Windows targets (cross-compile guard). } // ── Win32 globals (only used on Windows) ───────────────────────────────── -static OVERLAY_HWND: std::sync::atomic::AtomicIsize = - std::sync::atomic::AtomicIsize::new(0); -static CMD_RX_WIN: Mutex>> = Mutex::new(None); -static LAST_ZTICK: std::sync::atomic::AtomicU64 = - std::sync::atomic::AtomicU64::new(0); +static OVERLAY_HWND: std::sync::atomic::AtomicIsize = std::sync::atomic::AtomicIsize::new(0); +static CMD_RX_WIN: Mutex>> = Mutex::new(None); +static LAST_ZTICK: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); static Z_ORDER: OnceLock = OnceLock::new(); // ── Window procedure ────────────────────────────────────────────────────── @@ -346,34 +564,67 @@ unsafe extern "system" fn wnd_proc( .unwrap_or_default() .as_millis() as u64; - // Drain commands and tick animation. Measure real dt from last - // tick — Windows timer resolution defaults to 15ms so the - // hardcoded 8ms was running the animation at half speed. - let (pixmap, fire_arrival) = { + // ── Drain commands, tick all cursors, composite one pixmap ─────── + // Measure real dt from last tick — Windows timer resolution defaults + // to 15ms so the hardcoded 8ms ran the animation at half speed. + let (pixmap, arrived, pinned_wid) = { let mut guard = RENDER.lock().unwrap(); - if let Some(rs) = guard.as_mut() { - // Drain the channel. + if let Some(map) = guard.as_mut() { + // Drain the channel via get-or-create; track the last-touched + // key so the z-order pin follows the most-recent cursor. if let Ok(rx_guard) = CMD_RX_WIN.try_lock() { if let Some(ref rx) = *rx_guard { - while let Ok(cmd) = rx.try_recv() { - rs.apply_command(cmd); + while let Ok(m) = rx.try_recv() { + if let Some(k) = apply_msg(map, m) { + map.last_active = Some(k); + } } } } - let now = std::time::Instant::now(); - let dt = now.duration_since(rs.last_tick).as_secs_f64().clamp(0.0, 0.05); - rs.last_tick = now; - let arrived = rs.tick(dt); - (Some(cursor_overlay::render_frame( - &rs.core, - rs.virt_w.max(1) as u32, - rs.virt_h.max(1) as u32, - rs.virt_x as f64, - rs.virt_y as f64, - None, // focus-rect is macOS-only - )), arrived) + + let now = Instant::now(); + let dt = now + .duration_since(map.last_tick) + .as_secs_f64() + .clamp(0.0, 0.05); + map.last_tick = now; + + // Tick every cursor; record the ones that just arrived. + let mut arrived: Vec = Vec::new(); + for (k, rs) in map.cursors.iter_mut() { + if rs.tick(dt) { + arrived.push(k.clone()); + } + } + + // Composite every cursor into ONE virtual-screen pixmap. + // tiny-skia fills are alpha-over, so insertion order = + // paint/z-order; idle/hidden cursors early-return inside + // paint_cursor so an idle session costs ~nothing. + let w = map.virt_w.max(1) as u32; + let h = map.virt_h.max(1) as u32; + let mut pm = tiny_skia::Pixmap::new(w.max(1), h.max(1)) + .unwrap_or_else(|| tiny_skia::Pixmap::new(1, 1).unwrap()); + for (_k, rs) in &map.cursors { + cursor_overlay::paint_cursor( + &mut pm, + &rs.core, + map.virt_x as f64, + map.virt_y as f64, + None, // focus-rect is macOS-only + ); + } + + // Pin above the most-recently-touched cursor's target. + let pinned = map + .last_active + .as_ref() + .and_then(|k| map.cursors.get(k)) + .and_then(|rs| rs.core.pinned_wid); + + (Some(pm), arrived, pinned) } else { - (None, false) + (None, Vec::new(), None) } }; @@ -381,15 +632,11 @@ unsafe extern "system" fn wnd_proc( update_layered_window(hwnd, &pm); } - // Fire arrival oneshot the tick the path just ended — unblocks - // any `animate_cursor_to(...).await` so the click action only - // dispatches once the cursor has visually landed. - if fire_arrival { - if let Ok(mut guard) = ARRIVAL_TX.lock() { - if let Some(tx) = guard.take() { - let _ = tx.send(()); - } - } + // Fire arrival oneshots for cursors whose path just ended — unblocks + // each session's `animate_cursor_to(...).await` so the click action + // only dispatches once that cursor has visually landed. + for k in &arrived { + arrival_fire(k); } // Z-order maintenance every 80ms — delegate to the cross-platform @@ -398,8 +645,6 @@ unsafe extern "system" fn wnd_proc( let last = LAST_ZTICK.load(std::sync::atomic::Ordering::Relaxed); if now_ms.wrapping_sub(last) >= 80 { LAST_ZTICK.store(now_ms, std::sync::atomic::Ordering::Relaxed); - let pinned_wid = RENDER.lock().ok() - .and_then(|g| g.as_ref().and_then(|rs| rs.core.pinned_wid)); if let Some(enforcer) = Z_ORDER.get() { enforcer.reassert(pinned_wid); } @@ -428,7 +673,9 @@ unsafe fn update_layered_window( let w = pixmap.width() as i32; let h = pixmap.height() as i32; - if w <= 0 || h <= 0 { return; } + if w <= 0 || h <= 0 { + return; + } let hdc_screen = GetDC(None); let hdc_mem = CreateCompatibleDC(hdc_screen); @@ -448,14 +695,7 @@ unsafe fn update_layered_window( }; let mut bits_ptr = std::ptr::null_mut::(); - let hbmp = CreateDIBSection( - hdc_mem, - &bmi, - DIB_RGB_COLORS, - &mut bits_ptr, - None, - 0, - ); + let hbmp = CreateDIBSection(hdc_mem, &bmi, DIB_RGB_COLORS, &mut bits_ptr, None, 0); if hbmp.is_err() || bits_ptr.is_null() { let _ = DeleteDC(hdc_mem); ReleaseDC(None, hdc_screen); @@ -473,7 +713,7 @@ unsafe fn update_layered_window( let b = src[i * 4 + 2]; let a = src[i * 4 + 3]; // Swap R <-> B for BGRA. - dst[i * 4] = b; + dst[i * 4] = b; dst[i * 4 + 1] = g; dst[i * 4 + 2] = r; dst[i * 4 + 3] = a; @@ -484,9 +724,9 @@ unsafe fn update_layered_window( let virt_y; { let guard = RENDER.lock().unwrap(); - if let Some(rs) = &*guard { - virt_x = rs.virt_x; - virt_y = rs.virt_y; + if let Some(map) = &*guard { + virt_x = map.virt_x; + virt_y = map.virt_y; } else { virt_x = 0; virt_y = 0; @@ -495,15 +735,24 @@ unsafe fn update_layered_window( let pt_src = POINT { x: 0, y: 0 }; let pt_dst = POINT { x: virt_x, y: virt_y }; - let sz = SIZE { cx: w, cy: h }; - let blend = BLENDFUNCTION { - BlendOp: 0, // AC_SRC_OVER - BlendFlags: 0, + let sz = SIZE { cx: w, cy: h }; + let blend = BLENDFUNCTION { + BlendOp: 0, // AC_SRC_OVER + BlendFlags: 0, SourceConstantAlpha: 255, - AlphaFormat: 1, // AC_SRC_ALPHA + AlphaFormat: 1, // AC_SRC_ALPHA }; - let _ = UpdateLayeredWindow(hwnd, hdc_screen, Some(&pt_dst), Some(&sz), - hdc_mem, Some(&pt_src), COLORREF(0), Some(&blend), ULW_ALPHA); + let _ = UpdateLayeredWindow( + hwnd, + hdc_screen, + Some(&pt_dst), + Some(&sz), + hdc_mem, + Some(&pt_src), + COLORREF(0), + Some(&blend), + ULW_ALPHA, + ); let _ = DeleteObject(hbmp); let _ = DeleteDC(hdc_mem); @@ -566,7 +815,10 @@ impl ZOrderEnforcer for WinZOrderEnforcer { let _ = SetWindowPos( hwnd, HWND_NOTOPMOST, - 0, 0, 0, 0, + 0, + 0, + 0, + 0, SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_NOOWNERZORDER, ); @@ -580,7 +832,10 @@ impl ZOrderEnforcer for WinZOrderEnforcer { let _ = SetWindowPos( hwnd, insert_after, - 0, 0, 0, 0, + 0, + 0, + 0, + 0, SWP_NOMOVE | SWP_NOSIZE | SWP_NOACTIVATE | SWP_SHOWWINDOW | SWP_NOOWNERZORDER, ); @@ -588,3 +843,172 @@ impl ZOrderEnforcer for WinZOrderEnforcer { } } } + +// ── Headless unit tests for the keyed render collection ─────────────────── +// +// These prove the per-session ownership data model, the session_end removal +// lifecycle, the "default" guard, the resurrection tombstone, and the +// sentinel seed WITHOUT any Win32 window. The on-screen rendering +// (UpdateLayeredWindow) still needs a real display and is verified separately. + +#[cfg(test)] +mod tests { + use super::*; + + fn empty_map() -> RenderMap { + let mut cursors = IndexMap::new(); + cursors.insert("default".to_owned(), RenderState::new(CursorConfig::default())); + RenderMap { + cursors, + virt_x: 0, + virt_y: 0, + virt_w: 100, + virt_h: 100, + last_tick: Instant::now(), + template: CursorConfig::default(), + ended: HashSet::new(), + last_active: None, + } + } + + fn move_msg(key: &str, x: f64, y: f64) -> OverlayMsg { + OverlayMsg::Cmd(KeyedOverlayCommand { + key: key.to_owned(), + cmd: OverlayCommand::MoveTo { x, y, end_heading_radians: 0.0 }, + }) + } + + #[test] + fn two_sessions_produce_two_distinct_render_entries() { + let mut map = empty_map(); + apply_msg(&mut map, move_msg("sessA", 10.0, 10.0)); + apply_msg(&mut map, move_msg("sessB", 42.0, 24.0)); + // default + sessA + sessB = 3 distinct owned cursors. The pre-port + // regression: a single RenderState would clobber these to one cursor. + assert_eq!(map.cursors.len(), 3); + assert!(map.cursors.contains_key("sessA")); + assert!(map.cursors.contains_key("sessB")); + assert!(map.cursors.contains_key("default")); + } + + #[test] + fn session_end_removes_only_that_session() { + let mut map = empty_map(); + apply_msg(&mut map, move_msg("sessA", 10.0, 10.0)); + apply_msg(&mut map, move_msg("sessB", 20.0, 20.0)); + assert_eq!(map.cursors.len(), 3); + + // session_end(A): A gone, B + default retained. + apply_msg(&mut map, OverlayMsg::Remove("sessA".to_owned())); + assert!(!map.cursors.contains_key("sessA")); + assert!(map.cursors.contains_key("sessB")); + assert!(map.cursors.contains_key("default")); + assert_eq!(map.cursors.len(), 2); + + // Remove("default") is guarded — default survives. + apply_msg(&mut map, OverlayMsg::Remove("default".to_owned())); + assert!(map.cursors.contains_key("default")); + + // Remove of an absent key is a harmless no-op. + let before = map.cursors.len(); + apply_msg(&mut map, OverlayMsg::Remove("never-existed".to_owned())); + assert_eq!(map.cursors.len(), before); + } + + #[test] + fn lazily_created_cursors_get_distinct_palettes() { + let mut map = empty_map(); + apply_msg(&mut map, move_msg("sessA", 10.0, 10.0)); + apply_msg(&mut map, move_msg("sessB", 20.0, 20.0)); + let a = &map.cursors["sessA"].core.palette; + let b = &map.cursors["sessB"].core.palette; + let def = &map.cursors["default"].core.palette; + assert_ne!(a.name, def.name); + assert_ne!(b.name, def.name); + } + + #[test] + fn insertion_order_is_stable_z_order() { + let mut map = empty_map(); + apply_msg(&mut map, move_msg("first", 1.0, 1.0)); + apply_msg(&mut map, move_msg("second", 2.0, 2.0)); + // Re-touching "first" must NOT move it to the back (IndexMap keeps the + // original slot), so z-order is stable frame to frame. + apply_msg(&mut map, move_msg("first", 3.0, 3.0)); + let keys: Vec<&String> = map.cursors.keys().collect(); + assert_eq!(keys, vec!["default", "first", "second"]); + } + + #[test] + fn tombstone_blocks_resurrection_after_remove() { + let mut map = empty_map(); + apply_msg(&mut map, move_msg("sessA", 10.0, 10.0)); + assert_eq!(map.cursors.len(), 2); // default + sessA + + apply_msg(&mut map, OverlayMsg::Remove("sessA".to_owned())); + assert!(!map.cursors.contains_key("sessA")); + assert_eq!(map.cursors.len(), 1); + + // A late in-flight Cmd for the ended session must be dropped WITHOUT + // re-inserting (no get-or-create resurrection). + let resolved = apply_msg(&mut map, move_msg("sessA", 99.0, 99.0)); + assert!(resolved.is_none(), "ended-session Cmd must be dropped, not resolved"); + assert!(!map.cursors.contains_key("sessA"), "tombstone must block resurrection"); + assert_eq!(map.cursors.len(), 1); + } + + #[test] + fn default_is_never_tombstoned() { + let mut map = empty_map(); + apply_msg(&mut map, OverlayMsg::Remove("default".to_owned())); + assert!(map.cursors.contains_key("default")); + assert!(!map.ended.contains("default")); + + let resolved = apply_msg(&mut map, move_msg("default", 5.0, 5.0)); + assert_eq!(resolved.as_deref(), Some("default")); + assert!(map.cursors.contains_key("default")); + } + + #[test] + fn seed_moves_sentinel_cursor_on_screen_for_first_action() { + let mut map = empty_map(); // 100x100 frame at origin + // No "sessA" cursor exists yet — the seed must get-or-create it. + let seeded = seed_start_in_map(&mut map, &"sessA".to_owned(), 60.0, 60.0); + assert!(seeded, "sentinel cursor must be seeded"); + let pos = map.cursors["sessA"].core.pos; + assert!(pos.0 > -50.0 && pos.1 > -50.0, "seed must be on-screen, got {pos:?}"); + assert!( + (pos.0 - 60.0).abs() > 4.0 || (pos.1 - 60.0).abs() > 4.0, + "seed must differ from target to produce a visible glide, got {pos:?}" + ); + } + + #[test] + fn seed_is_noop_when_cursor_already_on_screen() { + let mut map = empty_map(); + seed_start_in_map(&mut map, &"sessA".to_owned(), 60.0, 60.0); + map.cursors.get_mut("sessA").unwrap().core.pos = (30.0, 30.0); + let seeded_again = seed_start_in_map(&mut map, &"sessA".to_owned(), 80.0, 80.0); + assert!(!seeded_again, "on-screen cursor must not be re-seeded"); + assert_eq!(map.cursors["sessA"].core.pos, (30.0, 30.0), "pos must be untouched"); + } + + #[test] + fn seed_does_not_resurrect_ended_session() { + let mut map = empty_map(); + map.ended.insert("sessA".to_owned()); + let seeded = seed_start_in_map(&mut map, &"sessA".to_owned(), 60.0, 60.0); + assert!(!seeded, "ended session must not be seeded"); + assert!(!map.cursors.contains_key("sessA"), "ended session must not be resurrected"); + } + + #[test] + fn remove_clears_last_active_for_that_key() { + let mut map = empty_map(); + let k = apply_msg(&mut map, move_msg("sessA", 10.0, 10.0)); + map.last_active = k; + assert_eq!(map.last_active.as_deref(), Some("sessA")); + apply_msg(&mut map, OverlayMsg::Remove("sessA".to_owned())); + assert_eq!(map.last_active, None, "removing the active cursor must clear last_active"); + } +} diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs index 05a388e04d..0542127abc 100644 --- a/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs +++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs @@ -11,14 +11,14 @@ use async_trait::async_trait; /// the host appears in the z-order. `GA_ROOT` normalises both inputs to the /// host so the overlay sits at z+1 of whatever is actually painted on screen. /// -/// No-op when the overlay is disabled; the command is just dropped by the -/// render thread in that case. -fn pin_overlay_above(hwnd: u64) { +/// No-op when the overlay is disabled or `key` is empty (anonymous, cursor-less +/// run); the command is just dropped by the render thread in that case. +fn pin_overlay_above(key: &str, hwnd: u64) { use windows::Win32::Foundation::HWND; use windows::Win32::UI::WindowsAndMessaging::{GetAncestor, GA_ROOT}; let root = unsafe { GetAncestor(HWND(hwnd as *mut _), GA_ROOT) }; let wid = if !root.0.is_null() { root.0 as u64 } else { hwnd }; - crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(wid)); + crate::overlay::send_command(key.to_owned(), cursor_overlay::OverlayCommand::PinAbove(wid)); } /// Convert (px, py) — "window-local screenshot pixels, top-left origin @@ -83,15 +83,20 @@ fn bitmap_to_screen(hwnd: u64, px: i32, py: i32) -> (i32, i32) { /// render thread's arrival oneshot — that's how we keep the click action /// from firing before the cursor has visually landed (the old heuristic /// `tokio::sleep(80..600 ms)` was racing the spring-physics glide). -async fn overlay_glide_to(sx: f64, sy: f64) { - if !crate::overlay::is_enabled() { return; } - let pos = crate::overlay::current_position(); +/// +/// `key` is the session's cursor key (see [`resolve_cursor_key`]). An empty key +/// (anonymous, no declared session) is cursor-less: every overlay op +/// short-circuits, so the action runs with no visible cursor. +async fn overlay_glide_to(key: &str, sx: f64, sy: f64) { + if key.is_empty() { return; } + if !crate::overlay::is_enabled(key) { return; } + let pos = crate::overlay::current_position(key); if pos.0 < 0.0 && pos.1 < 0.0 { // Snap to target on first use; no animation to wait for. - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy }); + crate::overlay::send_command(key.to_owned(), cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy }); return; } - crate::overlay::animate_cursor_to(sx, sy).await; + crate::overlay::animate_cursor_to(key.to_owned(), sx, sy).await; } use cua_driver_core::{protocol::ToolResult, tool::{Tool, ToolDef, ToolRegistry}}; use serde_json::{json, Value}; @@ -101,6 +106,32 @@ use crate::uia::ElementCache; use cursor_overlay::CursorRegistry; use windows::core::Interface as _; +/// The cursor key for an anonymous (cursor-less) call. A run opts into a cursor +/// by declaring a `session`; without one, every cursor op short-circuits on +/// this empty key (see `overlay::send_command` / `overlay_glide_to`). +pub(crate) const NO_CURSOR: &str = ""; + +/// Resolve the cursor key for a tool invocation, or [`NO_CURSOR`] (`""`) for an +/// anonymous call. +/// +/// A cursor is tied to a **caller-declared session**, never to the MCP +/// connection. Precedence: an explicit `session` arg, then its legacy alias +/// `cursor_id`. We deliberately do NOT fall back to the connection-injected +/// `_session_id` or to a seeded `"default"` cursor — `""` means "no session +/// declared → no cursor", while the underlying action (click/type/…) still +/// executes. Mirrors `platform_macos::tools::cursor_tools::resolve_cursor_key` +/// so the two platforms key cursors identically. +pub(crate) fn resolve_cursor_key(args: &Value) -> String { + for key in ["session", "cursor_id"] { + if let Some(v) = args.get(key).and_then(|v| v.as_str()) { + if !v.is_empty() { + return v.to_owned(); + } + } + } + NO_CURSOR.to_owned() +} + // ── DriverConfig + ResizeRegistry + ZoomRegistry ───────────────────────────── #[derive(Clone)] @@ -1785,6 +1816,7 @@ impl Tool for ClickTool { use cua_driver_core::tool_args::ArgsExt; use crate::input::dispatch::{DispatchMode, EventKind, background_unavailable_error}; use crate::uia::cache::SnapshotKind; + let cursor_key = resolve_cursor_key(&args); let pid = match args.require_u32("pid") { Ok(v) => v, Err(e) => return e }; let hwnd_opt = args.opt_u64("window_id"); let elem_idx = args.opt_u64("element_index").map(|v| v as usize); @@ -1869,9 +1901,9 @@ impl Tool for ClickTool { )), } }; - pin_overlay_above(hwnd); - overlay_glide_to(tx as f64, ty as f64).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { + pin_overlay_above(&cursor_key, hwnd); + overlay_glide_to(&cursor_key, tx as f64, ty as f64).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: tx as f64, y: ty as f64, }); let btn_fg = button.clone(); @@ -1898,10 +1930,10 @@ impl Tool for ClickTool { None => return ToolResult::error(format!("Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first.")), }; // Step 2: pin overlay to target window, then animate to screen coords. - pin_overlay_above(hwnd); - overlay_glide_to(cx as f64, cy as f64).await; + pin_overlay_above(&cursor_key, hwnd); + overlay_glide_to(&cursor_key, cx as f64, cy as f64).await; // Step 3: click pulse + actual click. - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64, }); let btn = button.clone(); @@ -2067,9 +2099,9 @@ impl Tool for ClickTool { // mapping (DWM-frame top-left + 1-px inset, NOT ClientToScreen). let (sx_i, sy_i) = bitmap_to_screen(hwnd, px as i32, py as i32); let (sx, sy) = (sx_i as f64, sy_i as f64); - pin_overlay_above(hwnd); - overlay_glide_to(sx, sy).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy }); + pin_overlay_above(&cursor_key, hwnd); + overlay_glide_to(&cursor_key, sx, sy).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy }); let btn = button.clone(); // Vision-mode (x, y) dispatch is **layered**, mirroring the // trope-cua reference impl @@ -2307,6 +2339,7 @@ impl Tool for TypeTextTool { let raw_pid = match args.require_i64("pid") { Ok(v) => v, Err(e) => return e }; let pid = raw_pid as u32; let text_raw = match args.require_str("text") { Ok(v) => v, Err(e) => return e }; + let cursor_key = resolve_cursor_key(&args); // Strip trailing agent-protocol closing tags before delivery — // catches the case where an LLM hallucinated its own tool- // invocation tags into the text param (see text_sanitize docs). @@ -2364,7 +2397,7 @@ impl Tool for TypeTextTool { // Pin the agent-cursor overlay above the target window so the synthetic // cursor stays sandwiched at z+1 of the type target for the full // duration of the keystrokes (both XAML/UIA and PostMessage paths). - pin_overlay_above(hwnd); + pin_overlay_above(&cursor_key, hwnd); // Glide the agent cursor onto the field being typed into, so the viewer // can see *where* the agent is typing — same visual feedback as a click. @@ -2372,8 +2405,8 @@ impl Tool for TypeTextTool { // the focused-element path has no resolvable position to point at. if let Some(idx) = elem_idx { if let Some((cx, cy)) = self.state.element_cache.get_element_center(pid, hwnd, idx as usize) { - overlay_glide_to(cx as f64, cy as f64).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { + overlay_glide_to(&cursor_key, cx as f64, cy as f64).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64, }); } @@ -2825,6 +2858,7 @@ impl Tool for SetValueTool { async fn invoke(&self, args: Value) -> ToolResult { // Swift's "Missing required integer fields pid, window_id, and element_index." let mut missing_ints: Vec<&str> = Vec::new(); + let cursor_key = resolve_cursor_key(&args); let raw_pid = args.get("pid").and_then(|v| v.as_i64()); if raw_pid.is_none() { missing_ints.push("pid"); } if args.get("window_id").and_then(|v| v.as_u64()).is_none() { missing_ints.push("window_id"); } @@ -2847,9 +2881,9 @@ impl Tool for SetValueTool { // the viewer can see *where* the agent is acting. No-op when the // overlay is disabled or the element has no cached center. if let Some((cx, cy)) = self.state.element_cache.get_element_center(pid, hwnd, idx) { - pin_overlay_above(hwnd); - overlay_glide_to(cx as f64, cy as f64).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { + pin_overlay_above(&cursor_key, hwnd); + overlay_glide_to(&cursor_key, cx as f64, cy as f64).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64, }); } @@ -3154,6 +3188,7 @@ impl Tool for DoubleClickTool { let x = args.opt_f64("x"); let y = args.opt_f64("y"); let dispatch = DispatchMode::from_args(&args); + let cursor_key = resolve_cursor_key(&args); // Swift validates "both x and y or neither" and "no element_index without window_id". let has_xy = x.is_some() && y.is_some(); let partial_xy = x.is_some() != y.is_some(); @@ -3190,9 +3225,9 @@ impl Tool for DoubleClickTool { Some(v) => v, None => return ToolResult::error(format!("Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first.")), }; - pin_overlay_above(hwnd); - overlay_glide_to(cx as f64, cy as f64).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64 }); + pin_overlay_above(&cursor_key, hwnd); + overlay_glide_to(&cursor_key, cx as f64, cy as f64).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64 }); // dispatch:"background" — reject if PostMessage would be silently dropped. if dispatch == DispatchMode::Background && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick) @@ -3244,9 +3279,9 @@ impl Tool for DoubleClickTool { // `bitmap_to_screen` doc for why ClientToScreen is wrong). let (sx_i, sy_i) = bitmap_to_screen(hwnd, px as i32, py as i32); let (sx, sy) = (sx_i as f64, sy_i as f64); - pin_overlay_above(hwnd); - overlay_glide_to(sx, sy).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy }); + pin_overlay_above(&cursor_key, hwnd); + overlay_glide_to(&cursor_key, sx, sy).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy }); // dispatch:"background" — reject if PostMessage would be silently dropped. if dispatch == DispatchMode::Background && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick) @@ -3345,6 +3380,7 @@ impl Tool for RightClickTool { let x = args.opt_f64("x"); let y = args.opt_f64("y"); let dispatch = DispatchMode::from_args(&args); + let cursor_key = resolve_cursor_key(&args); // Port Swift's full validation set. let has_xy = x.is_some() && y.is_some(); let partial_xy = x.is_some() != y.is_some(); @@ -3381,9 +3417,9 @@ impl Tool for RightClickTool { Some(v) => v, None => return ToolResult::error(format!("Element {idx} not in cache for hwnd={hwnd}. Call get_window_state first.")), }; - pin_overlay_above(hwnd); - overlay_glide_to(cx as f64, cy as f64).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64 }); + pin_overlay_above(&cursor_key, hwnd); + overlay_glide_to(&cursor_key, cx as f64, cy as f64).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: cx as f64, y: cy as f64 }); if dispatch == DispatchMode::Background && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick) { @@ -3434,9 +3470,9 @@ impl Tool for RightClickTool { // `bitmap_to_screen` doc). let (sx_i, sy_i) = bitmap_to_screen(hwnd, px as i32, py as i32); let (sx, sy) = (sx_i as f64, sy_i as f64); - pin_overlay_above(hwnd); - overlay_glide_to(sx, sy).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy }); + pin_overlay_above(&cursor_key, hwnd); + overlay_glide_to(&cursor_key, sx, sy).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: sx, y: sy }); if dispatch == DispatchMode::Background && crate::input::dispatch::would_be_silently_dropped(hwnd, EventKind::MouseClick) { @@ -3520,6 +3556,7 @@ impl Tool for DragTool { let dispatch = DispatchMode::from_args(&args); use cua_driver_core::tool_args::ArgsExt; + let cursor_key = resolve_cursor_key(&args); // Accepts numeric JSON as either float or integer — coerce both to f64. let coerce = |key: &str| -> Option { args.opt_f64(key).or_else(|| args.opt_i64(key).map(|i| i as f64)) @@ -3614,7 +3651,7 @@ impl Tool for DragTool { // cursor stays sandwiched at z+1 of the dragged window for the full // path. Drag stays within a single HWND, so one pin at the start is // sufficient — the 80 ms z-order tick keeps it asserted thereafter. - pin_overlay_above(hwnd); + pin_overlay_above(&cursor_key, hwnd); // Animate the agent cursor to the drag-start, fire a press pulse, // run the actual drag synthesis, then glide to the drag-end and @@ -3623,8 +3660,8 @@ impl Tool for DragTool { // the drag itself (the timing coordination would be invasive); // pre- and post-glides plus the press/release pulses are enough // signal for a user watching the agent operate. - overlay_glide_to(sx_from as f64, sy_from as f64).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { + overlay_glide_to(&cursor_key, sx_from as f64, sy_from as f64).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: sx_from as f64, y: sy_from as f64, }); @@ -3642,8 +3679,8 @@ impl Tool for DragTool { // pulse the release. Skipped on error so the cursor doesn't lie // about a successful endpoint. if matches!(&result, Ok(Ok(()))) { - overlay_glide_to(sx_to as f64, sy_to as f64).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { + overlay_glide_to(&cursor_key, sx_to as f64, sy_to as f64).await; + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::ClickPulse { x: sx_to as f64, y: sy_to as f64, }); } @@ -3748,16 +3785,20 @@ impl Tool for MoveCursorTool { use cua_driver_core::tool_args::ArgsExt; let x = args.f64_or("x", 0.0); let y = args.f64_or("y", 0.0); - let cursor_id_owned = args.str_or("cursor_id", "default"); - let cursor_id = cursor_id_owned.as_str(); - self.state.cursor_registry.update_position(cursor_id, x, y); + // Cursor key precedence: caller-declared `session` > legacy `cursor_id` + // > NO_CURSOR. An anonymous run (no session) has no cursor to move. + let cursor_key = resolve_cursor_key(&args); + if !cursor_key.is_empty() { + self.state.cursor_registry.update_position(&cursor_key, x, y); + } // End pointing upper-left (45°) — matches Swift's // `AgentCursor.animateAndWait(endAngleDegrees: 45)` convention so // the cursor settles to the natural macOS-style pose. - crate::overlay::send_command(cursor_overlay::OverlayCommand::MoveTo { + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::MoveTo { x, y, end_heading_radians: std::f64::consts::FRAC_PI_4, }); - ToolResult::text(format!("Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1}).")) + let shown = if cursor_key.is_empty() { "default" } else { cursor_key.as_str() }; + ToolResult::text(format!("Agent cursor '{shown}' moved to ({x:.1}, {y:.1}).")) } } @@ -3797,11 +3838,11 @@ impl Tool for SetAgentCursorEnabledTool { Some(v) => v, None => return ToolResult::error("Missing required boolean field `enabled`."), }; - use cua_driver_core::tool_args::ArgsExt; - let cursor_id_owned = args.str_or("cursor_id", "default"); - let cursor_id = cursor_id_owned.as_str(); - self.state.cursor_registry.set_enabled(cursor_id, enabled); - crate::overlay::send_command(cursor_overlay::OverlayCommand::SetEnabled(enabled)); + let cursor_key = resolve_cursor_key(&args); + if !cursor_key.is_empty() { + self.state.cursor_registry.set_enabled(&cursor_key, enabled); + } + crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::SetEnabled(enabled)); // Match Swift text format 1:1: `"✅ Agent cursor enabled."` // (or `"✅ Agent cursor disabled."`). ToolResult::text(if enabled { @@ -3867,7 +3908,8 @@ impl Tool for SetAgentCursorMotionTool { fn num(v: Option<&Value>) -> Option { v.and_then(|x| x.as_f64().or_else(|| x.as_i64().map(|i| i as f64))) } - let cursor_id = args.get("cursor_id").and_then(|v| v.as_str()).unwrap_or("default").to_owned(); + // Cursor key: caller-declared `session` > legacy `cursor_id` > NO_CURSOR. + let cursor_id = resolve_cursor_key(&args); // 1. Per-instance appearance fields (Rust-only). self.state.cursor_registry.update_config(&cursor_id, |cfg| { if let Some(v) = args.get("cursor_icon").and_then(|v| v.as_str()) { cfg.cursor_icon = Some(v.to_owned()); } @@ -3878,7 +3920,7 @@ impl Tool for SetAgentCursorMotionTool { }); // 2. Apply motion knobs to the live render state — was silently // dropped before; this is the Swift parity behavior. - let current = crate::overlay::current_motion(); + let current = crate::overlay::current_motion(&cursor_id); let updated = current.with_overrides( num(args.get("start_handle")), num(args.get("end_handle")), @@ -3890,7 +3932,7 @@ impl Tool for SetAgentCursorMotionTool { num(args.get("idle_hide_ms")), None, // press_duration_ms — not in Swift tool surface ); - crate::overlay::send_command(cursor_overlay::OverlayCommand::SetMotion(updated.clone())); + crate::overlay::send_command(cursor_id.clone(), cursor_overlay::OverlayCommand::SetMotion(updated.clone())); // Match Swift text format 1:1. let summary = format!( "cursor motion: startHandle={sh} endHandle={eh} arcSize={asz} arcFlow={af} \ @@ -3938,9 +3980,12 @@ impl Tool for GetAgentCursorStateTool { read_only: true, destructive: false, idempotent: true, open_world: false, }) } - async fn invoke(&self, _args: Value) -> ToolResult { - let enabled = crate::overlay::is_enabled(); - let motion = crate::overlay::current_motion(); + async fn invoke(&self, args: Value) -> ToolResult { + // Report THIS session's cursor (caller-declared `session` > `cursor_id` + // > "default"), mirroring macOS get_agent_cursor_state scoping. + let cursor_key = resolve_cursor_key(&args); + let enabled = crate::overlay::is_enabled(&cursor_key); + let motion = crate::overlay::current_motion(&cursor_key); // Swift text format 1:1: single-line camelCase key=value pairs. let summary = format!( "cursor: enabled={enabled} startHandle={sh} endHandle={eh} arcSize={asz} \ @@ -4025,7 +4070,8 @@ impl Tool for SetAgentCursorStyleTool { } async fn invoke(&self, args: Value) -> ToolResult { - let cursor_id = args.get("cursor_id").and_then(|v| v.as_str()).unwrap_or("default").to_owned(); + // Cursor key: caller-declared `session` > legacy `cursor_id` > NO_CURSOR. + let cursor_id = resolve_cursor_key(&args); // image_path let image_path = args.get("image_path").and_then(|v| v.as_str()); @@ -4084,12 +4130,12 @@ impl Tool for SetAgentCursorStyleTool { // Dispatch to overlay if let Some(cmd) = shape_cmd { - crate::overlay::send_command(cmd); + crate::overlay::send_command(cursor_id.clone(), cmd); } let gradient_provided = args.get("gradient_colors").is_some(); let bloom_provided = args.get("bloom_color").is_some(); if gradient_provided || bloom_provided { - crate::overlay::send_command(cursor_overlay::OverlayCommand::SetGradient { + crate::overlay::send_command(cursor_id.clone(), cursor_overlay::OverlayCommand::SetGradient { gradient_colors, bloom_color: bloom_color.flatten(), }); @@ -4322,15 +4368,15 @@ impl Tool for GetConfigTool { read_only: true, destructive: false, idempotent: true, open_world: false, }) } - async fn invoke(&self, _args: Value) -> ToolResult { + async fn invoke(&self, args: Value) -> ToolResult { let cfg = self.state.config.read().unwrap(); // Mirror the macOS agent's parity addition (commit adb9ecca): // nested `agent_cursor.enabled` block so Swift-shaped get_config // consumers can read the cursor's enabled state from one place. - let cursor_enabled = self.state.cursor_registry.all_states() - .first() - .map(|s| s.config.enabled) - .unwrap_or(true); + // Scope to the CALLING session's cursor (session > cursor_id > default) + // and read it from the overlay deterministically — `all_states().first()` + // was a nondeterministic HashMap read across sessions (macOS BUG 3). + let cursor_enabled = crate::overlay::is_enabled(&resolve_cursor_key(&args)); let (pip_enabled, pip_geometry) = pip_preview::read_pip_keys_from_file(); let payload = json!({ "schema_version": 1, @@ -5195,6 +5241,26 @@ pub fn build_registry(compat: bool) -> ToolRegistry { // Share the element cache with the recording-hook layer so it can // resolve element_index → window-local screenshot coords for click.png. crate::recording_hooks::set_element_cache(state.element_cache.clone()); + + // Drop a session's owned cursor on `session_end` (explicit end_session, the + // CLI `session end` verb, or the daemon idle-TTL sweep). The session id IS + // the cursor key (caller-declared `session`), so this prunes the metadata + // registry AND stops the overlay painting that session's cursor. Both paths + // guard "default" so the anonymous / one-shot cursor survives. Registering + // once per process (build_registry runs once in the daemon) is guarded so a + // repeated build in tests can't accumulate duplicate hooks. Mirrors the + // macOS `register_all` session_end hook (platform-macos/src/tools/mod.rs). + { + static HOOK_ONCE: std::sync::OnceLock<()> = std::sync::OnceLock::new(); + if HOOK_ONCE.set(()).is_ok() { + let cursor_registry = state.cursor_registry.clone(); + cua_driver_core::session::register_session_end_hook(move |session_id| { + cursor_registry.remove(session_id); + crate::overlay::remove_cursor(session_id.to_owned()); + }); + } + } + let mut r = ToolRegistry::new(); r.register(Box::new(ListAppsTool)); r.register(Box::new(ListWindowsTool)); @@ -5254,9 +5320,56 @@ pub fn build_registry(compat: bool) -> ToolRegistry { std::sync::Arc::new(super::page::WindowsPageBackend::new()), ))); r.register_recording_tools(); + r.register_session_tools(); r } +#[cfg(test)] +mod cursor_key_resolution_tests { + use super::{resolve_cursor_key, NO_CURSOR}; + use serde_json::json; + + #[test] + fn anonymous_resolves_to_no_cursor() { + // No session/cursor_id → NO_CURSOR (""): the action still runs but no + // cursor is shown. The connection-injected `_session_id` is NOT a cursor + // source — it stays the recording/config lifecycle key. + assert_eq!(resolve_cursor_key(&json!({})), NO_CURSOR); + assert_eq!(resolve_cursor_key(&json!({ "pid": 1 })), NO_CURSOR); + assert_eq!(resolve_cursor_key(&json!({ "_session_id": "mcp-1-2" })), NO_CURSOR); + } + + #[test] + fn explicit_session_owns_a_cursor() { + assert_eq!(resolve_cursor_key(&json!({ "session": "research-run" })), "research-run"); + } + + #[test] + fn cursor_id_is_a_legacy_alias_and_session_wins() { + assert_eq!(resolve_cursor_key(&json!({ "cursor_id": "user-handle" })), "user-handle"); + assert_eq!(resolve_cursor_key(&json!({ "session": "s1", "cursor_id": "c1" })), "s1"); + } + + #[test] + fn empty_strings_fall_through_to_no_cursor() { + // An empty `session` falls through to `cursor_id`; both empty → NO_CURSOR. + assert_eq!(resolve_cursor_key(&json!({ "session": "", "cursor_id": "c1" })), "c1"); + assert_eq!(resolve_cursor_key(&json!({ "session": "", "cursor_id": "" })), NO_CURSOR); + } + + #[test] + fn two_parallel_sessions_resolve_distinct_keys() { + // The regression this whole port fixes: two concurrent runs each declare + // their own `session`, so they resolve DISTINCT cursor keys and own + // separate overlay cursors instead of clobbering one shared cursor. + let a = resolve_cursor_key(&json!({ "pid": 10, "element_index": 1, "session": "calc-2plus1" })); + let b = resolve_cursor_key(&json!({ "pid": 20, "element_index": 1, "session": "calc-5plus6" })); + assert_eq!(a, "calc-2plus1"); + assert_eq!(b, "calc-5plus6"); + assert_ne!(a, b); + } +} + #[cfg(test)] mod launch_focus_restore_decision_tests { use super::{should_restore_foreground_after_launch, LaunchTargetShape}; diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs index 5557ceb53d..8d25a7c514 100644 --- a/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs +++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/page.rs @@ -228,10 +228,14 @@ impl PageBackend for WindowsPageBackend { use windows::Win32::UI::WindowsAndMessaging::{GetAncestor, GA_ROOT}; let root = unsafe { GetAncestor(HWND(hwnd as *mut _), GA_ROOT) }; let pin_wid = if !root.0.is_null() { root.0 as u64 } else { hwnd }; - crate::overlay::send_command(cursor_overlay::OverlayCommand::PinAbove(pin_wid)); + crate::overlay::send_command_default(cursor_overlay::OverlayCommand::PinAbove(pin_wid)); } - crate::overlay::animate_cursor_to(screen_x, screen_y).await; - crate::overlay::send_command(cursor_overlay::OverlayCommand::ClickPulse { + // The cross-platform `PageBackend::click_element` trait carries no + // caller `session`, so this drives the seeded `"default"` cursor rather + // than a per-session one. Threading session through the trait is a + // separate cross-platform change (tracked as a follow-up). + crate::overlay::animate_cursor_to("default".to_owned(), screen_x, screen_y).await; + crate::overlay::send_command_default(cursor_overlay::OverlayCommand::ClickPulse { x: screen_x, y: screen_y, }); diff --git a/libs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs b/libs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs index 465abebe6e..a7eb705411 100644 --- a/libs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs +++ b/libs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs @@ -164,7 +164,7 @@ mod move_cursor_m { let x = args.get("x").and_then(|v| v.as_f64()).unwrap_or(0.0); let y = args.get("y").and_then(|v| v.as_f64()).unwrap_or(0.0); let cursor_id = args.get("cursor_id").and_then(|v| v.as_str()).unwrap_or("default"); - crate::overlay::send_command(cursor_overlay::OverlayCommand::MoveTo { x, y, end_heading_radians: 0.0 }); + crate::overlay::send_command(cursor_id.to_owned(), cursor_overlay::OverlayCommand::MoveTo { x, y, end_heading_radians: 0.0 }); ToolResult::text(format!("Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1}).")) } } @@ -191,7 +191,7 @@ mod set_enabled_m { async fn invoke(&self, args: Value) -> ToolResult { let enabled = args.get("enabled").and_then(|v| v.as_bool()).unwrap_or(true); let cursor_id = args.get("cursor_id").and_then(|v| v.as_str()).unwrap_or("default"); - crate::overlay::send_command(cursor_overlay::OverlayCommand::SetEnabled(enabled)); + crate::overlay::send_command(cursor_id.to_owned(), cursor_overlay::OverlayCommand::SetEnabled(enabled)); ToolResult::text(format!("Agent cursor '{}' {}.", cursor_id, if enabled { "enabled" } else { "disabled" })) } } @@ -228,5 +228,6 @@ pub fn build_registry() -> cua_driver_core::tool::ToolRegistry { r.register(Box::new(ZoomTool)); r.register(Box::new(TypeTextCharsTool)); r.register_recording_tools(); + r.register_session_tools(); r } diff --git a/libs/cua-driver/scripts/_install-local-rust.sh b/libs/cua-driver/scripts/_install-local-rust.sh index a0c523b427..6e679cf87f 100755 --- a/libs/cua-driver/scripts/_install-local-rust.sh +++ b/libs/cua-driver/scripts/_install-local-rust.sh @@ -260,6 +260,53 @@ ln -sfn "$VERSIONED_DIR" "$CURRENT_LINK" echo "${GREEN}current -> $VERSIONED_DIR${NORMAL}" echo "" +# --- macOS: stable local code-signing identity (so TCC grants survive rebuilds) --- +# +# Ad-hoc signing (`codesign --sign -`) keys the TCC grant +# (Accessibility / Screen Recording) on the binary's *cdhash*, which changes +# on EVERY rebuild — so the grant silently invalidates on each install-local +# and the daemon re-prompts ("I already granted!"). Signing with a certificate +# keys the grant on the cert identity instead, which is stable across rebuilds. +# We create a self-signed code-signing cert once (idempotent, in the login +# keychain) and reuse it. Local dev only; releases are CI-signed. +# +# Echoes the `codesign --sign` argument: the cert CN when available, or "-" +# (ad-hoc) when it can't be created — no codesign/openssl, CI, locked keychain. +CUA_LOCAL_SIGN_CN="CuaDriver Local Signing (cua-driver-rs)" +ensure_local_signing_identity() { + { [ "$OS" = "Darwin" ] && command -v codesign >/dev/null 2>&1; } || { printf -- '-'; return; } + # Reuse if already present (find-certificate finds it regardless of trust; + # `find-identity -v` would miss an untrusted self-signed cert). + if security find-certificate -c "$CUA_LOCAL_SIGN_CN" >/dev/null 2>&1; then + printf '%s' "$CUA_LOCAL_SIGN_CN"; return + fi + command -v openssl >/dev/null 2>&1 || { printf -- '-'; return; } + local kc="$HOME/Library/Keychains/login.keychain-db" + [ -f "$kc" ] || kc="$HOME/Library/Keychains/login.keychain" + [ -f "$kc" ] || { printf -- '-'; return; } + local tmp; tmp="$(mktemp -d)" || { printf -- '-'; return; } + printf '[req]\ndistinguished_name=dn\nx509_extensions=ext\nprompt=no\n[dn]\nCN=%s\n[ext]\nbasicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=critical,codeSigning\n' \ + "$CUA_LOCAL_SIGN_CN" > "$tmp/req.cnf" + # Transient password for the p12 handoff (deleted right after import). + # Apple's `security` rejects the empty-password p12 that openssl 3.x emits + # by default ("MAC verification failed"), so use a real password + `-legacy` + # PBE (Apple-compatible). Fall back to non-legacy for LibreSSL/older openssl + # which lacks `-legacy` but already writes a compatible p12. + local pw="cua-local-$$" + if openssl req -x509 -newkey rsa:2048 -keyout "$tmp/key.pem" -out "$tmp/cert.pem" \ + -days 3650 -nodes -config "$tmp/req.cnf" >/dev/null 2>&1 \ + && { openssl pkcs12 -export -legacy -inkey "$tmp/key.pem" -in "$tmp/cert.pem" \ + -out "$tmp/id.p12" -passout pass:"$pw" -name "$CUA_LOCAL_SIGN_CN" >/dev/null 2>&1 \ + || openssl pkcs12 -export -inkey "$tmp/key.pem" -in "$tmp/cert.pem" \ + -out "$tmp/id.p12" -passout pass:"$pw" -name "$CUA_LOCAL_SIGN_CN" >/dev/null 2>&1; } \ + && security import "$tmp/id.p12" -k "$kc" -P "$pw" -A -T /usr/bin/codesign >/dev/null 2>&1; then + rm -rf "$tmp" + printf '%s' "$CUA_LOCAL_SIGN_CN"; return + fi + rm -rf "$tmp" + printf -- '-' +} + # --- macOS: wrap the binary in CuaDriver.app for a stable TCC identity --- # # TCC keys Accessibility / Screen-Recording grants on the bundle @@ -296,14 +343,63 @@ if [ "$OS" = "Darwin" ]; then # as install.sh). Replace any prior bundle wholesale. rm -rf "$APP_DEST" ditto "$APP_STAGE" "$APP_DEST" - # Ad-hoc re-sign the whole bundle (--deep covers the inner binary). - # Required on macOS 26+ where Taskgated rejects a copied binary's - # stale signature, and gives the bundle a consistent cdhash for TCC. + # Re-sign the whole bundle (--deep covers the inner binary). Required on + # macOS 26+ where Taskgated rejects a copied binary's stale signature. + # Prefer the STABLE self-signed identity so TCC grants survive rebuilds; + # fall back to ad-hoc (which works but resets grants on the next rebuild). if command -v codesign >/dev/null 2>&1; then - codesign --force --deep --sign - "$APP_DEST" 2>/dev/null \ - || echo "${YELLOW}warning: codesign of $APP_DEST failed; first run may hit a Gatekeeper/Taskgated prompt${NORMAL}" >&2 + SIGN_ID="$(ensure_local_signing_identity)" + if [ "$SIGN_ID" != "-" ] \ + && codesign --force --deep --sign "$SIGN_ID" "$APP_DEST" 2>/dev/null; then + echo "${GREEN}signed $APP_DEST with a stable local identity — TCC grants survive future install-local rebuilds${NORMAL}" + elif codesign --force --deep --sign - "$APP_DEST" 2>/dev/null; then + if [ "$SIGN_ID" != "-" ]; then + echo "${YELLOW}note: stable-identity signing failed; signed ad-hoc instead (Accessibility/Screen Recording will reset on the next rebuild)${NORMAL}" >&2 + fi + else + echo "${YELLOW}warning: codesign of $APP_DEST failed; first run may hit a Gatekeeper/Taskgated prompt${NORMAL}" >&2 + fi fi echo "${GREEN}installed $APP_DEST${NORMAL}" + + # --- Clear a TCC grant pinned to a PREVIOUS signing identity ----------- + # + # TCC pins each Accessibility / Screen-Recording grant to the app's + # designated requirement AT GRANT TIME. A user who granted while the app + # was ad-hoc signed has a grant whose csreq is a bare `cdhash H"..."` + # (changes every rebuild); a user who granted under a different cert has + # one pinned to that leaf. After we re-sign with the stable cert above, + # that old row survives with auth_value=allowed but a csreq that no longer + # matches THIS build — so the daemon reads "not granted" while System + # Settings still shows the toggle ON. That's a dead end: re-toggling + # doesn't help because the row already records a decision, so the grant + # prompt never re-fires. Detect a signing-identity change vs the last + # install and `tccutil reset` once, so the next `permissions grant` + # prompts cleanly and re-pins to the current (stable cert) identity — + # after which cert-pinned grants survive all future rebuilds. + # + # `tccutil reset` needs no sudo / Full Disk Access, and is a no-op when + # nothing was granted. We only reset when moving TO a cert identity (the + # case a clean re-grant durably fixes); an ad-hoc build churns its cdhash + # every rebuild regardless, so resetting it would just add friction. + if command -v tccutil >/dev/null 2>&1; then + IDENTITY_MARKER="$HOME_DIR/.tcc-signing-identity" + NEW_IDENTITY="$(codesign -d -r- "$APP_DEST" 2>&1 \ + | sed -n 's/.*certificate leaf = H"\([0-9a-fA-F]*\)".*/cert:\1/p' | head -1)" + [ -n "$NEW_IDENTITY" ] || NEW_IDENTITY="adhoc" + OLD_IDENTITY="$(cat "$IDENTITY_MARKER" 2>/dev/null || true)" + case "$NEW_IDENTITY" in + cert:*) + if [ "$NEW_IDENTITY" != "$OLD_IDENTITY" ]; then + tccutil reset Accessibility com.trycua.driver >/dev/null 2>&1 || true + tccutil reset ScreenCapture com.trycua.driver >/dev/null 2>&1 || true + echo "${BOLD}cleared any stale Accessibility / Screen-Recording grant pinned to a previous build.${NORMAL}" + echo " Grant once more (System Settings → Privacy & Security) and it will${BOLD} stick across every future rebuild${NORMAL} — the grant now pins to a stable signing certificate, not the per-build cdhash." + fi + ;; + esac + printf '%s\n' "$NEW_IDENTITY" > "$IDENTITY_MARKER" 2>/dev/null || true + fi fi # --- Visible-bin symlink ------------------------------------------------ diff --git a/libs/cua-driver/scripts/_install-rust.sh b/libs/cua-driver/scripts/_install-rust.sh index a55076db6b..1696afd9b9 100644 --- a/libs/cua-driver/scripts/_install-rust.sh +++ b/libs/cua-driver/scripts/_install-rust.sh @@ -28,9 +28,14 @@ # binary location # CUA_DRIVER_RS_BIN_DIR=PATH legacy alias for INSTALL_DIR # CUA_DRIVER_RS_HOME=PATH package home for versioned installs -# (default ~/.cua-driver-rs). Holds +# (default ~/.cua-driver). Holds # packages/releases/-/ and # packages/current/ on Linux/Windows. +# Renamed from ~/.cua-driver-rs in +# v0.2.16 / PR #1644 — this release +# installer was missed in that rename +# and is reconciled here; a stale +# ~/.cua-driver-rs is swept post-install. # CUA_DRIVER_RS_NO_MODIFY_PATH=1 same as --no-modify-path # CUA_DRIVER_RS_KEEP_VERSIONS=N keep the N most recent per-version # release dirs after install; older @@ -110,7 +115,17 @@ TAG_PREFIX="cua-driver-rs-v" # CUA_DRIVER_RS_INSTALL_DIR is the documented name; CUA_DRIVER_RS_BIN_DIR is # the legacy alias kept for users with the old env in their shell rc. BIN_DIR="${CUA_DRIVER_RS_INSTALL_DIR:-${CUA_DRIVER_RS_BIN_DIR:-$HOME/.local/bin}}" -HOME_DIR="${CUA_DRIVER_RS_HOME:-$HOME/.cua-driver-rs}" +# Canonical home is ~/.cua-driver (renamed from ~/.cua-driver-rs in v0.2.16 / +# PR #1644). The local installer (_install-local-rust.sh) and the runtime +# already default here; this release installer was missed in that rename and +# kept writing to the legacy ~/.cua-driver-rs, which is the root cause of the +# install collision (release wrote one home, install-local + runtime used the +# other). Reconcile the default here, keep accepting the CUA_DRIVER_RS_HOME +# override for back-compat, and sweep the stale legacy dir post-install below. +HOME_DIR="${CUA_DRIVER_RS_HOME:-$HOME/.cua-driver}" +# Pre-v0.2.16 home this installer used to write to. Swept after the new +# install is staged so a single rooted home (~/.cua-driver) is left behind. +LEGACY_HOME_DIR="$HOME/.cua-driver-rs" NO_MODIFY_PATH="${CUA_DRIVER_RS_NO_MODIFY_PATH:-0}" # Post-install GC: how many per-version release dirs to retain. Validated # below as a non-negative integer; 0 means "never GC". The dir that @@ -352,6 +367,84 @@ prune_old_releases() { printf '%s\0' "${to_prune[@]}" | xargs -0 rm -rf } +# --- Clean up a pre-existing LOCAL (install-local) install -------------- +# +# `install-local.sh` (`_install-local-rust.sh`) installs a dev build into the +# SAME canonical home this release installer now writes to (~/.cua-driver, see +# the HOME_DIR reconciliation above), under a `*-local-*` versioned release dir +# (VERSION_TAG="0.0.0-local-"). On macOS it also cert-signs the shared +# /Applications/CuaDriver.app with a self-signed identity recorded at +# `~/.cua-driver/.tcc-signing-identity`. +# +# A user who ran install-local and then runs this release installer would +# otherwise end up with the local artifacts lingering alongside the fresh +# release: the `*-local-*` release dir(s) sit in `packages/releases/` (the +# release `current` swap re-points away from them, but they're never removed +# explicitly here), and the stale `.tcc-signing-identity` marker survives even +# though the release bundle is CI-signed, not locally cert-signed. Follow the +# same logic install-local / uninstall.sh use: stop the daemon, then remove +# ONLY the unambiguously-local artifacts so the release install is the single +# authoritative one. +# +# Conservative by construction: we only ever remove `*-local-*` release dirs +# and the local signing-identity marker — never a real release dir, never the +# `current` symlink (the release branch owns that), never unrelated user state +# under the home. Every step is best-effort + idempotent; a machine with no +# prior local install is a clean no-op. +# +# TCC is preserved deliberately: we do NOT `tccutil reset` here. The bundle at +# /Applications/CuaDriver.app is shared (bundle id com.trycua.driver) and the +# subsequent release `ditto` re-points the binary in place; grants keyed on the +# bundle id survive (macOS may re-prompt once on the cdhash change, same as any +# upgrade). Churning the signing identity would gratuitously invalidate +# cert-pinned grants, so we leave it alone. +cleanup_prior_local_install() { + local releases_dir="$HOME_DIR/packages/releases" + local tcc_marker="$HOME_DIR/.tcc-signing-identity" + + # Collect the local-build release dirs (the unambiguous install-local + # signature — a release install never creates a `*-local-*` dir). + local local_dirs=() + local d + if [[ -d "$releases_dir" ]]; then + for d in "$releases_dir"/*-local-*/; do + [[ -d "$d" ]] && local_dirs+=("${d%/}") + done + fi + + # Nothing local on disk → clean no-op (no marker, no local dirs). + if [[ ${#local_dirs[@]} -eq 0 && ! -f "$tcc_marker" ]]; then + return 0 + fi + + log "detected a prior install-local build under $HOME_DIR — cleaning it up so this release install is authoritative" + + # Stop the local daemon BEFORE we yank its binary out from under it, + # mirroring the post-swap stop both installers already do. Best-effort. + stop_cua_driver_daemons + + # Remove the `*-local-*` release dirs. The release install stages into its + # own `-` dir and swaps `current` to it, so deleting the + # local dirs can't strand the active install. If `current` somehow still + # points into a local dir (e.g. a partial prior run), the release branch + # below re-creates `current` immediately after, so a transient dangling + # link is harmless. + if [[ ${#local_dirs[@]} -gt 0 ]]; then + for d in "${local_dirs[@]}"; do + rm -rf "$d" 2>/dev/null || true + log " removed local build dir ${d##*/}" + done + fi + + # Remove the local signing-identity marker — it describes the locally + # cert-signed bundle, which the release `ditto` is about to replace with + # the CI-signed one. Leaving it would misreport the bundle's identity. + if [[ -f "$tcc_marker" ]]; then + rm -f "$tcc_marker" 2>/dev/null || true + log " removed local signing-identity marker $tcc_marker" + fi +} + # --- Resolve OS/arch ---------------------------------------------------- OS=$(uname -s) @@ -419,7 +512,7 @@ done # the baked line hasn't been updated yet (dev / pre-release checkouts). # # ~~~ BAKED_VERSION: auto-updated by CD workflow after each release — do not edit ~~~ -CUA_DRIVER_RS_BAKED_VERSION="0.4.1" +CUA_DRIVER_RS_BAKED_VERSION="0.5.1" # ~~~ END_BAKED_VERSION ~~~ if [[ -n "${CUA_DRIVER_RS_VERSION:-}" ]]; then @@ -506,6 +599,11 @@ fi # --- Install ------------------------------------------------------------ +# Before staging the new release, sweep any prior install-local build that +# shares this home so the release install ends up authoritative (see the +# function definition above for the conservative marker-gated logic). +cleanup_prior_local_install + mkdir -p "$BIN_DIR" # macOS: install the .app to /Applications first, then symlink the @@ -641,6 +739,23 @@ else prune_old_releases "$RELEASES_DIR" "$CURRENT_LINK" "$TARGET" "$KEEP_VERSIONS" fi +# --- Sweep the legacy ~/.cua-driver-rs home ----------------------------- +# +# This release installer used to default HOME_DIR to ~/.cua-driver-rs (the +# pre-v0.2.16 name). Now that it writes to ~/.cua-driver like install-local +# and the runtime, a prior RELEASE install can have left a stale +# ~/.cua-driver-rs behind — the source of the two-homes collision this PR +# fixes. Sweep it now that the new install is fully staged under the canonical +# home, mirroring the same belt-and-braces sweep _install-local-rust.sh does. +# Runs AFTER staging so we never delete state before the replacement exists; +# skipped when the user pinned CUA_DRIVER_RS_HOME to the legacy path on +# purpose. Best-effort + idempotent. +if [[ -d "$LEGACY_HOME_DIR" && "$HOME_DIR" != "$LEGACY_HOME_DIR" ]]; then + rm -rf "$LEGACY_HOME_DIR" 2>/dev/null \ + && log "swept legacy package home $LEGACY_HOME_DIR (reconciled onto $HOME_DIR)" \ + || log "note: could not fully remove legacy package home $LEGACY_HOME_DIR (best-effort)" +fi + # --- Stop any pre-swap cua-driver daemons ------------------------------- # # Mirror of install.ps1's `Stop-CuaDriverDaemons` call sequence. The @@ -663,7 +778,7 @@ show_cua_driver_daemon_survivors # --- Fire the one-shot install telemetry ping --------------------------- # # Anonymous adoption signal — sends `cua_driver_install` to PostHog -# exactly once per install (guarded by ~/.cua-driver-rs/.installation_recorded +# exactly once per install (guarded by ~/.cua-driver/.installation_recorded # on the binary side). The Rust port keeps its install signal independent # of the Swift `cua-driver` install (separate marker dir + separate env var) # so users can opt out of one without affecting the other. diff --git a/libs/cua-driver/scripts/install.ps1 b/libs/cua-driver/scripts/install.ps1 index db4e3e0a7b..943117332e 100644 --- a/libs/cua-driver/scripts/install.ps1 +++ b/libs/cua-driver/scripts/install.ps1 @@ -114,7 +114,7 @@ $BinaryName = "cua-driver.exe" # where the baked line hasn't been updated yet. # # ~~~ BAKED_VERSION: auto-updated by CD workflow after each release — do not edit ~~~ -$Script:CuaDriverRsBakedVersion = "0.4.1" +$Script:CuaDriverRsBakedVersion = "0.5.1" # ~~~ END_BAKED_VERSION ~~~ # ---------- Path resolution ------------------------------------------------ diff --git a/libs/cua-driver/swift/Sources/CuaDriverCore/Input/SkyLightEventPost.swift b/libs/cua-driver/swift/Sources/CuaDriverCore/Input/SkyLightEventPost.swift index 7d753de960..d11b9c76fa 100644 --- a/libs/cua-driver/swift/Sources/CuaDriverCore/Input/SkyLightEventPost.swift +++ b/libs/cua-driver/swift/Sources/CuaDriverCore/Input/SkyLightEventPost.swift @@ -104,13 +104,22 @@ public enum SkyLightEventPost { "SLSEventAuthenticationMessage") else { return nil } + // macOS 14 (Sonoma) compatibility: the class exists on macOS 14 but + // `messageWithEventRecord:pid:version:` was added in macOS 15. + // `NSSelectorFromString` always succeeds (it interns the string), so + // verify the class responds before storing it — otherwise `objc_msgSend` + // raises NSInvalidArgumentException at runtime and crashes the daemon. + // When absent, return nil so callers skip the auth envelope. See #1503. + let factorySelector = NSSelectorFromString( + "messageWithEventRecord:pid:version:") + guard messageClass.responds(to: factorySelector) else { return nil } + return Resolved( postToPid: postToPid, setAuthMessage: setAuth, msgSendFactory: msgSend, messageClass: messageClass, - factorySelector: NSSelectorFromString( - "messageWithEventRecord:pid:version:") + factorySelector: factorySelector ) }()