fix(cua-driver-rs)(windows): per-session agent cursors (port macOS #1779) - #1801
Merged
Merged
Conversation
) The Windows overlay was a process-wide singleton (one `RenderState`), so concurrent MCP sessions clobbered each other last-writer-wins → one shared cursor. #1779 fixed this on macOS but explicitly left Windows/Linux on the old single-cursor model ("the key concept never reaches them"). Port the keyed render collection to platform-windows: - overlay.rs: `RenderMap { IndexMap<CursorKey, RenderState> }`; `send_command` now carries a `CursorKey`; the WM_TIMER tick drains keyed `OverlayMsg`s, ticks every cursor, and composites them all into the ONE layered window via `paint_cursor` (insertion order = stable z-order). Per-key arrival isolation, lazy per-key palette (`Palette::for_instance`), `remove_cursor` + render-side resurrection tombstone, and the sentinel seed — all mirroring platform-macos/src/cursor/overlay.rs. - tools/impl_.rs: `resolve_cursor_key` (session > cursor_id > NO_CURSOR, never the connection `_session_id`), threaded through `pin_overlay_above`, `overlay_glide_to`, every ClickPulse callsite, and the 5 cursor tools. A `session_end` hook (once-guarded) calls `remove_cursor`; `get_config`'s `cursor_enabled` is now session-scoped + deterministic (was a nondeterministic `all_states().first()` — macOS BUG 3). - cursor-overlay: `CursorRegistry::remove` (guards "default"). page.click_element keeps the seeded "default" cursor — the cross-platform `PageBackend` trait carries no caller session (separate follow-up). 15 new headless unit tests (two-session isolation, session_end removal, default guard, resurrection tombstone, sentinel seed, key resolution); full platform-windows lib suite green (49 tests), daemon builds warning-free. Verified live on Windows 11: two calculators driven by two sessions show two distinct-coloured cursors gliding in parallel; end_session removes each. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
📝 WalkthroughWalkthroughThe pull request refactors the Windows cursor overlay from a single global cursor to a per-session keyed model, enabling multiple independent cursor overlays to coexist in one layered window. The core change replaces singleton render state with an insertion-ordered map, updates tool and helper APIs to resolve and pass cursor keys, and adds session-end cleanup hooks that remove cursors from the shared registry. ChangesPer-session keyed cursor overlay with shared registry lifecycle
Sequence DiagramsequenceDiagram
participant WM_TIMER
participant RenderMap
participant Pixmap
participant UpdateLayered
WM_TIMER->>RenderMap: apply_msg (drain queued keyed cmds)
RenderMap->>RenderMap: tick all cursors, collect finished
RenderMap->>Pixmap: composite all cursor states
Pixmap->>UpdateLayered: push one composited bitmap
RenderMap->>RenderMap: fire per-key arrival oneshots
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning Tools execution failed with the following error: Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error) Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs`:
- Around line 109-133: resolve_cursor_key now prioritizes a caller-declared
"session" key, but several tool schemas (e.g., click, type_text, move_cursor,
set_agent_cursor_enabled, get_agent_cursor_state, get_config) omit "session" and
use additionalProperties: false, causing callers to be unable to send the
session key and falling back to NO_CURSOR. Update each affected tool definition
to add an optional "session" property (string) to the schema (keeping
additionalProperties: false) so schema-driven clients can include it, and verify
any schema validators/serializers for those tools accept and forward the
"session" field to the args consumed by resolve_cursor_key.
- Around line 4371-4379: The invoke function updates cursor_enabled using
crate::overlay::is_enabled(&resolve_cursor_key(&args)) but set_config still
reads agent_cursor.enabled from cursor_registry.all_states().first(), causing
inconsistent values; update set_config (and any code paths that echo
agent_cursor.enabled) to read from the same scoped overlay lookup used by
invoke/get_config by calling
crate::overlay::is_enabled(&resolve_cursor_key(&args)) (or a shared helper)
instead of using cursor_registry.all_states().first(), ensuring both get_config
and set_config use the same resolve_cursor_key()/overlay lookup for the calling
session.
- Around line 3788-3801: The code currently treats an empty cursor key as
"default" in the user-facing message while earlier logic skips updating the
registry and the overlay ignores commands for a NO_CURSOR key; change the
control flow to check resolve_cursor_key() (the cursor_key returned by
resolve_cursor_key) and if it's empty skip calling
crate::overlay::send_command(...) and return a clear no-op message (e.g., "No
cursor present; no movement performed.") instead of reporting "Agent cursor
'default' moved..."; otherwise keep the existing update via
self.state.cursor_registry.update_position(...) and send_command(...) and return
the success message using cursor_key.as_str().
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 31cffa23-0e35-41cd-aa61-d43f21922bf0
⛔ Files ignored due to path filters (1)
libs/cua-driver/rust/Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (6)
libs/cua-driver/rust/crates/cursor-overlay/src/lib.rslibs/cua-driver/rust/crates/platform-windows/Cargo.tomllibs/cua-driver/rust/crates/platform-windows/src/overlay.rslibs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rslibs/cua-driver/rust/crates/platform-windows/src/tools/page.rslibs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs
Comment on lines
+109
to
+133
| /// The cursor key for an anonymous (cursor-less) call. A run opts into a cursor | ||
| /// by declaring a `session`; without one, every cursor op short-circuits on | ||
| /// this empty key (see `overlay::send_command` / `overlay_glide_to`). | ||
| pub(crate) const NO_CURSOR: &str = ""; | ||
|
|
||
| /// Resolve the cursor key for a tool invocation, or [`NO_CURSOR`] (`""`) for an | ||
| /// anonymous call. | ||
| /// | ||
| /// A cursor is tied to a **caller-declared session**, never to the MCP | ||
| /// connection. Precedence: an explicit `session` arg, then its legacy alias | ||
| /// `cursor_id`. We deliberately do NOT fall back to the connection-injected | ||
| /// `_session_id` or to a seeded `"default"` cursor — `""` means "no session | ||
| /// declared → no cursor", while the underlying action (click/type/…) still | ||
| /// executes. Mirrors `platform_macos::tools::cursor_tools::resolve_cursor_key` | ||
| /// so the two platforms key cursors identically. | ||
| pub(crate) fn resolve_cursor_key(args: &Value) -> String { | ||
| for key in ["session", "cursor_id"] { | ||
| if let Some(v) = args.get(key).and_then(|v| v.as_str()) { | ||
| if !v.is_empty() { | ||
| return v.to_owned(); | ||
| } | ||
| } | ||
| } | ||
| NO_CURSOR.to_owned() | ||
| } |
Contributor
There was a problem hiding this comment.
Expose session anywhere this helper is now part of the API contract.
resolve_cursor_key() now makes caller-declared session the primary selector, but many tool defs below still use additionalProperties: false and omit session entirely (for example click, type_text, move_cursor, set_agent_cursor_enabled, get_agent_cursor_state, and get_config). Schema-driven clients won't be able to send the key this logic depends on, so those calls fall back to NO_CURSOR and lose the per-session behavior this PR is adding.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs` around lines
109 - 133, resolve_cursor_key now prioritizes a caller-declared "session" key,
but several tool schemas (e.g., click, type_text, move_cursor,
set_agent_cursor_enabled, get_agent_cursor_state, get_config) omit "session" and
use additionalProperties: false, causing callers to be unable to send the
session key and falling back to NO_CURSOR. Update each affected tool definition
to add an optional "session" property (string) to the schema (keeping
additionalProperties: false) so schema-driven clients can include it, and verify
any schema validators/serializers for those tools accept and forward the
"session" field to the args consumed by resolve_cursor_key.
Comment on lines
+3788
to
+3801
| // Cursor key precedence: caller-declared `session` > legacy `cursor_id` | ||
| // > NO_CURSOR. An anonymous run (no session) has no cursor to move. | ||
| let cursor_key = resolve_cursor_key(&args); | ||
| if !cursor_key.is_empty() { | ||
| self.state.cursor_registry.update_position(&cursor_key, x, y); | ||
| } | ||
| // End pointing upper-left (45°) — matches Swift's | ||
| // `AgentCursor.animateAndWait(endAngleDegrees: 45)` convention so | ||
| // the cursor settles to the natural macOS-style pose. | ||
| crate::overlay::send_command(cursor_overlay::OverlayCommand::MoveTo { | ||
| crate::overlay::send_command(cursor_key.clone(), cursor_overlay::OverlayCommand::MoveTo { | ||
| x, y, end_heading_radians: std::f64::consts::FRAC_PI_4, | ||
| }); | ||
| ToolResult::text(format!("Agent cursor '{cursor_id}' moved to ({x:.1}, {y:.1}).")) | ||
| let shown = if cursor_key.is_empty() { "default" } else { cursor_key.as_str() }; | ||
| ToolResult::text(format!("Agent cursor '{shown}' moved to ({x:.1}, {y:.1}).")) |
Contributor
There was a problem hiding this comment.
Don’t report a default cursor move when the call was cursor-less.
When resolve_cursor_key() returns NO_CURSOR, this path intentionally skips the registry update and the overlay thread drops the command, but the response still says Agent cursor 'default' moved.... That makes anonymous calls look successful even though no cursor instance exists or moved.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs` around lines
3788 - 3801, The code currently treats an empty cursor key as "default" in the
user-facing message while earlier logic skips updating the registry and the
overlay ignores commands for a NO_CURSOR key; change the control flow to check
resolve_cursor_key() (the cursor_key returned by resolve_cursor_key) and if it's
empty skip calling crate::overlay::send_command(...) and return a clear no-op
message (e.g., "No cursor present; no movement performed.") instead of reporting
"Agent cursor 'default' moved..."; otherwise keep the existing update via
self.state.cursor_registry.update_position(...) and send_command(...) and return
the success message using cursor_key.as_str().
Comment on lines
+4371
to
+4379
| async fn invoke(&self, args: Value) -> ToolResult { | ||
| let cfg = self.state.config.read().unwrap(); | ||
| // Mirror the macOS agent's parity addition (commit adb9ecca): | ||
| // nested `agent_cursor.enabled` block so Swift-shaped get_config | ||
| // consumers can read the cursor's enabled state from one place. | ||
| let cursor_enabled = self.state.cursor_registry.all_states() | ||
| .first() | ||
| .map(|s| s.config.enabled) | ||
| .unwrap_or(true); | ||
| // Scope to the CALLING session's cursor (session > cursor_id > default) | ||
| // and read it from the overlay deterministically — `all_states().first()` | ||
| // was a nondeterministic HashMap read across sessions (macOS BUG 3). | ||
| let cursor_enabled = crate::overlay::is_enabled(&resolve_cursor_key(&args)); |
Contributor
There was a problem hiding this comment.
Keep set_config and get_config on the same cursor-enabled lookup.
get_config() now reads agent_cursor.enabled from the scoped overlay state, but set_config() still echoes that field from cursor_registry.all_states().first(). After this change, the two endpoints can return different agent_cursor.enabled values for the same caller/session. Please route both through the same scoped source.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs` around lines
4371 - 4379, The invoke function updates cursor_enabled using
crate::overlay::is_enabled(&resolve_cursor_key(&args)) but set_config still
reads agent_cursor.enabled from cursor_registry.all_states().first(), causing
inconsistent values; update set_config (and any code paths that echo
agent_cursor.enabled) to read from the same scoped overlay lookup used by
invoke/get_config by calling
crate::overlay::is_enabled(&resolve_cursor_key(&args)) (or a shared helper)
instead of using cursor_registry.all_states().first(), ensuring both get_config
and set_config use the same resolve_cursor_key()/overlay lookup for the calling
session.
r33drichards
added a commit
that referenced
this pull request
Jun 3, 2026
![trycua-release[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
* docs(cua-driver): add changelog reference page (#1785)
Mirror the cua-driver-rs GitHub releases into the docs site so the
release history is discoverable on the docs site (not just GitHub),
matching the convention used by the other products (cua CLI, lume).
Wire it into the reference nav.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver)(macos): guard SkyLight auth-message selector for macOS 14 Sonoma (#1503) (#1782)
`hotkey`, `press_key`, and `scroll` crash the daemon on macOS 14 (Sonoma)
with `NSInvalidArgumentException: +[SLSEventAuthenticationMessage
messageWithEventRecord:pid:version:]: unrecognized selector sent to class`.
The class `SLSEventAuthenticationMessage` exists on macOS 14, but the
`messageWithEventRecord:pid:version:` factory selector was only added in
macOS 15 (Sequoia). The existing `!cls.is_null() && !sel.is_null()` guard
is insufficient: `sel_registerName` / `NSSelectorFromString` always succeed
(they just intern the string), so `objc_msgSend` still dispatches an
unimplemented selector and the ObjC runtime aborts the process.
Guard the dispatch with `class_respondsToSelector` (Rust) /
`messageClass.responds(to:)` (Swift), which actually checks the metaclass.
On macOS 14 it returns false, so we skip the auth envelope and fall through
to plain `SLEventPostToPid`. Chromium-class targets may not receive the
event on macOS 14, but the daemon no longer crashes — graceful degradation.
This re-applies the fix from #1579 (by @hippoley) onto the current
`libs/cua-driver/{rust,swift}/` layout — #1579 predates the #1674
directory restructure and no longer merges.
- rust: platform-macos/src/input/skylight.rs — class_responds_to_selector()
- swift: CuaDriverCore/Input/SkyLightEventPost.swift — responds(to:) guard
Verified: platform-macos + the full cua-driver binary build; the Swift
`responds(to:)` form compiles and returns true for an existing class method,
false for an absent one.
Closes #1503
Co-authored-by: hippoley <hippoley@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs)(macos): enable Chromium/Electron AX trees for get_window_state (#1756)
Chromium/Electron apps (Arc, VS Code, Electron shells) ship their web-content
accessibility tree off and only build it once an assistive client requests it.
Without enablement the first AX walk returns an empty/title-bar-only tree.
Flip AXManualAccessibility (modern, side-effect-free) on the application root,
falling back to AXEnhancedUserInterface when the modern attribute is
unsupported. When the flip actually takes, let the asynchronously-built tree
settle (~500ms run-loop pump) before walking. Cache per-pid so repeat snapshots
skip the settle. Native Cocoa apps reject the attribute and pay no cost.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bump cua-driver-rs to v0.4.2
* docs(cua-driver): add 0.4.2 changelog entry + fix 0.3.6 wording (#1786)
- Add 0.4.2: macOS 14 Sonoma SkyLight selector guard (#1782, #1503) and
Chromium/Electron AX trees via AXManualAccessibility (#1756).
- Fix the 0.3.6 entry, which described the permissions-status fix backwards:
it now reports the driver's grants (via the daemon), not the caller's.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* chore(cua-driver-rs): bake version 0.4.2 into install scripts [skip ci]
* fix(cua-driver-rs): wire/guide per-session cursors through the real mcp path + skills/docs (#1787)
* fix(cua-driver-rs): wire/guide per-session cursors through the real mcp path + update skills/docs
A user drove `cua-driver mcp --claude-code-computer-use-compat` (the
documented Claude Code install) and asked: (1) why no agent cursor even
on AX actions, (2) where is the session in the mcp calls, (3) did we
forget the CLI / MCP / skills wiring.
Investigation + fixes:
- Session IS wired (working as designed): the proxy path the user runs
mints one session_id per MCP connection and stamps it on every
forwarded request; the daemon injects it as `_session_id` into tool
args and strips it from the user-visible wire envelope. Per-session
cursor / config / recording are live on the compat proxy path —
verified headless (set_agent_cursor_enabled{false} in a session is
read back by get_config{enabled:false}, proving _session_id reached
the daemon).
- BUG (user-visible): no glide on a pure-AX run. A brand-new session
cursor sat at the off-screen sentinel; animate_cursor_to early-returned
so the first AX action only snapped a static arrow via ClickPulse —
easy to miss. Fix: seed the sentinel cursor on-screen (offset, clamped)
before animating so the FIRST action glides. Get-or-create + ended
tombstone guard so it never resurrects a reaped session. Unit-tested.
- BUG (latent wiring): `--claude-code-computer-use-compat` was silently
dropped on the proxy path (daemon hardcoded compat=false). Thread it
end-to-end: proxy forwards `serve --claude-code-computer-use-compat`,
the Serve arm honours it via build_macos_registry_with_compat. Today
this has no tool-surface effect (the compat screenshot tool was removed
in #1692) but the flag now travels for any future compat-gated tool.
- BUG (nondeterministic): get_config reported agent_cursor.enabled from a
HashMap .first(). Resolve the calling session's cursor by key
(cursor_id > _session_id > "default"). Unit-tested per-session.
Docs/skills (no default change — that is the user's call; see PR body):
SKILL.md (per-session model, session_end removal, AX no-glide caveat,
corrected the false "AX skips the overlay" claim), set_agent_cursor_enabled
description, protocol.rs server-instructions, CLI help (cursor flags +
overlay + compat), mcp-tools.mdx AX-snap caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver)(skills): correct the AX cursor caveat — short glide, not no glide
After the sentinel-seed fix the first AX action seeds the cursor on-screen
near the target and plays a brief glide + pulse (not "does not glide").
Reword the SKILL.md visibility caveat to match the actual behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(macos): run the agent-cursor overlay in the serve daemon (#1790)
The overlay NSWindow + AppKit render loop were only wired into the in-process
`mcp` arm. In the daemon-proxy setup users run (`mcp` relaunches
`open -n -g … serve` and proxies to it for correct TCC), the DAEMON performs
the clicks/AX presses but never inited or ran the overlay — its main thread
parked in `serve_handle.join()`. So `set_agent_cursor_enabled` flipped registry
flags and clicks sent OverlayCommands, but CMD_TX/RENDER were never set →
every cursor command was a silent no-op and the agent cursor never appeared.
Fix: the Serve arm now builds cursor_cfg, inits the overlay channel before
spawning the serve thread, and (when enabled) parks main in
`overlay::run_on_main_thread()` (mirrors the Mcp arm) instead of join. It
self-guards on has_graphic_access() and falls back to join when there's no
Window Server session, so headless serving is unaffected. PiP unchanged.
Verified via the REAL launch path: `open -n -g -a CuaDriver --args serve`
daemon's main thread now runs __CFRunLoopRun / -[NSApplication run] with
run_appkit + SkyLight + tiny_skia overlay rendering, and still serves.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(macos): stop the permissions gate spamming the TCC prompt on every re-exec (#1791)
`cua-driver permissions grant` (and any first-launch serve) raises the system
TCC prompt, then re-execs the daemon ~every 25s to refresh the per-process
AXIsProcessTrusted cache. Each re-exec'd process re-ran run_if_needed and
re-raised request_accessibility/request_screen_recording — so a fresh "Cua
Driver" dialog popped every ~25s. Worse, the 10-min deadline was anchored to
each process's own start, and since the re-exec fires (~25s) well before the
deadline, the deadline never triggered: the gate re-execed (and restarted the
whole daemon, now incl. the cursor overlay) forever whenever the grant read as
missing — including the stale-ad-hoc-cdhash case (Settings shows granted but
the rebuilt binary's hash no longer matches, so the live check returns false).
Fix:
- reexec_self sets CUA_DRIVER_RS_GATE_REEXEC=1; run_if_needed sees it and polls
SILENTLY (skips the prompts + panel) on re-exec'd processes. The prompt +
panel appear exactly once, on first launch.
- reexec_self persists the original gate start in CUA_DRIVER_RS_GATE_START_UNIX;
wait_for_grants anchors `start` to it so the deadline is cumulative across
re-execs and the gate actually gives up (and stops churning) after the
deadline, continuing to serve (tools fail with TCC errors until granted).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs)(install-local): sign the bundle with a stable self-signed identity so TCC grants survive rebuilds (#1792)
install-local ad-hoc-signed the bundle (`codesign --sign -`), which keys the
TCC grant (Accessibility / Screen Recording) on the binary's cdhash. The
cdhash changes on EVERY rebuild, so each install-local silently invalidated the
grant — System Settings still showed "CuaDriver ✅" (it's keyed on the bundle
id) while the live AXIsProcessTrusted check failed, and the daemon re-prompted
("I already granted!"). A genuinely miserable dev loop.
Fix: create a self-signed code-signing certificate once (idempotent, in the
login keychain) and sign the bundle with it. TCC then keys the grant on the
certificate leaf — stable across rebuilds — so the Designated Requirement
becomes `identifier "com.trycua.driver" and certificate leaf = H"..."` instead
of a cdhash pin. Grant once; every future install-local keeps it.
Robust + fail-soft: openssl 3.x needs `-legacy` PBE + a real p12 password for
Apple's `security import` (the empty-password default fails MAC verification);
falls back to non-legacy for LibreSSL. If the cert can't be created (no
openssl, locked keychain, CI), falls back to ad-hoc signing + a one-line note.
Local dev only — releases are CI-signed and already stable.
One-time migration: switching from ad-hoc to the cert changes the requirement
once, so the next grant after this lands is a single re-grant; stable after.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Bump cua-driver-rs to v0.4.3
* docs(cua-driver): add 0.4.3 changelog entry (#1793)
cursor overlay in the daemon (#1790), permissions-grant prompt no-spam (#1791),
and install-local stable signing identity (#1792).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* chore(cua-driver-rs): bake version 0.4.3 into install scripts [skip ci]
* fix(cua-driver-rs)(install-local): reset a TCC grant pinned to a previous signing identity (#1795)
#1792 made install-local sign CuaDriver.app with a stable self-signed cert so
Accessibility / Screen-Recording grants survive rebuilds — but only for grants
CREATED while cert-signed. A grant the user made earlier on an ad-hoc build is
pinned to that build's cdhash (the stored csreq is a bare `cdhash H"..."`), so it
survives reinstall with auth_value=allowed yet stops matching the new binary. The
daemon then reads "not granted" while System Settings still shows CuaDriver toggled
ON — a dead end, because the row already records a decision so re-toggling never
re-fires the prompt.
Record the signing identity (cert leaf, or "adhoc") in
~/.cua-driver/.tcc-signing-identity. When the installer signs with a cert identity
that differs from the last install, `tccutil reset` Accessibility + ScreenCapture
once so the next `permissions grant` prompts cleanly and re-pins to the stable
cert (after which grants survive every future rebuild). `tccutil reset` needs no
sudo/FDA and is a no-op when nothing was granted. We only reset when moving TO a
cert identity — an ad-hoc build churns its cdhash regardless, so resetting it would
add friction with no durable fix.
Docs: FAQ entry for "granted but reports NOT granted after a rebuild" + changelog.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(macos): retain cached AX element across action so concurrent sessions can't UAF-crash the daemon (#1796)
Two sessions driving the same window concurrently crashed the daemon with
EXC_BREAKPOINT (SIGTRAP) inside AXUIElementCopyActionNames → _AXUIElementValidate
→ CFGetTypeID — a use-after-free.
Root cause: the per-(pid, window_id) element cache (ax/cache.rs) handed out raw
AXUIElementRef pointers as usize. A tool (click/type_text/set_value/…) copied the
pointer out from under the cache lock and used it across await points and on a
blocking thread. Meanwhile another session's get_window_state called
ElementCache::update → ElementCacheCore::insert, which replaced the snapshot and
ran CachedSnapshot::drop on the old one — CFRelease-ing those exact pointers to
zero. The in-flight action then dereferenced freed memory.
Fix: replace get_element_ptr with get_element_retained, which CFRetains the
element while still holding the cache lock and returns a RetainedElement guard
(CFRelease on drop). An in-flight action holds the guard for its whole duration,
so a concurrent snapshot replace can't free the element under it. Migrated all
nine element-action call sites (click, right_click, double_click, type_text,
type_text_chars, press_key, scroll, set_value, recording_hooks).
Test: ax::cache::tests::retained_element_survives_concurrent_snapshot_replace
asserts the retain accounting — after a concurrent replace the guard's retain is
what keeps the element alive (count = base+1, not base). 74/74 platform-macos
lib tests pass.
Note: platform-windows has the same shape (uia/cache.rs::get_element_ptr hands
out raw IUIAutomationElement pointers); a mirrored AddRef-on-get fix is a
follow-up, not included here (untestable in this environment).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver-rs)(launch_app): surface creates_new_application_instance for concurrent multi-agent isolation (#1797)
launch_app is idempotent, so two sessions launching the same app get the same
instance — and on single-instance apps (Calculator, many utilities) the same
window — and clobber each other. The `creates_new_application_instance` param
already solves this (it maps to NSWorkspaceOpenConfiguration.createsNewApplicationInstance,
the programmatic `open -n`), but nothing told an agent to reach for it in the
concurrent case. Enrich the tool description, the MCP-tools doc, and the skill's
action-loop section to call out the concurrent-session use. No behavior change.
Verified end-to-end: two launch_app(name=Calculator, creates_new_application_instance=true)
calls return distinct pids + distinct window_ids.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs): caller-declared session identity + Streamable-HTTP transport for multi-agent parallelism (#1798)
* feat(cua-driver-rs): explicit session identity core + cursor explicit-required
- core/session.rs: touch_session/end_session/evict_idle + idle-TTL activity map
- serve.rs: apply_session_identity at the daemon boundary (explicit `session` →
_session_id; minted id is recording/config fallback only, not a cursor source)
- cursor: resolve_cursor_key returns NO_CURSOR("") when no session declared;
overlay + registry short-circuit the empty key (explicit-required cursor)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs): start_session/end_session tools + idle-TTL sweep + session schema
- core/session_tools.rs: start_session / end_session tools (cross-platform),
registered via ToolRegistry::register_session_tools on all 3 platforms
- serve.rs: spawn_session_idle_sweep — evict_idle every 30s (TTL default 300s,
CUA_DRIVER_RS_SESSION_IDLE_TTL_SECS override)
- inject session property into action-tool schemas; fix set_agent_cursor_enabled
description (cursor is explicit-required now, not auto-per-MCP-session)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver-rs): document explicit session identity (MCP instructions, SKILL, mcp-tools, changelog)
- MCP server instructions: add start_session step + explicit-session cursor model
- SKILL.md: canonical loop gains start_session/end_session; fix concurrent note
(cursor keyed on session, not (pid,window_id))
- mcp-tools.mdx: rewrite per-session cursor section; add start_session/end_session
- changelog: breaking session-identity entry
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs): stop a session's recording on session_end (end_session/idle-TTL/EOF)
Register a session_end hook that calls recording.stop_owner(Some(sid)) on a
detached thread, so end_session and the idle-TTL sweep tear down a session's
recording too (matching end_session's contract) — not just the EOF path. Safe:
stop_owner(Some) is a no-op unless that session owns the live recording, and the
detached thread keeps mp4 finalize off the synchronous fire_session_end caller.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(cua-driver-rs): unit-test apply_session_identity boundary (explicit/minted/anonymous)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(macos): move_cursor visibly moves the drawn cursor (seed sentinel like click)
move_cursor sent a raw MoveTo, which doesn't bring a brand-new session cursor
on-screen — it sits at the off-screen sentinel until a click seeds it, so the
DRAWN cursor never moved (only the reported position did). Use animate_cursor_to
(the same path click uses): it seeds the sentinel on-screen then glides in.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs)(macos): mark move_cursor read-only so MCP clients can parallelize cursor moves
move_cursor only nudges the agent-cursor overlay, never the target app, so it is
concurrency-safe. read_only:true emits readOnlyHint, which Claude Code's
isConcurrencySafe() uses to run cursor moves in parallel. Mutating tools
(click/type_text/press_key) stay read_only:false on purpose — parallelizing an
ordered intra-agent sequence would race.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs): Streamable-HTTP MCP transport on the daemon for parallel multi-agent (#1799)
Over stdio, one cua-driver mcp process is a single pipe, so a client's tool calls
(incl. multiple subagents) serialize. The daemon is already concurrent (task per
connection). This adds an HTTP MCP front-end so each agent opens its OWN
connection: per-connection FIFO keeps a single agent's ordered calls correct,
distinct connections run truly in parallel — safe because per-(pid,window) caches
+ per-session cursors make concurrent cross-connection actions non-colliding.
- mcp_http.rs: hand-rolled HTTP/1.1 (no new deps, mirrors the UDS line protocol),
POST -> cua_driver_core::server::handle_request (now pub) -> application/json
JSON-RPC. Task per TCP connection; honors Connection: close; mirrors the
"session" arg -> _session_id + touches idle-TTL so HTTP == stdio behavior.
- opt-in via CUA_DRIVER_RS_MCP_HTTP_PORT (loopback only); spawned from run_serve.
Proven: 10 list_apps over 10 concurrent connections = 3.6s vs 12.9s sequential
(3.6x). curl initialize/tools/list/tools/call all correct. 3 unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver-rs): document HTTP MCP transport + the concurrency model
- changelog: Streamable-HTTP transport + move_cursor readOnlyHint
- FAQ: "Concurrency & multiple agents" — why subagents serialize (shared stdio
pipe), and how to run agents truly in parallel (separate connections / the
CUA_DRIVER_RS_MCP_HTTP_PORT HTTP endpoint)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver-rs)(skill): note subagent serialization + HTTP transport for parallel agents
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(windows): per-session agent cursors (port macOS #1779) (#1801)
The Windows overlay was a process-wide singleton (one `RenderState`), so
concurrent MCP sessions clobbered each other last-writer-wins → one shared
cursor. #1779 fixed this on macOS but explicitly left Windows/Linux on the
old single-cursor model ("the key concept never reaches them").
Port the keyed render collection to platform-windows:
- overlay.rs: `RenderMap { IndexMap<CursorKey, RenderState> }`; `send_command`
now carries a `CursorKey`; the WM_TIMER tick drains keyed `OverlayMsg`s,
ticks every cursor, and composites them all into the ONE layered window via
`paint_cursor` (insertion order = stable z-order). Per-key arrival isolation,
lazy per-key palette (`Palette::for_instance`), `remove_cursor` + render-side
resurrection tombstone, and the sentinel seed — all mirroring
platform-macos/src/cursor/overlay.rs.
- tools/impl_.rs: `resolve_cursor_key` (session > cursor_id > NO_CURSOR, never
the connection `_session_id`), threaded through `pin_overlay_above`,
`overlay_glide_to`, every ClickPulse callsite, and the 5 cursor tools. A
`session_end` hook (once-guarded) calls `remove_cursor`; `get_config`'s
`cursor_enabled` is now session-scoped + deterministic (was a
nondeterministic `all_states().first()` — macOS BUG 3).
- cursor-overlay: `CursorRegistry::remove` (guards "default").
page.click_element keeps the seeded "default" cursor — the cross-platform
`PageBackend` trait carries no caller session (separate follow-up).
15 new headless unit tests (two-session isolation, session_end removal,
default guard, resurrection tombstone, sentinel seed, key resolution); full
platform-windows lib suite green (49 tests), daemon builds warning-free.
Verified live on Windows 11: two calculators driven by two sessions show two
distinct-coloured cursors gliding in parallel; end_session removes each.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Bump cua-driver-rs to v0.5.0
Release the caller-declared session identity + Streamable-HTTP multi-agent
transport (#1798) and Windows per-session cursors (#1801). Breaking: the agent
cursor is now opt-in (declare a `session`). Changelog Unreleased → 0.5.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(cua-driver-rs): bake version 0.5.0 into install scripts [skip ci]
* fix(cua-driver-rs): release installer unifies home on ~/.cua-driver + cleans up prior local install (#1803)
The release installer (install.sh → _install-rust.sh) defaulted its package
home to the legacy ~/.cua-driver-rs, but the local installer
(_install-local-rust.sh) and the runtime already use ~/.cua-driver (renamed in
v0.2.16 / PR #1644). That mismatch is the root cause of a two-install collision:
a user who ran install-local and then the release install.sh ended up with two
homes and two conflicting installs, with the local build's artifacts left
dangling.
Fixes in _install-rust.sh:
- Default HOME_DIR to ~/.cua-driver (still honoring CUA_DRIVER_RS_HOME for
back-compat), matching install-local + runtime.
- Before staging: cleanup_prior_local_install() stops the daemon and removes
the prior install-local artifacts under the shared home — the `*-local-*`
release dirs and the ~/.cua-driver/.tcc-signing-identity marker. Marker-gated
and conservative: never touches a real release dir, the `current` symlink, or
unrelated user state; best-effort + idempotent (no-op on a clean machine).
- After staging: sweep a stale ~/.cua-driver-rs left by an older release,
mirroring the belt-and-braces legacy-home sweep install-local already does.
- TCC grants preserved: /Applications/CuaDriver.app is replaced in place via
the existing release ditto (grants key on the shared com.trycua.driver bundle
id); no tccutil reset, so cert-pinned grants are not churned.
install.ps1 (Windows) already defaults to ~/.cua-driver and migrates the legacy
home, so it is unchanged.
Docs: reconcile the ~/.cua-driver-rs → ~/.cua-driver home references across the
installation + linux guides, document the local/legacy cleanup behavior, and add
an Unreleased changelog entry.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs)(linux): generalize background keyboard input via XTEST
The background-terminal work special-cased terminals: type_text and
press_key(Enter) detected a terminal process, found its /dev/pts tty, and
shoved bytes in with the legacy TIOCSTI ioctl. That only ever worked for
terminals, and TIOCSTI is exactly the mechanism modern kernels harden away
(CONFIG_LEGACY_TIOCSTI / dev.tty.legacy_tiocsti), so it would EPERM on many
systems. It also left the XTEST scaffold added alongside it as dead code.
Replace the terminal-specific path with a general one. Keyboard input now
goes through XTEST for every window: XSendEvent keystrokes carry the
send_event flag that xterm (and friends) deliberately ignore, which is why
typing into a background terminal silently did nothing; XTEST injects at the
server level with no such flag, so it lands on terminals and every other app
alike. Because XTEST targets the focused window, with_focus briefly focuses
the target, injects, and restores the prior focus — preserving the same
no-focus-steal contract the XSendEvent pointer path keeps.
- input/mod.rs: send_type_text / send_type_text_with_delay / send_key now
use XTEST (with real Shift presses for shifted chars and held modifiers),
wiring up the previously-dead xtest_* helpers. Pointer (click/drag) stays
on XSendEvent.
- impl_.rs: drop inject_terminal_input + is_terminal_process /
terminal_*_tty helpers and the TIOCSTI ioctl, and the type_text / press_key
branches that called them.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(cua-driver-rs)(linux): restore active window after XTEST injection
The background-terminal GIF test injected fine but failed its focus check:
typing landed in the inactive xterm, yet focus ended on the target instead
of returning to the control terminal. XTEST delivers to the focused window,
so with_focus moves focus to the target to inject — but the restore used a
bare SetInputFocus, and under an EWMH WM (openbox) `xdotool getactivewindow`
reads `_NET_ACTIVE_WINDOW`, which the WM owns and doesn't update from a raw
SetInputFocus. So focus never came back.
Restore cooperatively: capture `_NET_ACTIVE_WINDOW` up front and re-activate
it afterwards with a `_NET_ACTIVE_WINDOW` client message (source = 2, the same
nudge `xdotool windowactivate` sends), keeping SetInputFocus for the no-WM
case. Add a short settle after each focus/activation request so the
asynchronous WM acts before we inject or restore.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): restore Cargo.lock to keep cargoHash valid
A stray `cargo check` re-bumped the workspace crates in Cargo.lock from
0.4.0 to 0.4.1 (matching the manifests) and it got committed. Nixpkgs'
fetchCargoVendor hashes the vendored directory, which includes a copy of
Cargo.lock, so the changed lock invalidated the pinned cargoHash and broke
the cua-driver build — and with it every NixOS VM test that builds the
driver. Restore Cargo.lock to the base/known-good revision.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(cua-driver-rs)(linux): focus-free input — XSendEvent for GUI, pty master for terminals
Replaces the XTEST-with-temporary-focus approach (which broke the
cross-platform "no focus steal" contract that macOS SLEventPostToPid and
Windows PostMessage uphold) with two focus-free paths:
- GUI apps: XSendEvent, as before, but the typing path now resolves the
shift level from the keyboard map so uppercase / shifted symbols inject
correctly (previously "A" was sent as "a"). Removed the dead XTest scaffold.
- Terminals: instead of the legacy TIOCSTI ioctl (which dev.tty.legacy_tiocsti
disables on modern kernels), borrow the emulator's pty master fd via
pidfd_getfd(2) and write to it. The kernel delivers the bytes to the shell's
stdin exactly as typed — no X focus change, immune to the TIOCSTI sysctl.
pidfd_getfd needs ptrace-mode access, which under the default ptrace_scope=1
is granted for the caller's own descendants — i.e. terminals the driver
launched — with no root and no special capability. For terminals the driver
did not launch it returns Ok(false) and the caller falls back; injecting into
someone else's terminal unprivileged is what the kernel deliberately prevents.
New module crate::tty holds the master-borrow logic.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(nix): matrix background-GUI input coverage (chromium, firefox, tk)
Adds a parameterized NixOS VM test proving cua-driver types into a GUI window
via XSendEvent WITHOUT stealing focus — the general computer-use claim, beyond
terminals. Each app shows a focused text field that mirrors what it receives
into its X11 window title; the test types a known string into the *inactive*
app window (no click/focus first) and asserts the title became that string
(input landed) and a separate control terminal stayed active (no focus steal).
Wired as one independent matrix job per app (chromium, firefox, tk) in
flake.nix checks and the nix-build workflow, so coverage spans a Chromium web
engine, a Gecko web engine, and a native Tk toolkit.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): use python3 + tkinter for the tk GUI test (python3Full removed)
nixpkgs removed python3Full ("tkinter is available within the package set"),
which broke flake evaluation of the tk matrix job. Use
python3.withPackages (ps: [ ps.tkinter ]) instead.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): background-GUI test — file:// page, exec launchers, find window by name
Two harness bugs the matrix run surfaced (driver logic unaffected):
- The browser launch commands embedded a data: URL whose double quotes
collided with the testScript's Python/shell quoting, so the nixos test
driver rejected the script with "invalid-syntax". Serve the page from a
file:// URL written via writeText and move each launch into a writeShellScript
that exec's the app, so the testScript only ever embeds a quote-free path.
- Window discovery used `xdotool search --pid`, which needs _NET_WM_PID — Tk
doesn't set it and browser window pids differ from the launcher, so the
search hung to timeout. Give every app a known initial window title
("cua-initial") and discover by --name instead.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(cua-driver-rs)(linux): type into GUI apps via AT-SPI (focus-free)
X11 only routes keystrokes to the focused toplevel's focused widget, so
background XSendEvent typing never lands in an unfocused GUI window (confirmed
in CI against both Tk and Chromium: the type call "succeeds" but no text
appears). Terminals are the lone exception, handled below the toolkit via the
pty master.
For GUI apps, fill the editable field through AT-SPI EditableText instead —
focus-free and toolkit-agnostic. type_text now tries, in order: pty master
(terminals) -> AT-SPI insert into the focused/first editable element (GUI) ->
XSendEvent (last resort, e.g. apps with no a11y tree). New atspi::insert_text
holds the EditableText logic.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(nix): AT-SPI harness for background-GUI input (zenity, chromium, firefox)
Reworks the GUI matrix to validate the focus-free AT-SPI typing path the driver
now uses, rather than X11 keystroke injection (which can't reach an unfocused
GUI widget).
- Stand up a session D-Bus at a fixed address and an AT-SPI bus
(at-spi-bus-launcher), shared via a common env so cua-driver's pyatspi and the
apps register with the same registry.
- Swap the un-accessible Tk app for zenity (a GTK app exposing AT-SPI).
- Enable accessibility for the browsers (chromium --force-renderer-accessibility,
firefox GNOME_ACCESSIBILITY=1).
- Read the typed text back through AT-SPI (queryText) — self-consistent with how
the driver writes — and still assert focus never left the control terminal.
Matrix jobs renamed tk -> gtk accordingly.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): env-prefix must precede timeout in the GUI type step
`timeout 120 DISPLAY=:99 ... python3` made timeout try to exec "DISPLAY=:99"
as the command (failed instantly). Move the env assignments before timeout so
they apply to the command. The AT-SPI bus, zenity launch, and window discovery
already worked in CI; this unblocks the actual type/readback steps.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): add pygobject3 so pyatspi readback can import `gi`
The AT-SPI readback helper failed with `ModuleNotFoundError: No module
named 'gi'` — pyatspi is a thin wrapper over PyGObject and needs it at
import time. The env-prefix fix got us past the type step; this unblocks
the readback verification.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(linux): native AT-SPI over D-Bus, replacing the pyatspi subprocess
The Linux accessibility path shelled out to `python3 -c "import pyatspi"`
for every tree walk, text insert, value set, action, and bounds query. That
bridge needs Python + pyatspi + PyGObject + GI typelibs at runtime, and under
Nix it broke at `import pyatspi` (missing `gi`, then a missing `DBus-1.0`
typelib). Worse, `type_text` swallowed the failure (`insert_text(...).unwrap_or(false)`)
and silently fell back to X11 XSendEvent, so focus-free typing wasn't actually
working — only the readback surfaced it.
Link AT-SPI directly via the `atspi` crate (zbus, pure Rust). A new
`atspi::native` module reimplements walk_tree / insert_text / set_value /
perform_action / get_element_bounds over D-Bus: it resolves the target app by
matching pid via `org.freedesktop.DBus.GetConnectionUnixProcessID`, walks the
tree depth-first/pre-order (identical element indexing and markdown format so
downstream parsing is unchanged), and uses the EditableText/Text/Action/Value/
Component proxies. The public functions stay synchronous (callers use
`spawn_blocking`) and drive a shared Tokio runtime.
No Python, pyatspi, PyGObject, or GI typelibs are required at runtime anymore.
Test: the background-GUI test verifies the typed text via the driver's own
`page`/`get_text` (same native path), and drops pythonAtspi/pygobject3 and the
pyatspi readback entirely.
cargoHash is set to a placeholder; the nix build will report the real value.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): set cua-driver cargoHash for the atspi/zbus dependency set
The nix build reported the expected fixed-output vendor hash; pin it so the
driver (and the GUI test that builds it) compiles against the new native
AT-SPI dependencies.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(linux): capture Text-interface content + timeouts in native AT-SPI walk
First end-to-end run of the native walk surfaced two issues:
- get_text returned empty for the editable: an entry's typed text lives in
the AT-SPI Text interface, but the walk only emitted name/value/actions.
Now read bounded Text content and use it as the display name when the
widget has no accessible name, so typed text shows up in get_text.
- Chromium's large, lazily-built tree could hang the walk forever (zbus
calls have no timeout). Add a 3s per-call timeout (skip the node on
timeout), a 25s overall walk budget, and a 5000-node cap.
Also add CUA_ATSPI_DEBUG diagnostics (app/pid match + node counts to stderr)
and have the test print the raw get_text response, so CI shows what the walk
actually found.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* perf(linux): parallelize AT-SPI node reads; fix GTK app registration
Diagnostics from the first working native run:
- Chromium resolved its app by pid and walked 211 nodes, but each walk took
~9s (fully sequential D-Bus round-trips), so the readback loop blew the
timeout. Issue the four independent per-node reads (role, name, state,
children) concurrently via join!, and only touch interface proxies when the
node actually advertises that interface.
- GTK app (zenity) registered 0 applications: its atk-bridge module wasn't on
GTK_PATH, so it never joined the AT-SPI registry. Point GTK_PATH at
at-spi2-atk. (Chromium uses its own AT-SPI impl, hence it registered.)
Test: trim the readback retry loop (8x, 1s) and raise the script timeout to
200s to accommodate larger trees.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(linux): target web-document editable for focus-free typing
Browsers expose multiple editables: the address bar (omnibox) sorts first in
the AT-SPI tree, but the field a user/agent wants when typing into a browser
is the page input. Track a per-node `in_web_doc` flag (inherited from a
"document web"/document ancestor) and prioritize the insert target as:
focused editable -> editable inside web content -> first editable. This makes
focus-free typing drive the page field for browser control, while leaving
single-field apps (e.g. a GTK dialog entry) unchanged.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(linux): target page editable for browsers; help GTK load a11y bridge
Browser write path: focus-free insert_text sorted to the first editable in
the tree, which in a browser is the address bar, not the page field. Track a
per-node "in web document" flag (inherited from a "document web"/document
ancestor) and prefer, in order: a focused editable, an editable inside web
content (the page's input), then the first editable. Single-field apps (a GTK
dialog entry) are unaffected. This is what lets the driver type into a page to
control a browser, rather than into chrome.
GTK registration: zenity registered 0 applications because a GTK3 app dlopens
libatk-bridge-2.0.so by soname to join the AT-SPI bus, and it wasn't on the
loader path in the manual session. Add at-spi2-atk to LD_LIBRARY_PATH.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): enable AT-SPI status for GTK; log editable counts
Two diagnostics-driven changes after confirming the native walk works:
- GTK3 apps only export their accessible tree when org.a11y.Status.IsEnabled
is true on the session bus (GNOME sets this via gsettings). The hand-rolled
session left it false, so zenity registered nothing. Set IsEnabled=true via
dbus-send right after launching the a11y bus, before the app starts.
- insert_text now logs node/editable/entry-role counts. The chromium run
walked 211 nodes but found zero EditableText editables (despite two `entry`
nodes), indicating browsers don't expose EditableText for background
windows; this makes that explicit in the logs.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* revert(test): drop org.a11y.Status IsEnabled dbus-send
Poking org.a11y.Bus in setup triggered D-Bus activation of a second
at-spi-bus-launcher that conflicted with the manually-launched one, so the
driver could no longer reach the registry — both chromium and gtk fell back
to the X11 tree with zero AT-SPI nodes. Revert to the prior working setup
(chromium registers and the native walk reads its 211-node tree); the GTK
registration gate needs a different, non-conflicting fix.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): enable a11y via gsettings keyfile so GTK app registers
GTK3 only exports its accessible tree when toolkit-accessibility is enabled.
Set org.gnome.desktop.interface toolkit-accessibility=true once, before the
bus launcher and apps start, using the keyfile GSettings backend with a shared
XDG_CONFIG_HOME. This avoids poking org.a11y.Bus at runtime (which previously
D-Bus-activated a conflicting at-spi-bus-launcher and broke the registry).
Adds glib (gsettings) + gsettings-desktop-schemas to the VM. Targets the GTK
write path; browser write (CDP) is a separate follow-up.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): fix GSettings schema lookup; make a11y enable non-fatal
The gsettings call failed with schema-not-found because NixOS installs
compiled schemas under share/gsettings-schemas/<pkg>/glib-2.0/schemas, not the
bare share/glib-2.0/schemas that XDG_DATA_DIRS pointed at. Set
GSETTINGS_SCHEMA_DIR to the real compiled-schema path, and run the enable as a
non-fatal step (logging set+get) so AT-SPI registration diagnostics still
surface even if it errors.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): enable AT-SPI by setting IsEnabled on the owned bus launcher
Per at-spi-bus-launcher source, it reports a11y enabled only after an AT
client registers an event listener or IsEnabled is set explicitly; it does
NOT read toolkit-accessibility at startup (it only writes it). GTK3 apps check
IsEnabled at startup and stay silent when false, so gsettings had no effect.
Set IsEnabled directly, but first wait until our manually-launched launcher
actually OWNS org.a11y.Bus (via the bus driver's NameHasOwner, which does not
activate the name). The earlier attempt poked org.a11y.Bus before it was
owned, D-Bus-activating a second launcher that broke the registry for every
app. With single ownership guaranteed, the Set reaches the live launcher.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): add Qt (PyQt5) app to the background-GUI a11y matrix
Adds a non-GTK toolkit data point for focus-free AT-SPI typing: a minimal
PyQt5 window with a focused QLineEdit titled cua-initial. Qt exposes it over
AT-SPI (EditableText) under QT_ACCESSIBILITY=1, so it exercises the same
focus-free insert + readback path as the GTK case via a different toolkit.
Wires it through flake.nix (app list) and the nix-build.yml matrix.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* Bump cua-driver-rs to v0.5.1
Patch: release the installer fix (#1803) — release + local installers + runtime
all use ~/.cua-driver, and either installer cleans up a prior local install +
sweeps the stale legacy ~/.cua-driver-rs home. Changelog Unreleased → 0.5.1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(linux): surface target app stdout/stderr after launch
The qt job timed out finding the window because the PyQt5 app never showed
one (likely a Qt xcb platform-plugin load error). Log /tmp/target.log a few
seconds after launch so the real cause is visible rather than a bare
window-find timeout.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* chore(cua-driver-rs): bake version 0.5.1 into install scripts [skip ci]
* test(linux): point PyQt5 at qtbase's xcb platform plugin
The qt app failed to launch: `qt.qpa.plugin: Could not find the Qt platform
plugin "xcb" in ""`. A bare `python3` PyQt5 invocation doesn't inherit
qtbase's plugin path. Export QT_PLUGIN_PATH / QT_QPA_PLATFORM_PLUGIN_PATH from
qt5.qtbase's qtPluginPrefix so the xcb plugin is found and the window appears.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): read back IsEnabled + dump launcher log (diagnostic)
Both GTK and Qt apps launch fine but register 0 AT-SPI applications, even
after setting org.a11y.Status.IsEnabled. Read the property back (print-reply)
and dump the at-spi-bus-launcher log to determine whether the Set is taking
effect or the toolkit bridges simply aren't activating in this session.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): force Qt AT-SPI bridge on (QT_LINUX_ACCESSIBILITY_ALWAYS_ON)
IsEnabled is confirmed true on the a11y bus, yet the Qt app still registers 0
applications — Qt's bridge isn't activating from the bus handshake in this
headless session. Set QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1 (and QT_ACCESSIBILITY=1)
in the qt launch to force Qt to export its accessible tree.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* Delete JOURNAL.md
* Delete JOURNAL_VIDEO.md
* test(linux): validate AT-SPI read path; document focus-free write limit
Per investigation, focus-free WRITE into a *background, unfocused* toolkit
window isn't reliably supported: toolkits gate editable accessibility on
focus/activation (Chromium exposes fields read-only over AT-SPI; an unfocused
Qt window exposes only its top node; a GTK app's atk-bridge doesn't register
in this headless session). Chromium's own AT-SPI impl does expose a full
read-only tree.
So assert the proven READ path: the driver's get_text returns the background
window's accessibility/structure (a window/frame/document node) for every app
in the matrix — native tree for Chromium, at least the window node (native or
X11 fallback) for the others. type_text is still exercised but its readback is
no longer asserted; the write-needs-focus limitation is documented inline.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): add focus-gate confirmation run (diagnostic, non-fatal)
After the focus-free assertions, activate the target window and re-run the
driver, logging the focused get_text and whether the typed text now reads
back. This directly confirms the finding that toolkits expose the editable
only when the window is focused. Non-fatal: it's evidence in the logs, not a
gate (behaviour differs per toolkit).
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* ci(linux): temporarily disable firefox background-GUI matrix job
Firefox times out at launch under the emulated CI VM (no KVM) — it never
surfaces its window within the wait, so the job fails before any AT-SPI
subtest runs. This is an environmental launch issue, not a driver problem,
and the browser/AT-SPI read path is already covered by the chromium job.
Drop "firefox" from the flake check list and comment out its workflow matrix
entry; the app definition is kept so it can be re-enabled once launch is made
reliable (longer timeout + pre-seeded first-run-free profile).
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): add CDP focus-free write override + Electron matrix job
Chromium/Electron expose their fields read-only over AT-SPI, so the driver
can't write into a background browser window through it. Add an approved
Chromium/Electron-specific override using the Chrome DevTools Protocol:
Input.insertText targets the page's focused DOM element regardless of OS
window focus, so it lands in the unfocused background window.
- chromium/electron launch with --remote-debugging-port + --remote-allow-origins
- new asserting subtest drives a stdlib-only CDP client (HTTP target discovery
+ minimal RFC-6455 WebSocket) to insertText into the background window and
reads it back, while asserting the control terminal keeps X focus
- add a minimal Electron app (Chromium-backed BrowserWindow) as a new matrix
job; like chromium it's read-only over AT-SPI and writable via CDP
- wire "electron" into the flake matrix
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): expand background-GUI matrix with qt6, gtk4, tk
Broaden toolkit/version coverage of the background-GUI a11y suite:
- qt6 (PyQt6): same AT-SPI bridge as qt5 on the current Qt major; sets the
lib/qt-6 plugin path and libxcb-cursor (Qt 6.5+ needs it headless)
- gtk4 (compiled C GtkEntry): GTK4 talks AT-SPI directly (no atk-bridge
module), contrasting the GTK3/zenity bridge path; cairo renderer + x11
backend keep it headless-safe
- tk (tkinter): negative control — Tk has no AT-SPI bridge, so get_text
degrades to the X11 window node, proving graceful handling of
non-accessible toolkits
All wired into the flake matrix as independent jobs.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* ci: run electron/gtk4/qt6/tk background-GUI jobs
The nix-build matrix is hardcoded here (not derived from flake.nix), so the
new flake checks added for electron, gtk4, qt6 and tk never ran in CI. Add
them to the matrix so the expanded suite executes, including the CDP
focus-free-write assertion on electron.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): accept text/entry nodes in the read assertion
Qt6's AT-SPI bridge exposes the editable even while unfocused, so the
driver's focus-free write lands and get_text returns a bare `text "..."`
node rather than a frame/window/document. Broaden the read-back assertion to
accept text/entry nodes too (also future-proofs gtk4, which exposes the
entry directly). The narrow frame/window/document check was the only reason
the qt6 job failed — the read (and write) actually worked.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(linux): add GTK3 focus-free write fallback via X11 click+type
GTK3's AT-SPI bridge gates EditableText on window/widget focus, so unfocused
background windows expose entry nodes in the tree (reads work) but not the
EditableText interface (writes fail). Qt6 exposes EditableText unconditionally.
This commit adds a GTK3-specific fallback: when insert_text finds an entry/text
role with Component bounds but no EditableText, it:
1. Gets the entry widget's screen coordinates via Component.GetExtents
2. Translates to window-local coords
3. Sends an X11 click to the entry's center to establish widget focus
4. Types via XSendEvent (now accepted by the internally-focused widget)
The window remains unfocused (control terminal keeps X focus), but the widget
receives and processes the keystrokes. This unblocks the gtk job in the
background-GUI test matrix.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat(linux): focus-free Tk writes via send command
Tk has no AT-SPI bridge, so background writes use Tk's `send` IPC instead.
The test app registers as "cua-tk-target" and the driver injects text by
spawning `wish` to send Tcl commands. This is the Tk-specific override
(like CDP for Chromium), proving non-accessible toolkits can support
focus-free input with bespoke paths.
- Add inject_tk_send() in platform-linux/input/mod.rs
- Wire it into type_text tool after AT-SPI, before XSendEvent fallback
- Update Tk test app to register with tk appname + name entry widget
- Add tkSubtest that asserts the write lands and focus stays put
- Include pkgs.tk so wish is available in the test environment
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat(linux): add GTK3/GTK4 focus-free write fallback via X11 click+type
GTK3 and GTK4's AT-SPI bridge gates EditableText on window/widget focus, so unfocused
background windows expose entry nodes in the tree (reads work) but not the
EditableText interface (writes fail). Qt6 exposes EditableText unconditionally.
This commit adds a GTK fallback: when insert_text finds an entry/text
role with Component bounds but no EditableText, it:
1. Gets the entry widget's screen coordinates via Component.GetExtents
2. Translates to window-local coords
3. Sends an X11 click to the entry's center to establish widget focus
4. Types via XSendEvent (now accepted by the internally-focused widget)
The window remains unfocused (control terminal keeps X focus), but the widget
receives and processes the keystrokes. This unblocks the gtk3 and gtk4 jobs in the
background-GUI test matrix.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat(linux): use AT-SPI Component.GrabFocus for GTK4 focus-free writes
GTK4 gates EditableText on widget focus, unlike Qt6 which exposes it
regardless of focus state. When a GTK4 window is in the background, the
AT-SPI tree contains entry/text widgets (so reads work) but EditableText
is unavailable, blocking focus-free writes.
Call Component.GrabFocus on the target widget before accessing EditableText.
This gives the widget internal keyboard focus without activating its window,
allowing GTK4 to expose EditableText on the focused widget. The approach is:
1. Find target editable widget (same priority as before)
2. If it has Component interface, call GrabFocus on it
3. Proceed to call EditableText.InsertText as usual
Benefits:
- No window activation: GrabFocus works at widget level, not window level
- Toolkit-agnostic: Component.GrabFocus is standard AT-SPI
- Non-breaking: if GrabFocus fails/unavailable, still try EditableText (Qt6+)
- Diagnostic logging shows GrabFocus success/failure for debugging
This should allow the gtk4 background-GUI test to pass with true focus-free
writes: the control terminal stays active throughout, the GTK4 entry gains
internal focus via GrabFocus, and EditableText.InsertText succeeds.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* ci: generate GIF artifacts for all background GUI tests
- Set visual: true for gtk, qt, qt6, gtk4, chromium, electron, tk tests
- Add artifact_name for each test so GIFs are uploaded
- Update PR comment script to list all new artifacts
This will make it easy to visually verify focus-free writes work correctly
for each toolkit by watching the GIF showing the window staying unfocused.
* feat(linux): enable focus-free background writes for Qt5 via synthetic focus events
Adds three-tier typing strategy for Linux:
1. Native AT-SPI EditableText (Qt6, GTK4 focus-free)
2. Synthetic FocusIn → AT-SPI → FocusOut (Qt5 workaround)
3. X11 XSendEvent fallback (terminal/legacy apps)
The synthetic-focus path sends FocusIn to trigger Qt5's AT-SPI bridge
without changing the X11 active window, enabling focus-free writes.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: restore GTK3 fallback code after merge conflict resolution
The GTK3 widget click fallback was accidentally removed when resolving
the merge conflict for PR #1817. This restores the entry_find_window_xid
and screen_to_window_coords helpers and the GTK3 X11 click+type fallback
logic that enables focus-free writes for GTK3 (zenity).
* fix(platform-linux): qualify Command in atspi python fallback
The merge-conflict resolution that restored type_into_editable's pyatspi
fallback reintroduced `Command::new("python3")` without a
`use std::process::Command;` import, breaking the cua-driver build
(E0433: cannot find type `Command`) and thus every nix CI job. Fully-qualify
the call as `std::process::Command::new` (matching the style in tools/impl_.rs)
to restore compilation without touching imports.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
---------
Co-authored-by: Francesco Bonacci <f@trycua.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: hippoley <hippoley@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: trycua-release[bot] <trycua-release[bot]@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
r33drichards
added a commit
that referenced
this pull request
Jun 5, 2026
* docs(cua-driver): add changelog reference page (#1785)
Mirror the cua-driver-rs GitHub releases into the docs site so the
release history is discoverable on the docs site (not just GitHub),
matching the convention used by the other products (cua CLI, lume).
Wire it into the reference nav.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver)(macos): guard SkyLight auth-message selector for macOS 14 Sonoma (#1503) (#1782)
`hotkey`, `press_key`, and `scroll` crash the daemon on macOS 14 (Sonoma)
with `NSInvalidArgumentException: +[SLSEventAuthenticationMessage
messageWithEventRecord:pid:version:]: unrecognized selector sent to class`.
The class `SLSEventAuthenticationMessage` exists on macOS 14, but the
`messageWithEventRecord:pid:version:` factory selector was only added in
macOS 15 (Sequoia). The existing `!cls.is_null() && !sel.is_null()` guard
is insufficient: `sel_registerName` / `NSSelectorFromString` always succeed
(they just intern the string), so `objc_msgSend` still dispatches an
unimplemented selector and the ObjC runtime aborts the process.
Guard the dispatch with `class_respondsToSelector` (Rust) /
`messageClass.responds(to:)` (Swift), which actually checks the metaclass.
On macOS 14 it returns false, so we skip the auth envelope and fall through
to plain `SLEventPostToPid`. Chromium-class targets may not receive the
event on macOS 14, but the daemon no longer crashes — graceful degradation.
This re-applies the fix from #1579 (by @hippoley) onto the current
`libs/cua-driver/{rust,swift}/` layout — #1579 predates the #1674
directory restructure and no longer merges.
- rust: platform-macos/src/input/skylight.rs — class_responds_to_selector()
- swift: CuaDriverCore/Input/SkyLightEventPost.swift — responds(to:) guard
Verified: platform-macos + the full cua-driver binary build; the Swift
`responds(to:)` form compiles and returns true for an existing class method,
false for an absent one.
Closes #1503
Co-authored-by: hippoley <hippoley@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs)(macos): enable Chromium/Electron AX trees for get_window_state (#1756)
Chromium/Electron apps (Arc, VS Code, Electron shells) ship their web-content
accessibility tree off and only build it once an assistive client requests it.
Without enablement the first AX walk returns an empty/title-bar-only tree.
Flip AXManualAccessibility (modern, side-effect-free) on the application root,
falling back to AXEnhancedUserInterface when the modern attribute is
unsupported. When the flip actually takes, let the asynchronously-built tree
settle (~500ms run-loop pump) before walking. Cache per-pid so repeat snapshots
skip the settle. Native Cocoa apps reject the attribute and pay no cost.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bump cua-driver-rs to v0.4.2
* docs(cua-driver): add 0.4.2 changelog entry + fix 0.3.6 wording (#1786)
- Add 0.4.2: macOS 14 Sonoma SkyLight selector guard (#1782, #1503) and
Chromium/Electron AX trees via AXManualAccessibility (#1756).
- Fix the 0.3.6 entry, which described the permissions-status fix backwards:
it now reports the driver's grants (via the daemon), not the caller's.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* chore(cua-driver-rs): bake version 0.4.2 into install scripts [skip ci]
* fix(cua-driver-rs): wire/guide per-session cursors through the real mcp path + skills/docs (#1787)
* fix(cua-driver-rs): wire/guide per-session cursors through the real mcp path + update skills/docs
A user drove `cua-driver mcp --claude-code-computer-use-compat` (the
documented Claude Code install) and asked: (1) why no agent cursor even
on AX actions, (2) where is the session in the mcp calls, (3) did we
forget the CLI / MCP / skills wiring.
Investigation + fixes:
- Session IS wired (working as designed): the proxy path the user runs
mints one session_id per MCP connection and stamps it on every
forwarded request; the daemon injects it as `_session_id` into tool
args and strips it from the user-visible wire envelope. Per-session
cursor / config / recording are live on the compat proxy path —
verified headless (set_agent_cursor_enabled{false} in a session is
read back by get_config{enabled:false}, proving _session_id reached
the daemon).
- BUG (user-visible): no glide on a pure-AX run. A brand-new session
cursor sat at the off-screen sentinel; animate_cursor_to early-returned
so the first AX action only snapped a static arrow via ClickPulse —
easy to miss. Fix: seed the sentinel cursor on-screen (offset, clamped)
before animating so the FIRST action glides. Get-or-create + ended
tombstone guard so it never resurrects a reaped session. Unit-tested.
- BUG (latent wiring): `--claude-code-computer-use-compat` was silently
dropped on the proxy path (daemon hardcoded compat=false). Thread it
end-to-end: proxy forwards `serve --claude-code-computer-use-compat`,
the Serve arm honours it via build_macos_registry_with_compat. Today
this has no tool-surface effect (the compat screenshot tool was removed
in #1692) but the flag now travels for any future compat-gated tool.
- BUG (nondeterministic): get_config reported agent_cursor.enabled from a
HashMap .first(). Resolve the calling session's cursor by key
(cursor_id > _session_id > "default"). Unit-tested per-session.
Docs/skills (no default change — that is the user's call; see PR body):
SKILL.md (per-session model, session_end removal, AX no-glide caveat,
corrected the false "AX skips the overlay" claim), set_agent_cursor_enabled
description, protocol.rs server-instructions, CLI help (cursor flags +
overlay + compat), mcp-tools.mdx AX-snap caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver)(skills): correct the AX cursor caveat — short glide, not no glide
After the sentinel-seed fix the first AX action seeds the cursor on-screen
near the target and plays a brief glide + pulse (not "does not glide").
Reword the SKILL.md visibility caveat to match the actual behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(macos): run the agent-cursor overlay in the serve daemon (#1790)
The overlay NSWindow + AppKit render loop were only wired into the in-process
`mcp` arm. In the daemon-proxy setup users run (`mcp` relaunches
`open -n -g … serve` and proxies to it for correct TCC), the DAEMON performs
the clicks/AX presses but never inited or ran the overlay — its main thread
parked in `serve_handle.join()`. So `set_agent_cursor_enabled` flipped registry
flags and clicks sent OverlayCommands, but CMD_TX/RENDER were never set →
every cursor command was a silent no-op and the agent cursor never appeared.
Fix: the Serve arm now builds cursor_cfg, inits the overlay channel before
spawning the serve thread, and (when enabled) parks main in
`overlay::run_on_main_thread()` (mirrors the Mcp arm) instead of join. It
self-guards on has_graphic_access() and falls back to join when there's no
Window Server session, so headless serving is unaffected. PiP unchanged.
Verified via the REAL launch path: `open -n -g -a CuaDriver --args serve`
daemon's main thread now runs __CFRunLoopRun / -[NSApplication run] with
run_appkit + SkyLight + tiny_skia overlay rendering, and still serves.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(macos): stop the permissions gate spamming the TCC prompt on every re-exec (#1791)
`cua-driver permissions grant` (and any first-launch serve) raises the system
TCC prompt, then re-execs the daemon ~every 25s to refresh the per-process
AXIsProcessTrusted cache. Each re-exec'd process re-ran run_if_needed and
re-raised request_accessibility/request_screen_recording — so a fresh "Cua
Driver" dialog popped every ~25s. Worse, the 10-min deadline was anchored to
each process's own start, and since the re-exec fires (~25s) well before the
deadline, the deadline never triggered: the gate re-execed (and restarted the
whole daemon, now incl. the cursor overlay) forever whenever the grant read as
missing — including the stale-ad-hoc-cdhash case (Settings shows granted but
the rebuilt binary's hash no longer matches, so the live check returns false).
Fix:
- reexec_self sets CUA_DRIVER_RS_GATE_REEXEC=1; run_if_needed sees it and polls
SILENTLY (skips the prompts + panel) on re-exec'd processes. The prompt +
panel appear exactly once, on first launch.
- reexec_self persists the original gate start in CUA_DRIVER_RS_GATE_START_UNIX;
wait_for_grants anchors `start` to it so the deadline is cumulative across
re-execs and the gate actually gives up (and stops churning) after the
deadline, continuing to serve (tools fail with TCC errors until granted).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs)(install-local): sign the bundle with a stable self-signed identity so TCC grants survive rebuilds (#1792)
install-local ad-hoc-signed the bundle (`codesign --sign -`), which keys the
TCC grant (Accessibility / Screen Recording) on the binary's cdhash. The
cdhash changes on EVERY rebuild, so each install-local silently invalidated the
grant — System Settings still showed "CuaDriver ✅" (it's keyed on the bundle
id) while the live AXIsProcessTrusted check failed, and the daemon re-prompted
("I already granted!"). A genuinely miserable dev loop.
Fix: create a self-signed code-signing certificate once (idempotent, in the
login keychain) and sign the bundle with it. TCC then keys the grant on the
certificate leaf — stable across rebuilds — so the Designated Requirement
becomes `identifier "com.trycua.driver" and certificate leaf = H"..."` instead
of a cdhash pin. Grant once; every future install-local keeps it.
Robust + fail-soft: openssl 3.x needs `-legacy` PBE + a real p12 password for
Apple's `security import` (the empty-password default fails MAC verification);
falls back to non-legacy for LibreSSL. If the cert can't be created (no
openssl, locked keychain, CI), falls back to ad-hoc signing + a one-line note.
Local dev only — releases are CI-signed and already stable.
One-time migration: switching from ad-hoc to the cert changes the requirement
once, so the next grant after this lands is a single re-grant; stable after.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Bump cua-driver-rs to v0.4.3
* docs(cua-driver): add 0.4.3 changelog entry (#1793)
cursor overlay in the daemon (#1790), permissions-grant prompt no-spam (#1791),
and install-local stable signing identity (#1792).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* chore(cua-driver-rs): bake version 0.4.3 into install scripts [skip ci]
* fix(cua-driver-rs)(install-local): reset a TCC grant pinned to a previous signing identity (#1795)
Accessibility / Screen-Recording grants survive rebuilds — but only for grants
CREATED while cert-signed. A grant the user made earlier on an ad-hoc build is
pinned to that build's cdhash (the stored csreq is a bare `cdhash H"..."`), so it
survives reinstall with auth_value=allowed yet stops matching the new binary. The
daemon then reads "not granted" while System Settings still shows CuaDriver toggled
ON — a dead end, because the row already records a decision so re-toggling never
re-fires the prompt.
Record the signing identity (cert leaf, or "adhoc") in
~/.cua-driver/.tcc-signing-identity. When the installer signs with a cert identity
that differs from the last install, `tccutil reset` Accessibility + ScreenCapture
once so the next `permissions grant` prompts cleanly and re-pins to the stable
cert (after which grants survive every future rebuild). `tccutil reset` needs no
sudo/FDA and is a no-op when nothing was granted. We only reset when moving TO a
cert identity — an ad-hoc build churns its cdhash regardless, so resetting it would
add friction with no durable fix.
Docs: FAQ entry for "granted but reports NOT granted after a rebuild" + changelog.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(macos): retain cached AX element across action so concurrent sessions can't UAF-crash the daemon (#1796)
Two sessions driving the same window concurrently crashed the daemon with
EXC_BREAKPOINT (SIGTRAP) inside AXUIElementCopyActionNames → _AXUIElementValidate
→ CFGetTypeID — a use-after-free.
Root cause: the per-(pid, window_id) element cache (ax/cache.rs) handed out raw
AXUIElementRef pointers as usize. A tool (click/type_text/set_value/…) copied the
pointer out from under the cache lock and used it across await points and on a
blocking thread. Meanwhile another session's get_window_state called
ElementCache::update → ElementCacheCore::insert, which replaced the snapshot and
ran CachedSnapshot::drop on the old one — CFRelease-ing those exact pointers to
zero. The in-flight action then dereferenced freed memory.
Fix: replace get_element_ptr with get_element_retained, which CFRetains the
element while still holding the cache lock and returns a RetainedElement guard
(CFRelease on drop). An in-flight action holds the guard for its whole duration,
so a concurrent snapshot replace can't free the element under it. Migrated all
nine element-action call sites (click, right_click, double_click, type_text,
type_text_chars, press_key, scroll, set_value, recording_hooks).
Test: ax::cache::tests::retained_element_survives_concurrent_snapshot_replace
asserts the retain accounting — after a concurrent replace the guard's retain is
what keeps the element alive (count = base+1, not base). 74/74 platform-macos
lib tests pass.
Note: platform-windows has the same shape (uia/cache.rs::get_element_ptr hands
out raw IUIAutomationElement pointers); a mirrored AddRef-on-get fix is a
follow-up, not included here (untestable in this environment).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver-rs)(launch_app): surface creates_new_application_instance for concurrent multi-agent isolation (#1797)
launch_app is idempotent, so two sessions launching the same app get the same
instance — and on single-instance apps (Calculator, many utilities) the same
window — and clobber each other. The `creates_new_application_instance` param
already solves this (it maps to NSWorkspaceOpenConfiguration.createsNewApplicationInstance,
the programmatic `open -n`), but nothing told an agent to reach for it in the
concurrent case. Enrich the tool description, the MCP-tools doc, and the skill's
action-loop section to call out the concurrent-session use. No behavior change.
Verified end-to-end: two launch_app(name=Calculator, creates_new_application_instance=true)
calls return distinct pids + distinct window_ids.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs): caller-declared session identity + Streamable-HTTP transport for multi-agent parallelism (#1798)
* feat(cua-driver-rs): explicit session identity core + cursor explicit-required
- core/session.rs: touch_session/end_session/evict_idle + idle-TTL activity map
- serve.rs: apply_session_identity at the daemon boundary (explicit `session` →
_session_id; minted id is recording/config fallback only, not a cursor source)
- cursor: resolve_cursor_key returns NO_CURSOR("") when no session declared;
overlay + registry short-circuit the empty key (explicit-required cursor)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs): start_session/end_session tools + idle-TTL sweep + session schema
- core/session_tools.rs: start_session / end_session tools (cross-platform),
registered via ToolRegistry::register_session_tools on all 3 platforms
- serve.rs: spawn_session_idle_sweep — evict_idle every 30s (TTL default 300s,
CUA_DRIVER_RS_SESSION_IDLE_TTL_SECS override)
- inject session property into action-tool schemas; fix set_agent_cursor_enabled
description (cursor is explicit-required now, not auto-per-MCP-session)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver-rs): document explicit session identity (MCP instructions, SKILL, mcp-tools, changelog)
- MCP server instructions: add start_session step + explicit-session cursor model
- SKILL.md: canonical loop gains start_session/end_session; fix concurrent note
(cursor keyed on session, not (pid,window_id))
- mcp-tools.mdx: rewrite per-session cursor section; add start_session/end_session
- changelog: breaking session-identity entry
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs): stop a session's recording on session_end (end_session/idle-TTL/EOF)
Register a session_end hook that calls recording.stop_owner(Some(sid)) on a
detached thread, so end_session and the idle-TTL sweep tear down a session's
recording too (matching end_session's contract) — not just the EOF path. Safe:
stop_owner(Some) is a no-op unless that session owns the live recording, and the
detached thread keeps mp4 finalize off the synchronous fire_session_end caller.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(cua-driver-rs): unit-test apply_session_identity boundary (explicit/minted/anonymous)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(macos): move_cursor visibly moves the drawn cursor (seed sentinel like click)
move_cursor sent a raw MoveTo, which doesn't bring a brand-new session cursor
on-screen — it sits at the off-screen sentinel until a click seeds it, so the
DRAWN cursor never moved (only the reported position did). Use animate_cursor_to
(the same path click uses): it seeds the sentinel on-screen then glides in.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs)(macos): mark move_cursor read-only so MCP clients can parallelize cursor moves
move_cursor only nudges the agent-cursor overlay, never the target app, so it is
concurrency-safe. read_only:true emits readOnlyHint, which Claude Code's
isConcurrencySafe() uses to run cursor moves in parallel. Mutating tools
(click/type_text/press_key) stay read_only:false on purpose — parallelizing an
ordered intra-agent sequence would race.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs): Streamable-HTTP MCP transport on the daemon for parallel multi-agent (#1799)
Over stdio, one cua-driver mcp process is a single pipe, so a client's tool calls
(incl. multiple subagents) serialize. The daemon is already concurrent (task per
connection). This adds an HTTP MCP front-end so each agent opens its OWN
connection: per-connection FIFO keeps a single agent's ordered calls correct,
distinct connections run truly in parallel — safe because per-(pid,window) caches
+ per-session cursors make concurrent cross-connection actions non-colliding.
- mcp_http.rs: hand-rolled HTTP/1.1 (no new deps, mirrors the UDS line protocol),
POST -> cua_driver_core::server::handle_request (now pub) -> application/json
JSON-RPC. Task per TCP connection; honors Connection: close; mirrors the
"session" arg -> _session_id + touches idle-TTL so HTTP == stdio behavior.
- opt-in via CUA_DRIVER_RS_MCP_HTTP_PORT (loopback only); spawned from run_serve.
Proven: 10 list_apps over 10 concurrent connections = 3.6s vs 12.9s sequential
(3.6x). curl initialize/tools/list/tools/call all correct. 3 unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver-rs): document HTTP MCP transport + the concurrency model
- changelog: Streamable-HTTP transport + move_cursor readOnlyHint
- FAQ: "Concurrency & multiple agents" — why subagents serialize (shared stdio
pipe), and how to run agents truly in parallel (separate connections / the
CUA_DRIVER_RS_MCP_HTTP_PORT HTTP endpoint)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(cua-driver-rs)(skill): note subagent serialization + HTTP transport for parallel agents
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cua-driver-rs)(windows): per-session agent cursors (port macOS #1779) (#1801)
The Windows overlay was a process-wide singleton (one `RenderState`), so
concurrent MCP sessions clobbered each other last-writer-wins → one shared
cursor. #1779 fixed this on macOS but explicitly left Windows/Linux on the
old single-cursor model ("the key concept never reaches them").
Port the keyed render collection to platform-windows:
- overlay.rs: `RenderMap { IndexMap<CursorKey, RenderState> }`; `send_command`
now carries a `CursorKey`; the WM_TIMER tick drains keyed `OverlayMsg`s,
ticks every cursor, and composites them all into the ONE layered window via
`paint_cursor` (insertion order = stable z-order). Per-key arrival isolation,
lazy per-key palette (`Palette::for_instance`), `remove_cursor` + render-side
resurrection tombstone, and the sentinel seed — all mirroring
platform-macos/src/cursor/overlay.rs.
- tools/impl_.rs: `resolve_cursor_key` (session > cursor_id > NO_CURSOR, never
the connection `_session_id`), threaded through `pin_overlay_above`,
`overlay_glide_to`, every ClickPulse callsite, and the 5 cursor tools. A
`session_end` hook (once-guarded) calls `remove_cursor`; `get_config`'s
`cursor_enabled` is now session-scoped + deterministic (was a
nondeterministic `all_states().first()` — macOS BUG 3).
- cursor-overlay: `CursorRegistry::remove` (guards "default").
page.click_element keeps the seeded "default" cursor — the cross-platform
`PageBackend` trait carries no caller session (separate follow-up).
15 new headless unit tests (two-session isolation, session_end removal,
default guard, resurrection tombstone, sentinel seed, key resolution); full
platform-windows lib suite green (49 tests), daemon builds warning-free.
Verified live on Windows 11: two calculators driven by two sessions show two
distinct-coloured cursors gliding in parallel; end_session removes each.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Bump cua-driver-rs to v0.5.0
Release the caller-declared session identity + Streamable-HTTP multi-agent
transport (#1798) and Windows per-session cursors (#1801). Breaking: the agent
cursor is now opt-in (declare a `session`). Changelog Unreleased → 0.5.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(cua-driver-rs): bake version 0.5.0 into install scripts [skip ci]
* fix(cua-driver-rs): release installer unifies home on ~/.cua-driver + cleans up prior local install (#1803)
The release installer (install.sh → _install-rust.sh) defaulted its package
home to the legacy ~/.cua-driver-rs, but the local installer
(_install-local-rust.sh) and the runtime already use ~/.cua-driver (renamed in
v0.2.16 / PR #1644). That mismatch is the root cause of a two-install collision:
a user who ran install-local and then the release install.sh ended up with two
homes and two conflicting installs, with the local build's artifacts left
dangling.
Fixes in _install-rust.sh:
- Default HOME_DIR to ~/.cua-driver (still honoring CUA_DRIVER_RS_HOME for
back-compat), matching install-local + runtime.
- Before staging: cleanup_prior_local_install() stops the daemon and removes
the prior install-local artifacts under the shared home — the `*-local-*`
release dirs and the ~/.cua-driver/.tcc-signing-identity marker. Marker-gated
and conservative: never touches a real release dir, the `current` symlink, or
unrelated user state; best-effort + idempotent (no-op on a clean machine).
- After staging: sweep a stale ~/.cua-driver-rs left by an older release,
mirroring the belt-and-braces legacy-home sweep install-local already does.
- TCC grants preserved: /Applications/CuaDriver.app is replaced in place via
the existing release ditto (grants key on the shared com.trycua.driver bundle
id); no tccutil reset, so cert-pinned grants are not churned.
install.ps1 (Windows) already defaults to ~/.cua-driver and migrates the legacy
home, so it is unchanged.
Docs: reconcile the ~/.cua-driver-rs → ~/.cua-driver home references across the
installation + linux guides, document the local/legacy cleanup behavior, and add
an Unreleased changelog entry.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(cua-driver-rs)(linux): generalize background keyboard input via XTEST
The background-terminal work special-cased terminals: type_text and
press_key(Enter) detected a terminal process, found its /dev/pts tty, and
shoved bytes in with the legacy TIOCSTI ioctl. That only ever worked for
terminals, and TIOCSTI is exactly the mechanism modern kernels harden away
(CONFIG_LEGACY_TIOCSTI / dev.tty.legacy_tiocsti), so it would EPERM on many
systems. It also left the XTEST scaffold added alongside it as dead code.
Replace the terminal-specific path with a general one. Keyboard input now
goes through XTEST for every window: XSendEvent keystrokes carry the
send_event flag that xterm (and friends) deliberately ignore, which is why
typing into a background terminal silently did nothing; XTEST injects at the
server level with no such flag, so it lands on terminals and every other app
alike. Because XTEST targets the focused window, with_focus briefly focuses
the target, injects, and restores the prior focus — preserving the same
no-focus-steal contract the XSendEvent pointer path keeps.
- input/mod.rs: send_type_text / send_type_text_with_delay / send_key now
use XTEST (with real Shift presses for shifted chars and held modifiers),
wiring up the previously-dead xtest_* helpers. Pointer (click/drag) stays
on XSendEvent.
- impl_.rs: drop inject_terminal_input + is_terminal_process /
terminal_*_tty helpers and the TIOCSTI ioctl, and the type_text / press_key
branches that called them.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(cua-driver-rs)(linux): restore active window after XTEST injection
The background-terminal GIF test injected fine but failed its focus check:
typing landed in the inactive xterm, yet focus ended on the target instead
of returning to the control terminal. XTEST delivers to the focused window,
so with_focus moves focus to the target to inject — but the restore used a
bare SetInputFocus, and under an EWMH WM (openbox) `xdotool getactivewindow`
reads `_NET_ACTIVE_WINDOW`, which the WM owns and doesn't update from a raw
SetInputFocus. So focus never came back.
Restore cooperatively: capture `_NET_ACTIVE_WINDOW` up front and re-activate
it afterwards with a `_NET_ACTIVE_WINDOW` client message (source = 2, the same
nudge `xdotool windowactivate` sends), keeping SetInputFocus for the no-WM
case. Add a short settle after each focus/activation request so the
asynchronous WM acts before we inject or restore.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): restore Cargo.lock to keep cargoHash valid
A stray `cargo check` re-bumped the workspace crates in Cargo.lock from
0.4.0 to 0.4.1 (matching the manifests) and it got committed. Nixpkgs'
fetchCargoVendor hashes the vendored directory, which includes a copy of
Cargo.lock, so the changed lock invalidated the pinned cargoHash and broke
the cua-driver build — and with it every NixOS VM test that builds the
driver. Restore Cargo.lock to the base/known-good revision.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(cua-driver-rs)(linux): focus-free input — XSendEvent for GUI, pty master for terminals
Replaces the XTEST-with-temporary-focus approach (which broke the
cross-platform "no focus steal" contract that macOS SLEventPostToPid and
Windows PostMessage uphold) with two focus-free paths:
- GUI apps: XSendEvent, as before, but the typing path now resolves the
shift level from the keyboard map so uppercase / shifted symbols inject
correctly (previously "A" was sent as "a"). Removed the dead XTest scaffold.
- Terminals: instead of the legacy TIOCSTI ioctl (which dev.tty.legacy_tiocsti
disables on modern kernels), borrow the emulator's pty master fd via
pidfd_getfd(2) and write to it. The kernel delivers the bytes to the shell's
stdin exactly as typed — no X focus change, immune to the TIOCSTI sysctl.
pidfd_getfd needs ptrace-mode access, which under the default ptrace_scope=1
is granted for the caller's own descendants — i.e. terminals the driver
launched — with no root and no special capability. For terminals the driver
did not launch it returns Ok(false) and the caller falls back; injecting into
someone else's terminal unprivileged is what the kernel deliberately prevents.
New module crate::tty holds the master-borrow logic.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(nix): matrix background-GUI input coverage (chromium, firefox, tk)
Adds a parameterized NixOS VM test proving cua-driver types into a GUI window
via XSendEvent WITHOUT stealing focus — the general computer-use claim, beyond
terminals. Each app shows a focused text field that mirrors what it receives
into its X11 window title; the test types a known string into the *inactive*
app window (no click/focus first) and asserts the title became that string
(input landed) and a separate control terminal stayed active (no focus steal).
Wired as one independent matrix job per app (chromium, firefox, tk) in
flake.nix checks and the nix-build workflow, so coverage spans a Chromium web
engine, a Gecko web engine, and a native Tk toolkit.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): use python3 + tkinter for the tk GUI test (python3Full removed)
nixpkgs removed python3Full ("tkinter is available within the package set"),
which broke flake evaluation of the tk matrix job. Use
python3.withPackages (ps: [ ps.tkinter ]) instead.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): background-GUI test — file:// page, exec launchers, find window by name
Two harness bugs the matrix run surfaced (driver logic unaffected):
- The browser launch commands embedded a data: URL whose double quotes
collided with the testScript's Python/shell quoting, so the nixos test
driver rejected the script with "invalid-syntax". Serve the page from a
file:// URL written via writeText and move each launch into a writeShellScript
that exec's the app, so the testScript only ever embeds a quote-free path.
- Window discovery used `xdotool search --pid`, which needs _NET_WM_PID — Tk
doesn't set it and browser window pids differ from the launcher, so the
search hung to timeout. Give every app a known initial window title
("cua-initial") and discover by --name instead.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(cua-driver-rs)(linux): type into GUI apps via AT-SPI (focus-free)
X11 only routes keystrokes to the focused toplevel's focused widget, so
background XSendEvent typing never lands in an unfocused GUI window (confirmed
in CI against both Tk and Chromium: the type call "succeeds" but no text
appears). Terminals are the lone exception, handled below the toolkit via the
pty master.
For GUI apps, fill the editable field through AT-SPI EditableText instead —
focus-free and toolkit-agnostic. type_text now tries, in order: pty master
(terminals) -> AT-SPI insert into the focused/first editable element (GUI) ->
XSendEvent (last resort, e.g. apps with no a11y tree). New atspi::insert_text
holds the EditableText logic.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(nix): AT-SPI harness for background-GUI input (zenity, chromium, firefox)
Reworks the GUI matrix to validate the focus-free AT-SPI typing path the driver
now uses, rather than X11 keystroke injection (which can't reach an unfocused
GUI widget).
- Stand up a session D-Bus at a fixed address and an AT-SPI bus
(at-spi-bus-launcher), shared via a common env so cua-driver's pyatspi and the
apps register with the same registry.
- Swap the un-accessible Tk app for zenity (a GTK app exposing AT-SPI).
- Enable accessibility for the browsers (chromium --force-renderer-accessibility,
firefox GNOME_ACCESSIBILITY=1).
- Read the typed text back through AT-SPI (queryText) — self-consistent with how
the driver writes — and still assert focus never left the control terminal.
Matrix jobs renamed tk -> gtk accordingly.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): env-prefix must precede timeout in the GUI type step
`timeout 120 DISPLAY=:99 ... python3` made timeout try to exec "DISPLAY=:99"
as the command (failed instantly). Move the env assignments before timeout so
they apply to the command. The AT-SPI bus, zenity launch, and window discovery
already worked in CI; this unblocks the actual type/readback steps.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): add pygobject3 so pyatspi readback can import `gi`
The AT-SPI readback helper failed with `ModuleNotFoundError: No module
named 'gi'` — pyatspi is a thin wrapper over PyGObject and needs it at
import time. The env-prefix fix got us past the type step; this unblocks
the readback verification.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(linux): native AT-SPI over D-Bus, replacing the pyatspi subprocess
The Linux accessibility path shelled out to `python3 -c "import pyatspi"`
for every tree walk, text insert, value set, action, and bounds query. That
bridge needs Python + pyatspi + PyGObject + GI typelibs at runtime, and under
Nix it broke at `import pyatspi` (missing `gi`, then a missing `DBus-1.0`
typelib). Worse, `type_text` swallowed the failure (`insert_text(...).unwrap_or(false)`)
and silently fell back to X11 XSendEvent, so focus-free typing wasn't actually
working — only the readback surfaced it.
Link AT-SPI directly via the `atspi` crate (zbus, pure Rust). A new
`atspi::native` module reimplements walk_tree / insert_text / set_value /
perform_action / get_element_bounds over D-Bus: it resolves the target app by
matching pid via `org.freedesktop.DBus.GetConnectionUnixProcessID`, walks the
tree depth-first/pre-order (identical element indexing and markdown format so
downstream parsing is unchanged), and uses the EditableText/Text/Action/Value/
Component proxies. The public functions stay synchronous (callers use
`spawn_blocking`) and drive a shared Tokio runtime.
No Python, pyatspi, PyGObject, or GI typelibs are required at runtime anymore.
Test: the background-GUI test verifies the typed text via the driver's own
`page`/`get_text` (same native path), and drops pythonAtspi/pygobject3 and the
pyatspi readback entirely.
cargoHash is set to a placeholder; the nix build will report the real value.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(nix): set cua-driver cargoHash for the atspi/zbus dependency set
The nix build reported the expected fixed-output vendor hash; pin it so the
driver (and the GUI test that builds it) compiles against the new native
AT-SPI dependencies.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* fix(linux): capture Text-interface content + timeouts in native AT-SPI walk
First end-to-end run of the native walk surfaced two issues:
- get_text returned empty for the editable: an entry's typed text lives in
the AT-SPI Text interface, but the walk only emitted name/value/actions.
Now read bounded Text content and use it as the display name when the
widget has no accessible name, so typed text shows up in get_text.
- Chromium's large, lazily-built tree could hang the walk forever (zbus
calls have no timeout). Add a 3s per-call timeout (skip the node on
timeout), a 25s overall walk budget, and a 5000-node cap.
Also add CUA_ATSPI_DEBUG diagnostics (app/pid match + node counts to stderr)
and have the test print the raw get_text response, so CI shows what the walk
actually found.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* perf(linux): parallelize AT-SPI node reads; fix GTK app registration
Diagnostics from the first working native run:
- Chromium resolved its app by pid and walked 211 nodes, but each walk took
~9s (fully sequential D-Bus round-trips), so the readback loop blew the
timeout. Issue the four independent per-node reads (role, name, state,
children) concurrently via join!, and only touch interface proxies when the
node actually advertises that interface.
- GTK app (zenity) registered 0 applications: its atk-bridge module wasn't on
GTK_PATH, so it never joined the AT-SPI registry. Point GTK_PATH at
at-spi2-atk. (Chromium uses its own AT-SPI impl, hence it registered.)
Test: trim the readback retry loop (8x, 1s) and raise the script timeout to
200s to accommodate larger trees.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(linux): target web-document editable for focus-free typing
Browsers expose multiple editables: the address bar (omnibox) sorts first in
the AT-SPI tree, but the field a user/agent wants when typing into a browser
is the page input. Track a per-node `in_web_doc` flag (inherited from a
"document web"/document ancestor) and prioritize the insert target as:
focused editable -> editable inside web content -> first editable. This makes
focus-free typing drive the page field for browser control, while leaving
single-field apps (e.g. a GTK dialog entry) unchanged.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(linux): target page editable for browsers; help GTK load a11y bridge
Browser write path: focus-free insert_text sorted to the first editable in
the tree, which in a browser is the address bar, not the page field. Track a
per-node "in web document" flag (inherited from a "document web"/document
ancestor) and prefer, in order: a focused editable, an editable inside web
content (the page's input), then the first editable. Single-field apps (a GTK
dialog entry) are unaffected. This is what lets the driver type into a page to
control a browser, rather than into chrome.
GTK registration: zenity registered 0 applications because a GTK3 app dlopens
libatk-bridge-2.0.so by soname to join the AT-SPI bus, and it wasn't on the
loader path in the manual session. Add at-spi2-atk to LD_LIBRARY_PATH.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): enable AT-SPI status for GTK; log editable counts
Two diagnostics-driven changes after confirming the native walk works:
- GTK3 apps only export their accessible tree when org.a11y.Status.IsEnabled
is true on the session bus (GNOME sets this via gsettings). The hand-rolled
session left it false, so zenity registered nothing. Set IsEnabled=true via
dbus-send right after launching the a11y bus, before the app starts.
- insert_text now logs node/editable/entry-role counts. The chromium run
walked 211 nodes but found zero EditableText editables (despite two `entry`
nodes), indicating browsers don't expose EditableText for background
windows; this makes that explicit in the logs.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* revert(test): drop org.a11y.Status IsEnabled dbus-send
Poking org.a11y.Bus in setup triggered D-Bus activation of a second
at-spi-bus-launcher that conflicted with the manually-launched one, so the
driver could no longer reach the registry — both chromium and gtk fell back
to the X11 tree with zero AT-SPI nodes. Revert to the prior working setup
(chromium registers and the native walk reads its 211-node tree); the GTK
registration gate needs a different, non-conflicting fix.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): enable a11y via gsettings keyfile so GTK app registers
GTK3 only exports its accessible tree when toolkit-accessibility is enabled.
Set org.gnome.desktop.interface toolkit-accessibility=true once, before the
bus launcher and apps start, using the keyfile GSettings backend with a shared
XDG_CONFIG_HOME. This avoids poking org.a11y.Bus at runtime (which previously
D-Bus-activated a conflicting at-spi-bus-launcher and broke the registry).
Adds glib (gsettings) + gsettings-desktop-schemas to the VM. Targets the GTK
write path; browser write (CDP) is a separate follow-up.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): fix GSettings schema lookup; make a11y enable non-fatal
The gsettings call failed with schema-not-found because NixOS installs
compiled schemas under share/gsettings-schemas/<pkg>/glib-2.0/schemas, not the
bare share/glib-2.0/schemas that XDG_DATA_DIRS pointed at. Set
GSETTINGS_SCHEMA_DIR to the real compiled-schema path, and run the enable as a
non-fatal step (logging set+get) so AT-SPI registration diagnostics still
surface even if it errors.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): enable AT-SPI by setting IsEnabled on the owned bus launcher
Per at-spi-bus-launcher source, it reports a11y enabled only after an AT
client registers an event listener or IsEnabled is set explicitly; it does
NOT read toolkit-accessibility at startup (it only writes it). GTK3 apps check
IsEnabled at startup and stay silent when false, so gsettings had no effect.
Set IsEnabled directly, but first wait until our manually-launched launcher
actually OWNS org.a11y.Bus (via the bus driver's NameHasOwner, which does not
activate the name). The earlier attempt poked org.a11y.Bus before it was
owned, D-Bus-activating a second launcher that broke the registry for every
app. With single ownership guaranteed, the Set reaches the live launcher.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): add Qt (PyQt5) app to the background-GUI a11y matrix
Adds a non-GTK toolkit data point for focus-free AT-SPI typing: a minimal
PyQt5 window with a focused QLineEdit titled cua-initial. Qt exposes it over
AT-SPI (EditableText) under QT_ACCESSIBILITY=1, so it exercises the same
focus-free insert + readback path as the GTK case via a different toolkit.
Wires it through flake.nix (app list) and the nix-build.yml matrix.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* Bump cua-driver-rs to v0.5.1
Patch: release the installer fix (#1803) — release + local installers + runtime
all use ~/.cua-driver, and either installer cleans up a prior local install +
sweeps the stale legacy ~/.cua-driver-rs home. Changelog Unreleased → 0.5.1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(linux): surface target app stdout/stderr after launch
The qt job timed out finding the window because the PyQt5 app never showed
one (likely a Qt xcb platform-plugin load error). Log /tmp/target.log a few
seconds after launch so the real cause is visible rather than a bare
window-find timeout.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* chore(cua-driver-rs): bake version 0.5.1 into install scripts [skip ci]
* test(linux): point PyQt5 at qtbase's xcb platform plugin
The qt app failed to launch: `qt.qpa.plugin: Could not find the Qt platform
plugin "xcb" in ""`. A bare `python3` PyQt5 invocation doesn't inherit
qtbase's plugin path. Export QT_PLUGIN_PATH / QT_QPA_PLATFORM_PLUGIN_PATH from
qt5.qtbase's qtPluginPrefix so the xcb plugin is found and the window appears.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): read back IsEnabled + dump launcher log (diagnostic)
Both GTK and Qt apps launch fine but register 0 AT-SPI applications, even
after setting org.a11y.Status.IsEnabled. Read the property back (print-reply)
and dump the at-spi-bus-launcher log to determine whether the Set is taking
effect or the toolkit bridges simply aren't activating in this session.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): force Qt AT-SPI bridge on (QT_LINUX_ACCESSIBILITY_ALWAYS_ON)
IsEnabled is confirmed true on the a11y bus, yet the Qt app still registers 0
applications — Qt's bridge isn't activating from the bus handshake in this
headless session. Set QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1 (and QT_ACCESSIBILITY=1)
in the qt launch to force Qt to export its accessible tree.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* Delete JOURNAL.md
* Delete JOURNAL_VIDEO.md
* test(linux): validate AT-SPI read path; document focus-free write limit
Per investigation, focus-free WRITE into a *background, unfocused* toolkit
window isn't reliably supported: toolkits gate editable accessibility on
focus/activation (Chromium exposes fields read-only over AT-SPI; an unfocused
Qt window exposes only its top node; a GTK app's atk-bridge doesn't register
in this headless session). Chromium's own AT-SPI impl does expose a full
read-only tree.
So assert the proven READ path: the driver's get_text returns the background
window's accessibility/structure (a window/frame/document node) for every app
in the matrix — native tree for Chromium, at least the window node (native or
X11 fallback) for the others. type_text is still exercised but its readback is
no longer asserted; the write-needs-focus limitation is documented inline.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): add focus-gate confirmation run (diagnostic, non-fatal)
After the focus-free assertions, activate the target window and re-run the
driver, logging the focused get_text and whether the typed text now reads
back. This directly confirms the finding that toolkits expose the editable
only when the window is focused. Non-fatal: it's evidence in the logs, not a
gate (behaviour differs per toolkit).
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* ci(linux): temporarily disable firefox background-GUI matrix job
Firefox times out at launch under the emulated CI VM (no KVM) — it never
surfaces its window within the wait, so the job fails before any AT-SPI
subtest runs. This is an environmental launch issue, not a driver problem,
and the browser/AT-SPI read path is already covered by the chromium job.
Drop "firefox" from the flake check list and comment out its workflow matrix
entry; the app definition is kept so it can be re-enabled once launch is made
reliable (longer timeout + pre-seeded first-run-free profile).
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): add CDP focus-free write override + Electron matrix job
Chromium/Electron expose their fields read-only over AT-SPI, so the driver
can't write into a background browser window through it. Add an approved
Chromium/Electron-specific override using the Chrome DevTools Protocol:
Input.insertText targets the page's focused DOM element regardless of OS
window focus, so it lands in the unfocused background window.
- chromium/electron launch with --remote-debugging-port + --remote-allow-origins
- new asserting subtest drives a stdlib-only CDP client (HTTP target discovery
+ minimal RFC-6455 WebSocket) to insertText into the background window and
reads it back, while asserting the control terminal keeps X focus
- add a minimal Electron app (Chromium-backed BrowserWindow) as a new matrix
job; like chromium it's read-only over AT-SPI and writable via CDP
- wire "electron" into the flake matrix
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): expand background-GUI matrix with qt6, gtk4, tk
Broaden toolkit/version coverage of the background-GUI a11y suite:
- qt6 (PyQt6): same AT-SPI bridge as qt5 on the current Qt major; sets the
lib/qt-6 plugin path and libxcb-cursor (Qt 6.5+ needs it headless)
- gtk4 (compiled C GtkEntry): GTK4 talks AT-SPI directly (no atk-bridge
module), contrasting the GTK3/zenity bridge path; cairo renderer + x11
backend keep it headless-safe
- tk (tkinter): negative control — Tk has no AT-SPI bridge, so get_text
degrades to the X11 window node, proving graceful handling of
non-accessible toolkits
All wired into the flake matrix as independent jobs.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* ci: run electron/gtk4/qt6/tk background-GUI jobs
The nix-build matrix is hardcoded here (not derived from flake.nix), so the
new flake checks added for electron, gtk4, qt6 and tk never ran in CI. Add
them to the matrix so the expanded suite executes, including the CDP
focus-free-write assertion on electron.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* test(linux): accept text/entry nodes in the read assertion
Qt6's AT-SPI bridge exposes the editable even while unfocused, so the
driver's focus-free write lands and get_text returns a bare `text "..."`
node rather than a frame/window/document. Broaden the read-back assertion to
accept text/entry nodes too (also future-proofs gtk4, which exposes the
entry directly). The narrow frame/window/document check was the only reason
the qt6 job failed — the read (and write) actually worked.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
* feat(linux): add GTK3 focus-free write fallback via X11 click+type
GTK3's AT-SPI bridge gates EditableText on window/widget focus, so unfocused
background windows expose entry nodes in the tree (reads work) but not the
EditableText interface (writes fail). Qt6 exposes EditableText unconditionally.
This commit adds a GTK3-specific fallback: when insert_text finds an entry/text
role with Component bounds but no EditableText, it:
1. Gets the entry widget's screen coordinates via Component.GetExtents
2. Translates to window-local coords
3. Sends an X11 click to the entry's center to establish widget focus
4. Types via XSendEvent (now accepted by the internally-focused widget)
The window remains unfocused (control terminal keeps X focus), but the widget
receives and processes the keystrokes. This unblocks the gtk job in the
background-GUI test matrix.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat(linux): focus-free Tk writes via send command
Tk has no AT-SPI bridge, so background writes use Tk's `send` IPC instead.
The test app registers as "cua-tk-target" and the driver injects text by
spawning `wish` to send Tcl commands. This is the Tk-specific override
(like CDP for Chromium), proving non-accessible toolkits can support
focus-free input with bespoke paths.
- Add inject_tk_send() in platform-linux/input/mod.rs
- Wire it into type_text tool after AT-SPI, before XSendEvent fallback
- Update Tk test app to register with tk appname + name entry widget
- Add tkSubtest that asserts the write lands and focus stays put
- Include pkgs.tk so wish is available in the test environment
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat(linux): add GTK3/GTK4 focus-free write fallback via X11 click+type
GTK3 and GTK4's AT-SPI bridge gates EditableText on window/widget focus, so unfocused
background windows expose entry nodes in the tree (reads work) but not the
EditableText interface (writes fail). Qt6 exposes EditableText unconditionally.
This commit adds a GTK fallback: when insert_text finds an entry/text
role with Component bounds but no EditableText, it:
1. Gets the entry widget's screen coordinates via Component.GetExtents
2. Translates to window-local coords
3. Sends an X11 click to the entry's center to establish widget focus
4. Types via XSendEvent (now accepted by the internally-focused widget)
The window remains unfocused (control terminal keeps X focus), but the widget
receives and processes the keystrokes. This unblocks the gtk3 and gtk4 jobs in the
background-GUI test matrix.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* feat(linux): use AT-SPI Component.GrabFocus for GTK4 focus-free writes
GTK4 gates EditableText on widget focus, unlike Qt6 which exposes it
regardless of focus state. When a GTK4 window is in the background, the
AT-SPI tree contains entry/text widgets (so reads work) but EditableText
is unavailable, blocking focus-free writes.
Call Component.GrabFocus on the target widget before accessing EditableText.
This gives the widget internal keyboard focus without activating its window,
allowing GTK4 to expose EditableText on the focused widget. The approach is:
1. Find target editable widget (same priority as before)
2. If it has Component interface, call GrabFocus on it
3. Proceed to call EditableText.InsertText as usual
Benefits:
- No window activation: GrabFocus works at widget level, not window level
- Toolkit-agnostic: Component.GrabFocus is standard AT-SPI
- Non-breaking: if GrabFocus fails/unavailable, still try EditableText (Qt6+)
- Diagnostic logging shows GrabFocus success/failure for debugging
This should allow the gtk4 background-GUI test to pass with true focus-free
writes: the control terminal stays active throughout, the GTK4 entry gains
internal focus via GrabFocus, and EditableText.InsertText succeeds.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* ci: generate GIF artifacts for all background GUI tests
- Set visual: true for gtk, qt, qt6, gtk4, chromium, electron, tk tests
- Add artifact_name for each test so GIFs are uploaded
- Update PR comment script to list all new artifacts
This will make it easy to visually verify focus-free writes work correctly
for each toolkit by watching the GIF showing the window staying unfocused.
* feat(linux): enable focus-free background writes for Qt5 via synthetic focus events
Adds three-tier typing strategy for Linux:
1. Native AT-SPI EditableText (Qt6, GTK4 focus-free)
2. Synthetic FocusIn → AT-SPI → FocusOut (Qt5 workaround)
3. X11 XSendEvent fallback (terminal/legacy apps)
The synthetic-focus path sends FocusIn to trigger Qt5's AT-SPI bridge
without changing the X11 active window, enabling focus-free writes.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: restore GTK3 fallback code after merge conflict resolution
The GTK3 widget click fallback was accidentally removed when resolving
the merge conflict for PR #1817. This restores the entry_find_window_xid
and screen_to_window_coords helpers and the GTK3 X11 click+type fallback
logic that enables focus-free writes for GTK3 (zenity).
* fix(platform-linux): qualify Command in atspi python fallback
The merge-conflict resolution that restored type_into_editable's pyatspi
fallback reintroduced `Command::new("python3")` without a
`use std::process::Command;` import, breaking the cua-driver build
(E0433: cannot find type `Command`) and thus every nix CI job. Fully-qualify
the call as `std::process::Command::new` (matching the style in tools/impl_.rs)
to restore compilation without touching imports.
https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd
---------
Co-authored-by: Francesco Bonacci <f@trycua.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: hippoley <hippoley@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: trycua-release[bot] <trycua-release[bot]@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
r33drichards
added a commit
that referenced
this pull request
Jun 5, 2026
…als (#1789) * feat(cua-driver-rs)(linux): show cursor and type in background terminals * fix(nix): refresh cua-driver vendoring metadata * fix(nix): set cua-driver cargo hash * ci(nix): parallelize checks with matrix * fix(platform-linux): import request connection trait * test(nix): track xterm windows by pid * test(nix): wait for ffmpeg recorders to finish * test(nix): simplify ffmpeg gif encoding * test(nix): log ffmpeg output on gif failures * test(nix): record gifs with imagemagick * test(nix): relax linux cursor focus assertion * test(nix): align linux cursor gif assertion * Switch Linux keyboard input from XSendEvent to XTEST injection (#1805) * docs(cua-driver): add changelog reference page (#1785) Mirror the cua-driver-rs GitHub releases into the docs site so the release history is discoverable on the docs site (not just GitHub), matching the convention used by the other products (cua CLI, lume). Wire it into the reference nav. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(cua-driver)(macos): guard SkyLight auth-message selector for macOS 14 Sonoma (#1503) (#1782) `hotkey`, `press_key`, and `scroll` crash the daemon on macOS 14 (Sonoma) with `NSInvalidArgumentException: +[SLSEventAuthenticationMessage messageWithEventRecord:pid:version:]: unrecognized selector sent to class`. The class `SLSEventAuthenticationMessage` exists on macOS 14, but the `messageWithEventRecord:pid:version:` factory selector was only added in macOS 15 (Sequoia). The existing `!cls.is_null() && !sel.is_null()` guard is insufficient: `sel_registerName` / `NSSelectorFromString` always succeed (they just intern the string), so `objc_msgSend` still dispatches an unimplemented selector and the ObjC runtime aborts the process. Guard the dispatch with `class_respondsToSelector` (Rust) / `messageClass.responds(to:)` (Swift), which actually checks the metaclass. On macOS 14 it returns false, so we skip the auth envelope and fall through to plain `SLEventPostToPid`. Chromium-class targets may not receive the event on macOS 14, but the daemon no longer crashes — graceful degradation. This re-applies the fix from #1579 (by @hippoley) onto the current `libs/cua-driver/{rust,swift}/` layout — #1579 predates the #1674 directory restructure and no longer merges. - rust: platform-macos/src/input/skylight.rs — class_responds_to_selector() - swift: CuaDriverCore/Input/SkyLightEventPost.swift — responds(to:) guard Verified: platform-macos + the full cua-driver binary build; the Swift `responds(to:)` form compiles and returns true for an existing class method, false for an absent one. Closes #1503 Co-authored-by: hippoley <hippoley@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(cua-driver-rs)(macos): enable Chromium/Electron AX trees for get_window_state (#1756) Chromium/Electron apps (Arc, VS Code, Electron shells) ship their web-content accessibility tree off and only build it once an assistive client requests it. Without enablement the first AX walk returns an empty/title-bar-only tree. Flip AXManualAccessibility (modern, side-effect-free) on the application root, falling back to AXEnhancedUserInterface when the modern attribute is unsupported. When the flip actually takes, let the asynchronously-built tree settle (~500ms run-loop pump) before walking. Cache per-pid so repeat snapshots skip the settle. Native Cocoa apps reject the attribute and pay no cost. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Bump cua-driver-rs to v0.4.2 * docs(cua-driver): add 0.4.2 changelog entry + fix 0.3.6 wording (#1786) - Add 0.4.2: macOS 14 Sonoma SkyLight selector guard (#1782, #1503) and Chromium/Electron AX trees via AXManualAccessibility (#1756). - Fix the 0.3.6 entry, which described the permissions-status fix backwards: it now reports the driver's grants (via the daemon), not the caller's. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(cua-driver-rs): bake version 0.4.2 into install scripts [skip ci] * fix(cua-driver-rs): wire/guide per-session cursors through the real mcp path + skills/docs (#1787) * fix(cua-driver-rs): wire/guide per-session cursors through the real mcp path + update skills/docs A user drove `cua-driver mcp --claude-code-computer-use-compat` (the documented Claude Code install) and asked: (1) why no agent cursor even on AX actions, (2) where is the session in the mcp calls, (3) did we forget the CLI / MCP / skills wiring. Investigation + fixes: - Session IS wired (working as designed): the proxy path the user runs mints one session_id per MCP connection and stamps it on every forwarded request; the daemon injects it as `_session_id` into tool args and strips it from the user-visible wire envelope. Per-session cursor / config / recording are live on the compat proxy path — verified headless (set_agent_cursor_enabled{false} in a session is read back by get_config{enabled:false}, proving _session_id reached the daemon). - BUG (user-visible): no glide on a pure-AX run. A brand-new session cursor sat at the off-screen sentinel; animate_cursor_to early-returned so the first AX action only snapped a static arrow via ClickPulse — easy to miss. Fix: seed the sentinel cursor on-screen (offset, clamped) before animating so the FIRST action glides. Get-or-create + ended tombstone guard so it never resurrects a reaped session. Unit-tested. - BUG (latent wiring): `--claude-code-computer-use-compat` was silently dropped on the proxy path (daemon hardcoded compat=false). Thread it end-to-end: proxy forwards `serve --claude-code-computer-use-compat`, the Serve arm honours it via build_macos_registry_with_compat. Today this has no tool-surface effect (the compat screenshot tool was removed in #1692) but the flag now travels for any future compat-gated tool. - BUG (nondeterministic): get_config reported agent_cursor.enabled from a HashMap .first(). Resolve the calling session's cursor by key (cursor_id > _session_id > "default"). Unit-tested per-session. Docs/skills (no default change — that is the user's call; see PR body): SKILL.md (per-session model, session_end removal, AX no-glide caveat, corrected the false "AX skips the overlay" claim), set_agent_cursor_enabled description, protocol.rs server-instructions, CLI help (cursor flags + overlay + compat), mcp-tools.mdx AX-snap caveat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(cua-driver)(skills): correct the AX cursor caveat — short glide, not no glide After the sentinel-seed fix the first AX action seeds the cursor on-screen near the target and plays a brief glide + pulse (not "does not glide"). Reword the SKILL.md visibility caveat to match the actual behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(cua-driver-rs)(macos): run the agent-cursor overlay in the serve daemon (#1790) The overlay NSWindow + AppKit render loop were only wired into the in-process `mcp` arm. In the daemon-proxy setup users run (`mcp` relaunches `open -n -g … serve` and proxies to it for correct TCC), the DAEMON performs the clicks/AX presses but never inited or ran the overlay — its main thread parked in `serve_handle.join()`. So `set_agent_cursor_enabled` flipped registry flags and clicks sent OverlayCommands, but CMD_TX/RENDER were never set → every cursor command was a silent no-op and the agent cursor never appeared. Fix: the Serve arm now builds cursor_cfg, inits the overlay channel before spawning the serve thread, and (when enabled) parks main in `overlay::run_on_main_thread()` (mirrors the Mcp arm) instead of join. It self-guards on has_graphic_access() and falls back to join when there's no Window Server session, so headless serving is unaffected. PiP unchanged. Verified via the REAL launch path: `open -n -g -a CuaDriver --args serve` daemon's main thread now runs __CFRunLoopRun / -[NSApplication run] with run_appkit + SkyLight + tiny_skia overlay rendering, and still serves. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(cua-driver-rs)(macos): stop the permissions gate spamming the TCC prompt on every re-exec (#1791) `cua-driver permissions grant` (and any first-launch serve) raises the system TCC prompt, then re-execs the daemon ~every 25s to refresh the per-process AXIsProcessTrusted cache. Each re-exec'd process re-ran run_if_needed and re-raised request_accessibility/request_screen_recording — so a fresh "Cua Driver" dialog popped every ~25s. Worse, the 10-min deadline was anchored to each process's own start, and since the re-exec fires (~25s) well before the deadline, the deadline never triggered: the gate re-execed (and restarted the whole daemon, now incl. the cursor overlay) forever whenever the grant read as missing — including the stale-ad-hoc-cdhash case (Settings shows granted but the rebuilt binary's hash no longer matches, so the live check returns false). Fix: - reexec_self sets CUA_DRIVER_RS_GATE_REEXEC=1; run_if_needed sees it and polls SILENTLY (skips the prompts + panel) on re-exec'd processes. The prompt + panel appear exactly once, on first launch. - reexec_self persists the original gate start in CUA_DRIVER_RS_GATE_START_UNIX; wait_for_grants anchors `start` to it so the deadline is cumulative across re-execs and the gate actually gives up (and stops churning) after the deadline, continuing to serve (tools fail with TCC errors until granted). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(cua-driver-rs)(install-local): sign the bundle with a stable self-signed identity so TCC grants survive rebuilds (#1792) install-local ad-hoc-signed the bundle (`codesign --sign -`), which keys the TCC grant (Accessibility / Screen Recording) on the binary's cdhash. The cdhash changes on EVERY rebuild, so each install-local silently invalidated the grant — System Settings still showed "CuaDriver ✅" (it's keyed on the bundle id) while the live AXIsProcessTrusted check failed, and the daemon re-prompted ("I already granted!"). A genuinely miserable dev loop. Fix: create a self-signed code-signing certificate once (idempotent, in the login keychain) and sign the bundle with it. TCC then keys the grant on the certificate leaf — stable across rebuilds — so the Designated Requirement becomes `identifier "com.trycua.driver" and certificate leaf = H"..."` instead of a cdhash pin. Grant once; every future install-local keeps it. Robust + fail-soft: openssl 3.x needs `-legacy` PBE + a real p12 password for Apple's `security import` (the empty-password default fails MAC verification); falls back to non-legacy for LibreSSL. If the cert can't be created (no openssl, locked keychain, CI), falls back to ad-hoc signing + a one-line note. Local dev only — releases are CI-signed and already stable. One-time migration: switching from ad-hoc to the cert changes the requirement once, so the next grant after this lands is a single re-grant; stable after. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Bump cua-driver-rs to v0.4.3 * docs(cua-driver): add 0.4.3 changelog entry (#1793) cursor overlay in the daemon (#1790), permissions-grant prompt no-spam (#1791), and install-local stable signing identity (#1792). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(cua-driver-rs): bake version 0.4.3 into install scripts [skip ci] * fix(cua-driver-rs)(install-local): reset a TCC grant pinned to a previous signing identity (#1795) Accessibility / Screen-Recording grants survive rebuilds — but only for grants CREATED while cert-signed. A grant the user made earlier on an ad-hoc build is pinned to that build's cdhash (the stored csreq is a bare `cdhash H"..."`), so it survives reinstall with auth_value=allowed yet stops matching the new binary. The daemon then reads "not granted" while System Settings still shows CuaDriver toggled ON — a dead end, because the row already records a decision so re-toggling never re-fires the prompt. Record the signing identity (cert leaf, or "adhoc") in ~/.cua-driver/.tcc-signing-identity. When the installer signs with a cert identity that differs from the last install, `tccutil reset` Accessibility + ScreenCapture once so the next `permissions grant` prompts cleanly and re-pins to the stable cert (after which grants survive every future rebuild). `tccutil reset` needs no sudo/FDA and is a no-op when nothing was granted. We only reset when moving TO a cert identity — an ad-hoc build churns its cdhash regardless, so resetting it would add friction with no durable fix. Docs: FAQ entry for "granted but reports NOT granted after a rebuild" + changelog. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(cua-driver-rs)(macos): retain cached AX element across action so concurrent sessions can't UAF-crash the daemon (#1796) Two sessions driving the same window concurrently crashed the daemon with EXC_BREAKPOINT (SIGTRAP) inside AXUIElementCopyActionNames → _AXUIElementValidate → CFGetTypeID — a use-after-free. Root cause: the per-(pid, window_id) element cache (ax/cache.rs) handed out raw AXUIElementRef pointers as usize. A tool (click/type_text/set_value/…) copied the pointer out from under the cache lock and used it across await points and on a blocking thread. Meanwhile another session's get_window_state called ElementCache::update → ElementCacheCore::insert, which replaced the snapshot and ran CachedSnapshot::drop on the old one — CFRelease-ing those exact pointers to zero. The in-flight action then dereferenced freed memory. Fix: replace get_element_ptr with get_element_retained, which CFRetains the element while still holding the cache lock and returns a RetainedElement guard (CFRelease on drop). An in-flight action holds the guard for its whole duration, so a concurrent snapshot replace can't free the element under it. Migrated all nine element-action call sites (click, right_click, double_click, type_text, type_text_chars, press_key, scroll, set_value, recording_hooks). Test: ax::cache::tests::retained_element_survives_concurrent_snapshot_replace asserts the retain accounting — after a concurrent replace the guard's retain is what keeps the element alive (count = base+1, not base). 74/74 platform-macos lib tests pass. Note: platform-windows has the same shape (uia/cache.rs::get_element_ptr hands out raw IUIAutomationElement pointers); a mirrored AddRef-on-get fix is a follow-up, not included here (untestable in this environment). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * docs(cua-driver-rs)(launch_app): surface creates_new_application_instance for concurrent multi-agent isolation (#1797) launch_app is idempotent, so two sessions launching the same app get the same instance — and on single-instance apps (Calculator, many utilities) the same window — and clobber each other. The `creates_new_application_instance` param already solves this (it maps to NSWorkspaceOpenConfiguration.createsNewApplicationInstance, the programmatic `open -n`), but nothing told an agent to reach for it in the concurrent case. Enrich the tool description, the MCP-tools doc, and the skill's action-loop section to call out the concurrent-session use. No behavior change. Verified end-to-end: two launch_app(name=Calculator, creates_new_application_instance=true) calls return distinct pids + distinct window_ids. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(cua-driver-rs): caller-declared session identity + Streamable-HTTP transport for multi-agent parallelism (#1798) * feat(cua-driver-rs): explicit session identity core + cursor explicit-required - core/session.rs: touch_session/end_session/evict_idle + idle-TTL activity map - serve.rs: apply_session_identity at the daemon boundary (explicit `session` → _session_id; minted id is recording/config fallback only, not a cursor source) - cursor: resolve_cursor_key returns NO_CURSOR("") when no session declared; overlay + registry short-circuit the empty key (explicit-required cursor) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cua-driver-rs): start_session/end_session tools + idle-TTL sweep + session schema - core/session_tools.rs: start_session / end_session tools (cross-platform), registered via ToolRegistry::register_session_tools on all 3 platforms - serve.rs: spawn_session_idle_sweep — evict_idle every 30s (TTL default 300s, CUA_DRIVER_RS_SESSION_IDLE_TTL_SECS override) - inject session property into action-tool schemas; fix set_agent_cursor_enabled description (cursor is explicit-required now, not auto-per-MCP-session) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(cua-driver-rs): document explicit session identity (MCP instructions, SKILL, mcp-tools, changelog) - MCP server instructions: add start_session step + explicit-session cursor model - SKILL.md: canonical loop gains start_session/end_session; fix concurrent note (cursor keyed on session, not (pid,window_id)) - mcp-tools.mdx: rewrite per-session cursor section; add start_session/end_session - changelog: breaking session-identity entry Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cua-driver-rs): stop a session's recording on session_end (end_session/idle-TTL/EOF) Register a session_end hook that calls recording.stop_owner(Some(sid)) on a detached thread, so end_session and the idle-TTL sweep tear down a session's recording too (matching end_session's contract) — not just the EOF path. Safe: stop_owner(Some) is a no-op unless that session owns the live recording, and the detached thread keeps mp4 finalize off the synchronous fire_session_end caller. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cua-driver-rs): unit-test apply_session_identity boundary (explicit/minted/anonymous) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cua-driver-rs)(macos): move_cursor visibly moves the drawn cursor (seed sentinel like click) move_cursor sent a raw MoveTo, which doesn't bring a brand-new session cursor on-screen — it sits at the off-screen sentinel until a click seeds it, so the DRAWN cursor never moved (only the reported position did). Use animate_cursor_to (the same path click uses): it seeds the sentinel on-screen then glides in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cua-driver-rs)(macos): mark move_cursor read-only so MCP clients can parallelize cursor moves move_cursor only nudges the agent-cursor overlay, never the target app, so it is concurrency-safe. read_only:true emits readOnlyHint, which Claude Code's isConcurrencySafe() uses to run cursor moves in parallel. Mutating tools (click/type_text/press_key) stay read_only:false on purpose — parallelizing an ordered intra-agent sequence would race. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(cua-driver-rs): Streamable-HTTP MCP transport on the daemon for parallel multi-agent (#1799) Over stdio, one cua-driver mcp process is a single pipe, so a client's tool calls (incl. multiple subagents) serialize. The daemon is already concurrent (task per connection). This adds an HTTP MCP front-end so each agent opens its OWN connection: per-connection FIFO keeps a single agent's ordered calls correct, distinct connections run truly in parallel — safe because per-(pid,window) caches + per-session cursors make concurrent cross-connection actions non-colliding. - mcp_http.rs: hand-rolled HTTP/1.1 (no new deps, mirrors the UDS line protocol), POST -> cua_driver_core::server::handle_request (now pub) -> application/json JSON-RPC. Task per TCP connection; honors Connection: close; mirrors the "session" arg -> _session_id + touches idle-TTL so HTTP == stdio behavior. - opt-in via CUA_DRIVER_RS_MCP_HTTP_PORT (loopback only); spawned from run_serve. Proven: 10 list_apps over 10 concurrent connections = 3.6s vs 12.9s sequential (3.6x). curl initialize/tools/list/tools/call all correct. 3 unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(cua-driver-rs): document HTTP MCP transport + the concurrency model - changelog: Streamable-HTTP transport + move_cursor readOnlyHint - FAQ: "Concurrency & multiple agents" — why subagents serialize (shared stdio pipe), and how to run agents truly in parallel (separate connections / the CUA_DRIVER_RS_MCP_HTTP_PORT HTTP endpoint) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(cua-driver-rs)(skill): note subagent serialization + HTTP transport for parallel agents Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(cua-driver-rs)(windows): per-session agent cursors (port macOS #1779) (#1801) The Windows overlay was a process-wide singleton (one `RenderState`), so concurrent MCP sessions clobbered each other last-writer-wins → one shared cursor. #1779 fixed this on macOS but explicitly left Windows/Linux on the old single-cursor model ("the key concept never reaches them"). Port the keyed render collection to platform-windows: - overlay.rs: `RenderMap { IndexMap<CursorKey, RenderState> }`; `send_command` now carries a `CursorKey`; the WM_TIMER tick drains keyed `OverlayMsg`s, ticks every cursor, and composites them all into the ONE layered window via `paint_cursor` (insertion order = stable z-order). Per-key arrival isolation, lazy per-key palette (`Palette::for_instance`), `remove_cursor` + render-side resurrection tombstone, and the sentinel seed — all mirroring platform-macos/src/cursor/overlay.rs. - tools/impl_.rs: `resolve_cursor_key` (session > cursor_id > NO_CURSOR, never the connection `_session_id`), threaded through `pin_overlay_above`, `overlay_glide_to`, every ClickPulse callsite, and the 5 cursor tools. A `session_end` hook (once-guarded) calls `remove_cursor`; `get_config`'s `cursor_enabled` is now session-scoped + deterministic (was a nondeterministic `all_states().first()` — macOS BUG 3). - cursor-overlay: `CursorRegistry::remove` (guards "default"). page.click_element keeps the seeded "default" cursor — the cross-platform `PageBackend` trait carries no caller session (separate follow-up). 15 new headless unit tests (two-session isolation, session_end removal, default guard, resurrection tombstone, sentinel seed, key resolution); full platform-windows lib suite green (49 tests), daemon builds warning-free. Verified live on Windows 11: two calculators driven by two sessions show two distinct-coloured cursors gliding in parallel; end_session removes each. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Bump cua-driver-rs to v0.5.0 Release the caller-declared session identity + Streamable-HTTP multi-agent transport (#1798) and Windows per-session cursors (#1801). Breaking: the agent cursor is now opt-in (declare a `session`). Changelog Unreleased → 0.5.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(cua-driver-rs): bake version 0.5.0 into install scripts [skip ci] * fix(cua-driver-rs): release installer unifies home on ~/.cua-driver + cleans up prior local install (#1803) The release installer (install.sh → _install-rust.sh) defaulted its package home to the legacy ~/.cua-driver-rs, but the local installer (_install-local-rust.sh) and the runtime already use ~/.cua-driver (renamed in v0.2.16 / PR #1644). That mismatch is the root cause of a two-install collision: a user who ran install-local and then the release install.sh ended up with two homes and two conflicting installs, with the local build's artifacts left dangling. Fixes in _install-rust.sh: - Default HOME_DIR to ~/.cua-driver (still honoring CUA_DRIVER_RS_HOME for back-compat), matching install-local + runtime. - Before staging: cleanup_prior_local_install() stops the daemon and removes the prior install-local artifacts under the shared home — the `*-local-*` release dirs and the ~/.cua-driver/.tcc-signing-identity marker. Marker-gated and conservative: never touches a real release dir, the `current` symlink, or unrelated user state; best-effort + idempotent (no-op on a clean machine). - After staging: sweep a stale ~/.cua-driver-rs left by an older release, mirroring the belt-and-braces legacy-home sweep install-local already does. - TCC grants preserved: /Applications/CuaDriver.app is replaced in place via the existing release ditto (grants key on the shared com.trycua.driver bundle id); no tccutil reset, so cert-pinned grants are not churned. install.ps1 (Windows) already defaults to ~/.cua-driver and migrates the legacy home, so it is unchanged. Docs: reconcile the ~/.cua-driver-rs → ~/.cua-driver home references across the installation + linux guides, document the local/legacy cleanup behavior, and add an Unreleased changelog entry. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(cua-driver-rs)(linux): generalize background keyboard input via XTEST The background-terminal work special-cased terminals: type_text and press_key(Enter) detected a terminal process, found its /dev/pts tty, and shoved bytes in with the legacy TIOCSTI ioctl. That only ever worked for terminals, and TIOCSTI is exactly the mechanism modern kernels harden away (CONFIG_LEGACY_TIOCSTI / dev.tty.legacy_tiocsti), so it would EPERM on many systems. It also left the XTEST scaffold added alongside it as dead code. Replace the terminal-specific path with a general one. Keyboard input now goes through XTEST for every window: XSendEvent keystrokes carry the send_event flag that xterm (and friends) deliberately ignore, which is why typing into a background terminal silently did nothing; XTEST injects at the server level with no such flag, so it lands on terminals and every other app alike. Because XTEST targets the focused window, with_focus briefly focuses the target, injects, and restores the prior focus — preserving the same no-focus-steal contract the XSendEvent pointer path keeps. - input/mod.rs: send_type_text / send_type_text_with_delay / send_key now use XTEST (with real Shift presses for shifted chars and held modifiers), wiring up the previously-dead xtest_* helpers. Pointer (click/drag) stays on XSendEvent. - impl_.rs: drop inject_terminal_input + is_terminal_process / terminal_*_tty helpers and the TIOCSTI ioctl, and the type_text / press_key branches that called them. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * fix(cua-driver-rs)(linux): restore active window after XTEST injection The background-terminal GIF test injected fine but failed its focus check: typing landed in the inactive xterm, yet focus ended on the target instead of returning to the control terminal. XTEST delivers to the focused window, so with_focus moves focus to the target to inject — but the restore used a bare SetInputFocus, and under an EWMH WM (openbox) `xdotool getactivewindow` reads `_NET_ACTIVE_WINDOW`, which the WM owns and doesn't update from a raw SetInputFocus. So focus never came back. Restore cooperatively: capture `_NET_ACTIVE_WINDOW` up front and re-activate it afterwards with a `_NET_ACTIVE_WINDOW` client message (source = 2, the same nudge `xdotool windowactivate` sends), keeping SetInputFocus for the no-WM case. Add a short settle after each focus/activation request so the asynchronous WM acts before we inject or restore. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * fix(nix): restore Cargo.lock to keep cargoHash valid A stray `cargo check` re-bumped the workspace crates in Cargo.lock from 0.4.0 to 0.4.1 (matching the manifests) and it got committed. Nixpkgs' fetchCargoVendor hashes the vendored directory, which includes a copy of Cargo.lock, so the changed lock invalidated the pinned cargoHash and broke the cua-driver build — and with it every NixOS VM test that builds the driver. Restore Cargo.lock to the base/known-good revision. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * feat(cua-driver-rs)(linux): focus-free input — XSendEvent for GUI, pty master for terminals Replaces the XTEST-with-temporary-focus approach (which broke the cross-platform "no focus steal" contract that macOS SLEventPostToPid and Windows PostMessage uphold) with two focus-free paths: - GUI apps: XSendEvent, as before, but the typing path now resolves the shift level from the keyboard map so uppercase / shifted symbols inject correctly (previously "A" was sent as "a"). Removed the dead XTest scaffold. - Terminals: instead of the legacy TIOCSTI ioctl (which dev.tty.legacy_tiocsti disables on modern kernels), borrow the emulator's pty master fd via pidfd_getfd(2) and write to it. The kernel delivers the bytes to the shell's stdin exactly as typed — no X focus change, immune to the TIOCSTI sysctl. pidfd_getfd needs ptrace-mode access, which under the default ptrace_scope=1 is granted for the caller's own descendants — i.e. terminals the driver launched — with no root and no special capability. For terminals the driver did not launch it returns Ok(false) and the caller falls back; injecting into someone else's terminal unprivileged is what the kernel deliberately prevents. New module crate::tty holds the master-borrow logic. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(nix): matrix background-GUI input coverage (chromium, firefox, tk) Adds a parameterized NixOS VM test proving cua-driver types into a GUI window via XSendEvent WITHOUT stealing focus — the general computer-use claim, beyond terminals. Each app shows a focused text field that mirrors what it receives into its X11 window title; the test types a known string into the *inactive* app window (no click/focus first) and asserts the title became that string (input landed) and a separate control terminal stayed active (no focus steal). Wired as one independent matrix job per app (chromium, firefox, tk) in flake.nix checks and the nix-build workflow, so coverage spans a Chromium web engine, a Gecko web engine, and a native Tk toolkit. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * fix(nix): use python3 + tkinter for the tk GUI test (python3Full removed) nixpkgs removed python3Full ("tkinter is available within the package set"), which broke flake evaluation of the tk matrix job. Use python3.withPackages (ps: [ ps.tkinter ]) instead. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * fix(nix): background-GUI test — file:// page, exec launchers, find window by name Two harness bugs the matrix run surfaced (driver logic unaffected): - The browser launch commands embedded a data: URL whose double quotes collided with the testScript's Python/shell quoting, so the nixos test driver rejected the script with "invalid-syntax". Serve the page from a file:// URL written via writeText and move each launch into a writeShellScript that exec's the app, so the testScript only ever embeds a quote-free path. - Window discovery used `xdotool search --pid`, which needs _NET_WM_PID — Tk doesn't set it and browser window pids differ from the launcher, so the search hung to timeout. Give every app a known initial window title ("cua-initial") and discover by --name instead. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * feat(cua-driver-rs)(linux): type into GUI apps via AT-SPI (focus-free) X11 only routes keystrokes to the focused toplevel's focused widget, so background XSendEvent typing never lands in an unfocused GUI window (confirmed in CI against both Tk and Chromium: the type call "succeeds" but no text appears). Terminals are the lone exception, handled below the toolkit via the pty master. For GUI apps, fill the editable field through AT-SPI EditableText instead — focus-free and toolkit-agnostic. type_text now tries, in order: pty master (terminals) -> AT-SPI insert into the focused/first editable element (GUI) -> XSendEvent (last resort, e.g. apps with no a11y tree). New atspi::insert_text holds the EditableText logic. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(nix): AT-SPI harness for background-GUI input (zenity, chromium, firefox) Reworks the GUI matrix to validate the focus-free AT-SPI typing path the driver now uses, rather than X11 keystroke injection (which can't reach an unfocused GUI widget). - Stand up a session D-Bus at a fixed address and an AT-SPI bus (at-spi-bus-launcher), shared via a common env so cua-driver's pyatspi and the apps register with the same registry. - Swap the un-accessible Tk app for zenity (a GTK app exposing AT-SPI). - Enable accessibility for the browsers (chromium --force-renderer-accessibility, firefox GNOME_ACCESSIBILITY=1). - Read the typed text back through AT-SPI (queryText) — self-consistent with how the driver writes — and still assert focus never left the control terminal. Matrix jobs renamed tk -> gtk accordingly. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * fix(nix): env-prefix must precede timeout in the GUI type step `timeout 120 DISPLAY=:99 ... python3` made timeout try to exec "DISPLAY=:99" as the command (failed instantly). Move the env assignments before timeout so they apply to the command. The AT-SPI bus, zenity launch, and window discovery already worked in CI; this unblocks the actual type/readback steps. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * fix(nix): add pygobject3 so pyatspi readback can import `gi` The AT-SPI readback helper failed with `ModuleNotFoundError: No module named 'gi'` — pyatspi is a thin wrapper over PyGObject and needs it at import time. The env-prefix fix got us past the type step; this unblocks the readback verification. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * feat(linux): native AT-SPI over D-Bus, replacing the pyatspi subprocess The Linux accessibility path shelled out to `python3 -c "import pyatspi"` for every tree walk, text insert, value set, action, and bounds query. That bridge needs Python + pyatspi + PyGObject + GI typelibs at runtime, and under Nix it broke at `import pyatspi` (missing `gi`, then a missing `DBus-1.0` typelib). Worse, `type_text` swallowed the failure (`insert_text(...).unwrap_or(false)`) and silently fell back to X11 XSendEvent, so focus-free typing wasn't actually working — only the readback surfaced it. Link AT-SPI directly via the `atspi` crate (zbus, pure Rust). A new `atspi::native` module reimplements walk_tree / insert_text / set_value / perform_action / get_element_bounds over D-Bus: it resolves the target app by matching pid via `org.freedesktop.DBus.GetConnectionUnixProcessID`, walks the tree depth-first/pre-order (identical element indexing and markdown format so downstream parsing is unchanged), and uses the EditableText/Text/Action/Value/ Component proxies. The public functions stay synchronous (callers use `spawn_blocking`) and drive a shared Tokio runtime. No Python, pyatspi, PyGObject, or GI typelibs are required at runtime anymore. Test: the background-GUI test verifies the typed text via the driver's own `page`/`get_text` (same native path), and drops pythonAtspi/pygobject3 and the pyatspi readback entirely. cargoHash is set to a placeholder; the nix build will report the real value. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * fix(nix): set cua-driver cargoHash for the atspi/zbus dependency set The nix build reported the expected fixed-output vendor hash; pin it so the driver (and the GUI test that builds it) compiles against the new native AT-SPI dependencies. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * fix(linux): capture Text-interface content + timeouts in native AT-SPI walk First end-to-end run of the native walk surfaced two issues: - get_text returned empty for the editable: an entry's typed text lives in the AT-SPI Text interface, but the walk only emitted name/value/actions. Now read bounded Text content and use it as the display name when the widget has no accessible name, so typed text shows up in get_text. - Chromium's large, lazily-built tree could hang the walk forever (zbus calls have no timeout). Add a 3s per-call timeout (skip the node on timeout), a 25s overall walk budget, and a 5000-node cap. Also add CUA_ATSPI_DEBUG diagnostics (app/pid match + node counts to stderr) and have the test print the raw get_text response, so CI shows what the walk actually found. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * perf(linux): parallelize AT-SPI node reads; fix GTK app registration Diagnostics from the first working native run: - Chromium resolved its app by pid and walked 211 nodes, but each walk took ~9s (fully sequential D-Bus round-trips), so the readback loop blew the timeout. Issue the four independent per-node reads (role, name, state, children) concurrently via join!, and only touch interface proxies when the node actually advertises that interface. - GTK app (zenity) registered 0 applications: its atk-bridge module wasn't on GTK_PATH, so it never joined the AT-SPI registry. Point GTK_PATH at at-spi2-atk. (Chromium uses its own AT-SPI impl, hence it registered.) Test: trim the readback retry loop (8x, 1s) and raise the script timeout to 200s to accommodate larger trees. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * feat(linux): target web-document editable for focus-free typing Browsers expose multiple editables: the address bar (omnibox) sorts first in the AT-SPI tree, but the field a user/agent wants when typing into a browser is the page input. Track a per-node `in_web_doc` flag (inherited from a "document web"/document ancestor) and prioritize the insert target as: focused editable -> editable inside web content -> first editable. This makes focus-free typing drive the page field for browser control, while leaving single-field apps (e.g. a GTK dialog entry) unchanged. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * feat(linux): target page editable for browsers; help GTK load a11y bridge Browser write path: focus-free insert_text sorted to the first editable in the tree, which in a browser is the address bar, not the page field. Track a per-node "in web document" flag (inherited from a "document web"/document ancestor) and prefer, in order: a focused editable, an editable inside web content (the page's input), then the first editable. Single-field apps (a GTK dialog entry) are unaffected. This is what lets the driver type into a page to control a browser, rather than into chrome. GTK registration: zenity registered 0 applications because a GTK3 app dlopens libatk-bridge-2.0.so by soname to join the AT-SPI bus, and it wasn't on the loader path in the manual session. Add at-spi2-atk to LD_LIBRARY_PATH. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): enable AT-SPI status for GTK; log editable counts Two diagnostics-driven changes after confirming the native walk works: - GTK3 apps only export their accessible tree when org.a11y.Status.IsEnabled is true on the session bus (GNOME sets this via gsettings). The hand-rolled session left it false, so zenity registered nothing. Set IsEnabled=true via dbus-send right after launching the a11y bus, before the app starts. - insert_text now logs node/editable/entry-role counts. The chromium run walked 211 nodes but found zero EditableText editables (despite two `entry` nodes), indicating browsers don't expose EditableText for background windows; this makes that explicit in the logs. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * revert(test): drop org.a11y.Status IsEnabled dbus-send Poking org.a11y.Bus in setup triggered D-Bus activation of a second at-spi-bus-launcher that conflicted with the manually-launched one, so the driver could no longer reach the registry — both chromium and gtk fell back to the X11 tree with zero AT-SPI nodes. Revert to the prior working setup (chromium registers and the native walk reads its 211-node tree); the GTK registration gate needs a different, non-conflicting fix. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): enable a11y via gsettings keyfile so GTK app registers GTK3 only exports its accessible tree when toolkit-accessibility is enabled. Set org.gnome.desktop.interface toolkit-accessibility=true once, before the bus launcher and apps start, using the keyfile GSettings backend with a shared XDG_CONFIG_HOME. This avoids poking org.a11y.Bus at runtime (which previously D-Bus-activated a conflicting at-spi-bus-launcher and broke the registry). Adds glib (gsettings) + gsettings-desktop-schemas to the VM. Targets the GTK write path; browser write (CDP) is a separate follow-up. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): fix GSettings schema lookup; make a11y enable non-fatal The gsettings call failed with schema-not-found because NixOS installs compiled schemas under share/gsettings-schemas/<pkg>/glib-2.0/schemas, not the bare share/glib-2.0/schemas that XDG_DATA_DIRS pointed at. Set GSETTINGS_SCHEMA_DIR to the real compiled-schema path, and run the enable as a non-fatal step (logging set+get) so AT-SPI registration diagnostics still surface even if it errors. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): enable AT-SPI by setting IsEnabled on the owned bus launcher Per at-spi-bus-launcher source, it reports a11y enabled only after an AT client registers an event listener or IsEnabled is set explicitly; it does NOT read toolkit-accessibility at startup (it only writes it). GTK3 apps check IsEnabled at startup and stay silent when false, so gsettings had no effect. Set IsEnabled directly, but first wait until our manually-launched launcher actually OWNS org.a11y.Bus (via the bus driver's NameHasOwner, which does not activate the name). The earlier attempt poked org.a11y.Bus before it was owned, D-Bus-activating a second launcher that broke the registry for every app. With single ownership guaranteed, the Set reaches the live launcher. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): add Qt (PyQt5) app to the background-GUI a11y matrix Adds a non-GTK toolkit data point for focus-free AT-SPI typing: a minimal PyQt5 window with a focused QLineEdit titled cua-initial. Qt exposes it over AT-SPI (EditableText) under QT_ACCESSIBILITY=1, so it exercises the same focus-free insert + readback path as the GTK case via a different toolkit. Wires it through flake.nix (app list) and the nix-build.yml matrix. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * Bump cua-driver-rs to v0.5.1 Patch: release the installer fix (#1803) — release + local installers + runtime all use ~/.cua-driver, and either installer cleans up a prior local install + sweeps the stale legacy ~/.cua-driver-rs home. Changelog Unreleased → 0.5.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(linux): surface target app stdout/stderr after launch The qt job timed out finding the window because the PyQt5 app never showed one (likely a Qt xcb platform-plugin load error). Log /tmp/target.log a few seconds after launch so the real cause is visible rather than a bare window-find timeout. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * chore(cua-driver-rs): bake version 0.5.1 into install scripts [skip ci] * test(linux): point PyQt5 at qtbase's xcb platform plugin The qt app failed to launch: `qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in ""`. A bare `python3` PyQt5 invocation doesn't inherit qtbase's plugin path. Export QT_PLUGIN_PATH / QT_QPA_PLATFORM_PLUGIN_PATH from qt5.qtbase's qtPluginPrefix so the xcb plugin is found and the window appears. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): read back IsEnabled + dump launcher log (diagnostic) Both GTK and Qt apps launch fine but register 0 AT-SPI applications, even after setting org.a11y.Status.IsEnabled. Read the property back (print-reply) and dump the at-spi-bus-launcher log to determine whether the Set is taking effect or the toolkit bridges simply aren't activating in this session. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): force Qt AT-SPI bridge on (QT_LINUX_ACCESSIBILITY_ALWAYS_ON) IsEnabled is confirmed true on the a11y bus, yet the Qt app still registers 0 applications — Qt's bridge isn't activating from the bus handshake in this headless session. Set QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1 (and QT_ACCESSIBILITY=1) in the qt launch to force Qt to export its accessible tree. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * Delete JOURNAL.md * Delete JOURNAL_VIDEO.md * test(linux): validate AT-SPI read path; document focus-free write limit Per investigation, focus-free WRITE into a *background, unfocused* toolkit window isn't reliably supported: toolkits gate editable accessibility on focus/activation (Chromium exposes fields read-only over AT-SPI; an unfocused Qt window exposes only its top node; a GTK app's atk-bridge doesn't register in this headless session). Chromium's own AT-SPI impl does expose a full read-only tree. So assert the proven READ path: the driver's get_text returns the background window's accessibility/structure (a window/frame/document node) for every app in the matrix — native tree for Chromium, at least the window node (native or X11 fallback) for the others. type_text is still exercised but its readback is no longer asserted; the write-needs-focus limitation is documented inline. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): add focus-gate confirmation run (diagnostic, non-fatal) After the focus-free assertions, activate the target window and re-run the driver, logging the focused get_text and whether the typed text now reads back. This directly confirms the finding that toolkits expose the editable only when the window is focused. Non-fatal: it's evidence in the logs, not a gate (behaviour differs per toolkit). https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * ci(linux): temporarily disable firefox background-GUI matrix job Firefox times out at launch under the emulated CI VM (no KVM) — it never surfaces its window within the wait, so the job fails before any AT-SPI subtest runs. This is an environmental launch issue, not a driver problem, and the browser/AT-SPI read path is already covered by the chromium job. Drop "firefox" from the flake check list and comment out its workflow matrix entry; the app definition is kept so it can be re-enabled once launch is made reliable (longer timeout + pre-seeded first-run-free profile). https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): add CDP focus-free write override + Electron matrix job Chromium/Electron expose their fields read-only over AT-SPI, so the driver can't write into a background browser window through it. Add an approved Chromium/Electron-specific override using the Chrome DevTools Protocol: Input.insertText targets the page's focused DOM element regardless of OS window focus, so it lands in the unfocused background window. - chromium/electron launch with --remote-debugging-port + --remote-allow-origins - new asserting subtest drives a stdlib-only CDP client (HTTP target discovery + minimal RFC-6455 WebSocket) to insertText into the background window and reads it back, while asserting the control terminal keeps X focus - add a minimal Electron app (Chromium-backed BrowserWindow) as a new matrix job; like chromium it's read-only over AT-SPI and writable via CDP - wire "electron" into the flake matrix https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): expand background-GUI matrix with qt6, gtk4, tk Broaden toolkit/version coverage of the background-GUI a11y suite: - qt6 (PyQt6): same AT-SPI bridge as qt5 on the current Qt major; sets the lib/qt-6 plugin path and libxcb-cursor (Qt 6.5+ needs it headless) - gtk4 (compiled C GtkEntry): GTK4 talks AT-SPI directly (no atk-bridge module), contrasting the GTK3/zenity bridge path; cairo renderer + x11 backend keep it headless-safe - tk (tkinter): negative control — Tk has no AT-SPI bridge, so get_text degrades to the X11 window node, proving graceful handling of non-accessible toolkits All wired into the flake matrix as independent jobs. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * ci: run electron/gtk4/qt6/tk background-GUI jobs The nix-build matrix is hardcoded here (not derived from flake.nix), so the new flake checks added for electron, gtk4, qt6 and tk never ran in CI. Add them to the matrix so the expanded suite executes, including the CDP focus-free-write assertion on electron. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * test(linux): accept text/entry nodes in the read assertion Qt6's AT-SPI bridge exposes the editable even while unfocused, so the driver's focus-free write lands and get_text returns a bare `text "..."` node rather than a frame/window/document. Broaden the read-back assertion to accept text/entry nodes too (also future-proofs gtk4, which exposes the entry directly). The narrow frame/window/document check was the only reason the qt6 job failed — the read (and write) actually worked. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd * feat(linux): add GTK3 focus-free write fallback via X11 click+type GTK3's AT-SPI bridge gates EditableText on window/widget focus, so unfocused background windows expose entry nodes in the tree (reads work) but not the EditableText interface (writes fail). Qt6 exposes EditableText unconditionally. This commit adds a GTK3-specific fallback: when insert_text finds an entry/text role with Component bounds but no EditableText, it: 1. Gets the entry widget's screen coordinates via Component.GetExtents 2. Translates to window-local coords 3. Sends an X11 click to the entry's center to establish widget focus 4. Types via XSendEvent (now accepted by the internally-focused widget) The window remains unfocused (control terminal keeps X focus), but the widget receives and processes the keystrokes. This unblocks the gtk job in the background-GUI test matrix. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(linux): focus-free Tk writes via send command Tk has no AT-SPI bridge, so background writes use Tk's `send` IPC instead. The test app registers as "cua-tk-target" and the driver injects text by spawning `wish` to send Tcl commands. This is the Tk-specific override (like CDP for Chromium), proving non-accessible toolkits can support focus-free input with bespoke paths. - Add inject_tk_send() in platform-linux/input/mod.rs - Wire it into type_text tool after AT-SPI, before XSendEvent fallback - Update Tk test app to register with tk appname + name entry widget - Add tkSubtest that asserts the write lands and focus stays put - Include pkgs.tk so wish is available in the test environment Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(linux): add GTK3/GTK4 focus-free write fallback via X11 click+type GTK3 and GTK4's AT-SPI bridge gates EditableText on window/widget focus, so unfocused background windows expose entry nodes in the tree (reads work) but not the EditableText interface (writes fail). Qt6 exposes EditableText unconditionally. This commit adds a GTK fallback: when insert_text finds an entry/text role with Component bounds but no EditableText, it: 1. Gets the entry widget's screen coordinates via Component.GetExtents 2. Translates to window-local coords 3. Sends an X11 click to the entry's center to establish widget focus 4. Types via XSendEvent (now accepted by the internally-focused widget) The window remains unfocused (control terminal keeps X focus), but the widget receives and processes the keystrokes. This unblocks the gtk3 and gtk4 jobs in the background-GUI test matrix. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(linux): use AT-SPI Component.GrabFocus for GTK4 focus-free writes GTK4 gates EditableText on widget focus, unlike Qt6 which exposes it regardless of focus state. When a GTK4 window is in the background, the AT-SPI tree contains entry/text widgets (so reads work) but EditableText is unavailable, blocking focus-free writes. Call Component.GrabFocus on the target widget before accessing EditableText. This gives the widget internal keyboard focus without activating its window, allowing GTK4 to expose EditableText on the focused widget. The approach is: 1. Find target editable widget (same priority as before) 2. If it has Component interface, call GrabFocus on it 3. Proceed to call EditableText.InsertText as usual Benefits: - No window activation: GrabFocus works at widget level, not window level - Toolkit-agnostic: Component.GrabFocus is standard AT-SPI - Non-breaking: if GrabFocus fails/unavailable, still try EditableText (Qt6+) - Diagnostic logging shows GrabFocus success/failure for debugging This should allow the gtk4 background-GUI test to pass with true focus-free writes: the control terminal stays active throughout, the GTK4 entry gains internal focus via GrabFocus, and EditableText.InsertText succeeds. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * ci: generate GIF artifacts for all background GUI tests - Set visual: true for gtk, qt, qt6, gtk4, chromium, electron, tk tests - Add artifact_name for each test so GIFs are uploaded - Update PR comment script to list all new artifacts This will make it easy to visually verify focus-free writes work correctly for each toolkit by watching the GIF showing the window staying unfocused. * feat(linux): enable focus-free background writes for Qt5 via synthetic focus events Adds three-tier typing strategy for Linux: 1. Native AT-SPI EditableText (Qt6, GTK4 focus-free) 2. Synthetic FocusIn → AT-SPI → FocusOut (Qt5 workaround) 3. X11 XSendEvent fallback (terminal/legacy apps) The synthetic-focus path sends FocusIn to trigger Qt5's AT-SPI bridge without changing the X11 active window, enabling focus-free writes. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: restore GTK3 fallback code after merge conflict resolution The GTK3 widget click fallback was accidentally removed when resolving the merge conflict for PR #1817. This restores the entry_find_window_xid and screen_to_window_coords helpers and the GTK3 X11 click+type fallback logic that enables focus-free writes for GTK3 (zenity). * fix(platform-linux): qualify Command in atspi python fallback The merge-conflict resolution that restored type_into_editable's pyatspi fallback reintroduced `Command::new("python3")` without a `use std::process::Command;` import, breaking the cua-driver build (E0433: cannot find type `Command`) and thus every nix CI job. Fully-qualify the call as `std::process::Command::new` (matching the style in tools/impl_.rs) to restore compilation without touching imports. https://claude.ai/code/session_01MFLNL9q7v5xd3rXqZuvsgd --------- Co-authored-by: Francesco Bonacci <f@trycua.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: hippoley <hippoley@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: trycua-release[bot] <trycua-release[bot]@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com> * fix(nix): bump cua-driver to 0.5.1 and refresh cargoHash The main merge bumped the workspace to 0.5.1 and changed Cargo.lock, so the vendored-deps cargoHash was stale, failing all Linux/NixOS Nix tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(linux): bound Tk `send` so the tk background-GUI test can't hang Tk's `send` is synchronous: it blocks the sender until the target's Tcl event loop replies, and the X server must permit it. In the headless openbox/Xvfb session the tk job wedged in the "Tk send focus-free write" subtest with no timeout anywhere, so the GitHub job timed out at 15 min. Two unbounded waits caused the hang: 1. Driver `inject_tk_send` spawned `wish` and called `wait_with_output()` with no timeout — a blocked `send` wedged the driver task forever. 2. The test readback `wish /tmp/tk-get-value.tcl` ran with no `timeout`; a blocking synchronous `send` hung the whole NixOS test. Fixes: - Driver: issue the write with `send -async` (keeps the local event loop live) guarded by a Tcl `after` timer, and add a Rust wall-clock backstop that polls `try_wait()` and hard-kills `wish` after 15s, falling back to XSendEvent. The driver task can no longer hang. - Test: wrap the readback `wish` in `timeout 30` (hard backstop) and make the readback Tcl self-terminating with an `after` timer + catch that emits clear diagnostics. The subtest now passes when the write lands or fails fast with diagnostics instead of hanging 15 min. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(platform-linux): stop Qt5 AT-SPI segfault by disabling property cache The "Linux background GUI test (qt)" job crashed: when cua-driver walked the background Qt5 (PyQt5) window over AT-SPI, the Qt5 app segfaulted in libQt5Core (AtSpiAdaptor::handleMessage -> QVariant::toString), so the typed text never landed and the readback assertion failed. qt6 passed. Root cause: our AccessibleProxy in `accessible_for` was built with the zbus default `CacheProperties::Lazily`. The first property read (`acc.name()`) makes zbus issue `org.freedesktop.DBus.Properties.GetAll`, a one-argument call. Qt5's AtSpiAdaptor::handleMessage assumes every Properties message is Get/Set and unconditionally reads `message.arguments().at(1)`; for GetAll that index is out of range, and the following `QVariant::toString()` dereferences garbage -> SIGSEGV inside the Qt5 app. Qt6's bridge handles GetAll, which is why only Qt5 crashed. Fix: build the AccessibleProxy with `CacheProperties::No`, so zbus issues per-property `Get` calls (two arguments) that Qt5 handles correctly. The window can then be walked and written without killing the app. The sub-interface proxies from `proxies()` already used `CacheProperties::No`; this aligns the top-level Accessible proxy. No behavior change for other toolkits (they already tolerate GetAll). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(nix): record a GIF artifact in every Linux background GUI matrix job The 7 Linux background GUI matrix jobs (gtk, gtk4, qt, qt6, chromium, electron, tk) ran with `visual: true` but recorded nothing, so the workflow's `find -L "<result>/" -name '*.gif'` and `actions/upload-artifact` step warned "no files found". Add X11 screen-recording of display :99 to linux-background-gui.nix: start the recorder before the AT-SPI drive subtest, stop it and copy the per-app GIF (/tmp/cua-driver-linux-background-gui-<app>.gif) into the test derivation's $out *before* any toolkit assertion can fail, so even the failing jobs (qt, tk) still upload a GIF. The drive step now uses machine.execute instead of machine.succeed so a non-zero driver exit can't abort the test before the GIF is copied out. Adds pkgs.imagemagick to the GUI test's systemPackages. Factor the duplicated recordGifScript out of linux-cursor-click-gif.nix and linux-background-terminal-gif.nix into a shared record-x11-gif.nix imported by all three tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(platform-linux): land GTK4 focus-free write via generic AT-SPI path The "Linux background GUI test (gtk4)" job only passed because it did not assert a write: headless, the GTK4 app exposed only its top window node over AT-SPI (no GtkEntry child), so the driver's tree walk found nothing editable to write into. This is a tree-exposure problem, not a missing write technique — the generic AT-SPI EditableText + Component.GrabFocus path in atspi::native::insert_text already targets GTK4. Two complementary fixes, both through the generic path: App/launch (nix test): GTK4 talks AT-SPI directly but only builds/exports its accessible tree when it selects the AT-SPI accessibility backend at startup. In the hand-rolled headless session GTK4's auto-detection picks the "none" backend, leaving the tree empty. Force it on with GTK_A11Y=atspi so the GtkEntry is exposed with EditableText. Driver (native.rs): generalize the Qt5 synthetic-focus workaround into a toolkit-agnostic "expose-via-synthetic-focus" fallback inside insert_text. When the walk finds no editable, send a synthetic FocusIn (XSendEvent — does not move the X11 active window, so the no-focus-steal contract holds), let the toolkit rebuild its subtree, re-walk, and retry the EditableText write, then always FocusOut. Factored the editable-pick + GrabFocus + write into pick_editable/write_into_editable helpers so both the primary and re-walk attempts share one code path. Test: extend the "Input landed" typed-text assertion to include gtk4 (was qt/qt6 only). gtk (zenity/GTK3) stays read-only with a precise comment: GTK3 joins the bus via libatk-bridge, which reads org.a11y.Status IsEnabled once at startup; that handshake is racy here so registration is not reliably achievable in this CI session (not fundamentally impossible). Validation: cargo check -p platform-linux --target x86_64-unknown-linux-gnu passes (clean, no new warnings); nix-instantiate --parse of the test file passes. platform-linux is cfg(target_os="linux")-gated and cannot be built on the macOS dev host; CI runs the real gtk4 nixos test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Remove synthetic-focus-nudge fallback from AT-SPI insert_text GTK4 focus-free write now relies solely on GTK_A11Y=atspi exposing the GtkEntry plus the GrabFocus inside write_into_editable; drop the generic expose-via-synthetic-focus (FocusIn/re-walk/FocusOut) fallback. The Qt5 synthetic-focus workaround in tools/impl_.rs is unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(linux-background-gui): real-app READ-ONLY skeleton matrix (5 per toolkit) Replace the toy gtk/gtk4/qt/qt6/electron entries with a matrix of REAL desktop applications — 5 per toolkit category — run as a lenient, read-only smoke test. Keep chromium (CDP focus-free-write override) and tk (Tk `send` override) as full entries; skip Tk-family expansion. Skeleton entries (skeleton = true) find the app window via a per-app xdotool matcher (with PID / newest-window fallback + 120s timeout), drive cua-driver `page get_text` (read only), and assert: (a) the window appeared, (b) get_text returned a non-error accessibility response (no role required), (c) focus stayed on the control terminal, (d) a GIF was produced and copied out. Focus-free WRITE / typed-text assertions are intentionally OUT OF SCOPE here and added later per-app via trajectories. App matrix (verified to exist in the pin): - GTK3: gedit, mousepad, geany, scite(SciTE), abiword - GTK4: gnome-text-editor, gnome-characters, gnome-console(kgx), gnome-contacts, gnome-calendar - Qt5 (qtbase 5.15.x): manuskript(PyQt5), klog, wsjtx, qsstv, openambit - Qt6 (qtbase 6.x): kdePackages.{kate,kcalc,okular,ghostwriter}, qownnotes (kwrite is not packaged separately in the pin, so qownnotes takes its slot) - Electron: marktext, zettlr, vscodium(codium), joplin-desktop, logseq Wire all 27 keys into flake.nix, add a matrix.include job per app in nix-build.yml (25-min timeout for Electron, 15 otherwise) and list the new artifacts in the comment-linux-visual-artifacts job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(linux-background-gui): make windowFindCmd a script path, not inline string The multi-line windowFindCmd shell snippet was interpolated into the Python testScript as a "..." argument to wait_until_succeeds, whose embedded newlines broke the string literal — failing the NixOS testScript type-check for every GUI job (chromium/tk included) before any VM booted. Emit it as a writeShellScript store path (one safe token) instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(linux-background-gui): drop the 9 apps that fail headless, keep the 18 green Remove the GUI skeleton entries that failed CI (run 26923526185): GNOME GTK4 text-editor/console/contacts/calendar, qt5 wsjtx/qsstv, qt6 ghostwriter, electron marktext/vscodium — they either never surfaced a window within 120s or stole focus on launch. Keeps the 18 passing jobs (GTK3 x5, gtk4-characters, qt5 manuskript/klog/openambit, qt6 kate/kcalc/okular/qownnotes, electron zettlr/joplin/logseq, chromium, tk) across the test apps set, flake check list, and CI matrix + artifact list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(linux): AT-SPI + Set-of-Marks annotated screenshots for skeleton matrix (#1832) * feat(linux): emit AT-SPI + Set-of-Marks annotated screenshots for skeleton matrix For each read-only skeleton app in the background-GUI NixOS test matrix, emit two annotated screenshots as CI artifacts: `<app>-atspi.png` (AT-SPI element boxes + screen coords) and `<app>-som.png` (cua Set-of-Marks). ch…
This was referenced Jun 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
On Windows the agent cursor overlay was a process-wide singleton — one
RenderStatebehind the single layered window. Every cursor op (move_cursor, theClickPulseof click/double_click/right_click/drag,set_agent_cursor_*) drove that one state with no session discriminator, so concurrent MCP sessions clobbered each other last-writer-wins → one shared cursor for all sessions.#1779 fixed exactly this on macOS but scoped itself macOS-only: "Windows (
platform-windows/src/overlay.rs) and Linux compile and behave exactly as before (single cursor; the key concept never reaches them)." The 0.4.x daemon already mints per-session identity (caller-declaredsession,session_endhook, idle-TTL) — this PR uses it to give each Windows session its own cursor, mirroring the macOS data model 1:1.What changed
platform-windows/src/overlay.rs— the keyed render collection:static RENDER: Mutex<Option<RenderState>>→Mutex<Option<RenderMap>>whereRenderMap { cursors: IndexMap<CursorKey, RenderState>, … }.IndexMap= deterministic insertion-ordered iteration = stable per-session z-order frame to frame.send_command(key, cmd)carries aCursorKey; theWM_TIMERtick drains keyedOverlayMsgs via get-or-create, ticks every cursor, fires per-key arrival oneshots, and composites all cursors into the ONE layered window viapaint_cursor(alpha-over, insertion order = paint order). Idle/hidden cursors early-return.Palette::for_instance(key)→ each session a distinct colour automatically.remove_cursor+ render-sideendedtombstone (resurrection guard);"default"guarded against removal so the anonymous / one-shotcua-driver callpath is unchanged. Per-key arrivalHashMapso concurrent sessions don't cross-cancel glide-waits. Sentinel-seed so a session's first AX action glides instead of snapping.platform-windows/src/tools/impl_.rs:resolve_cursor_key— precedencesession>cursor_id>NO_CURSOR(never the connection-injected_session_id), identical toplatform_macos::tools::cursor_tools::resolve_cursor_key.pin_overlay_above,overlay_glide_to, everyClickPulsecallsite, and the 5 cursor tools (move_cursor,set_agent_cursor_enabled/_motion/_style,get_agent_cursor_state).session_endhook →overlay::remove_cursor(+cursor_registry.remove), so explicitend_session, the CLI verb, and the idle-TTL sweep all drop a session's cursor.get_config'scursor_enabledis now session-scoped + deterministic (wasall_states().first()over a HashMap — macOS BUG 3).cursor-overlay/src/lib.rs—CursorRegistry::remove(guards"default").Cargo.toml—indexmap.page.rs/stubs.rs— updated to the keyed API.Known limitation (follow-up)
page.click_elementdrives the seeded"default"cursor: the cross-platformPageBackendtrait carries no callersession. Threading session through the trait is a separate cross-platform change (macOS doesn't implementclick_elementat all today).Verification
cargo build -p platform-windows+-p cua-driver— clean, zero warnings.cargo test -p platform-windows --lib— 49 passed (15 new: two-session isolation,session_endremoval, default-guard, resurrection tombstone, sentinel seed, key-resolution precedence, two-parallel-session distinct keys).install-local, daemon restarted): launched two Calculator instances, drove2+1and5+6from two declared sessions interleaved → two distinct-coloured cursors gliding/pulsing in parallel (confirmed by eye); results 3 and 11;end_sessionremoved each cursor.Advances #1777 (the Win/Linux half of the cursor session-scoping). Linux (
platform-linux/src/overlay.rs) is still single-cursor — a natural follow-up using the same sharedcursor_overlay::{CursorKey, KeyedOverlayCommand, OverlayMsg}types.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Refactor