cua-driver 0.7.0 — action-time modality, honest verification, browser + Wayland (Rust-only) by f-trycua · Pull Request #2077 · trycua/cua

f-trycua · 2026-06-30T18:30:45Z

cua-driver 0.7.0 — action-time modality, honest verification, browser + Wayland

This release reshapes the modality model, makes every action honest about whether it actually happened, extends the driver into the browser DOM, and completes the Rust-only consolidation — with a full docs overhaul to match.

Release: this PR is the 0.7.0 milestone. Cut the version after merge via the release-bump-version workflow (service=cua-driver-rs, bump_type=minor) → tags cua-driver-rs-v0.7.0. No version files are touched here.

The model: perception is always both; modality is chosen at action time

get_window_state now always returns the accessibility tree and a screenshot in one call. capture_mode is deprecated and ignored (accepted for back-compat; som/screenshot decode to nothing). Perf opt-out: include_screenshot:false.
You no longer pick a capture mode — you pick how you address the target on the action call: element_index/element_token → the AX rung; x,y → the px rung. This includes the keyboard family (type_text/press_key/hotkey accept x,y = pixel-focus then type).
delivery_mode: background (default, no focus steal) | foreground (explicit escalation). Legacy auto removed. Background-first is mandatory — the driver returns background_unavailable when it truly can't, rather than agents fronting on a guess.

Honest verification (the headline)

Every action returns verified / path / effect / escalation:

path — the rung/primitive that actually ran (ax, cgevent, key_events, x11_atspi, pixel, msaa, …).
verified — true only when the driver read the effect back (AX read-back); else the caller confirms via the screenshot in the same response.
effect — confirmed / unverifiable / suspected_noop.
escalation — {recommended: "px" | "foreground" | "page", reason}.

Surface-aware verification ("the tree lies"): Electron echoes a set_value/type against its AX shim, Catalyst reports null AXValue, browser web content echoes DOM writes it never applied. The driver detects the web surface at the element level (AXWebArea ancestor) and refuses a false verified:true — it returns effect:"unverifiable" + an escalation to px (Electron) or page (browser tab). This kills the systemic false-success pattern.

Browser / web (`page` tool)

CDP-native typing rungs: insert_text (fast DOM insert) and type_keystrokes (per-character key events that fire JS handlers), plus get_text / query_dom / click_element / execute_javascript. This is the escalation:"page" target when AX typing echoes but the DOM doesn't observe it.
CDP target resolution re-resolves every call (a reopened tab reusing a closed tab's URL no longer swallows insert_text into a dead session).

Per platform

macOS — foreground rungs for click/type; set_value CFNumber + AXIncrement/Decrement stepping; pixel right-click / wheel / windowless click; honest verified:false+path on drag/scroll; element px keyboard actions.
Windows — delivery_mode alignment; focus-independent read-back verification; off-screen click guard; WinUI3 double/right-click; UWP → ApplicationFrameHost host window; element-cache UAF fix.
Linux X11 — Xvnc foreground keyboard fix (XTEST round-trip + shift levels); GTK left/right/double/middle-click + scroll; GTK4 GetExtents(SCREEN)=(0,0) coordinate reconstruction; session-bus auto-discovery; real EWMH bring_to_front.
Linux Wayland — background AX clicks (coordinate-free doAction); vision pixel-clicks land via screen-pixel → covering element → role-aware doAction (no pointer injection); pixel coordinates + the real gradient-arrow agent cursor via the bundled org.cua.WinRects GNOME Shell helper. Residual gap (documented): native-Wayland raw keyboard.

Rust is the only implementation

The Swift backend and its release flow are retired (f58d4b00). CuaDriver.app is unchanged for users (it's now the Rust bundle); install-local/install route to Rust only.

Agent integrations

mcp-config --client gains Qwen Code, Factory Droid, and ZCode (Z.ai) alongside the existing 8 (Claude Code, Codex, Cursor, Antigravity, OpenClaw, OpenCode, Hermes, Pi) — with post-install hints and per-client docs. (cua-driver isn't yet listed in their docs — a separate upstream step.)

Docs

Reference: the response-signal contract (verified/path/effect/escalation), the page tool's full action set, corrected capture_mode/permission claims, the full Modality Test Suite + harness reference.
Explanation: capture-and-dispatch modalities, the no-foreground contract, "when the tree lies", and a new Linux & Wayland page.
How-to: choose a capture and dispatch mode, and a new drive a web page (browser-use) guide.
Tutorial: a mocked Claude Code / Codex terminal session showing an agent driving the driver in plain English.
Docs app: slimmed to a local MDX preview only (production cua.ai/docs is served from the website) — removed the CopilotKit assistant, PostHog analytics, prompt-digest cron, search index, llms/sitemap routes, custom branding, and the landing page.

Test harnesses

Cross-platform modality recorder + per-action effect verifier (WPF / WinUI3 / WebView2 / Electron / GTK3 / AppKit / SwiftUI / WKWebView); Linux container + VM lanes (modality_matrix.sh, derec.sh with a WAYLAND=1 lane + vclick); session lifecycle/disposal coverage; a cross-platform tool-schema consistency CI gate.

🤖 Generated with Claude Code

https://claude.ai/code/session_011BJPGdRQxiBPTQhjwZsQ8G

…fier (WPF + GTK3) Adds the single-modality recording tooling used to capture the no-foreground contract across the action matrix on Windows (WPF) and Linux (GTK3), plus a per-action EFFECT verifier (does the action change the app, not just whether it steals focus). FINDINGS.md documents the driver behavior the verifier surfaced: right_click landing on the taskbar, scroll/type no-ops, and the GTK XSendEvent limitation — to be triaged and fixed in the driver, then re-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

When an element_index resolves to a cached center outside its own window (scrolled out of a ScrollViewer, or pushed off-screen by a tall form on a small display), the click/double_click/right_click tools previously injected the synthetic tap at the raw coordinate — landing on whatever was there (the taskbar, the desktop), e.g. opening the Windows taskbar context menu instead of the target's. Add point_in_window_bounds() and return a clear error in that case instead of clicking the wrong target. Surfaced by the modality recordings: a right-click on an off-screen harness button opened the taskbar menu. Recorder: focus the text box before type_text (type targets the focused control; set_value doesn't focus, so type was a no-op without this). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

… shipped + verifier results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…Electron) Parameterize the recorder by -Toolkit and run an ax-bg pass on each Windows toolkit harness. Key finding: the no-foreground contract holds on WPF/WinUI3/ WebView2 but breaks on Electron (7/8 actions stole focus — Chromium self- foregrounds). Effect coverage varies by harness (WinUI3 thinnest at 2/7). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…elivery paths only The first off-screen guard sat before the click dispatch branch, so it also blocked the background UIA Invoke path — which fires an element's handler with no coordinates and works fine for off-screen elements. That regressed opening (and then listing via list_windows) the WPF legacy modals from their off-screen buttons. Move the guard to the foreground SendInput tap and the background coordinate-injection fallback only; UIA Invoke runs unguarded. Confirmed: bg click → UIA Invoke opens Harness MessageBox / Owned Popup / Layered Popup from off-screen buttons, and list_windows enumerates them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…a dead session fail loudly end_session tombstoned a session id permanently: every later tool call carrying that id hit the daemon's resurrection guard and got a benign "session ended; tool call ignored" (isError:false) — a silent trap that looked like success while doing nothing. Re-declaring the session did not help, forcing callers (e.g. the modality recorder) to mint a fresh unique id per run. Two changes make the lifecycle behave as a caller expects: - start_session with a previously-ended id now REVIVES it (session::revive_session clears the ENDED_SESSIONS tombstone), so explicitly re-declaring a recycled id works. The lifecycle tools (start_session/end_session) are exempt from the resurrection guard so start_session can run to revive and end_session stays idempotent. - A non-lifecycle action on an ended-and-not-revived id is now rejected LOUDLY (isError) with a message telling the caller to start_session to revive — instead of the silent no-op. The original safety (a stray late action can't silently resurrect a dead session) is preserved: reviving requires an explicit start_session re-declare. Adds unit tests for revive_session and updates the daemon gate_tests integration test to assert loud rejection + revive-then-run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ss launches The AppKit harness window opened at slightly different heights between launches (observed 832 vs 858 pt), shifting screenshot dimensions and breaking tests with hardcoded pixel coords. Root cause is the harness app, not the driver: the window was .resizable with macOS state restoration ON (NSWindow.isRestorable defaults to true) and no pinned content size, so the OS persisted and restored a drifted frame on the next launch. Fix in the harness: drop .resizable, set isRestorable = false, clear the frame autosave name, and re-pin a fixed content size (720x800) on every launch so each run is byte-identical. Rebuild via test-harness/build/macos.sh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…dalities Bring the cross-platform Electron test harness to fruition on Linux (Chromium via AT-SPI) and record its 5 modality modes — the Linux analogue of the Windows Electron lane. Build/launch glue: - electron/build.sh: Linux analogue of build.ps1 (stage shared web/, npm install the prebuilt linux-x64 electron runtime, optional flat-stage with a deterministic CuaTestHarness.Electron binary name). - electron/package.json: add start:linux (electron . --no-sandbox --disable-gpu --force-renderer-accessibility — the flags Chromium needs under Xvfb for the web-AX tree to register on AT-SPI) and a linux electron-builder dir target. Recorder (modality-recordings/linux/): - lin-rec-electron.py: launches the Electron app, resolves the harness window by the "CuaTestHarness Electron [cdp=NNNN]" title, and reads BOTH control geometry and harness state from the web-AX tree (get_window_state capture_mode=ax) — same per-action verifier as Windows, AT-SPI text instead of UIA. - lin-run-electron.sh / lin-dashboard-electron.html / lin-all-electron.sh. Headline (FINDINGS.md): Linux Electron breaks the no-foreground contract LESS than Windows Electron. Windows stole 7/8 in ax-bg; Linux steals 3/8 in ax-bg (only set_value/type/press-key; AX pointer actions all held) and 6/7 in vision-bg (pixel coordinate-injection foregrounds Chromium). All 5 modes frame-verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

Add the controls the 8-action matrix needs and that GTK3 lacked: click-target (btn-clicktarget, left/right/double → last_action=/clicks=), slider (sld-value), checkbox (chk-agree → agreed=), context-menu (btn-context → menu_action=), and a tall scroll-target (scroll-tall → scroll_offset=), matching the WPF harness's AutomationIds + status-label format. Keeps all existing controls the Rust harness_gtk3_test asserts (btn-increment/reset, txt-input, btn-open-popover, btn-exit, HARNESS_TEXT_MARKER_v1, counter=0, POPOVER_MARKER_v1). Verified on the Linux VM: btn-clicktarget/chk-agree/btn-context resolve via AT-SPI; slider + scroll are present but (as documented) not surfaced by Linux get_window_state. Updated scenarios.json gtk3 section to the unified set. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…n matrix parity Brings the WinUI3 test harness to control parity with the WPF harness so the cua-driver action matrix (click/double/right/drag/scroll/set_value/type/press) can be exercised on WinUI3. Adds, mirroring the WPF harness AutomationIds + label formats so the shared modality recorder resolves them identically: - click-target: Button (AutomationId border-click-target, "Click target (left / right / double)") with lbl-last-action (last_action=left_click|double_click|right_click) and lbl-click-count (clicks=N). Click fires per tap (timing promotes the 2nd to double_click); RightTapped/DoubleTapped cover secondary + double gestures. - scroll-target: ScrollViewer (AutomationId scroll-tall) with tall SCROLL_TOP_MARKER_v1..SCROLL_BOTTOM_MARKER_v1 content and lbl-scroll-offset (scroll_offset=N) bound to VerticalOffset. The window subclasses its HWND to translate the driver's posted WM_VSCROLL into ScrollViewer.ChangeView (WinUI3 routes input through composition and does not consume WM_VSCROLL natively) — same rationale as the WPF harness HwndSource hook. scenarios.json: adds click_target + scroll_target entries to the winui3 section so the source-of-truth reflects parity. Verified on the Windows VM: rebuilt harness, get_window_state shows border-click-target + scroll-tall (+ labels/markers) in the tree; ax-bg modality recorder effects-landed rose 2/7 -> 3/7 (scroll now lands). The click-target handlers are confirmed correct via foreground SendInput (double_click -> last_action=double_click clicks=2; right_click -> last_action=right_click). double/right do NOT land under ax-bg background dispatch because the driver does not yet detect WinUI3 in would_be_silently_dropped, so clicks fall through to PostMessage (dropped by WinUI3 composition input) instead of inject_click_screen — driver-side follow-up, tracked separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

Add CuaTestHarness.WKWebView — the macOS analogue of the Windows WebView2 harness. A minimal AppKit NSWindow hosting a WKWebView (Apple WebKit, not Chromium) that loads the SAME shared web DOM (test-harness/shared/web/ index.html) the WebView2 and Electron harnesses use. Fixed content size + isRestorable=false so the window geometry doesn't drift between launches. build/macos.sh: add a `wkwebview` target (links -framework WebKit and copies shared/web/index.html into the app bundle's Resources so the harness is self-contained and reuses the canonical content rather than duplicating source). The .app bundle stays gitignored under rust/test-apps/. electron/package.json: add `start:macos` (electron . --force-renderer- accessibility — the flag needed to expose Chromium web-AX to cua-driver on macOS) and a `build:macos` electron-builder target, so the existing cross-platform Electron harness has a documented macOS launch/build path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…t not a simple WPF-style arm) The WinUI3 harness parity work surfaced that background double-click/right-click don't land on WinUI3. Tried mirroring the WPF arm in would_be_silently_dropped (route mouse clicks through the coordinate injector) — it did NOT work: WinUI3's composition input-site doesn't consume the synthetic pen the way WPF's stylus stack does, so double/right still no-op AND the no-foreground contract regressed (ax-bg 0/8 -> 8/8 stolen). Reverted; left an explanatory comment so the next attempt knows it needs a WinUI3-specific input path, not the WPF treatment. Single left-click already works via UIA Invoke. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…rix parity The shared web DOM (loaded by WebView2 + Electron on Win/Linux/macOS + the new macOS WKWebView) had click-target/slider/checkbox/text but no scrollable region, so the scroll action couldn't be exercised on any web surface. Add a scroll_target fieldset: a tall overflow:auto div (scroll-tall) between SCROLL_TOP/BOTTOM markers with a scroll_offset= label that updates on scroll — matching the WPF harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…WPF/GTK3 Bring the macOS harnesses to control parity so the 8-action matrix can be exercised on macOS via the same controls as every other platform. AppKit (apps/macos/appkit/main.swift): - click_target: replace the non-AX NSView with a real NSButton id=btn-clicktarget (AX-addressable via AXPress) → last_action=click|double_click|right_click, clicks=N - slider id=sld-value (NSSlider 0..100) → slider_value=N - checkable_controls id=chk-agree (NSButton checkbox) → agreed=true|false - context_menu id=btn-context (NSButton + NSMenu Cut/Copy/Paste) → menu_action=... - scroll_target: add SCROLL_TOP_MARKER_v1; scroll-tall id moved to the document AXTextArea (the surfaced scrollable node); offset label → scroll_offset=N - all existing controls (counter, text_body, text_input, ns_menubar, exit) kept SwiftUI (apps/macos/swiftui/main.swift): - click_target id=btn-clicktarget (Button + count:2 gesture) → last_action/clicks - slider id=sld-value (Slider) → slider_value=N - checkable_controls id=chk-agree (Toggle .checkbox) → agreed=true|false - context_menu id=btn-context (Button + .contextMenu) → menu_action=... - scroll_target id=scroll-tall (ScrollView, GeometryReader offset) → scroll_offset=N - existing controls (counter, text_body, text_input, popover, exit) kept scenarios.json: add slider/checkable_controls/context_menu (+ updated click_target and scroll_target) to the appkit section; add click_target/slider/checkable_controls/ context_menu/scroll_target to the swiftui section. Surgical edits only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…e controls NSSlider/AXStepper expose a numeric AXValue and reject a CFString write with -25201 (illegal argument) — so set_value on a slider failed. Add set_number_attr and, in the default AXValue path, try a CFNumber when the value parses as a number, falling back to a CFString (text fields still take a string). Surfaced by the macOS harness control-parity work. Compile-verified; runtime check on a live NSSlider deferred (Mac in use). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…order Wire the Linux Electron modality recorder to the new shared web scroll-target: - resolve `scr` to the scroll-tall clipped viewport (section ~260x120) in the Chromium web-AX tree instead of leaving it None - aim pixel/desktop scroll at the resolved region (AX path uses element_index) - verify scroll via the now-exposed `scroll_offset=` web-AX label (ok/fail instead of na) Measured on Linux Electron: scroll-tall resolves in all 5 modes, but the scroll action is a no-op everywhere (scroll_offset stays 0) — same AT-SPI XSendEvent limitation as the other Linux Electron pointer actions. ax-bg focus contract unchanged at 3/8 stole. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…surfaces Linux Electron scroll-target re-record (resolves all 5 modes, no-op via AT-SPI — contract still 3/8 stole in ax-bg), WebView2 harness restore + deferred full-parity recording, macOS 4-surface set + the numeric-set_value CFNumber driver fix and the two macOS session/geometry gotchas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…roll + set_value SwiftUI harness exposes all 6 WPF-parity controls (AX-actionable; checkbox press flips agreed=); set_value on the SwiftUI AXSlider reproduces the -25200 CFString-write failure the CFNumber fix targets (broaden the set_number_attr comment to note -25200 as well as -25201); WKWebView page-scroll works via keystroke PageDown but a nested overflow div without tabindex is a no-op. Live "after" verification of the fix needs a re-signed reinstall + TCC re-grant (Developer-ID/cdhash-pinned), recorded as such. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…t NSSlider Post install-local reinstall (post-fix daemon, TCC re-granted): set_value on the AppKit NSSlider now succeeds, slider_value 0->50 (frame-verified). The SwiftUI slider failure is separate — its AXSlider exposes only increment/decrement and rejects any AXValue write (-25200), so the value-type fix cannot help there; needs AXIncrement/AXDecrement (future work). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…AXValue is unsettable SwiftUI's AXSlider exposes a readable-but-unsettable AXValue plus increment/decrement actions, rejecting every direct AXValue write with -25200. Add a bounded AXIncrement/AXDecrement stepping fallback that runs only after both the CFNumber and CFString writes fail for a numeric target. - bindings.rs: add copy_number_attr() to read the control's current AXValue as f64 (mirrors copy_string_attr, downcasts to CFNumber). - set_value.rs: step_to_value() reads current value, steps toward target, stops within half the last observed step, breaks if a step doesn't move the value, hard-capped at 500 iterations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

Replace the live Azure blob URL with a generic reference before the branch goes up as a PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

Port the macOS #1796 retain-under-lock fix to platform-windows. The element cache handed out a bare COM vtable pointer; under concurrent sessions a get_window_state snapshot-replace could Release the element to zero while an in-flight click/type/set_value still dereferenced it (use-after-free → daemon crash), the Windows analogue of the macOS AXUIElementCopyActionNames SIGTRAP. New RetainedElement guard AddRefs the element inside with_snapshot (under the same mutex insert takes), so the refcount can't reach zero between lookup and AddRef; the guard's +1 keeps the object alive across a concurrent replace and Releases on drop. ClickTool / TypeTextTool / SetValueTool hold the guard for the whole blocking action. Compile-verified on the Windows VM (cargo build -p platform-windows, clean); the concurrency race was not runtime-reproduced (timing-dependent dual-session). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…-only widgets Two AT-SPI gaps surfaced by the modality verifier: 1. Pixel clicks were silent no-ops on GTK. GTK3/4 take input via XInput2, so the background XSendEvent path (synthetic, send_event=True) is dropped, and XTEST core events don't reach an XI2-only client (measured on the Xvfb stack: xinput sees the event, the GTK client sees zero). Route a plain left single-click through AT-SPI instead: hit-test the window-local pixel to the smallest-area actionable element (GetExtents(CoordType::Window)) and doAction it — lands without activating/raising the window (no-foreground contract preserved, verified). Scoped to left single-clicks; right/double/ middle and points with no actionable element fall back to XSendEvent. 2. Sliders / scroll bars never appeared in get_window_state. The walk indexed only nodes advertising Actions; GtkScale and scroll bars expose the Value interface but no Action, so they were dropped from `elements` (though they showed in tree_markdown). New is_indexable = !actions.is_empty() || has_value, applied identically in render and every action_nodes filter to keep the index space consistent. element_count 8->13 on the GTK3 harness; set_value on the slider (0->50) and scroll bar (scroll_offset 0->300) now work. Compile-verified on the Linux VM (cargo build -p platform-linux clean, 32 platform-linux unit tests pass). Honest scope: right/double-click pixel actions on GTK still can't land on this Xvfb (no AT-SPI doAction equivalent) and were left unchanged rather than shipping a focus-stealing XTEST hack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…+ fix MP4 pipeline The cross-toolkit recorder under-recorded WinUI3/WebView2 and produced empty MP4s. Fixes: - WebView2: set WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS=--force-renderer- accessibility so the web DOM surfaces in get_window_state (without it only the chrome frame appeared → every web action resolved to nothing → SIZE=0). - ffmpeg: prepend the Gyan.FFmpeg bin to PATH before serve. The daemon probes cuademo's WinGet LOCALAPPDATA but the task runs as fbonacci, so the probe missed ffmpeg → video present:false. - Resolver now finds the new parity controls across toolkits: scroll-tall by role Pane|Group, click-target by label or name, checkbox by role or "agree", context-menu falls back to the click-target (WinUI3/web record last_action=right_click); verifier accepts last_action=right_click. - Honest: WebView2's checkbox sits below the fold and web content can't be scrolled by the driver in ax mode (AX-scroll/WM_MOUSEWHEEL/PageDown all no-ops — the host HWND doesn't route scroll to the Chromium renderer), so its left-click step reports a true no-op rather than being faked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

… fixes WinUI3 (3/7) + WebView2 (4/7) re-recorded with parity controls (frame-verified, 0 focus steals); WebView2 needed --force-renderer-accessibility + ffmpeg-on-PATH + an attached Session 2. WinUI3 double/right-click confirmed not WPF-path-fixable (needs a composition-input-site path). Plus the Windows UAF, Linux GTK left-click/value-widget, and macOS numeric-set_value fixes from this round. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…ia uinput MPX Generalize the existing no-focus-steal uinput + XInput2-MPX + shield-grab drag pipeline to all pixel pointer actions, so they land on XInput2 toolkits (GTK3/4) that silently drop synthetic XSendEvent pointer events. - send_virtual_pointer_click: shield-grab → warp → press/replay/release train (count presses, sub-250ms cadence for double-click), tear down + restore focus. Lands right/middle/double-click on GTK without stealing foreground. - send_virtual_pointer_scroll: warp + REL_WHEEL/REL_HWHEEL detents (libinput → XI2 smooth-scroll GTK consumes); no shield needed (WMs don't focus on wheel). - real_pointer_input_available(): cheap capability probe (X vendor + Xvfb check) so callers skip the MPX path where uinput never becomes an X slave (Xvfb/VNC) without paying the slave-bind timeout. Auto-detected, no flag. - x11_pixel_click_no_focus_steal() wired at all 5 pixel-click sites; tier order is AT-SPI doAction (left single-click) → MPX (right/middle/double) → XSendEvent. Evaluated + rejected an AT-SPI right-click path: do_action(0) only fires the first advertised action and ATK exposes no standardized show-context-menu action. Compile-verified on the Linux VM (cargo build -p platform-linux clean). Runtime deferred: this Xvfb VM can't exercise uinput (no udev/libinput hotplug → uinput never becomes an X slave), so the MPX path needs a real Xorg host to confirm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…oll + windowless buttons Reconciles three macOS input fixes (the Rust port had diverged from the Swift reference, dropping the window-routing + priming that made AppKit delivery work): - Right-click: thread window-number + button-number (f3=1) through right_click_at_xy_inner with a mouseMoved primer + 28ms gap, so rightMouseDown reaches a backgrounded NSView (previously wid=None → WindowServer hit-test skipped non-key windows). post_mouse_event now takes an explicit button_number (was hardcoded left even for right/middle). - NSButton click: add a mouseMoved primer + wider down→up gap in click_at_xy_inner (AppKit's modal mouseDown tracking ignored the too-tight synthetic pair). - Scroll: new background-safe pixel-wheel scroll_wheel_at_xy posting per-pid (no foreground, no cursor warp) at a target point, wired into the scroll tool for element_index/element_token (→ element_screen_center) and window-local x/y — drives an unfocused NSScrollView and nested web overflow regions that the keystroke path can't reach. Keystroke mode unchanged when no coordinate target. - Windowless desktop-scope click now honors button: right/middle no longer silently degrade to left. Compile-verified on-host (cargo build -p platform-macos + cua-driver-core clean). Runtime deferred (host was screen-recording): confirm wheel sign + session-vs-HID tap + primer timings live. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…serving click scroll: add the coordinate/element-targeted pixel-wheel path (macOS) for nested overflow / unfocused scrollers, alongside the keystroke page-scroll; document the new element_token / x / y params. click: note desktop-scope screen-absolute clicks preserve the button (windowless right/middle no longer downgrade to left). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…f-screen scroll-into-view + UAF repro Reconciles four platform-windows fixes (compose on disjoint impl_.rs ranges): - WinUI3: background double-click now LANDS while holding the no-foreground contract — a double UIA Invoke under a WS_EX_NOACTIVATE guard fires the XAML Click handler twice. Empirically ruled out posting WM_*BUTTON to the content island (measured no-land) and pen injection (steals foreground). Right-click and drag have no contract-safe path → now return a structured background_unavailable error (retry with dispatch:foreground) instead of a silent false-success no-op. is_winui3_target_window detector added. - Electron foreground steal: extend the proven UWP EnableWindow shield gate to Chromium (is_xaml_host_hwnd || is_chromium_target_window), wrap UIA SetValue in run_with_uwp_bypass, and add a force_foreground_attached reclaim to inject_click_screen. - Off-screen controls: UIA ScrollItemPattern.ScrollIntoView + recenter before a coordinate action when the element resolves outside the window (background- safe via the bypass); falls back to the existing clean off-screen error. - UAF: a deterministic #[cfg(test)] reproduction proving the retain-under-lock fix — pre-fix path takes a real access violation in windows-core vtable dispatch; fixed path survives concurrent replace + a 6-thread stress loop. Compile-verified on the Windows VM (cargo build -p platform-windows clean); cache_uaf_repro tests pass (3 passed). WinUI3 double-click + contract runtime-verified live on the harness; Electron contract + off-screen actuation not re-measured (honest). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

…type_keystrokes Adds two new `page` tool actions for rich-text contenteditable editors where execCommand-based writes get silently discarded on the next render: insert_text (single CDP Input.insertText call) and type_keystrokes (real per-character Input.dispatchKeyEvent keyDown/char/keyUp sequences). Both reach a browser via a launch-time --remote-debugging-port on a dedicated profile, or via chrome://inspect's remote-debugging toggle on the user's real profile (which needs a browser-endpoint Target.attachToTarget fallback since it doesn't serve the classic /json HTTP discovery). Renamed the launch_app param electron_debugging_port -> cdp_debugging_port since it isn't Electron-specific. Also fixes several latent bugs found while building and live-testing this: - listening_ports()'s lsof call was missing -a, so it silently returned every listening port on the system instead of just the target pid's. - http_get_json used read_to_end, which hangs forever against Chrome's CDP server (it ignores Connection: close and replies via Content-Length instead) — now parses Content-Length properly with a hard timeout. - Setting the `text` field on the keyDown event (not just `char`) made Chrome double-insert every character. - CdpSessionCache now reuses one open connection per port across calls instead of reconnecting every time — Chrome's "Allow remote debugging?" popup fires on every new connection, not once per session, so reconnecting per call meant re-prompting on every single action. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nH7srSSrCyhSDaLCeyH29

…pture-mode claims Close the doc gaps from the modality-rework audit (0.7.0 model): Reference (mcp-tools.mdx): new "Action response shape" section — the verified / path / effect / escalation fields with their real value sets (path = rung primitive names: ax, cgevent, key_events, x11_atspi, pixel, msaa, …). Added the `x,y` px form note to the keyboard tools. Explanation (capture-and-dispatch-modalities.mdx): new "Response signals" section (verification/effect/escalation — the why), and expanded "When the tree lies" (Electron echo, Catalyst null AXValue, Chromium/WebKit web content; the AXWebArea element-level probe; surfaces as verified:false + effect:unverifiable + escalation). How-to (choose-a-modality.mdx): px keyboard fallback steps + the escalation.recommended:"page" → page/CDP branch. Stale fixes: contracts.mdx (drop capture_mode from live config keys), limits.mdx (Screen Recording no longer gates "som/vision capture modes"), linux-and-wayland.mdx (ax is verifiable *without a screenshot*; px gets an honest effect/escalation verdict). Kept quadrant-pure (value tables in reference, why in explanation, steps in how-to). Reference-page rewrites drafted via codex against the code-accurate contract; `pnpm build` + docs:check-links pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011BJPGdRQxiBPTQhjwZsQ8G

…Code Add three coding agents to the mcp-config client set, matching how the existing clients are wired: - `mcp-config --client qwen` → `qwen mcp add cua-driver <bin> mcp` (Qwen Code; also settings.json mcpServers) - `mcp-config --client droid` (alias `factory`) → `droid mcp add cua-driver "<bin> mcp"` (Factory Droid; also .factory/mcp.json) - `mcp-config --client zcode` → GUI paste JSON (ZCode by Z.ai — no CLI; the standalone `zai` CLI one-liner is noted as an alternative) Also: post-install-hints.txt gains the three, the invalid-client error + the --client help/discussion lists are kept in sync, and connect-your-agent.mdx gets a section per client. Config formats verified against each tool's official docs. `cargo check` + `pnpm build` + docs:check-links pass; --client outputs smoke-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011BJPGdRQxiBPTQhjwZsQ8G

…ge tool) how-to Two gaps in the getting-started + how-to docs: - Tutorial (drive-your-first-app): new "Ask your agent" step with a mocked Claude Code / Codex session — the user prompting in plain English ("Using the cua-computer-use MCP …" / "Using the Cua Driver skill …") and the agent driving the tool sequence to compute 6×7. Replaces the older prose-only step. - New how-to (drive-a-web-page): the browser-use section built on the `page` tool — get_text / query_dom to read, click_element / insert_text / type_keystrokes to act, execute_javascript for arbitrary JS; the CDP / --remote-debugging-port notes; and the tie-in to the honest-verification escalation:"page" rung. Registered in the driver how-to nav. - Reference (mcp-tools #page): add the insert_text and type_keystrokes actions (shipped in 6d56532) + their text/cdp_port/target_url_contains args, which the table was missing. Written via codex against the code-accurate page-tool API (7 actions, no navigate). `pnpm build` + docs:check-links pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011BJPGdRQxiBPTQhjwZsQ8G

…kground-first delivery - macOS CDP (cdp_client.rs): ensure_target always re-resolves + reattaches on the same connection instead of skipping when current_target_url already matches the hint. A reopened tab can carry the exact URL of one just closed, so the URL-only match kept sending Input.insertText into a dead target's session — no error, the text just never appeared. Confirmed live. - Windows delivery_mode (delivery.rs schema + ClickTool description): make explicit that 'background' is the mandatory first attempt, not a hint to weigh. Agents must not front preemptively because a target "looks like" GTK/Chromium/Electron — the driver decides when background is impossible and signals background_unavailable. - Skills (SKILL.md, WEB_APPS.md): same background-first + web-content typing guidance.

…ndows-support overclaim drive-a-web-page.mdx incorrectly listed Windows as supporting insert_text/ type_keystrokes via CDP — those actions aren't implemented there yet (WindowsPageBackend has no override, falls through to the default "not implemented" trait error). Corrected the backend-support table and added an explicit macOS-only callout to both docs pages and the internal skill doc, linking to #2084 (scoping the Windows/Linux port). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nH7srSSrCyhSDaLCeyH29

…dicts + correctness fixes Resolves the 13 CodeRabbit findings on #2077: Honest path/verified contract (macOS): - click: report path `cgevent` (not `cgevent_fg`) when with_foreground_assist ran the click WITHOUT actually fronting (the Ok(false) was discarded). - type_text: same for the keyboard family — the no-window / SPI-unavailable fallback now reports PATH_KEY_EVENTS, not PATH_KEY_EVENTS_FG. - type_text: verify the AX write against the intended text via verify_typed (before/after/text) instead of just "AXValue non-empty" — a pre-filled field whose AXSelectedText silently no-ops no longer reports verified:true. Removed the now-subsumed is_silent_accept helper + tests. - set_value: seed the slider tolerance to f64::EPSILON so it can't declare the target reached before performing a real AXIncrement/AXDecrement. - bring_to_front: validate pid with pid_t::try_from (no lossy cast) and propagate the activateWithOptions BOOL (no false activated:true when Cocoa rejects). Correctness (macOS): - double_click: error instead of stamping screen coords as window-local when window_bounds_by_id returns None. - scroll: fail a stale/unretained element_index instead of silently scrolling the focused/page scroller; require window_id for window-local x,y scrolling. Windows: - keyboard: foreground text synthesis maps line breaks to VK_RETURN (matching the background path) instead of emitting \r/\n as Unicode. - mouse: preserve a target's existing WS_EX_TOPMOST — only demote (HWND_NOTOPMOST) windows this path actually promoted. - uia/scroll: CoInitializeEx at the start of scroll_into_view_and_recenter so the first UIA call can't hit CO_E_NOTINITIALIZED on worker threads. Harness: - run-one.ps1: launch $PSScriptRoot\wpf-recorder.ps1 (the real recorder), not the never-created wpf-v7.ps1 alias. cargo check -p platform-macos passes; the platform-windows edits are verified by inspection (Windows-only crate — CI compiles it). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011BJPGdRQxiBPTQhjwZsQ8G

…_mode/Swift references Following the Swift-backend retirement and the capture_mode deprecation, clean up the markdown that no longer makes sense (audit found the source-of-truth docs — root README + Skills/*.md + the fumadocs content — already correct): Deleted (obsolete): - rust/PARITY.md — a 2.4k-line Rust-vs-retired-Swift parity audit citing dead Sources/**/*.swift paths; its behavioral contracts live in the fumadocs reference now. Removed the 6 code-comment citations of it. - test-harness/MODALITY_TEST_SUITE.md — a near-duplicate of the canonical reference/cua-driver/modality-test-suite.mdx that still presented ax/vision/som as live capture modes; the MDX is the single source of truth. De-staled (capture_mode is deprecated/ignored; Swift retired; point to cua.ai/docs): - cua-driver-fixtures/README.md — drop the "both ports (Swift + Rust)" framing. - test-harness/{TEST_SUITE,TEST_HARNESS_STRUCTURE}.md, vision-agent-test/README.md — capture_mode reworded to deprecated/ignored (tree+screenshot always returned). - test-harness/modality-recordings/{README,FINDINGS}.md — superseded banners on the old "GTK pixel clicks never land" / recorder-baseline-artifact figures. - cua-driver/python/README.md — trim duplicated install/usage; point at cua.ai/docs. - docs/README.md — note production docs are served from cua.ai/docs (this app is a local MDX preview). cargo check -p cua-driver -p platform-macos passes (comment-only rust edits). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011BJPGdRQxiBPTQhjwZsQ8G

f-trycua and others added 30 commits June 27, 2026 11:54

docs(cua-driver): update modality findings — Windows off-screen guard…

3ff50a2

… shipped + verifier results Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

docs: update modality index with Windows verifier numbers

9b6b36b

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

docs(cua-driver): keep the demo blob URL out of the public repo

a470b74

Replace the live Azure blob URL with a generic reference before the branch goes up as a PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CdJawS4PRvgn3egsa56KyR

f-trycua and others added 2 commits June 30, 2026 20:13

vercel Bot had a problem deploying to Preview July 1, 2026 03:32 Failure

vercel Bot had a problem deploying to Preview July 1, 2026 03:39 Failure

vercel Bot had a problem deploying to Preview July 1, 2026 04:08 Failure

f-trycua changed the title ~~cua-driver 0.7.0 — the modality ladder: honest, effect-verified automation across macOS, Windows, Linux & Wayland~~ cua-driver 0.7.0 — action-time modality, honest verification, browser + Wayland (Rust-only) Jul 1, 2026

vercel Bot had a problem deploying to Preview July 1, 2026 04:32 Failure

f-trycua and others added 2 commits June 30, 2026 21:43

vercel Bot had a problem deploying to Preview July 1, 2026 04:52 Failure

This was referenced Jul 1, 2026

docs: fix stale/dead links in SDK READMEs → point to cua.ai/docs #2085

Open

docs: trim redundant SDK READMEs to pointers at cua.ai/docs #2086

Open

f-trycua merged commit f73c62b into main Jul 1, 2026
69 of 72 checks passed

This was referenced Jul 1, 2026

docs(cua-driver): audit fixes across macOS/Linux/Windows #2089

Merged

docs(cua-driver): update delivery_mode wording in comments #2092

Merged

docs(cua-driver): clarify delivery_mode fallback #2095

Merged

This was referenced Jul 20, 2026

fix(cua-driver): percent-encode non-ASCII paths in launch_app urls #1574

Closed

perf(cua-driver): use on-screen-only SCShareableContent query for permission probe #1378

Closed

coderabbitai Bot mentioned this pull request Aug 21, 2026

fix(linux): harden Ubuntu upgrades and Xorg local control milind-soni/OpenMausBot#346

Open

Navigation Menu

Uh oh!

Uh oh!

cua-driver 0.7.0 — action-time modality, honest verification, browser + Wayland (Rust-only) - #2077

cua-driver 0.7.0 — action-time modality, honest verification, browser + Wayland (Rust-only)#2077
f-trycua merged 135 commits into
mainfrom
cua-driver/modality-ladder-p0-fixes

f-trycua commented Jun 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

f-trycua commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!