test(cua-driver): converge Rust desktop E2E contracts by f-trycua · Pull Request #2161 · trycua/cua

f-trycua · 2026-07-10T23:34:12Z

Why

The desktop suites mixed unit checks, shell-generated summaries, one-off guards, and GUI behavior tests. Missing cells could disappear, setup failures could produce reduced green runs, and a successful tool response could be mistaken for input delivery.

This PR makes the Rust harnesses the source of truth for desktop behavior. Repo-local applications expose observable state, and every action must prove either its external effect or an exact structured refusal. Background actions also prove that they did not steal focus, raise the target, move the real cursor, or leak input.

Canonical test model

Unit and protocol tests stay desktop-independent and video-free.
Electron and Tauri share one 36-cell catalog per app. macOS also runs a native WKWebView host.
Native harness owners use the same typed result and evidence contract.
AX/PX targeting and foreground/background delivery are dimensions of each action.
Focus, z-order, cursor, leaked-input, capture, and desktop-scope checks attach to the actions that need them.
Contributors run all; CI partitions shared, native, and capture work only as internal jobs.

Each result records platform, display server, harness, action, targeting, delivery, scope, backend route, expected and observed behavior, independent oracles, duration, and evidence. The reporter rejects missing, duplicate, undeclared, contradictory, or evidence-less rows.

Driver and harness fixes

Evidence and reporting

Added source, fixture, accessibility, capture, GUI-session, and playable-video preflights.
Resolves one immutable source SHA per workflow and records it in every summary and environment record.
Added a physically foreground, fully occluding sentinel for background actions.
Requires an allowed structured refusal plus clean desktop-side-effect observations.
Records a full-desktop MP4 for each canonical cell and before/after evidence for targeted actions.
Added a compact coverage grid and artifact-linked row for each cell.
Isolated process-global recording hook phases so parallel unit tests cannot consume another test's before/after capture.

Linux

Replaced X11 false-success paths with foreground XTest delivery, semantic background delivery where proven, and exact refusals elsewhere.
Added generic ext_foreign_toplevel_list_v1 discovery, native wlroots capture and recording, stable window identity, compositor-aware geometry, and strict session preflights.
Added GNOME WinRects geometry and activation plus persistent portal/libei input with revocable grants.
Preserves 64-bit Wayland identities, resolves connection-local toplevel IDs by stable process and title data, and reconstructs renderer origins for targeted PX input.
Packages libxkbcommon, portal/libei support, the nested compositor, and source-owned Rust tests through Nix.
Restored the optional cua-compositor as a separate experimental environment with a versioned acknowledged protocol, PID targeting, exact resource errors, focus and z-order observation, scene geometry queries, direct pointer and keyboard routes, and typed evidence.
Makes nested pointer hit-testing target-subtree-local so an occluding sentinel cannot intercept a background action. Renderer child surfaces reuse their bound pointer resource or fall back to the owning root.
Keeps standard Wayland and the private nested route separate in result metadata and public documentation.

Windows

Restored occluded Electron AX/PX single-left clicks through the proven background-safe PostMessage route.
Proved occluded Electron child-window delivery; unsupported Chromium gestures retain exact refusals.
Corrected foreground PX activation and renderer keyboard targeting.
Added canonical WPF and WebView2 PX background-left-click rows with fixture-owned state and desktop-side-effect oracles.
Added WPF background combo selection and an exact WPF background-drag refusal.
Made start_minimized fail closed before spawning when Windows cannot guarantee no activation.
Reports minimized windows truthfully, filters them from on_screen_only, and restores them through the existing bring_to_front tool.

macOS

Restored PX background left-click delivery through AX hit-testing for Electron, Tauri, and AppKit without focus changes.
Fixed the first-character loss in foreground PX typing.
Converts silent background scroll and drag no-effects into exact background_unavailable refusals.
Added the native WKWebView host to the complete shared catalog.
Added AppKit scroll, right/double click, slider drag, and background-drag contracts.
Added SwiftUI background AX click/value rows and foreground popover state verification.
Fixed desktop-scope HID delivery, fixture stability, per-connection daemon sessions, and structured refusal propagation.
Preserves the installed app's TCC identity during local installation.

Cleanup

Removed the synthetic calculator fixture and duplicate calculator tests.
Removed old guard, modality, delivery, and legacy shared owners after mapping useful assertions into typed action rows.
Removed shell and PowerShell behavioral inference from Cargo output.
Keeps software-dependent checks such as LibreOffice outside canonical all.
Added libs/cua-driver/docs/action-support.md, an evidence journal, and the public Platform Support reference.\n- Split the validation model into How Cua Driver Is Validated, covering source-built harnesses, independent oracles, retained evidence, and maintainer-triggered release validation.\n- Added a cross-OS Platform Roadmap that separates evidence gaps, engineering work, experimental environments, and hard OS boundaries without promising dates.

Validation

Linux X11: run 29194456173 passed all 108 declared outcomes: 71 delivered, 37 exact refusals, 0 failures, 0 skips.
Linux Sway native and capture: run 29195551765 passed native GTK 31/31 and capture/scope 5/5, with playable per-cell video.
Linux Sway shared: run 29199656600 passed 29/36 Electron rows, up from 23/36. Focused run 29200827296 then passed all 9/9 Electron PX-background rows at the replacement source. The same 9 Tauri rows remained environment-limited by the hosted DRM/EGL setup.
Nested compositor: run 29197643541 passed native GTK 31/31; run 29197887387 passed capture/scope 5/5 with exact LinuxCuaCompositorInject metadata. Full shared run 29199596935 passed 26/36 Electron rows before focused repairs. Run 29200462634 then passed 7/9 occluded PX-background rows. Dedicated scroll run 29200854535 remained red, so the environment stays experimental.
GNOME: a representative GNOME 46 session passed 31 native GTK rows, 3 capture contracts, and 2 desktop-scope contracts.
KDE: Plasma 6 reached session startup, GTK AT-SPI discovery, generic discovery, and portal-interface preflight. No KDE behavioral matrix is accepted.
Packaging and unit gates: Nix source run 29200852035 passed at a78f525a. Linux unit, distribution, docs, script, Windows compile, and the local 35-test testkit suite are green.
Windows: run 29166454913 passed all 113 typed rows: 92 delivered, 21 exact refusals, 0 failures, 0 skips.
macOS: the complete local matrix passed all 130 declared rows: 124 delivered, 6 exact refusals, 0 failures, 0 skips.

Remaining explicit gaps

Standard Wayland cannot route arbitrary raw focus-bound pointer or keyboard input to an occluded, unfocused surface. Unsupported paths return exact structured refusals.
Sway Electron still lacks foreground AX right click, foreground AX double click, and foreground AX/PX hotkey delivery.
Hosted Sway cannot expose a representative Tauri/WebKitGTK renderer accessibility tree without DRM/EGL. The native frame can exist while the renderer tree is absent.
The nested cua-compositor remains experimental. Renderer left/double click, one AX right-click timeout, and renderer scroll are not stable across focused runs. Unicode text and a canonical parallel-drag row remain unproven.
KDE lacks a target-addressable KWin activation adapter and an accepted behavioral matrix.
GNOME and KDE portal-backed MP4 parity, representative shared renderer runs, and real-Xorg MPX/uinput evidence remain open.
SwiftUI transient-panel AX enumeration, Windows elevated-integrity refusal coverage, and several native control combinations remain unproven.

github-actions · 2026-07-10T23:34:22Z