Skip to content

refactor(cua-driver-testkit): shared integration-test harness + transport abstraction - #2039

Merged
f-trycua merged 1 commit into
mainfrom
refactor/cua-driver-testkit
Jun 26, 2026
Merged

refactor(cua-driver-testkit): shared integration-test harness + transport abstraction#2039
f-trycua merged 1 commit into
mainfrom
refactor/cua-driver-testkit

Conversation

@f-trycua

@f-trycua f-trycua commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Stacks on #2038 (base is its branch so the diff is just this refactor). This is PR #1 of the test-suite cleanup — extract the duplicated harness machinery; renames/coverage come in follow-ups.

Why

tests/ had no shared module — every file re-implemented the same code:

Duplicated Copies
workspace_root() / driver_binary() ×9 (4 divergent path spellings)
JSON-RPC-over-stdio client (send/call/init) ×10
Windows kill-on-close Job Object reaper ×2
result_text / is_error accessors ×2

What

New crates/cua-driver-testkit (dev-dependency only, never shipped):

  • Driver trait + two transportsMcpDriver (long-lived server, session-scoped state) and CliDriver (stateless cua-driver call). They normalize their different payloads (the CLI prints structuredContent directly; MCP returns the JSON-RPC envelope) into one ToolResponse (text() / structured() / is_error()). This is what makes the CLI-vs-MCP transport axis testable instead of MCP-only.
  • ChildReaper — cross-platform child kill-on-drop; on Windows assigns to a JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE job (no orphaned windows / held ports on panic or SIGKILL).
  • paths — one driver_binary() (.exe via cfg!), workspace_root(), harness_app().

Proof-of-shape migrations (no behavior change)

  • focus_check_test.rs (macOS) — ✅ compiles locally.
  • harness_desktop_scope_test.rs (Windows) — drops its entire Job Object + JSON-RPC block for the testkit.

Verification

  • ✅ macOS: cargo test --no-run -p cua-driver --test focus_check_test (testkit + test compile).
  • ✅ Windows VM: cargo clean -p cua-driver-testkit then cargo test --no-run --test harness_desktop_scope_test → recompiles the testkit Windows (Job Object) path, EXIT=0, binary produced.
  • Tests are #[ignore] (real desktop) — live --ignored runs unchanged from before.

Follow-ups (per the plan): migrate the remaining 11 files, then the 4-family rename, then the coverage matrix + transport_config_persistence_test.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added a shared test harness for driver interactions, supporting both command-line and long-running server workflows.
  • Bug Fixes

    • Improved integration test reliability with better process cleanup and timeout handling across platforms.
    • Updated desktop and focus-related tests to use the shared harness, making behavior validation more consistent.

@vercel

vercel Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Ignored Ignored Preview Jun 26, 2026 6:11am

Request Review

@coderabbitai

coderabbitai Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1a3999fd-5fb5-4664-97cc-25ed97e3679e

📥 Commits

Reviewing files that changed from the base of the PR and between ff3d038 and d19ff18.

⛔ Files ignored due to path filters (1)
  • libs/cua-driver/rust/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (12)
  • libs/cua-driver/rust/Cargo.toml
  • libs/cua-driver/rust/crates/cua-driver-testkit/Cargo.toml
  • libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs
  • libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs
  • libs/cua-driver/rust/crates/cua-driver-testkit/src/lib.rs
  • libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs
  • libs/cua-driver/rust/crates/cua-driver-testkit/src/paths.rs
  • libs/cua-driver/rust/crates/cua-driver-testkit/src/reaper.rs
  • libs/cua-driver/rust/crates/cua-driver-testkit/src/response.rs
  • libs/cua-driver/rust/crates/cua-driver/Cargo.toml
  • libs/cua-driver/rust/crates/cua-driver/tests/focus_check_test.rs
  • libs/cua-driver/rust/crates/cua-driver/tests/harness_desktop_scope_test.rs

📝 Walkthrough

Walkthrough

Adds a shared cua-driver-testkit crate with transport, response, path, and process helpers, then updates the focus and desktop-scope integration tests to use the new MCP-based harness.

Changes

Shared testkit crate and core API

Layer / File(s) Summary
Crate contract and exports
libs/cua-driver/rust/Cargo.toml, libs/cua-driver/rust/crates/cua-driver-testkit/Cargo.toml, libs/cua-driver/rust/crates/cua-driver/Cargo.toml, libs/cua-driver/rust/crates/cua-driver-testkit/src/lib.rs, libs/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs, libs/cua-driver/rust/crates/cua-driver-testkit/src/response.rs, libs/cua-driver/rust/crates/cua-driver-testkit/src/paths.rs
Adds the workspace member, dev-dependency wiring, shared Driver/ToolResponse APIs, runtime path helpers, and crate re-exports.
Process reaping
libs/cua-driver/rust/crates/cua-driver-testkit/Cargo.toml, libs/cua-driver/rust/crates/cua-driver-testkit/src/reaper.rs
Adds child-process tracking, tree termination, and Windows job-object support for tracked driver and harness processes.
MCP and CLI drivers
libs/cua-driver/rust/crates/cua-driver-testkit/src/lib.rs, libs/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs, libs/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs
Adds the long-lived MCP driver, request/response plumbing, timeout-bounded tool calls, and the stateless CLI driver.

Integration test migration

Layer / File(s) Summary
Focus test migration
libs/cua-driver/rust/crates/cua-driver/tests/focus_check_test.rs
Rewrites the macOS focus integration test to use McpDriver for app lookup, window state, click, and text entry.
Desktop-scope test migration
libs/cua-driver/rust/crates/cua-driver/tests/harness_desktop_scope_test.rs
Reworks the desktop-scope tests to use McpDriver, shared path helpers, and the new capture-scope assertions.

Sequence Diagram(s)

sequenceDiagram
  participant focus_check_test
  participant McpDriver
  participant ChildReaper
  participant cua_driver as cua-driver
  participant ToolResponse

  focus_check_test->>McpDriver: McpDriver::spawn()
  McpDriver->>ChildReaper: track spawned child
  McpDriver->>cua_driver: initialize JSON-RPC over stdio
  focus_check_test->>McpDriver: call(tool, args)
  McpDriver->>cua_driver: tools/call request
  cua_driver-->>McpDriver: JSON-RPC response
  McpDriver->>ToolResponse: ToolResponse::from_mcp(raw)
  McpDriver-->>focus_check_test: ToolResponse
Loading

Estimated review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • trycua/cua#2019: Refactors the same integration-test harness to drive the updated desktop-scope APIs through cua_driver_testkit::McpDriver::call.
  • trycua/cua#2038: Updates harness_desktop_scope_test.rs to use the new cua-driver-testkit McpDriver and shared harness helpers.

Poem

I hopped through MCP with a thump and a grin,
Kept CLI and JSON all neatly within.
My paws on the harness, my nose in the glue,
The child jobs stayed tidy; the tests leapt through.
🐇✨

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/cua-driver-testkit

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Linux visual regression artifacts

Matrix jobs now run independently. Download visual artifacts from this workflow run.
Each background-GUI job uploads a .gif of the interaction plus two annotated PNGs (<app>.png raw, <app>-atspi.png with AT-SPI element boxes); the cua-driver-linux-som-overlays artifact adds <app>-som.png cua Set-of-Marks overlays:

  • cua-driver-linux-cursor-click-gif
  • cua-driver-linux-background-terminal-gif
  • cua-driver-linux-parallel-drag-xserver
  • cua-driver-linux-background-gui-chromium
  • cua-driver-linux-background-gui-tk
  • cua-driver-linux-background-gui-gtk3-gedit
  • cua-driver-linux-background-gui-gtk3-mousepad
  • cua-driver-linux-background-gui-gtk3-scite
  • cua-driver-linux-background-gui-gtk4-characters
  • cua-driver-linux-background-gui-qt5-manuskript
  • cua-driver-linux-background-gui-qt5-klog
  • cua-driver-linux-background-gui-qt5-openambit
  • cua-driver-linux-background-gui-qt6-kate
  • cua-driver-linux-background-gui-qt6-kcalc
  • cua-driver-linux-background-gui-qt6-okular
  • cua-driver-linux-background-gui-qt6-qownnotes
  • cua-driver-linux-background-gui-electron-zettlr
  • cua-driver-linux-background-gui-electron-joplin
  • cua-driver-linux-background-gui-electron-logseq
  • cua-driver-linux-som-overlays

Open workflow run and download artifacts

…ests

Introduces crates/cua-driver-testkit — the single home for machinery that was
copy-pasted across tests/ (workspace_root/driver_binary ×9, the JSON-RPC stdio
client ×10, the Windows kill-on-close Job Object reaper ×2, result accessors).

Key design: a Driver trait with two transports — McpDriver (long-lived server,
session-scoped state) and CliDriver (stateless cua-driver call) — that normalize
their differing payloads (the CLI prints structuredContent directly, MCP returns
the JSON-RPC envelope) into one ToolResponse {text(), structured(), is_error()}.
This makes the CLI-vs-MCP transport axis testable instead of MCP-only.

Proof-of-shape migrations (no behavior change):
- focus_check_test.rs (macOS) — verified compiles locally.
- harness_desktop_scope_test.rs (Windows) — drops its whole Job Object +
  JSON-RPC block for the testkit; pending Windows compile-verify.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@f-trycua
f-trycua force-pushed the refactor/cua-driver-testkit branch from bb18f12 to d19ff18 Compare June 26, 2026 06:10
@f-trycua
f-trycua changed the base branch from test/desktop-scope-harness to main June 26, 2026 06:10
@f-trycua
f-trycua merged commit a92db98 into main Jun 26, 2026
118 of 122 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant