test(cua-driver): cross-platform capture_mode matrix coverage + modality docs - #2053
Conversation
Add `modality_capture_mode_test` — the cross-platform spine of the `capture_mode` axis (ax / vision / som). It drives each platform's native controlled harness (WPF / AppKit / GTK3) and asserts the contract: ax → accessibility tree present (btn-increment), no image vision → screenshot present, no tree som → both (the default) The oracle reads the canonical structured fields (`tree_markdown`, `screenshot_png_b64`) so it is transport-agnostic, with a bounded settle-retry for cold-start AX registration and a new-instance window resolver (pid-diff) so the three tests stay deterministic under `--test-threads=1`. Vision/AX grants that are absent skip-with-note rather than false-fail. This closes the audited "capture_mode only on Windows" gap (the macOS/Linux native vision path was previously untested). Verified green on macOS against the AppKit harness; Windows/Linux run via the same `#[ignore]` lane on a VM. Docs: new explanation page `capture-and-dispatch-modalities.mdx` documenting the three modality axes (capture_mode × dispatch × capture_scope) and the validity matrix (window scope = any combo; desktop = vision + foreground only), wired into the Explanation nav + index. TEST_SUITE.md gains a per-platform modality-matrix coverage table with an honest remaining-gaps note (window/vision/foreground untested; desktop-scope actuator Windows-only). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdded a new capture-and-dispatch modalities explanation page, linked it from the explanation index, and updated modality coverage notes. Added a cross-platform Rust integration test that checks capture-mode output across ChangesCapture modality docs and test coverage
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/content/docs/explanation/capture-and-dispatch-modalities.mdx`:
- Line 30: The internal docs links in capture-and-dispatch-modalities.mdx are
using the wrong route prefix and do not resolve in CI. Update the broken
`/docs/...` links at the referenced places to match the site’s existing docs
route shape, using the same pattern as the other navigation links (for example
`/explanation/...`). Keep the changes limited to the affected markdown links in
this doc.
In `@libs/cua-driver/rust/crates/cua-driver/tests/modality_capture_mode_test.rs`:
- Line 1: The new test file is missing the project’s SPDX license header. Add
the standard SPDX header at the very top of modality_capture_mode_test.rs before
the existing module/doc comment, matching the header style used by other files
under libs/cua-driver so CI’s SPDX check passes.
In `@libs/cua-driver/test-harness/TEST_SUITE.md`:
- Line 57: The family inventory count is stale after adding
modality_capture_mode_test, so the section header in TEST_SUITE.md should be
updated to match the new total. Adjust the “Inventory by family” heading to
reflect 19 files/rows, keeping the table count consistent with the entry list
and the modality_capture_mode_test row.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e528fc01-b305-441f-b45d-eed171e77312
📒 Files selected for processing (5)
docs/content/docs/explanation/capture-and-dispatch-modalities.mdxdocs/content/docs/explanation/index.mdxdocs/content/docs/explanation/meta.jsonlibs/cua-driver/rust/crates/cua-driver/tests/modality_capture_mode_test.rslibs/cua-driver/test-harness/TEST_SUITE.md
|
|
||
| | `dispatch` | Behavior | | ||
| |---|---| | ||
| | `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched. This is the [no-foreground contract](/docs/explanation/the-no-foreground-contract). | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Fix the internal doc routes.
Line 30, Line 69, and Line 73 link to /docs/... URLs that do not resolve, and docs CI is already failing on them. These should use the same route shape as the rest of the docs navigation (for example /explanation/... instead of /docs/explanation/...).
Suggested patch
-| `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched. This is the [no-foreground contract](/docs/explanation/the-no-foreground-contract). |
+| `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched. This is the [no-foreground contract](/explanation/the-no-foreground-contract). |
@@
-| `dispatch: background` (the contract) | ✅ | ✅ | ✅ (X11/AT-SPI; native Wayland input is a [known gap](/docs/explanation/the-no-foreground-contract#linux)) |
+| `dispatch: background` (the contract) | ✅ | ✅ | ✅ (X11/AT-SPI; native Wayland input is a [known gap](/explanation/the-no-foreground-contract#linux)) |
@@
-`capture_mode` is recognized everywhere, so window-scope automation — the four window-scope rows of the matrix — works on all three platforms. The desktop-scope loop (`get_desktop_state` plus window-less screen-absolute input via hit-testing) is complete on Windows and rolling out to macOS and Linux; on those platforms a window-less action under window scope is still rejected. See the [MCP tool reference](/docs/reference/cua-driver/mcp-tools) for per-tool parameters and the [no-foreground contract](/docs/explanation/the-no-foreground-contract) for how background dispatch is implemented on each OS.
+`capture_mode` is recognized everywhere, so window-scope automation — the four window-scope rows of the matrix — works on all three platforms. The desktop-scope loop (`get_desktop_state` plus window-less screen-absolute input via hit-testing) is complete on Windows and rolling out to macOS and Linux; on those platforms a window-less action under window scope is still rejected. See the [MCP tool reference](/reference/cua-driver/mcp-tools) for per-tool parameters and the [no-foreground contract](/explanation/the-no-foreground-contract) for how background dispatch is implemented on each OS.Also applies to: 69-69, 73-73
🧰 Tools
🪛 GitHub Actions: CI: Check Docs Links / 0_Check Internal Links (next-validate-link).txt
[error] 30-30: docs:check-links (tsx scripts/check-links.ts) failed: Invalid URL/not-found reference at /docs/explanation/the-no-foreground-contract.
🪛 GitHub Actions: CI: Check Docs Links / Check Internal Links (next-validate-link)
[error] 30-30: docs:check-links reported invalid URL: /docs/explanation/the-no-foreground-contract not-found at content/docs/explanation/capture-and-dispatch-modalities.mdx:30:176
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/content/docs/explanation/capture-and-dispatch-modalities.mdx` at line
30, The internal docs links in capture-and-dispatch-modalities.mdx are using the
wrong route prefix and do not resolve in CI. Update the broken `/docs/...` links
at the referenced places to match the site’s existing docs route shape, using
the same pattern as the other navigation links (for example `/explanation/...`).
Keep the changes limited to the affected markdown links in this doc.
Source: Pipeline failures
| @@ -0,0 +1,257 @@ | |||
| //! modality_capture_mode_test — the **capture_mode** axis (`ax` / `vision` / | |||
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add the SPDX header to this new file.
CI's SPDX check is failing for files under libs/cua-driver; as a newly added source file, this one should carry the project's standard SPDX header at the top to avoid adding to the warn-only violation count.
🧰 Tools
🪛 GitHub Actions: CI: SPDX Headers / 0_Check SPDX headers (warn-only).txt
[error] 1-1: spdx-headers.py --check failed: 349 file(s) missing SPDX header under /home/runner/work/cua/cua/libs/cua-driver. Process completed with exit code 1.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/rust/crates/cua-driver/tests/modality_capture_mode_test.rs`
at line 1, The new test file is missing the project’s SPDX license header. Add
the standard SPDX header at the very top of modality_capture_mode_test.rs before
the existing module/doc comment, matching the header style used by other files
under libs/cua-driver so CI’s SPDX check passes.
Source: Pipeline failures
| | | `harness_appkit_test` | mac | AppKit: AX tree, AXPress, NSTextField, NSScrollView, NSMenu | | ||
| | | `harness_swiftui_test` | mac | SwiftUI: AX tree, `.popover()` | | ||
| | **modality_** | `modality_background_test` | win | background-modality / no-focus-steal sentinel + `capture_mode` ax/vision/som | | ||
| | | `modality_capture_mode_test` | mac+win+linux | `capture_mode` axis on each native harness: `ax`→tree-only, `vision`→image-only, `som`→both | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Update the inventory count.
Adding modality_capture_mode_test makes the family table 19 rows, but the section header still reads "Inventory by family (18 files)". Bump it to 19 to keep the count accurate.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/test-harness/TEST_SUITE.md` at line 57, The family inventory
count is stale after adding modality_capture_mode_test, so the section header in
TEST_SUITE.md should be updated to match the new total. Adjust the “Inventory by
family” heading to reflect 19 files/rows, keeping the table count consistent
with the entry list and the modality_capture_mode_test row.
Linux visual regression artifactsMatrix jobs now run independently. Download visual artifacts from this workflow run.
|
Linux VM verification surfaced a cold-start race: a freshly launched GTK3 harness needs the AT-SPI atk-bridge to register on the a11y bus, which on a cold bus can take several seconds — longer than the prior 6×400ms (~2.4s) settle budget, so the first cold run env-skipped the `ax`/`som` tree assertions (still green, but not exercised). Widen to 16×500ms (~8s) so the tree-bearing modes reliably exercise their assertions on cold CI boots. No added latency when the tree is present immediately (macOS stays ~2.4s). Cross-platform verification of this test: macOS (AppKit) and Linux (GTK3, Ubuntu 24.04) both runtime-green with real assertions; Windows compiles + graceful-skips under Session 0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cross-platform verification completeRan the new
So the oracle is runtime-proven on 2 of 3 platforms with real assertions, and compile + graceful-skip verified on Windows. Follow-ups surfaced during verification
|
Windows runtime-green ✅ — all three platforms now verified with real assertionsCompleted the real Windows run in an interactive Session 2 desktop (user Final verification matrix
The earlier Session-0 skip was just the SSH session having no desktop; the cross-user ACL concern did not actually block running the |
New reference page `reference/cua-driver/contracts.mdx` — the cross-cutting contract map behind the two surfaces: CLI vs MCP transport state, where set_config lands (anonymous→disk global vs session→in-memory override), the capture-scope contract (window needs pid/window_id; desktop is x,y-only, screen-absolute, skips window_id/list_windows), the action-routing decision (element → window-local pixel → screen-absolute → desktop_scope_disabled gate), and per-platform support. Tables + cross-links to Process model and Capture and dispatch modalities rather than re-explaining them. Wired into the Cua Driver reference nav. Also fixes the internal-link convention: `capture-and-dispatch-modalities.mdx` (and the new page) used a `/docs/...` prefix that doesn't match the rest of the docs (`/explanation/...`, `/reference/...`). Normalized to the prefix-less form so the cross-links resolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Added a consolidated Interface Contracts reference page ( |
1 participant
What
Two halves of one ask: document the modality test matrix and cover it across platforms.
Docs (fumadocs)
New explanation page Capture and Dispatch Modalities (
docs/content/docs/explanation/capture-and-dispatch-modalities.mdx), wired into the Explanation nav + index. It documents the three independent axes of every action and the validity matrix:ax/vision/som(what the agent observes)background(default) /foreground(how input lands)window(default) /desktop(the coordinate space)Validity: window scope supports any combination; desktop scope is vision + foreground only (no window handle → no AX tree → vision; screen-absolute input → must own the desktop → foreground). The two rejected combinations (
desktop+ax/background, and window-less click underwindowscope →desktop_scope_disabled) are called out, with an honest per-platform support table.Tests
New
modality_capture_mode_test— the cross-platform spine of thecapture_modeaxis. It drives each platform's native controlled harness (WPF / AppKit / GTK3) and asserts:axbtn-increment), no imagevisionsomThe oracle reads the canonical structured fields (
tree_markdown,screenshot_png_b64) so it is transport-agnostic. A bounded settle-retry handles cold-start AX registration, and a pid-diff new-instance window resolver keeps the three tests deterministic under--test-threads=1. Missing OS grants (Screen Recording / Accessibility) skip-with-note rather than false-fail.This closes the audited "
capture_modeonly on Windows" gap — the macOS/Linux native vision path was previously untested.TEST_SUITE.mdgains a per-platform modality-matrix coverage table with an honest remaining-gaps note.Verification — runtime-green with real assertions on all three platforms
axtree-only ·visionimage-only ·somboth; 3× deterministicpyatspi-confirmedtest result: ok. 3 passed; finished in 6.01sSee thread comments for the full per-platform run logs.
Honest remaining gaps (documented, not closed here)
window/vision/foregroundhas no explicit test on any platform.clickstill requirespid), so the desktop cell and its negative gate stay Windows-only until that loop lands on macOS/Linux.🤖 Generated with Claude Code
Summary by CodeRabbit
Documentation
Tests