fix(windows-click): hit-test inside target HWND's UIA subtree for vision (x,y)

[f-trycua]

Summary

The (x,y) click path's UIA Invoke fallback shipped in #1549 used desktop-wide IUIAutomation::ElementFromPoint, which silently failed for vision-mode clicks against UWP / packaged-app frames on the Win11 VM:

1. Occlusion — any other window covering the target at (sx, sy) (terminal, browser, etc.) made ElementFromPoint return that window's element. Caller already names the intended HWND; we should respect it.
2. UWP cross-process hosting — ApplicationFrameHost.exe hosts UWP content (CalculatorApp, Notepad-Win11, Settings, …) in a separate process. ElementFromPoint was observed returning the frame's outer Pane (no InvokePattern) instead of descending. The fallback PostMessage(WM_LBUTTONDOWN) silently no-ops on UWP because UWP routes input via Windows.UI.Input, not raw WM_LBUTTONDOWN.

Net: vision-mode clicks on Calculator returned Posted click to pid N (PostMessage path) and the calculator never updated.

Fix

New helper uia::windows_enum::try_invoke_in_window_at_point(hwnd, sx, sy) mirrors macOS vision-mode semantics (CGEvent delivered to a specific pid regardless of z-order):

• Roots the hit test in hwnd via ElementFromHandle
• Enumerates the subtree with FindAll(TreeScope_Subtree, TrueCondition)
• Picks the smallest-area descendant whose CurrentBoundingRectangle contains the point AND which exposes InvokePattern
• Calls Invoke()

The (x,y) click path now uses this instead of the desktop-wide hit-test. The original try_invoke_at_point stays in place (docstring updated to flag its desktop-wide nature) for any future callers without an HWND in scope.

Result

Validated live on the Win11 VM (Calculator UWP) under tscon console-attached state:

• Pre-fix: 6 vision-mode clicks reported Posted click to pid N (PostMessage fallback); display stayed at 0.
• Post-fix: 6 clicks return Performed UIA Invoke at (sx,sy); display reads 391; History pane shows 17 x 23 =; no foreground change.

Test plan

• cargo build --release on Win11 VM
• vision-mode 17x23 on Calculator (UWP) via cua-driver call click '{pid, window_id, x, y}'
• verify display + history pane show 391
• verify success message changed from PostMessage path to UIA Invoke path
• follow-up: extend to non-UWP targets (Notepad, Edge, Spotify) before flipping cua-driver-rs BETA -> GA

Summary by CodeRabbit

• Bug Fixes
  • Improved click targeting precision. Clicks now correctly resolve to the target element within the specified window rather than potentially selecting outer containers.
  • Enhanced reliability when clicking within packaged applications, particularly when target elements are occluded or nested within complex hierarchies.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Ignored	Preview	May 18, 2026 11:00am

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 113129d8-6131-4b63-b05b-68a99a9a3875

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR refactors UIA click invocation in the Windows platform layer to use window-scoped element hit-testing instead of desktop-wide lookup. A new helper try_invoke_in_window_at_point searches a UIA subtree rooted at a target hwnd, filtering for the smallest element containing the click point and exposing InvokePattern, then invokes it. The click handler now uses this window-scoped approach to avoid selecting outer containers that lack the pattern.

Changes

Window-scoped UIA click invoke

Layer / File(s)	Summary
Window-scoped invoke helper and documentation libs/cua-driver-rs/crates/platform-windows/src/uia/windows_enum.rs	Add try_invoke_in_window_at_point to perform hwnd-rooted UIA subtree enumeration, filtering by bounding rect and InvokePattern, selecting the smallest-area match, and calling Invoke(). Update try_invoke_at_point documentation to clarify desktop-wide hit-test limitations for UWP/packaged-app frames. Reorganize imports.
Click integration with window-scoped invoke libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs	Update click() (x, y) path to call try_invoke_in_window_at_point(hwnd, sx, sy) for single left/middle clicks, targeting UIA elements within the named window.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• trycua/cua#1549: Both PRs modify the Windows click(x, y) UIA-invoke path in impl_.rs and the corresponding uia/windows_enum.rs helpers—fix(windows-click): UIA Invoke for element_index + ElementFromPoint for (x,y) #1549 introduces a screen-wide try_invoke_at_point via ElementFromPoint, while the main PR refines it to a window-scoped try_invoke_in_window_at_point rooted at hwnd (changing which UIA element gets invoked).

Poem

🐇 A rabbit hops through UIA trees,
Finds the smallest match with ease,
No more desktop-wide confusion—
Window-scoped provides the solution,
Click lands true in every frame! 🎯

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: implementing a window-scoped UIA hit-test instead of a desktop-wide approach for vision-mode clicks on Windows.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/vision-click-window-walk

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[f-trycua]

Validated a second pass: launched Calculator fresh in the background (not foreground), did 3 + 5 = 8 via pixel clicks against the larger Win11 Calc layout (1024×801, History pane expanded, 62 UIA descendants under the AppFrame subtree).

==> click 3  wc=(441,671)  ->  ✅ Performed UIA Invoke at (689,671) for pid 6004.
==> click +  wc=(616,671)  ->  ✅ Performed UIA Invoke at (864,671) for pid 6004.
==> click 5  wc=(267,586)  ->  ✅ Performed UIA Invoke at (515,586) for pid 6004.
==> click =  wc=(616,756)  ->  ✅ Performed UIA Invoke at (864,756) for pid 6004.

Display shows 8, History pane shows 3 + 5 = 8. No foreground change. The windowed UIA walk reliably resolves each button across the cross-process ApplicationFrameHost → CalculatorApp boundary.