fix(cua-driver-rs)(windows): bitmap-pixel click/drag coords + dispatch:background UIA hit-test docs

[f-trycua]

This branch bundles two related Windows changes.

1. Fix: click(x, y) / drag coord-space mismatch (the actual ship-blocker)

tools/impl_.rs previously called ClientToScreen(px, py) on the bitmap pixels returned by get_window_state. But ClientToScreen interprets its argument as CLIENT-area-relative — top-left of the client area, EXCLUDING the title bar — while the bitmap covers the full window (top-left = DWM-cropped frame top-left, INCLUDING the title bar). The title-bar height (~30 px on Win11 dialogs) got double-counted, so every pixel click landed ~30 px below the agent's intended spot.

Concretely caught when trying to click Save on the LibreOffice Document Recovery dialog: the crosshair marker on the screenshot landed cleanly on the button, but the actual click went to the empty space just below it. PR #1697's DWM crop made the mismatch visible (was previously masked by the 7 px shadow margin partially absorbing the title-bar offset).

Fix: new bitmap_to_screen(hwnd, px, py) helper in tools/impl_.rs that uses DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS) + DWM_CROP_INSET_PX to compute the bitmap's actual screen origin, then adds the caller's (px, py). Falls back to GetWindowRect.top-left + (px, py) when the DWM call fails (matching the capture path's fallback).

Replaces 4 ClientToScreen sites:

• click (x, y) mode
• double_click (x, y) mode
• right_click (x, y) mode
• drag (from_x, from_y) / (to_x, to_y)

Also switches the PostMessage path in those tools from post_click (which re-applies ClientToScreen internally) to post_click_screen so the bitmap → screen mapping happens exactly once.

Verified end-to-end against the LO Document Recovery dialog: pixel click at bitmap (470, 244) now correctly maps to screen (988, 575) (the Save button's center) and the dialog dismisses.

2. Docs: dispatch:\"background\" already does a UIA hit-test for pixel clicks

The click tool's (x, y) background path runs try_invoke_in_window_at_point BEFORE PostMessage delivery: if the deepest invokable element at the resolved screen position exposes InvokePattern, the click is delivered through the UIA accessibility channel (same path as element_index mode — no foreground swap, no flash). PostMessage is only the fallback.

That means pixel clicks on UWP / WinUI3 / Win11 packaged apps (Calculator, modern Notepad, etc.) work flash-free out of the box with the default dispatch:\"background\" — no need to escalate to \"foreground\".

The schema description and WINDOWS.md previously didn't mention this hit-test fallback at all — agents reading the docs would conclude they have to use dispatch:\"foreground\" for any XAML host and accept the visible flash. Both updated:

• tools/impl_.rs click schema docstring — (x, y) paragraph now explains the UIA hit-test fallback and recommends background as the default even on XAML hosts.
• Skills/cua-driver/WINDOWS.md — dispatch table's \"background\" row mentions the hit-test; new "Always try dispatch:'background' first" section with the recommendation and the empirical UWP-Calculator result.

Empirical evidence (verified in this session): 4 pixel clicks against the UWP Calculator numpad with dispatch:\"background\" produced \"✅ Performed UIA Invoke at (sx,sy) for pid X.\" with zero visible flash, vs. the same coords with dispatch:\"foreground\" flashing the Calculator window for ~40 ms each.

Test plan

• LO Document Recovery dialog: pixel click at bitmap (470, 244) dismisses (was off by ~30 px before)
• UWP Calculator: 2 + 1 = via pixel clicks with dispatch:\"background\" — all four resolved as UIA Invoke at (sx,sy), display = 3, no flash
• cargo build --bin cua-driver clean

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Fixed coordinate conversion for click, double-click, right-click, and drag operations to accurately translate screenshot pixel coordinates to screen coordinates, improving compatibility across Windows application types.

• Documentation

  • Updated dispatch contract documentation with improved descriptions of background and foreground behavior, including error handling and guidance for coordinate-based operations.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Ignored		May 26, 2026 10:48am

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR unifies Windows screenshot-to-screen coordinate conversion by introducing a DWM-based mapping helper and applying it across click, double-click, right-click, and drag tools. The dispatch contract documentation is clarified to specify UIA hit-test-first behavior with conditional PostMessage fallback and structured error handling.

Changes

Bitmap-to-Screen Coordinate Mapping and Dispatch Contract

Layer / File(s)	Summary
Dispatch contract and guidance documentation libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md	Updated dispatch:"background" to specify UIA hit-test at the resolved screen point with InvokePattern invocation, conditional PostMessage fallback on UIA miss, and structured background_unavailable error when the fallback is known to fail. Guidance updated to prefer dispatch:"background" first, escalate to dispatch:"foreground" only after background_unavailable via bring_to_front flow.
DWM-based coordinate conversion helper libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs	Introduced bitmap_to_screen(hwnd, px, py) that maps screenshot bitmap pixels to screen coordinates using DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS) plus 1px crop inset, with fallback to GetWindowRect on DWM query failure.
Click tool with bitmap-to-screen mapping libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs	Updated click description to document the new screenshot-bitmap-to-screen mapping and UIA-first dispatch layering. Replaced ClientToScreen with bitmap_to_screen in pixel mode; replaced post_click with post_click_screen in the non-UIA PostMessage fallback path to use DWM-mapped screen coordinates.
Double-click and right-click coordinate mapping libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs	Applied bitmap_to_screen mapping to double_click and right_click pixel modes, replacing ClientToScreen. Updated dispatch:"foreground" SendInput paths to use DWM-mapped screen coordinates and changed PostMessage fallbacks to call post_click_screen with the converted coordinates.
Drag endpoint coordinate mapping libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs	Updated drag pixel-coordinate endpoint computation to use bitmap_to_screen for both start and end positions, aligning drag synthesis with the screenshot bitmap coordinate origin.

Sequence Diagram(s)

No sequence diagram generated; the changes are coordinate mapping refactors and documentation updates without multi-component interactions.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• trycua/cua#1697: Both PRs build on the DWMWA_EXTENDED_FRAME_BOUNDS-based "inner rect" concept—fix(cua-driver-rs)(windows): crop screenshots to DWMWA_EXTENDED_FRAME_BOUNDS to remove invisible-shadow black trim #1697 crops the captured bitmap to that rect, while this PR adds bitmap_to_screen using the same bounds and applies it to click/drag coordinate mapping so screenshot pixels align with the cropped capture.
• trycua/cua#1622: This PR changes how click(x,y) translates screenshot bitmap pixels into screen coordinates for dispatch, while #1622 changes when click(x,y) should avoid UIA Invoke() on certain control types—both affect pixel-accurate routing of coordinate clicks.
• trycua/cua#1669: Both PRs modify the Windows ClickTool flow in platform-windows/src/tools/impl_.rs—this PR handles pixel (x,y) to screen coordinate conversion via bitmap_to_screen, while #1669 addresses Chromium click synthesis with foreground-restore polling.

Poem

🐰 Pixels dance from bitmap's frame,
Through DWM's lens, they find their aim—
Screenshot coordinates align with screen,
No more confusion in between!
Click, drag, dispatch: harmonized,
A coordinate truth, crystallized! 🎯

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures both main changes: the bitmap-pixel coordinate fix and the dispatch:background UIA hit-test documentation clarification, using precise technical terminology.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cua-driver-rs-docs-bg-uia-hittest

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}