feat(cua-driver-rs/windows): type_text → UIA ValuePattern routing for XAML targets + debug_window_info tool - #1597
Merged
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThis PR enhances Windows keyboard and text input handling for modern XAML/UWP hosts. It adds explicit XAML host detection via window class names and process executables, updates TypeTextTool to route text entry through UI Automation's ValuePattern when targeting XAML hosts, and introduces a diagnostic tool to inspect window and UIA element state. ChangesWindows XAML/UWP Support Enhancements
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… XAML targets + debug_window_info tool
Two changes for CUA-543 (type_text/hotkey doesn't work on modern XAML
hosts like Win11 Notepad, Calculator, Settings):
1. **`debug_window_info` MCP tool** (new) — dumps everything the daemon
sees about a target pid's top-level windows: Win32 class names,
owning .exe basename + path, focused-element name/class/control_type
from UIA (when CUIAutomation succeeds), the list of UIA patterns
that element supports (Value, Invoke, Text, Toggle,
ExpandCollapse, SelectionItem, LegacyIAccessible), and which of
the routing predicates match (`xaml_class_match`, `exe_xaml_match`,
`xaml_routing_recommended`). Runs in the daemon (Session 2 via
autostart kick), so it sees the actual cross-session HWND state
that SSH-side PowerShell probes from Session 0 can't reach.
2. **`type_text` UIA routing** for XAML / UWP targets. The new
`input::is_xaml_host_hwnd` predicate ORs two signals:
- Top-level window class name matches a known XAML host class
(`ApplicationFrameWindow`, `WinUIDesktopWin32WindowClass`,
`Windows.UI.Core.CoreWindow`,
`Microsoft.UI.Content.DesktopChildSiteBridge`).
- Owning process .exe basename matches a known XAML-hosted .exe
(`notepad.exe`, `calculatorapp.exe`, `applicationframehost.exe`,
`photos.exe`, `systemsettings.exe`).
When the predicate hits AND `element_index` is supplied, `type_text`
routes through `IUIAutomationValuePattern::SetValue` — same code path
the existing `set_value` tool uses, verified live to work on modern
Notepad. When the predicate hits but no `element_index` is supplied,
`type_text` returns a clear actionable error pointing at
`get_window_state` (so agents don't get a misleading "✅ Typed"
message followed by no characters in the editor). Legacy Win32 stays
on the unchanged PostMessage path, preserving the no-focus-steal
property.
Why not SendInput: an earlier iteration tried SendInput with
`AttachThreadInput` + `SetForegroundWindow` and hit
`ERROR_ACCESS_DENIED` from UWP AppContainer targets — UIPI blocks
input injection even at the same integrity level. UIA Patterns are
the right abstraction for XAML / UWP automation; see the CUA-543
ticket for the full investigation log.
`hotkey` (Ctrl+S → Save dialog and similar) is intentionally NOT
covered here. It's a different routing problem — keyboard shortcuts
need to map to UIA `InvokePattern` invocations on the right command
element (e.g. the Save button in Notepad's command bar), which is
per-app discovery work. Deferred to a follow-up iteration.
Diagnostic data behind the routing decisions (captured live on the
Windows VM via `debug_window_info`):
- Modern Notepad's top-level class is the plain string `"Notepad"`,
same as legacy Notepad. EXE-name (`notepad.exe`) is the
distinguishing signal. The XAML class list above stays for the
apps that DO use those classes (`ApplicationFrameWindow` for
older UWP frame hosts, `WinUIDesktopWin32WindowClass` for some
WinUI3 desktop apps).
- The daemon's `IUIAutomation::GetFocusedElement` returns the OS
desktop ("Desktop 1", class `#32769`) when no app has system
focus — so the no-`element_index` path can't transparently
discover the right element. Hence the actionable error rather
than a silent fallback.
Tested live on the VM:
- `type_text(pid, window_id, element_index=0, text="UIA-ROUTING-FIX-CONFIRMED")`
on modern Notepad → driver returns
`✅ Wrote 25 char(s) via UIA ValuePattern`, follow-up
`get_window_state` shows the text in the Document tree.
- `type_text(pid, window_id, text="…")` (no element_index) on the
same Notepad → returns the actionable error message pointing at
`get_window_state`.
- Legacy Win32 path untouched; previously-working `type_text` on
classic apps still uses PostMessage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
f-trycua
force-pushed
the
feat/cua-driver-rs-type-text-uia-routing
branch
from
May 20, 2026 11:06
f5445ad to
75e50a7
Compare
This was referenced May 20, 2026
f-trycua
added a commit
that referenced
this pull request
May 20, 2026
…tBlt for XAML/UWP targets (#1599) Partial fix for CUA-542 — `PrintWindow` doesn't capture DirectComposition-backed surfaces (modern UWP / WinUI3 apps render directly to the compositor, no GDI back buffer). For known XAML host targets (detected via the existing `input::is_xaml_host_hwnd` predicate), the screenshot tool now skips PrintWindow and copies the window's on-screen bounds from the desktop DC. Mirrors the screen- capture approach the Windows Snipping Tool uses for its "Window" mode. Coverage: - Visible-at-real-size UWP / WinUI3 apps (Edge, Photos with content open, Settings, modern Notepad when foreground) — fixed. Their GetWindowRect returns real bounds and the desktop DC has their composited pixels. - Legacy Win32 apps — unchanged, still PrintWindow. - Background-collapsed UWP (notably Windows 11 Calculator launched via SW_SHOWNOACTIVATE) — NOT fixed. Calculator's top-level window persists at 120×30 px at (0, 1) even after SetWindowPos + ShowWindow(SW_RESTORE) + SetForegroundWindow attempts; the real Calculator UI is hosted in a child CoreWindow managed by the compositor with no addressable HWND. Capturing this case requires Windows.Graphics.Capture against the CoreWindow's GraphicsCapture- Item — left tracked on CUA-542 as the proper followup. Implementation notes: - `is_xaml_host_hwnd` is the predicate from PR #1597 — checks top- level window class against a list and falls back to the owning .exe basename. Verified live via `debug_window_info` (which also shipped in #1597) that Calculator's `xaml_class_match: true` and `xaml_routing_recommended: true` predicates fire. - `screenshot_via_screen_region` opens NULL-HWND desktop DC, BitBlts from screen coordinates returned by `GetWindowRect`, then GetDIBits to BGRA buffer. Same encode path as the PrintWindow success branch. - The mostly-black sentinel heuristic (`is_mostly_black_bgra`) is kept as a safety net for the rare legacy-Win32-app-using-D3D case, but the primary trigger is now the XAML host predicate. Tested live on the Windows VM: - Foreground Calculator window (forced via SetForegroundWindow): still 120×30 due to Calculator's compositor-managed UI; no improvement visible in this specific case. Limitation documented above. - Build clean; no regression on the legacy Win32 capture path. Closes part of CUA-542 (the visible-UWP case). The Calculator-style case is reopened on the same ticket; WGC implementation is the proper followup and is well-scoped from the work in this PR. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
f-trycua
added a commit
that referenced
this pull request
May 20, 2026
…loses #1607) (#1611) * fix(cua-driver-rs/windows): hotkey UIA routing for XAML/UWP targets (closes #1607) The `hotkey` tool was silently no-op'ing on modern XAML / UWP hosts (modern Notepad, Settings) because `PostMessage(WM_KEYDOWN/UP)` lands in the window's message queue but the CoreInput dispatcher these apps use only reads from the system input queue. Mirrors the fix in #1597 that routed `type_text` through UIA `ValuePattern.SetValue` for the same architectural reason — `hotkey` now routes through UIA accelerator discovery + pattern activation. ## How the routing works When the target HWND is a XAML host (`is_xaml_host_hwnd`): 1. **AcceleratorKey scan**: walk the UIA subtree from the window root, read each element's `UIA_AcceleratorKeyPropertyId`. If an element advertises an accelerator matching the requested combo (case- insensitive, normalized modifier order), use it. 2. **Name-pattern fallback**: many shipping XAML apps don't set AcceleratorKey at all and instead encode the shortcut in the visible element name as a parenthetical hint — e.g. modern Notepad ships `Button "Bold (Ctrl+B)"` rather than setting AcceleratorKey="Ctrl+B". When AcceleratorKey is empty we scan the Name property for that pattern (requires at least one modifier-like token inside the parens to avoid matching arbitrary parentheticals like "(2)" or "(beta)"). 3. **Pattern activation chain**: try `InvokePattern.Invoke()` first (conventional shortcut handler), then `TogglePattern.Toggle()` (Bold/Italic/Underline-style toolbar buttons). If neither pattern is supported on the matched element, surface an actionable error. 4. **Fall back to PostMessage** for non-XAML targets — non-UWP apps keep the existing fast path with zero behavior change. ## Empirical evidence (Windows 11 VM, modern Notepad) Before this PR: `cua-driver call hotkey {pid, key: "ctrl+b"}` returned "✅ Pressed ctrl+b on pid …" but Bold never toggled (PostMessage ignored by CoreInput dispatcher). After this PR: - `ctrl+b` → matched Name "Bold (Ctrl+B)" → TogglePattern.Toggle → Bold actually toggles ✅ - `ctrl+s` → no UIA AcceleratorKey + no element with "(Ctrl+S)" in name (Save is nested behind File menu) → returns actionable error pointing at `get_window_state` for inspection. Much better than silently no-op'ing. ## Known limitation Menu-nested actions (Save, "Save as", Print, etc. in modern Notepad) don't expose accelerators outside the closed menu's subtree. The fix for this is a richer matcher that walks menus by name + invokes the matching item directly. Tracked as a follow-up; the actionable error this PR produces is honest about the limit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cua-driver-rs/windows): cap hotkey UIA scan at 4s (CodeRabbit feedback) Per CodeRabbit on #1611: the new XAML hotkey path does a full cross-process UIA subtree walk + per-element property reads. Without a timeout, a hung UIA provider would wedge the entire `hotkey` call indefinitely instead of returning a deterministic error to the caller — exactly the failure mode `get_window_state` already guards against in the same file. Wrap the `spawn_blocking` invocation of `try_invoke_accelerator_in_window` in a 4-second `tokio::time::timeout`. On expiry the caller gets an actionable error that names the unresponsive provider's pid + hwnd rather than a daemon hang. 4 s matches the budget the rest of the file uses for similar UIA scans. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the bulk of CUA-543 — `type_text` now works on modern XAML / WinUI3 / UWP targets (Win 11 Notepad, Calculator, Settings) by routing to UI Automation's `ValuePattern.SetValue` when the target is detected as XAML and an `element_index` is provided. Legacy Win32 apps keep the existing PostMessage path so no focus-steal regression. Also adds `debug_window_info`, the diagnostic tool that unblocked the design.
Live-verified on the Windows VM today:
What's in it
1. `is_xaml_host_hwnd` predicate in `input::keyboard`
Two signals, OR'd:
EXE-name is the more reliable signal — modern Win11 Notepad uses the bare `"Notepad"` window class same as legacy, but its EXE lives under `C:\Program Files\WindowsApps\Microsoft.WindowsNotepad_*\`. Verified via the new `debug_window_info` tool.
2. `TypeTextTool` routing in `tools::impl_`
When `is_xaml_host_hwnd(hwnd)` returns true AND the caller supplied `element_index`, route through `IUIAutomationValuePattern::SetValue` — same code path the existing `set_value` tool uses. When XAML target but NO `element_index`, return an actionable error pointing the caller at `get_window_state` first (instead of lying with a "✅ Typed" message). Legacy Win32 path unchanged.
3. `debug_window_info` MCP tool (new)
Diagnostic that dumps everything cua-driver sees about a target pid's top-level windows from the daemon's Session-2 perspective: Win32 class names, owning .exe basename + full path, focused element + supported UIA patterns, and which routing predicates match. Runs in the daemon, so it sees cross-session HWND state that SSH-side PowerShell probes from Session 0 can't reach. Used to design + verify this PR's routing logic; useful permanently for future XAML-related work (CUA-544 will need it for the UIA-tree-empty-after-snapshot investigation).
Why not SendInput
Earlier iteration tried `SendInput` with `AttachThreadInput` + `SetForegroundWindow` to take focus, then unicode keystrokes. Result: `ERROR_ACCESS_DENIED` against UWP AppContainer'd targets even with daemon and Notepad in the same Session 2. UIPI (User Interface Privilege Isolation) blocks input injection into AppContainer'd UWP apps regardless of integrity level — would need `uiAccess="true"` in a signed manifest + UAC elevation to bypass, which significantly impacts install ergonomics. UIA Patterns are the right abstraction.
What's NOT in this PR
Test plan
Depends on
Stacked on top of #1596 (kill_app). Merge order: #1596 → this. If you merge this first you'll get a small conflict on the registration line in `build_registry` — trivial to resolve.
🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes
New Features
Improvements