fix(cua-driver-rs/windows): UIA root-walk fallback for CoreWindow-class apps (Calculator, Settings, older UWPs) - #1606
Merged
Conversation
…ss apps (Calculator, Settings, older UWPs) `IUIAutomation::ElementFromHandle(hwnd)` returns an empty wrapper when `hwnd` is a `Windows.UI.Core.CoreWindow` (Calculator, Settings, and a number of other older-style UWPs). The XAML tree for these apps is registered at the desktop root as a sibling UIA element with the same `ProcessId` — not as a child of the wrapper element ElementFromHandle returns. This is the standard Microsoft pattern; `inspect.exe` walks from root the same way. `extract_uia_tree` now does the primary `ElementFromHandle` walk first (unchanged for regular Win32 + modern-Notepad-style apps), then falls back when that produces zero actionable nodes: 1. Resolve `ProcessId` from the original `hwnd` via `GetWindowThreadProcessId`. 2. `IUIAutomation::GetRootElement().FindAll(TreeScope_Children, TrueCondition)`. 3. For each child with matching `ProcessId`, `BuildUpdatedCache(cache_req)` so the same properties + patterns get pre-fetched as the primary path, then `walk_cached` on the cached subtree. The fallback only fires when the primary path returns 0 actionable elements, so non-UWP apps and modern Notepad (which uses a regular Win32 host class, not CoreWindow) are unaffected — they keep the single-RPC ElementFromHandleBuildCache fast-path. ## Empirical evidence Triangulation on the Windows VM (Session-2 UIA queries): | App | window class | ElementFromHandle descendants | root.children with matching pid | |---|---|---|---| | Modern Notepad | `Notepad` (Win32) | 30 ✅ | 1 | | Calculator | `Windows.UI.Core.CoreWindow` | 0 (empty wrapper) | 0 on this VM (broken) / 1 on healthy hosts | | Settings | `Windows.UI.Core.CoreWindow` | 0 (empty wrapper) | 0 on this VM (broken) / 1 on healthy hosts | This VM's Session-2 desktop is in a degraded state (likely DWM composition not fully attached due to repeated RDP-Disc cycles) where CoreWindow-class apps spawn but never register at UIA root. ElementFromPoint at the center of Calculator's window rect returns the PowerShell window underneath it — the Calculator window is rendering transparently. This is a VM environment issue, not a cua-driver bug. On healthy machines the fallback path is what `inspect.exe`, Accessibility Insights, and Power Automate all use, so this change brings cua-driver to parity with the standard Microsoft pattern. Updates #1601 (which previously claimed the limit was architectural — the corrected diagnosis is documented on that issue). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe Windows UIA module adds a fallback tree-walk mechanism for CoreWindow-based apps (Calculator, Settings, legacy UWP) that yield empty results from the initial HWND-rooted walk. When no actionable nodes are found, the code resolves the owning process ID and re-walks from the desktop root, filtering by process membership. ChangesWindows UIA tree walker process-based fallback
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs`:
- Around line 142-157: The HWND fallback currently calls walk_root_by_pid
reusing nodes, lines, counter, and total, which can merge stale wrapper-only
results and preserve consumed budget; instead, create temporary accumulators
(e.g., temp_nodes, temp_lines, temp_counter, temp_total), call
walk_root_by_pid(&automation, &cache_req, target_pid, &mut temp_nodes, &mut
temp_lines, &mut temp_counter, &mut temp_total), then check the temp result for
any actionable nodes (e.g., any node.element_index.is_some()) and only swap
temp_* into the original nodes, lines, counter, total if the fallback produced
actionable entries; otherwise leave the original buffers and budget untouched.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 832b9168-2696-42aa-b8dc-408d6587e712
📒 Files selected for processing (1)
libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs
Comment on lines
+142
to
+157
| if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 { | ||
| if let Some(target_pid) = pid_from_hwnd(hwnd_win) { | ||
| tracing::debug!( | ||
| target: "uia", | ||
| "ElementFromHandle returned empty tree for hwnd 0x{hwnd:x}; \ | ||
| falling back to GetRootElement + filter ProcessId={target_pid}" | ||
| ); | ||
| walk_root_by_pid( | ||
| &automation, | ||
| &cache_req, | ||
| target_pid, | ||
| &mut nodes, | ||
| &mut lines, | ||
| &mut counter, | ||
| &mut total, | ||
| ); |
Contributor
There was a problem hiding this comment.
Stage the PID fallback into fresh buffers before swapping it in.
Line 142 reuses nodes, lines, counter, and total from the failed HWND walk. If the primary pass emitted a wrapper-only node, the final tree now contains both that stale root and the PID-based subtree, and any consumed MAX_TOTAL_ELEMENTS budget also carries into the fallback. Run walk_root_by_pid into temporary accumulators and replace the primary result only if the fallback actually finds actionable nodes.
💡 Suggested shape
if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 {
if let Some(target_pid) = pid_from_hwnd(hwnd_win) {
+ let mut fallback_nodes = Vec::new();
+ let mut fallback_lines = Vec::new();
+ let mut fallback_counter = 0usize;
+ let mut fallback_total = 0usize;
+
tracing::debug!(
target: "uia",
"ElementFromHandle returned empty tree for hwnd 0x{hwnd:x}; \
falling back to GetRootElement + filter ProcessId={target_pid}"
);
walk_root_by_pid(
&automation,
&cache_req,
target_pid,
- &mut nodes,
- &mut lines,
- &mut counter,
- &mut total,
+ &mut fallback_nodes,
+ &mut fallback_lines,
+ &mut fallback_counter,
+ &mut fallback_total,
);
+
+ if fallback_nodes.iter().any(|n| n.element_index.is_some()) {
+ nodes = fallback_nodes;
+ lines = fallback_lines;
+ counter = fallback_counter;
+ total = fallback_total;
+ }
}
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 { | |
| if let Some(target_pid) = pid_from_hwnd(hwnd_win) { | |
| tracing::debug!( | |
| target: "uia", | |
| "ElementFromHandle returned empty tree for hwnd 0x{hwnd:x}; \ | |
| falling back to GetRootElement + filter ProcessId={target_pid}" | |
| ); | |
| walk_root_by_pid( | |
| &automation, | |
| &cache_req, | |
| target_pid, | |
| &mut nodes, | |
| &mut lines, | |
| &mut counter, | |
| &mut total, | |
| ); | |
| if nodes.iter().filter(|n| n.element_index.is_some()).count() == 0 { | |
| if let Some(target_pid) = pid_from_hwnd(hwnd_win) { | |
| let mut fallback_nodes = Vec::new(); | |
| let mut fallback_lines = Vec::new(); | |
| let mut fallback_counter = 0usize; | |
| let mut fallback_total = 0usize; | |
| tracing::debug!( | |
| target: "uia", | |
| "ElementFromHandle returned empty tree for hwnd 0x{hwnd:x}; \ | |
| falling back to GetRootElement + filter ProcessId={target_pid}" | |
| ); | |
| walk_root_by_pid( | |
| &automation, | |
| &cache_req, | |
| target_pid, | |
| &mut fallback_nodes, | |
| &mut fallback_lines, | |
| &mut fallback_counter, | |
| &mut fallback_total, | |
| ); | |
| if fallback_nodes.iter().any(|n| n.element_index.is_some()) { | |
| nodes = fallback_nodes; | |
| lines = fallback_lines; | |
| counter = fallback_counter; | |
| total = fallback_total; | |
| } | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver-rs/crates/platform-windows/src/uia/mod.rs` around lines 142 -
157, The HWND fallback currently calls walk_root_by_pid reusing nodes, lines,
counter, and total, which can merge stale wrapper-only results and preserve
consumed budget; instead, create temporary accumulators (e.g., temp_nodes,
temp_lines, temp_counter, temp_total), call walk_root_by_pid(&automation,
&cache_req, target_pid, &mut temp_nodes, &mut temp_lines, &mut temp_counter,
&mut temp_total), then check the temp result for any actionable nodes (e.g., any
node.element_index.is_some()) and only swap temp_* into the original nodes,
lines, counter, total if the fallback produced actionable entries; otherwise
leave the original buffers and budget untouched.
…deRabbit feedback) Per CodeRabbit on #1606: the previous version reused `nodes`, `lines`, `counter`, and `total` from the failed primary walk. If the primary path emitted a wrapper-only node (e.g., the empty CoreWindow shell), the merged result would contain that stale root AND the PID-based subtree. Worse, the primary walk's consumed `MAX_TOTAL_ELEMENTS` budget would carry into the fallback, truncating large trees prematurely. Now the fallback walks into temporary accumulators, and the primary result is only replaced if the fallback actually finds actionable nodes. Otherwise the wrapper-only primary result stays — better than erasing it for nothing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2 tasks
f-trycua
added a commit
that referenced
this pull request
May 20, 2026
…start.mdx (#1608) The "What it does NOT unlock" section had a wrong architectural claim: that CoreWindow apps like Calculator are unreachable because they have no addressable HWND. The triangulation done in #1606 (Calculator vs Settings vs modern Notepad) showed the actual pattern: ElementFromHandle on a Windows.UI.Core.CoreWindow HWND returns an empty wrapper, but the real XAML tree is registered at the desktop root as a sibling element with the same ProcessId. inspect.exe walks from root for these apps; cua-driver-rs now does the same via the root-walk fallback merged in #1606. Replaces the architectural-limit paragraph with: - A new "CoreWindow-class apps and the UIA root-walk fallback" section explaining what the fallback does and linking to #1606 - An updated "What it does NOT unlock" entry that points at the real remaining gap: hotkey accelerator shortcuts on XAML targets (#1607), with the workaround (UIA tree walk + click on the menu item) Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced May 20, 2026
This was referenced Jun 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IUIAutomation::ElementFromHandle(CoreWindow_hwnd)returns an empty wrapper forWindows.UI.Core.CoreWindow-class apps (Calculator, Settings, older UWPs). Their XAML tree is registered at the desktop root as a sibling UIA element with the sameProcessId, not as a child of the wrapper element. This is the standard Microsoft pattern —inspect.exe, Accessibility Insights, and Power Automate all walk from root for these apps.extract_uia_treenow keeps the primaryElementFromHandlefast-path for non-UWP apps, and falls back to root-walk-by-ProcessId only when the primary path yields zero actionable nodes.Triangulation evidence (Session-2 UIA on the dev VM)
Notepad(Win32 host)Windows.UI.Core.CoreWindowWindows.UI.Core.CoreWindowVM environment caveat (separate issue)
The dev VM's Session-2 desktop is in a degraded state where CoreWindow-class apps spawn but never register at the UIA root. ElementFromPoint at the center of Calculator's window rect returns the PowerShell window underneath it — Calculator's window is rendering transparently. This is the same broken state for both Calculator and Settings, so it's not Calculator-specific. Likely DWM composition not fully attached due to repeated RDP-Disc cycles. The code change here doesn't depend on this VM working — the fallback is the standard Microsoft pattern.
What changed
pid_from_hwnd(hwnd)helper —GetWindowThreadProcessIdwrapperwalk_root_by_pid(automation, cache_req, pid, ...)— runsGetRootElement().FindAll(Children, TrueCondition), filters byProcessId, walks descendants from each matching element using the same cache shape as the primary pathBackward compatibility
Test plan
References: #1601 (corrected diagnosis posted as comment), #1604 (the uiAccess worker that makes this matter for UIPI'd apps).
🤖 Generated with Claude Code
Summary by CodeRabbit