Improve Windows DevFlow dialog automation

[Redth]

Windows DevFlow currently depends on the app UI dispatcher for tree/query/action routes, which can block when synchronous native Windows dialogs are open. This change ports the relevant Ailoha Windows dialog automation work so WinUI and WPF agents can discover and interact with native UIA dialogs off the UI thread.

Summary

• Adds a Windows native UIA probe for dialog/window discovery, including native element ids, bounds, labels, state, and supported action metadata.
• Merges native Windows UIA results into DevFlow tree/query responses and adds native: fast paths for tap, fill, clear, focus, and scroll.
• Makes Windows and WPF button taps schedule native invocation before inline MAUI click dispatch so modal dialog handlers do not block the action response.
• Backports Windows driver UIAutomationInterop improvements for window/title/handle metadata, named dialog buttons, value/focus/scroll helpers, and UIA fallback actions.
• Adds Windows-only integration coverage for native dialog inspection/action paths and gives the WPF sample dialog controls automation ids.

Notes

The MAUI WinUI Windows target compiles through CoreCompile on macOS, then fails at the expected Windows-only MakePri.exe step. Live Windows dialog integration still needs a Windows host or CI run.

Validation

• dotnet test src/DevFlow/Microsoft.Maui.DevFlow.Tests/Microsoft.Maui.DevFlow.Tests.csproj
• dotnet build src/DevFlow/Microsoft.Maui.DevFlow.Agent.IntegrationTests/Microsoft.Maui.DevFlow.Agent.IntegrationTests.csproj --no-restore
• dotnet build src/DevFlow/Microsoft.Maui.DevFlow.Agent.WPF/Microsoft.Maui.DevFlow.Agent.WPF.csproj -p:EnableWindowsTargeting=true
• dotnet build src/DevFlow/Microsoft.Maui.DevFlow.Driver/Microsoft.Maui.DevFlow.Driver.csproj
• git diff --check

[github-actions]

Expert Code Review: 6 findings posted inline (1 critical, 4 moderate, 1 minor). See the summary comment for full methodology and discarded findings.

Generated by Expert Code Review (auto) for issue #284 · ● 30.2M

[Copilot]

Pull request overview

This PR improves DevFlow’s Windows automation reliability when synchronous native dialogs are open by adding a native UI Automation (UIA) probe that can inspect and interact with native dialog UI off the MAUI UI thread, then merging those native results into the existing tree/query/action flows.

Changes:

• Add a Windows-native UIA probe (NativeWindowProbe) and integrate native element discovery/actions into the agent’s tree/query/element/action endpoints.
• Extend the Windows driver UIA interop and add native fallbacks/fast paths for tap/fill/clear/focus/scroll plus a Win32+Skia screenshot fallback.
• Add Windows-only integration tests for native dialog discovery/actions and improve WPF sample automation IDs.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/DevFlow/Microsoft.Maui.DevFlow.Driver/WindowsAppDriver.cs	Adds UIA/Win32 fallback paths for actions and screenshots; updates alert handling to use resolved windows.
src/DevFlow/Microsoft.Maui.DevFlow.Driver/Windows/UIAutomationInterop.cs	Expands UIA helpers (bounds, focus/value/scroll, window lookup, element search, label normalization).
src/DevFlow/Microsoft.Maui.DevFlow.Driver/AppDriverFactory.cs	Allows selecting Windows driver via wpf platform alias.
src/DevFlow/Microsoft.Maui.DevFlow.Driver/AppDriverBase.cs	Makes core action methods virtual so platform drivers can override.
src/DevFlow/Microsoft.Maui.DevFlow.Agent/Windows/NativeWindowProbe.cs	New native UIA discovery + action helper for dialogs/native windows.
src/DevFlow/Microsoft.Maui.DevFlow.Agent/VisualTreeWalker.cs	WinUI tree walker now supports native elements via NativeWindowProbe.
src/DevFlow/Microsoft.Maui.DevFlow.Agent/Microsoft.Maui.DevFlow.Agent.csproj	Adds Microsoft.WindowsDesktop.App framework reference for Windows UIA types.
src/DevFlow/Microsoft.Maui.DevFlow.Agent/DevFlowAgentService.cs	Schedules native invoke for button taps before MAUI inline click dispatch on Windows.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.WPF/WpfVisualTreeWalker.cs	Adds native element support to WPF walker using the shared probe.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.WPF/WpfAgentService.cs	Schedules native invoke/toggle/click dispatch prior to managed click handling in WPF.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.WPF/Microsoft.Maui.DevFlow.Agent.WPF.csproj	Links NativeWindowProbe into the WPF agent build.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.IntegrationTests/UiInspectionTests.cs	Adds Windows-only integration test validating native dialog elements appear in tree.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.IntegrationTests/UiActionTests.cs	Adds Windows-only integration test validating tapping a native dialog button works.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.Core/VisualTreeWalker.cs	Introduces native-element discovery/query/action abstraction points for platform walkers.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.Core/DevFlowAgentService.cs	Merges native probe results into tree/query and adds native: fast paths for actions.
platforms/Windows.WPF/samples/Windows.WPF.Sample/Pages/AlertsPage.cs	Adds AutomationIds to dialog sample controls for testability.

[github-actions]

Expert Code Review — PR #284

Methodology: 3 independent reviewers with adversarial consensus

Findings Summary

8 inline comments posted (1 🔴 critical, 5 🟡 moderate, 1 🟢 minor):

#	Severity	Consensus	File	Finding
1	🔴	3/3	WpfAgentService.cs:151, Agent/DevFlowAgentService.cs:545	Fire-and-forget BeginInvoke/TryEnqueue — unhandled exceptions crash host app + false-positive success
2	🟡	2/3	NativeWindowProbe.cs:59	int NativeWindowHandle sign-extension mismatch vs long comparison
3	🟡	2/3	NativeWindowProbe.cs:234	FindDialogCandidates O(n2) cross-process UIA calls on every request
4	🟡	2/3	Agent/VisualTreeWalker.cs:668	WalkNativeTree(Array.Empty) tags all windows as dialog
5	🟡	3/3†	Core/DevFlowAgentService.cs:816	Unobserved task exception when uiTask abandoned after timeout
6	🟡	2/3†	NativeWindowProbe.cs:595	BuildId/Sanitize silent ID collisions
7	🟢	2/3	WindowsAppDriver.cs:715	SetCursorPos+SendInput race and cursor side-effect

† = consensus reached via follow-up dispute resolution

Discarded Findings (single-reviewer, no consensus)

• Timer leak from uncancelled Task.Delay in CaptureUiOrNativeAsync (1/3)
• Width * Height * 4 integer overflow on high-DPI (1/3)
• Cumulative 3000ms timeout in CaptureUiOrNativeAsync (1/3)
• TryScheduleNativeTapFirst fires too broadly for any ButtonBase (1/3, disputed — 2/2 disagreed)
• FindFirstByAutomationIdOrName loads entire UIA subtree (1/3)
• Concurrent WalkNativeTree calls overwrite _nativeObjects (1/3)
• Duplicated WaitForNativeButtonAsync test helper (1/3)
• .WaitAsync(TimeSpan) throws instead of returning false in tests (1/3)
• Over-classification of non-dialog windows as "dialog" (1/3)

CI Status

✅ All CI checks passed — build (windows-latest) ×2, build (macos-latest), license/cla all green.

Test Coverage

The PR adds integration tests (UiActionTests.cs +41 lines, UiInspectionTests.cs +52 lines) for native dialog inspection and action paths. These are Windows-only tests that require a Windows host. Unit test coverage for the core NativeWindowProbe logic (ID building, sanitization, collision handling) is not present — consider adding platform-independent unit tests for BuildId/Sanitize.

---

Generated by Expert Code Review · 3 independent reviewers with adversarial consensus

Generated by Expert Code Review (auto) for issue #284 · ● 28.8M · ◷

[github-actions]

Expert Code Review — Native Windows UIA Dialog Automation

Reviewed the full source for NativeWindowProbe.cs, WindowsAppDriver.cs, UIAutomationInterop.cs, DevFlowAgentService.cs (Core and platform), WpfVisualTreeWalker.cs, and PlatformVisualTreeWalker.cs.

Findings

---

1. 🟡 MODERATE — Race condition: TryScheduleNativeTapFirst returns "ok" without confirming the action completed

• File: src/DevFlow/Microsoft.Maui.DevFlow.Agent.Core/DevFlowAgentService.cs, line ~1532
• Scenario: TryScheduleNativeTapFirst(btn) posts an asynchronous BeginInvoke/TryEnqueue and returns true immediately. The handler returns "ok" to the caller (HTTP response). If the enqueued invoke fails (e.g., element disposed, or the enqueue is dropped because the dispatcher is shutting down), the caller believes the tap succeeded but it never fired. On WPF, BeginInvoke can silently swallow exceptions.
• Recommendation: Consider returning a sentinel like "ok:scheduled" so callers can distinguish confirmed vs. fire-and-forget actions, or log a diagnostic when the scheduled action fails internally (wrap the delegate body in try/catch with diagnostics).

---

2. 🟡 MODERATE — CaptureUiOrNativeAsync leaks unobserved Task.Delay timers

• File: src/DevFlow/Microsoft.Maui.DevFlow.Agent.Core/DevFlowAgentService.cs, lines 800 and 816
• Scenario: Two Task.Delay(NativeUiProbeTimeoutMs) calls create timers. When the real task completes before the timeout, the delay timer continues running and its Task is never awaited. While this won't throw (unobserved empty delays), it creates timer pressure under high call frequency. On a hot tree/query loop, thousands of abandoned timers accumulate until GC collects them.
• Recommendation: Use CancellationTokenSource + cancel the delay when the real task completes, or use a shared timer pattern. Example:

  using var cts = new CancellationTokenSource();
  var winner = await Task.WhenAny(hwndSource.Task, Task.Delay(NativeUiProbeTimeoutMs, cts.Token));
  cts.Cancel(); // cancel the unused delay

---

3. 🟡 MODERATE — NativeWindowHandle is int (32-bit) in System.Windows.Automation.AutomationElement.Current, compared to IntPtr.ToInt64() (64-bit)

• File: src/DevFlow/Microsoft.Maui.DevFlow.Agent/Windows/NativeWindowProbe.cs, line 275
• Scenario: current.NativeWindowHandle is typed int in the .NET UIA API. On 64-bit Windows, window handles above 0x7FFFFFFF will appear as negative integers. The comparison current.NativeWindowHandle == rootHwnd.ToInt64() will fail to match if the handle's int representation is negative (e.g., int value = -1234 vs. long value = 0xFFFFF...), causing the dialog to not be excluded from candidates.
• Recommendation: Compare using: (long)(int)current.NativeWindowHandle == rootHwnd.ToInt64() — or better, cast both to the same unsigned representation: (uint)current.NativeWindowHandle == (uint)(int)rootHwnd.ToInt32().

---

4. 🟡 MODERATE — IsAncestor can be O(depth2) per candidate × O(n) candidates = potentially quadratic in FindDialogCandidates

• File: src/DevFlow/Microsoft.Maui.DevFlow.Agent/Windows/NativeWindowProbe.cs, lines 247–258
• Scenario: For each element in descendants (potentially thousands for a complex app), IsAncestor walks up the tree for each existing candidate. This nested loop over UIA cross-process calls can cause multi-second freezes when a large app with many UI elements has an open dialog. All of this runs on a Task.Run worker but still blocks the response until nativeTask completes (bounded by the 1500ms timeout, which provides some mitigation).
• Recommendation: This is partially mitigated by the timeout. For future hardening, consider limiting descendants to elements of specific control types (Window, Pane, Dialog, Group) rather than TrueCondition, or use a tree scope of Children recursed only a few levels deep.

---

5. 🟡 MODERATE — CaptureScreenRectangle potential integer overflow on very large DPI-scaled windows

• File: src/DevFlow/Microsoft.Maui.DevFlow.Driver/WindowsAppDriver.cs, line 657
• Scenario: var bytes = new byte[rect.Width * rect.Height * 4] — for a 4K monitor (3840×2160), this is ~33MB which is fine. But at high DPI scales (e.g., 300%), Width and Height from GetWindowRect are in physical pixels and could be 7680 * 4320 * 4 = ~132MB. The multiplication rect.Width * rect.Height * 4 uses int arithmetic and could overflow if the combined pixel count exceeds int.MaxValue / 4. This would produce a small or negative buffer, then Marshal.Copy would corrupt memory or throw.
• Recommendation: Add a sanity check: if ((long)rect.Width * rect.Height * 4 > int.MaxValue) return null; or use checked arithmetic.

---

6. 🟢 MINOR — ClickPoint uses SetCursorPos + SendInput without coordinate normalization

• File: src/DevFlow/Microsoft.Maui.DevFlow.Driver/WindowsAppDriver.cs, lines 713–723
• Scenario: SetCursorPos(x, y) uses pixel coordinates from UIA's BoundingRectangle (which reports in physical pixels). SendInput with MOUSEEVENTF_LEFTDOWN without MOUSEEVENTF_ABSOLUTE sends a relative mouse-down at the current cursor position. If another process moves the cursor between SetCursorPos and SendInput, the click lands at the wrong location. This is a known TOCTOU race in screen automation but could cause flaky test results.
• Recommendation: Use MOUSEEVENTF_ABSOLUTE | MOUSEEVENTF_LEFTDOWN with normalized coordinates (65535-based) in the MOUSEINPUT.dx/dy fields for atomic positioned clicks. This eliminates the race between SetCursorPos and SendInput.

---

7. 🟢 MINOR — FindFirstByAutomationIdOrName uses CreateTrueCondition for full-tree enumeration

• File: src/DevFlow/Microsoft.Maui.DevFlow.Driver/Windows/UIAutomationInterop.cs, line 289
• Scenario: This loads the entire UIA subtree of each window into memory just to linear-search for a matching AutomationId or Name. For large apps (e.g., Visual Studio), this can return tens of thousands of elements per window and take seconds.
• Recommendation: Use FindFirst with an OrCondition combining a property condition for AutomationId and one for Name. This lets the UIA provider optimize the search server-side rather than returning everything to the client.

---

8. 🟢 MINOR — _nativeObjects dictionary reference replacement under lock is safe but WalkNativeTree is not thread-safe with itself

• File: src/DevFlow/Microsoft.Maui.DevFlow.Agent/VisualTreeWalker.cs, lines 648–658 (and WPF equivalent)
• Scenario: If two concurrent HTTP requests trigger GetNativeElementById simultaneously, both may call WalkNativeTree(Array.Empty<IntPtr>()). Each creates its own nativeObjects dictionary and assigns it under lock. The last writer wins, potentially losing entries from the other's probe. This could make an element found by one probe unfindable by a subsequent lookup.
• Recommendation: This is low-severity since the window of concurrent access is small and the element can be re-found on retry. No action needed now, but could add a simple gate (e.g., Interlocked.CompareExchange pattern) to skip redundant walks.

---

Summary

No critical (🔴) issues found. The code is well-structured with good defensive exception handling around UIA COM calls. The most impactful items are the timer leak under load (#2) and the TryScheduleNativeTapFirst fire-and-forget semantics (#1). The IntPtr/int comparison mismatch (#3) is a correctness bug that could manifest on systems with high window handle values.

Generated by Expert Code Review (auto) for issue #284 · ● 28.8M