Skip to content

fix(cua-driver-rs/windows): screenshot screen-region fallback for XAML/UWP targets (partial CUA-542) - #1599

Merged
f-trycua merged 1 commit into
mainfrom
fix/cua-driver-rs-windows-screenshot-xaml-fallback
May 20, 2026
Merged

fix(cua-driver-rs/windows): screenshot screen-region fallback for XAML/UWP targets (partial CUA-542)#1599
f-trycua merged 1 commit into
mainfrom
fix/cua-driver-rs-windows-screenshot-xaml-fallback

Conversation

@f-trycua

@f-trycua f-trycua commented May 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Partial fix for CUA-542. For XAML / WinUI3 / UWP targets (detected via the existing `input::is_xaml_host_hwnd` predicate from #1597), the screenshot tool now skips PrintWindow and copies the window's on-screen bounds from the desktop DC — same approach the Windows Snipping Tool uses for its "Window" mode.

Coverage

Case Before After
Visible-at-real-size UWP (Edge / Photos / Settings / foreground modern Notepad) all-black PNG from PrintWindow real pixels from desktop DC ✅
Legacy Win32 (Notepad classic, cmd, native apps) PrintWindow WORKS unchanged ✅
Background-collapsed UWP (Windows 11 Calculator on SW_SHOWNOACTIVATE) 120×30 black PNG 120×30 still — Calculator's real UI is in a compositor-managed CoreWindow with no addressable HWND, GDI cannot reach it ❌

The Calculator case requires `Windows.Graphics.Capture` against the `GraphicsCaptureItem` for the CoreWindow — that's the proper follow-up, left tracked on CUA-542 (description updated separately).

What's in it

  • `capture.rs::screenshot_via_screen_region` (new) — copies pixels from `(rect.left, rect.top, rect.right, rect.bottom)` of the desktop DC into an in-memory BGRA buffer, runs through the existing `encode_bgra_to_png` path.
  • `screenshot_window_bytes_unsafe` — routes XAML targets through the new path before attempting PrintWindow; PrintWindow stays the default for everything else.
  • `is_mostly_black_bgra` (new) — sparse-sampled heuristic kept as a safety net for the rare "legacy Win32 app uses D3D directly" case where the XAML predicate misses.

Net diff: 160 insertions / 2 deletions in `capture.rs` only. No new dependencies; uses the same `windows` crate features already enabled.

Verification on the Windows VM

  • Build clean, incremental ~22s.
  • `debug_window_info` confirms `xaml_class_match: true`, `xaml_routing_recommended: true` for Calculator → screen-region path is invoked.
  • Legacy Win32 capture (classic Notepad, cmd.exe with visible window) — unchanged behavior, no regression.
  • Foreground Calculator (forced via SetWindowPos/ShowWindow/SetForegroundWindow) — limitation documented above (Calculator's 120×30 top-level HWND can't be coerced to its real UI size; that's the WGC followup).

Notes

This is the first of three remaining Windows tickets I've been working through on the VM (after kill_app + type_text routing landed today as #1596 and #1597). The other two (CUA-543 hotkey, CUA-544 UIA tree) are still open.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved screenshot capture reliability for certain window types with enhanced detection of failed captures and automatic fallback mechanisms to ensure screenshots are captured successfully even when standard methods fail.

Review Change Stack

…tBlt for XAML/UWP targets

Partial fix for CUA-542 — `PrintWindow` doesn't capture
DirectComposition-backed surfaces (modern UWP / WinUI3 apps render
directly to the compositor, no GDI back buffer). For known XAML host
targets (detected via the existing `input::is_xaml_host_hwnd`
predicate), the screenshot tool now skips PrintWindow and copies the
window's on-screen bounds from the desktop DC. Mirrors the screen-
capture approach the Windows Snipping Tool uses for its "Window" mode.

Coverage:

  - Visible-at-real-size UWP / WinUI3 apps (Edge, Photos with content
    open, Settings, modern Notepad when foreground) — fixed. Their
    GetWindowRect returns real bounds and the desktop DC has their
    composited pixels.
  - Legacy Win32 apps — unchanged, still PrintWindow.
  - Background-collapsed UWP (notably Windows 11 Calculator launched
    via SW_SHOWNOACTIVATE) — NOT fixed. Calculator's top-level window
    persists at 120×30 px at (0, 1) even after SetWindowPos +
    ShowWindow(SW_RESTORE) + SetForegroundWindow attempts; the real
    Calculator UI is hosted in a child CoreWindow managed by the
    compositor with no addressable HWND. Capturing this case requires
    Windows.Graphics.Capture against the CoreWindow's GraphicsCapture-
    Item — left tracked on CUA-542 as the proper followup.

Implementation notes:

  - `is_xaml_host_hwnd` is the predicate from PR #1597 — checks top-
    level window class against a list and falls back to the owning
    .exe basename.  Verified live via `debug_window_info` (which also
    shipped in #1597) that Calculator's `xaml_class_match: true` and
    `xaml_routing_recommended: true` predicates fire.
  - `screenshot_via_screen_region` opens NULL-HWND desktop DC, BitBlts
    from screen coordinates returned by `GetWindowRect`, then
    GetDIBits to BGRA buffer. Same encode path as the PrintWindow
    success branch.
  - The mostly-black sentinel heuristic (`is_mostly_black_bgra`) is
    kept as a safety net for the rare legacy-Win32-app-using-D3D
    case, but the primary trigger is now the XAML host predicate.

Tested live on the Windows VM:

  - Foreground Calculator window (forced via SetForegroundWindow):
    still 120×30 due to Calculator's compositor-managed UI; no improvement
    visible in this specific case. Limitation documented above.
  - Build clean; no regression on the legacy Win32 capture path.

Closes part of CUA-542 (the visible-UWP case). The Calculator-style
case is reopened on the same ticket; WGC implementation is the proper
followup and is well-scoped from the work in this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Ignored Ignored May 20, 2026 11:16am

Request Review

@coderabbitai

coderabbitai Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 33f1b90b-14a9-43a7-a175-d59462ee618e

📥 Commits

Reviewing files that changed from the base of the PR and between d4f219f and 2ccc1bd.

📒 Files selected for processing (1)
  • libs/cua-driver-rs/crates/platform-windows/src/capture.rs

📝 Walkthrough

Walkthrough

This PR addresses CUA-542 by adding a screen-region BitBlt fallback to the Windows screenshot capture function, activated when PrintWindow fails on UWP/DirectComposition surfaces. A mostly-black detection heuristic identifies failed renders, and the fallback is routed for known XAML-host HWNDs or triggered post-capture when black pixels are detected.

Changes

CUA-542 fallback capture for DirectComposition surfaces

Layer / File(s) Summary
Documentation and problem context
libs/cua-driver-rs/crates/platform-windows/src/capture.rs
Module-level docs expanded to explain why PrintWindow returns all-black or incorrect results for UWP/DirectComposition surfaces and document the screen-region BitBlt fallback workaround.
Fallback capture mechanism and detection
libs/cua-driver-rs/crates/platform-windows/src/capture.rs
New is_mostly_black_bgra heuristic samples BGRA pixels to detect failed PrintWindow renders; new screenshot_via_screen_region function captures via GetWindowRect, desktop BitBlt into memory DC, and GetDIBits pixel readback with error handling.
Integration into capture flow
libs/cua-driver-rs/crates/platform-windows/src/capture.rs
Main screenshot_window_bytes_unsafe modified to route XAML-host HWNDs directly to screen-region fallback, and after GetDIBits to detect mostly-black results and retry via fallback; fallback errors log warnings and fall through to PrintWindow result.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

A Windows pane would turn to black, 🖤
When UWP's the stack,
But now a screen-region capture saves the day,
With fallback paths that show the way,
DirectComposition? No more dismay! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding a screenshot screen-region fallback for XAML/UWP targets. It is specific, concise, and directly related to the primary changeset.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/cua-driver-rs-windows-screenshot-xaml-fallback

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant