fix(cua-driver)(windows): bail on minimized window screenshot - #1974
Merged
Conversation
`screenshot_window_bytes_with_occlusion_unsafe` (the GDI / PrintWindow + screen-region BitBlt path) only guarded `w <= 0 || h <= 0`. A minimized Win32 window has `GetWindowRect` returning the off-screen iconic position (typically `(-32000, -32000, -31840, -31972)` — width 160, height 28, both positive) so the check passes, `PrintWindow` paints nothing into the bitmap, and the result is a heavily-compressed all-black ~300-byte PNG that upstream agents can't distinguish from a real 'blank screen' capture. The sibling WGC path at `wgc.rs:58` already short-circuits iconic windows, but on a non-XAML target (most apps) WGC is never tried — the code goes straight to the GDI path. And on XAML targets, when WGC bails the code falls through to `screenshot_via_screen_region` which has the same bug. Add a single `IsIconic` check at the top of `screenshot_window_bytes_with_occlusion_unsafe` so every downstream capture path (WGC / GDI PrintWindow / screen-region BitBlt) is guarded by a single source of truth, and the error matches the shape WGC already emits so callers can call list_windows / raise_window and retry. Closes #1973.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds an ChangesMinimized window guard in GDI capture path
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related issues
Possibly related PRs
Poem
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Linux visual regression artifactsMatrix jobs now run independently. Download visual artifacts from this workflow run.
|
3 tasks
f-trycua
added a commit
that referenced
this pull request
Jun 22, 2026
…1977) Follow-up to #1974. The iconic-window bail from `screenshot_window_bytes` correctly fires for minimized targets, but `get_window_state` swallowed the error at impl_.rs:708: Err(_) => None, So the call returned an empty response (no image, no error) and the upstream agent had no signal that the window was minimized — it just saw 'no screenshot' and kept retrying, exactly the loop pattern from the original Hermes bug report. Capture the screenshot error alongside the (optional) image, and on the no-image branch surface it in BOTH: - the human-readable `content` stream as 'screenshot unavailable: <err>' so models with content-only parsing see it next to the UIA tree they did get; - `structuredContent.screenshot_error` for MCP clients that consume the structured side (Hermes' cua_backend.py, downstream wrappers). Now a minimized-window `get_window_state` returns the typed bail ('cannot capture minimized window 0x… : it has no rendered content. Restore the window first via list_windows / raise_window…'), making the issue actionable end-to-end instead of stopping at a silent empty response. Single file change, +25 / -6.
f-trycua
pushed a commit
to biztex/cua
that referenced
this pull request
Aug 4, 2026
![trycua-release[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
Element actions against a minimized window were dispatched to the Win32 iconic position and silently did nothing, while the tool reported success. Windows parks a minimized window at (-32000, -32000), so GetWindowRect returns (-32000,-32000)-(-31840,-31972) -- width 160, height 28, both positive. UIA mirrors that same sentinel into the BoundingRectangle of every element inside the window, so the cached element center is a sentinel point and the window rect is a sentinel rect. point_in_window_bounds then compares one against the other. Because both sides are poisoned identically, the sentinel center tests as *inside* the sentinel window rect and the guard passes. The click is posted to coordinates no monitor covers; nothing happens and no error is raised. The guard that exists to catch off-screen centers is structurally unable to catch this case. Refuse it in resolve_onscreen_point_with_scroll, before the containment check, on either signal: the resolved point being in the sentinel region, or IsIconic on the target window. The point test is kept alongside the authoritative IsIconic call because the reports in trycua#2015 include elements carrying sentinel bounds while their window appeared visible on a secondary monitor -- IsIconic alone would not cover that shape. The error names the remedy (bring_to_front, then re-snapshot), matching the wording the capture path already uses for the same condition. The capture and WGC paths have refused iconic windows since trycua#1973/trycua#1974; this brings the input path in line. Scope: this fixes the silent no-op. get_window_state still reports the raw sentinel bounds in elements[].frame -- suppressing or annotating those changes the response shape, so it is left for a follow-up. Refs trycua#2015, trycua#1979, trycua#1981
f-trycua
pushed a commit
to biztex/cua
that referenced
this pull request
Aug 4, 2026
Element actions against a minimized window were dispatched to the Win32 iconic position and silently did nothing, while the tool reported success. Windows parks a minimized window at (-32000, -32000), so GetWindowRect returns (-32000,-32000)-(-31840,-31972) -- width 160, height 28, both positive. UIA mirrors that same sentinel into the BoundingRectangle of every element inside the window, so the cached element center is a sentinel point and the window rect is a sentinel rect. point_in_window_bounds then compares one against the other. Because both sides are poisoned identically, the sentinel center tests as *inside* the sentinel window rect and the guard passes. The click is posted to coordinates no monitor covers; nothing happens and no error is raised. The guard that exists to catch off-screen centers is structurally unable to catch this case. Refuse it in resolve_onscreen_point_with_scroll, before the containment check, on either signal: the resolved point being in the sentinel region, or IsIconic on the target window. The point test is kept alongside the authoritative IsIconic call because the reports in trycua#2015 include elements carrying sentinel bounds while their window appeared visible on a secondary monitor -- IsIconic alone would not cover that shape. The error names the remedy (bring_to_front, then re-snapshot), matching the wording the capture path already uses for the same condition. The capture and WGC paths have refused iconic windows since trycua#1973/trycua#1974; this brings the input path in line. Scope: this fixes the silent no-op. get_window_state still reports the raw sentinel bounds in elements[].frame -- suppressing or annotating those changes the response shape, so it is left for a follow-up. Refs trycua#2015, trycua#1979, trycua#1981
f-trycua
pushed a commit
that referenced
this pull request
Aug 4, 2026
* fix(cua-driver/windows): refuse iconic-sentinel element actions Element actions against a minimized window were dispatched to the Win32 iconic position and silently did nothing, while the tool reported success. Windows parks a minimized window at (-32000, -32000), so GetWindowRect returns (-32000,-32000)-(-31840,-31972) -- width 160, height 28, both positive. UIA mirrors that same sentinel into the BoundingRectangle of every element inside the window, so the cached element center is a sentinel point and the window rect is a sentinel rect. point_in_window_bounds then compares one against the other. Because both sides are poisoned identically, the sentinel center tests as *inside* the sentinel window rect and the guard passes. The click is posted to coordinates no monitor covers; nothing happens and no error is raised. The guard that exists to catch off-screen centers is structurally unable to catch this case. Refuse it in resolve_onscreen_point_with_scroll, before the containment check, on either signal: the resolved point being in the sentinel region, or IsIconic on the target window. The point test is kept alongside the authoritative IsIconic call because the reports in #2015 include elements carrying sentinel bounds while their window appeared visible on a secondary monitor -- IsIconic alone would not cover that shape. The error names the remedy (bring_to_front, then re-snapshot), matching the wording the capture path already uses for the same condition. The capture and WGC paths have refused iconic windows since #1973/#1974; this brings the input path in line. Scope: this fixes the silent no-op. get_window_state still reports the raw sentinel bounds in elements[].frame -- suppressing or annotating those changes the response shape, so it is left for a follow-up. Refs #2015, #1979, #1981 * fix(cua-driver/windows): type minimized element refusals Replace the fixed negative-coordinate heuristic with live virtual-desktop bounds, fail before every element click route, and expose window_minimized as a structured refusal. Add a real minimized WPF public-driver cell with fixture, focus, z-order, cursor, and leaked-input oracles, plus evidence-validator coverage. * fix(cua-driver): record refusals without click markers --------- Co-authored-by: trycua-release[bot] <trycua-release[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1973.
Summary
screenshot_window_bytes_with_occlusion_unsafe(the GDI / PrintWindow + screen-region BitBlt path) only guardedw <= 0 || h <= 0. A minimized Win32 window hasGetWindowRectreturning the off-screen iconic position (typically(-32000, -32000, -31840, -31972)— width 160, height 28, both positive), so the check passes,PrintWindowpaints nothing into the bitmap, and the result is a heavily-compressed all-black ~300-byte PNG that an upstream agent can't distinguish from a real "blank screen" capture.The WGC sibling path at
wgc.rs:58already short-circuits iconic windows, but:screenshot_via_screen_region, which has the same bug ((-32000, -32000)BitBlts a black tile of the off-screen region).Fix
Single
IsIconiccheck at the top ofscreenshot_window_bytes_with_occlusion_unsafeso every downstream capture path (WGC / GDI PrintWindow / screen-region BitBlt) is guarded by one source of truth, with an error matching the WGC error shape so callers can calllist_windows/raise_windowand retry.+25 / -1on a single file.Test plan
cargo check -p cua-driverclean on macOS host.screenshotvia MCP with its window_id, observe typed error instead of a 300-byte PNG. Restore Notepad, retry, observe a valid full-window capture.cua_backend.py, cua-agent) propagate the error instead of caching the degenerate image.Notes
_active_window_idcaching in MCP-client wrappers like Hermes (that's a client-side concern), but the clear error from cua-driver makes it self-correcting from the model's perspective — the next turn the model can calllist_windowsand re-resolve.Summary by CodeRabbit