fix(cua-driver)(windows): surface get_window_state screenshot error by f-trycua · Pull Request #1977 · trycua/cua

f-trycua · 2026-06-22T19:48:07Z

Follow-up to #1974 / #1973.

Problem

PR #1974's IsIconic guard in screenshot_window_bytes_with_occlusion_unsafe correctly bails on minimized targets, but get_window_state discarded the error at impl_.rs:708:

Err(_) => None,

End-to-end behaviour:

Version	Minimized-window `get_window_state` response	Agent UX
v0.6.2	degenerate 300-byte all-black PNG	"screen looks blank" → burned turns
v0.6.3 (with #1974)	empty `content:[]`, no error	"no screenshot" → still confused
this PR	typed bail surfaced in content + structuredContent	model picks `raise_window` / `list_windows` and retries

Reproduced live on the Windows VM after PR #1974 shipped as v0.6.3: a minimized Notepad's get_window_state (capture_mode=vision) returns {"content":[],"structuredContent":{"pid":…, "window_id":…}} — the IsIconic error is firing inside screenshot_window_bytes but get_window_state was throwing it away.

Fix

Capture the screenshot error alongside the (optional) image in the spawn_blocking closure, and on the no-image branch surface it in BOTH:

the human-readable content stream (so models with content-only parsing see "screenshot unavailable: cannot capture minimized window 0x…" next to any UIA tree they got);
structuredContent.screenshot_error (so MCP clients that consume the structured side — cua_backend.py and downstream wrappers — can detect and act on it).

Single-file change in crates/platform-windows/src/tools/impl_.rs: +25 / -6.

Test plan

cargo check -p cua-driver clean.
On the Windows VM with v0.6.4: spawn Notepad, minimize, call get_window_state { pid, window_id, capture_mode: "vision" } and confirm structuredContent.screenshot_error contains the iconic-bail message + content[0].text carries "screenshot unavailable: cannot capture minimized window 0x…".
Restore Notepad, retry, confirm a normal screenshot path with no screenshot_error field set.

Out of scope

Linux / macOS platform crates have the same pattern but their screenshot paths don't have the iconic-window failure mode; deferring symmetry until there's a real failure to surface.

Summary by CodeRabbit

Bug Fixes
- Screenshot capture errors are now properly reported with clear error messages instead of being silently discarded. When screenshots cannot be captured, users receive transparent error feedback and diagnostic information in the response payload, enabling them to identify and troubleshoot underlying issues more effectively.

Follow-up to #1974. The iconic-window bail from `screenshot_window_bytes` correctly fires for minimized targets, but `get_window_state` swallowed the error at impl_.rs:708: Err(_) => None, So the call returned an empty response (no image, no error) and the upstream agent had no signal that the window was minimized — it just saw 'no screenshot' and kept retrying, exactly the loop pattern from the original Hermes bug report. Capture the screenshot error alongside the (optional) image, and on the no-image branch surface it in BOTH: - the human-readable `content` stream as 'screenshot unavailable: <err>' so models with content-only parsing see it next to the UIA tree they did get; - `structuredContent.screenshot_error` for MCP clients that consume the structured side (Hermes' cua_backend.py, downstream wrappers). Now a minimized-window `get_window_state` returns the typed bail ('cannot capture minimized window 0x… : it has no rendered content. Restore the window first via list_windows / raise_window…'), making the issue actionable end-to-end instead of stopping at a silent empty response. Single file change, +25 / -6.

vercel · 2026-06-22T19:48:11Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Ignored	Preview	Jun 22, 2026 7:48pm

coderabbitai · 2026-06-22T19:48:33Z

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0f4aa3b5-e1c3-44d1-91b1-e6b36d305692

📥 Commits

Reviewing files that changed from the base of the PR and between fd38eb3 and 252b011.

📒 Files selected for processing (1)

libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs

📝 Walkthrough

Walkthrough

In GetWindowStateTool::invoke, the screenshot capture branch is changed to return a 3-tuple (tree_opt, screenshot_opt, screenshot_err). Capture failures now propagate an error string instead of being silently discarded. The response builder is updated to emit a screenshot unavailable: {err} text message and set structured["screenshot_error"] when a screenshot is absent but an error is present.

Changes

Screenshot error propagation in GetWindowStateTool

Layer / File(s)	Summary
Screenshot error capture, return shape, and response emission `libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs`	`screenshot_window_bytes` failure now yields an error string alongside `None` instead of dropping it. The success-path destructure is extended to a 3-tuple including `screenshot_err`. A new response branch emits a `"screenshot unavailable: {err}"` content line and sets `structured["screenshot_error"]` when no screenshot payload is available.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐇 Hop hop, no more silent despair,
When screenshots vanish into thin air!
The error now surfaces, clear as can be,
A message appears for the debugger to see.
No failures forgotten, no clues left behind —
This bunny leaves no broken captures unsigned! 🖼️

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/get-window-state-surface-screenshot-error

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-22T20:01:46Z

Linux visual regression artifacts

Matrix jobs now run independently. Download visual artifacts from this workflow run.
Each background-GUI job uploads a .gif of the interaction plus two annotated PNGs (<app>.png raw, <app>-atspi.png with AT-SPI element boxes); the cua-driver-linux-som-overlays artifact adds <app>-som.png cua Set-of-Marks overlays:

cua-driver-linux-cursor-click-gif
cua-driver-linux-background-terminal-gif
cua-driver-linux-parallel-drag-xserver
cua-driver-linux-background-gui-chromium
cua-driver-linux-background-gui-tk
cua-driver-linux-background-gui-gtk3-gedit
cua-driver-linux-background-gui-gtk3-mousepad
cua-driver-linux-background-gui-gtk3-scite
cua-driver-linux-background-gui-gtk4-characters
cua-driver-linux-background-gui-qt5-manuskript
cua-driver-linux-background-gui-qt5-klog
cua-driver-linux-background-gui-qt5-openambit
cua-driver-linux-background-gui-qt6-kate
cua-driver-linux-background-gui-qt6-kcalc
cua-driver-linux-background-gui-qt6-okular
cua-driver-linux-background-gui-qt6-qownnotes
cua-driver-linux-background-gui-electron-zettlr
cua-driver-linux-background-gui-electron-joplin
cua-driver-linux-background-gui-electron-logseq
cua-driver-linux-som-overlays

Open workflow run and download artifacts

f-trycua merged commit c0d453c into main Jun 22, 2026
63 of 68 checks passed

f-trycua deleted the fix/get-window-state-surface-screenshot-error branch June 22, 2026 19:48

coderabbitai Bot mentioned this pull request Jul 3, 2026

fix(cua-driver-rs)(macos): surface display-asleep state in capture errors and action results #2104

Closed

coderabbitai Bot mentioned this pull request Aug 4, 2026

feat(desktop-use): control engine pin, Atmos Desktop Use host, unified AppShot TCC AruNi-01/atmos#202

Merged

17 tasks

Navigation Menu

Uh oh!

Uh oh!

fix(cua-driver)(windows): surface get_window_state screenshot error - #1977

fix(cua-driver)(windows): surface get_window_state screenshot error#1977
f-trycua merged 1 commit into
mainfrom
fix/get-window-state-surface-screenshot-error

f-trycua commented Jun 22, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

vercel Bot commented Jun 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 22, 2026 •

edited

Loading

Review failed

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

github-actions Bot commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

f-trycua commented Jun 22, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Test plan

Out of scope

Summary by CodeRabbit

Uh oh!

vercel Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

github-actions Bot commented Jun 22, 2026

Linux visual regression artifacts

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

f-trycua commented Jun 22, 2026 •

edited by coderabbitai Bot

Loading

vercel Bot commented Jun 22, 2026 •

edited

Loading

coderabbitai Bot commented Jun 22, 2026 •

edited

Loading