Skip to content

fix(computer_use): use get_window_state for vision/som captures (#39242) - #39342

Closed
kyssta-exe wants to merge 1 commit into
NousResearch:mainfrom
kyssta-exe:auto-fix/issue-39242
Closed

kyssta-exe wants to merge 1 commit into
NousResearch:mainfrom
kyssta-exe:auto-fix/issue-39242

Conversation

@kyssta-exe

Copy link
Copy Markdown

Fixes #39242.

Replace the non-existent cua-driver screenshot tool call with get_window_state(capture_mode="vision") for vision-mode captures, and pass capture_mode to get_window_state for som/ax modes so SOM captures also get their screenshot payload back.

Changes in tools/computer_use/cua_backend.py:

  • Vision mode: calls get_window_state with capture_mode="vision" instead of the non-existent screenshot tool; extracts width/height from structuredContent.
  • Non-vision modes: passes capture_mode ("som" or "ax") to get_window_state so SOM captures include the screenshot image alongside the AX tree.
  • Updated docstring to reflect the corrected tool mapping.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists tool/vision Vision analysis and image generation duplicate This issue or pull request already exists labels Jun 4, 2026
@alt-glitch

Copy link
Copy Markdown

Duplicate of #39262, which makes the same fix (route cua-driver vision/SOM captures through get_window_state(capture_mode=...) instead of the non-existent screenshot tool) for the same issue #39242.

@alaamohanad169-ship-it

Copy link
Copy Markdown

Note: This is NOT a duplicate of #39262. The referenced PRs were closed/superseded earlier. This version has been cherry-picked onto the latest origin/main with a clean diff containing only the fix changes. No conflicts. CI green. Ready for review.

@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for the focused report and patch. The underlying cua-driver vision-capture regression is already fixed on current main, so this PR is now redundant.

  • tools/computer_use/cua_backend.py:1373–1418 uses get_window_state when a modern driver no longer advertises screenshot, while retaining compatibility with older drivers.
  • tests/tools/test_computer_use.py:2360–2407 covers the dropped-screenshot case and verifies that vision capture returns a PNG with no AX-element noise.
  • The implementation shipped via merge commit 760fd9513e7d7350a672c76369ba632ca1ed1448 (PR fix(computer-use): vision capture returns an image on cua-driver >=0.5.x #51078), included in v2026.7.1.

Automated hermes-sweeper review.

@teknium1 teknium1 closed this Jul 14, 2026
@teknium1 teknium1 added sweeper:implemented-on-main Sweeper: behavior already present on current main sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:implemented-on-main Sweeper: behavior already present on current main sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

computer_use(capture, mode='vision') calls non-existent cua-driver 'screenshot' tool and returns 0x0 result

4 participants