fix(computer-use): vision capture returns an image on cua-driver >=0.5.x - #51078
Merged
Merged
Conversation
Vision mode called a `screenshot` MCP tool that cua-driver dropped in 0.5.x (full-window PNG capture was folded into `get_window_state`). The driver replied "Unknown tool: screenshot", so `images` came back empty, `png_b64` stayed None, and capture returned a 0x0 result with no image on every call. `som`/`ax` were unaffected because they already use `get_window_state`, which masked the regression. Route vision by capability: - driver advertises `screenshot` (older builds) -> use it (no AX walk) - otherwise -> call `get_window_state` but discard the AX tree/elements, returning only the PNG so vision stays free of element noise - capabilities not yet discovered -> try `screenshot`, fall back to `get_window_state` on an empty image, so the path self-heals Add `_image_from_tool_result` to pull the PNG from either an MCP image content-part or `structuredContent.screenshot_png_b64`, and use it on the som path too so the image won't silently drop on driver builds that deliver it via structuredContent instead of a content part. Verified live (vision: 1568x954, 0 elements; som: image + 527 elements) and with unit coverage of all four routing cases.
🔎 Lint report:
|
6 tasks
|
Duplicate of #39262 (the earliest open fix PR for #39242) — both route |
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
…-capture fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
habarmc1223-sudo
pushed a commit
to habarmc1223-sudo/hermes-agent-fluxmem
that referenced
this pull request
Jul 8, 2026
…-capture fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
kumaxs
pushed a commit
to kumaxs/hermes-agent
that referenced
this pull request
Jul 12, 2026
…om captures cua-driver 0.5.x+ requires capture_mode in get_window_state calls to return a screenshot image. Without it, the driver returns only the AX tree, causing computer_use capture (both vision and SOM modes) to silently return an empty 0x0 result. Add capture_mode to all 4 get_window_state call sites: - vision MCP fallback path → capture_mode='vision' - vision CLI re-fetch path → capture_mode='vision' - som/ax MCP path → capture_mode=mode - som/ax CLI re-fetch path → capture_mode=mode Fixes NousResearch#39242 (partial — the screenshot tool routing was already fixed by PR NousResearch#50994 / NousResearch#51078, but the missing capture_mode parameter continued to produce empty captures on the fallback paths).
santhreal
pushed a commit
to santhreal/hermes-agent
that referenced
this pull request
Jul 13, 2026
…-capture fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
Gravezzz
pushed a commit
to Gravezzz/hermes-agent
that referenced
this pull request
Jul 21, 2026
…-capture fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
leewenjie
pushed a commit
to leewenjie/hermes-agent
that referenced
this pull request
Aug 7, 2026
…-capture fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
…-capture fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
computer_usecapture(mode='vision')returned a0x0result with no image on every call:somandaxmodes worked fine, which masked the regression.Root cause
The vision branch called a
screenshotMCP tool that cua-driver dropped in 0.5.x — full-window PNG capture was folded intoget_window_state(its description even notes "Also captures a PNG screenshot of the specified window"). The driver repliesUnknown tool: screenshot, so the flattened result carried noimagepart,imagescame back empty,png_b64stayedNone, and capture returned0x0.Verified against cua-driver 0.5.1:
som/axwere unaffected because they already useget_window_state.Fix
Route vision by capability (
tools/computer_use/cua_backend.py):screenshot(older builds) → use it (cheapest, no AX walk)get_window_statebut discard the AX tree/elements, returning only the PNG so vision stays free of element noise (that's the whole point of vision mode)screenshot, fall back toget_window_stateon an empty image, so the path self-heals on any driver versionAlso added
_image_from_tool_result, which pulls the PNG from either an MCPimagecontent-part orstructuredContent.screenshot_png_b64. Applied on thesompath too, so the image won't silently drop on driver builds that deliver it via structuredContent instead of a content part.Verification
1568x954, 0 elements; som → image + 527 elements ✅_image_from_tool_resultshape handlingtests/computer_use— 24 passed