Skip to content

fix(computer-use): vision capture returns an image on cua-driver >=0.5.x - #51078

Merged
OutThisLife merged 1 commit into
mainfrom
bb/fix-vision-capture
Jun 22, 2026
Merged

OutThisLife merged 1 commit into
mainfrom
bb/fix-vision-capture

Conversation

@OutThisLife

Copy link
Copy Markdown

Problem

computer_use capture(mode='vision') returned a 0x0 result with no image on every call:

{"mode":"vision","width":0,"height":0,"elements":[],"total_elements":0}

som and ax modes worked fine, which masked the regression.

Root cause

The vision branch called a screenshot MCP tool that cua-driver dropped in 0.5.x — full-window PNG capture was folded into get_window_state (its description even notes "Also captures a PNG screenshot of the specified window"). The driver replies Unknown tool: screenshot, so the flattened result carried no image part, images came back empty, png_b64 stayed None, and capture returned 0x0.

Verified against cua-driver 0.5.1:

$ cua-driver call screenshot '{"window_id":...}'
Unknown tool: screenshot
$ cua-driver list-tools | grep screenshot   # (no match)

som/ax were unaffected because they already use get_window_state.

Fix

Route vision by capability (tools/computer_use/cua_backend.py):

  • driver advertises screenshot (older builds) → use it (cheapest, no AX walk)
  • otherwise → call get_window_state but discard the AX tree/elements, returning only the PNG so vision stays free of element noise (that's the whole point of vision mode)
  • capabilities not yet discovered → try screenshot, fall back to get_window_state on an empty image, so the path self-heals on any driver version

Also added _image_from_tool_result, which pulls the PNG from either an MCP image content-part or structuredContent.screenshot_png_b64. Applied on the som path too, so the image won't silently drop on driver builds that deliver it via structuredContent instead of a content part.

Verification

  • Live (cua-driver 0.5.1): vision → 1568x954, 0 elements; som → image + 527 elements ✅
  • Unit: all four routing cases (0.5.x no-screenshot, older with-screenshot, caps-undiscovered fallback, som unchanged) + the _image_from_tool_result shape handling
  • tests/computer_use — 24 passed

Vision mode called a `screenshot` MCP tool that cua-driver dropped in
0.5.x (full-window PNG capture was folded into `get_window_state`). The
driver replied "Unknown tool: screenshot", so `images` came back empty,
`png_b64` stayed None, and capture returned a 0x0 result with no image
on every call. `som`/`ax` were unaffected because they already use
`get_window_state`, which masked the regression.

Route vision by capability:
- driver advertises `screenshot` (older builds) -> use it (no AX walk)
- otherwise -> call `get_window_state` but discard the AX tree/elements,
  returning only the PNG so vision stays free of element noise
- capabilities not yet discovered -> try `screenshot`, fall back to
  `get_window_state` on an empty image, so the path self-heals

Add `_image_from_tool_result` to pull the PNG from either an MCP image
content-part or `structuredContent.screenshot_png_b64`, and use it on
the som path too so the image won't silently drop on driver builds that
deliver it via structuredContent instead of a content part.

Verified live (vision: 1568x954, 0 elements; som: image + 527 elements)
and with unit coverage of all four routing cases.
@github-actions

Copy link
Copy Markdown

🔎 Lint report: bb/fix-vision-capture vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11461 on HEAD, 11461 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 6038 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@alt-glitch alt-glitch added type/bug Something isn't working comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists duplicate This issue or pull request already exists labels Jun 22, 2026
@alt-glitch

Copy link
Copy Markdown

Duplicate of #39262 (the earliest open fix PR for #39242) — both route computer_use(capture, mode='vision') through get_window_state instead of the dropped screenshot MCP tool on cua-driver 0.5.x. Same cua-capture cluster as #44715. Marking duplicate so reviewers can compare against the canonical PR; the capability-fork + _image_from_tool_result structuredContent handling here is a useful refinement worth folding in.

@OutThisLife
OutThisLife merged commit 760fd95 into main Jun 22, 2026
35 checks passed
@OutThisLife
OutThisLife deleted the bb/fix-vision-capture branch June 22, 2026 23:37
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…-capture

fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
…-capture

fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
kumaxs pushed a commit to kumaxs/hermes-agent that referenced this pull request Jul 12, 2026
…om captures

cua-driver 0.5.x+ requires capture_mode in get_window_state calls to
return a screenshot image. Without it, the driver returns only the AX
tree, causing computer_use capture (both vision and SOM modes) to
silently return an empty 0x0 result.

Add capture_mode to all 4 get_window_state call sites:
- vision MCP fallback path → capture_mode='vision'
- vision CLI re-fetch path → capture_mode='vision'
- som/ax MCP path → capture_mode=mode
- som/ax CLI re-fetch path → capture_mode=mode

Fixes NousResearch#39242 (partial — the screenshot tool routing was already fixed
by PR NousResearch#50994 / NousResearch#51078, but the missing capture_mode parameter
continued to produce empty captures on the fallback paths).
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…-capture

fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…-capture

fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…-capture

fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…-capture

fix(computer-use): vision capture returns an image on cua-driver >=0.5.x
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants