Skip to content

fix(computer-use): route cua screenshots through aux vision - #42158

Open
im-khang wants to merge 1 commit into
NousResearch:mainfrom
im-khang:fix/computer-use-cua-vision-routing
Open

im-khang wants to merge 1 commit into
NousResearch:mainfrom
im-khang:fix/computer-use-cua-vision-routing

Conversation

@im-khang

@im-khang im-khang commented Jun 8, 2026

Copy link
Copy Markdown

Summary

  • make the Cua computer-use backend enumerate all windows, then prefer on-screen candidates, so capture/focus still work when cua-driver marks a capturable app window off-screen or on another Space
  • route som, vision, and ax captures through get_window_state with explicit capture_mode, and read dimensions/tree data from structuredContent
  • create the aux-vision cache directory before writing captured screenshots, and ignore failed vision_analyze envelopes instead of treating their error text as valid analysis

Bug fixed

computer_use with cua-driver could return 0x0 geometry, use the wrong screenshot path for vision, miss app-filtered windows, or fall back to multimodal screenshot output when aux vision routing failed because $HERMES_HOME/cache/vision did not exist. For text-only main models, that fallback caused the active provider to reject the tool result image.

Verification

  • python -m pytest tests/tools/test_computer_use_capture_routing.py tests/tools/test_computer_use.py -q -o 'addopts='
  • local live smoke on macOS with Calculator:
    • capture app="Calculator" mode="vision" returned width=460 height=816, vision_analysis_routed_via=auxiliary.vision, no error
    • capture app="Calculator" mode="som" returned width=460 height=816, vision_analysis_routed_via=auxiliary.vision, no error

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists tool/vision Vision analysis and image generation labels Jun 8, 2026
@Morad37

Morad37 commented Jun 8, 2026

Copy link
Copy Markdown

Clean unification of the three capture paths. One thing I noticed: the _split_tree_text fallback path at line ~442 still runs even when structuredContent provides tree_markdown, so you're parsing the AX tree twice on every som/ax capture that has structured content. Moving the parsed_tree parse inside the if not structured_tree branch would save the regex work on cua-driver versions that already ship the structured tree.

@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for the focused capture-routing work. The auxiliary-vision and cache-directory portions are now already present on current main: tools/computer_use/tool.py:778-832 creates the cache directory and returns the auxiliary.vision text result. Current main also has capability-aware vision capture routing at tools/computer_use/cua_backend.py:1373-1418, with regression coverage at tests/tools/test_computer_use.py:2360-2407.

Problems

  • The remaining change is not a clean replay: this PR changes window discovery to include off-screen windows, while current main explicitly defines both capture() and focus_app() as on-screen selection paths (tools/computer_use/cua_backend.py:1247-1264, :1738-1776).
  • Replacing current routing with unconditional get_window_state would bypass the current legacy-screenshot capability path (tools/computer_use/cua_backend.py:1387-1418).

Suggested changes

  • Rebase the remaining off-screen-window case onto current CuaDriverBackend and prove it with a targeted driver reproduction plus mixed-inventory tests, preserving the current capability and CLI-fallback paths.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants