You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(computer-use): route cua screenshots through aux vision - #42158
make the Cua computer-use backend enumerate all windows, then prefer on-screen candidates, so capture/focus still work when cua-driver marks a capturable app window off-screen or on another Space
route som, vision, and ax captures through get_window_state with explicit capture_mode, and read dimensions/tree data from structuredContent
create the aux-vision cache directory before writing captured screenshots, and ignore failed vision_analyze envelopes instead of treating their error text as valid analysis
Bug fixed
computer_use with cua-driver could return 0x0 geometry, use the wrong screenshot path for vision, miss app-filtered windows, or fall back to multimodal screenshot output when aux vision routing failed because $HERMES_HOME/cache/vision did not exist. For text-only main models, that fallback caused the active provider to reject the tool result image.
Clean unification of the three capture paths. One thing I noticed: the _split_tree_text fallback path at line ~442 still runs even when structuredContent provides tree_markdown, so you're parsing the AX tree twice on every som/ax capture that has structured content. Moving the parsed_tree parse inside the if not structured_tree branch would save the regex work on cua-driver versions that already ship the structured tree.
Thanks for the focused capture-routing work. The auxiliary-vision and cache-directory portions are now already present on current main: tools/computer_use/tool.py:778-832 creates the cache directory and returns the auxiliary.vision text result. Current main also has capability-aware vision capture routing at tools/computer_use/cua_backend.py:1373-1418, with regression coverage at tests/tools/test_computer_use.py:2360-2407.
Problems
The remaining change is not a clean replay: this PR changes window discovery to include off-screen windows, while current main explicitly defines both capture() and focus_app() as on-screen selection paths (tools/computer_use/cua_backend.py:1247-1264, :1738-1776).
Replacing current routing with unconditional get_window_state would bypass the current legacy-screenshot capability path (tools/computer_use/cua_backend.py:1387-1418).
Suggested changes
Rebase the remaining off-screen-window case onto current CuaDriverBackend and prove it with a targeted driver reproduction plus mixed-inventory tests, preserving the current capability and CLI-fallback paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
P2Medium — degraded but workaround existssweeper:blast-moderateSweeper blast radius: moderate — a subsystem or single platformsweeper:risk-compatibilitySweeper risk: may break existing users, config, migrations, defaults, or upgradessweeper:risk-platform-windowsSweeper risk: may break or behave differently on native Windowstool/visionVision analysis and image generationtype/bugSomething isn't working
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
som,vision, andaxcaptures throughget_window_statewith explicitcapture_mode, and read dimensions/tree data fromstructuredContentvision_analyzeenvelopes instead of treating their error text as valid analysisBug fixed
computer_usewith cua-driver could return0x0geometry, use the wrong screenshot path forvision, miss app-filtered windows, or fall back to multimodal screenshot output when aux vision routing failed because$HERMES_HOME/cache/visiondid not exist. For text-only main models, that fallback caused the active provider to reject the tool result image.Verification
python -m pytest tests/tools/test_computer_use_capture_routing.py tests/tools/test_computer_use.py -q -o 'addopts='capture app="Calculator" mode="vision"returnedwidth=460 height=816,vision_analysis_routed_via=auxiliary.vision, no errorcapture app="Calculator" mode="som"returnedwidth=460 height=816,vision_analysis_routed_via=auxiliary.vision, no error