fix(computer_use): adapt cua-driver backend to 0.6.x capture API

[vision-drakon]

Symptom

On macOS with cua-driver 0.6.x installed, the computer_use toolset returns broken captures:

• mode="vision" → width:0 height:0, no screenshot at all.
• mode="som" → screenshot present but every element has bounds (0,0,0,0) and empty labels, breaking coordinate clicks and spatial reasoning.

This happens regardless of Screen Recording / Accessibility permissions being correctly granted to com.trycua.driver (verified screen_recording_capturable: true).

Root cause

tools/computer_use/cua_backend.py is pinned to and written for cua-driver 0.5.0 (PINNED_CUA_DRIVER_VERSION = "0.5.0"), but 0.6.x changed the capture surface:

1. The standalone screenshot tool was removed. capture(mode="vision") still calls it (self._session.call_tool("screenshot", ...)), so it returns nothing → 0×0.
2. Capture was unified into get_window_state with a new capture_mode (som|vision|ax) parameter.
3. The screenshot now arrives as an MCP image content-part (and/or structuredContent.screenshot_png_b64), not from the removed tool.
4. Elements are returned in structuredContent.elements, each with a real frame: {x,y,w,h}. The existing _parse_elements_from_tree only parses the markdown tree, which carries no coordinates — hence the hard-coded bounds=(0,0,0,0).

Fix

• Route all three modes through get_window_state, passing capture_mode=<mode>.
• Read the screenshot from images[0], falling back to structuredContent.screenshot_png_b64.
• Add _parse_elements_from_structured() that reads frame → real (x,y,w,h) bounds; prefer it over the markdown parser, keeping the markdown path as a 0.5.x fallback.
• Read screenshot_width/height from structuredContent when present; only sniff PNG/JPEG bytes as a fallback.
• Extract window title from the first structured AXWindow element (markdown scan fallback).
• Bump PINNED_CUA_DRIVER_VERSION default to 0.6.5 to match the current installer.

The markdown-tree parsing path is retained as a fallback so older 0.5.x daemons keep working.

Verification

Verified live on macOS with cua-driver 0.6.5 via the real MCP stdio path (not mocks):

mode	before	after
som	screenshot, 0/N real bounds	screenshot + elements with real bounds & labels
vision	0×0, no PNG	full-resolution screenshot
ax	elements only	elements only (unchanged)

Direct-backend probe (CuaDriverBackend().capture(mode=...)) confirms som/vision now return real width×height + populated bounds; 568/818 Calendar elements carry non-zero frames where before all were zero.

Adds TestStructuredElementParsing (real-bounds parsing, missing-frame default, index-less skip). Full computer_use suite passes:

tests/tools/test_computer_use.py
tests/tools/test_computer_use_capture_routing.py
tests/tools/test_computer_use_vision_routing.py
127 passed

[f-trycua]

Thanks for digging into this @vision-drakon — the symptom you describe (0×0 vision captures, (0,0,0,0) bounds in som mode) was a real bug, but it's already fixed on main. Concretely:

1. PINNED_CUA_DRIVER_VERSION — was removed in f2e3754 ("cross-platform cua-driver", Jun 21). It's intentionally gone: the comment in its place explains "the upstream installer always fetches the latest release, so a HERMES_CUA_DRIVER_VERSION env var would only have looked like it pinned. For a reproducible version, point HERMES_CUA_DRIVER_CMD at a specific binary instead." Re-introducing the pin would regress that design decision.
2. _parse_elements_from_structured — same commit, lives at tools/computer_use/cua_backend.py:344. Reads frame: {x, y, w, h} exactly the way your patch does. The som path at cua_backend.py:~1202 already prefers it over the markdown tree fallback (Surface 2 of Track: decouple Hermes' computer_use wrapper from cua-driver internals #47072 / feat(cua-driver): cursor overhaul + Hermes-decoupling MCP surface trycua/cua#1961).
3. Vision route + screenshot tool fallback — capture() checks _has_tool("screenshot") first; on "Unknown tool: screenshot" (which is what cua-driver ≥ 0.5.x returns) it falls through to get_window_state and reads the image from the MCP image-part. See the use_screenshot branch around cua_backend.py:~1139.
4. Image from images[0] / structuredContent.screenshot_png_b64 — handled by _image_from_tool_result(), used by both code paths.

The only legitimate delta your PR adds is passing capture_mode=<mode> explicitly to get_window_state. cua-driver 0.6.x defaults capture_mode to "som" (platform-macos/src/tools/mod.rs:126), so for som/ax modes the default already does the right thing; for vision, the wrapper takes the screenshot fast path first and only falls back on missing-tool. So passing it doesn't unblock anything today.

Could you git pull main, reinstall (so the python side picks up f2e3754+) and retest with a 0.6.x driver? If you can still repro the 0×0 vision / zeroed-bounds symptom on current main, that's a different bug we'd love to chase — please share hermes computer-use doctor output and cua-driver --version. Worth checking HERMES_CUA_DRIVER_CMD too — if it points at an older binary, you'd see exactly the symptoms you described.

Going to leave this open for a moment to give you a chance to confirm. Thanks again for taking the time to file a thorough report.

[teknium1]

Thanks for this @vision-drakon — the analysis was correct against the code you branched from, but main has since landed a superset of this exact fix independently, so I'm closing as already-implemented.

What's already on main (via the #47072 work + commits 525033586 / 807b69629):

• _parse_elements_from_structured() — reads structuredContent.elements with real frame bounds, same as your patch (main's also parses element_token and has stricter per-row typing).
• capture() routes vision/som through get_window_state; the standalone screenshot tool is no longer assumed. Main probes _has_tool("screenshot") so it self-heals on both 0.5.x (tool present) and 0.6.x (tool gone) drivers.
• Reads screenshot_png_b64 + screenshot_width/screenshot_height from structuredContent, with PNG/JPEG byte-sniffing as fallback.

One note for the record: this PR also re-introduces PINNED_CUA_DRIVER_VERSION = "0.6.5", but main intentionally removed that pin (commit f2e37549c) in favor of cua-driver's native check_for_update MCP tool — there's deliberately no version-pin knob anymore. So that part would have been a regression.

Your reasoning was sound; it just got overtaken by parallel work. Appreciate the detailed write-up and the live-verified before/after table.