fix(computer-use): merge refs+content_refs in _ref_map for cua-driver 0.17

[weisiwu]

Summary

Fixes typed-browser mutations (cua_browser_click / cua_browser_type / cua_browser_pointer) failing with browser_ref_stale on every ref after upgrading cua-driver to >= 0.17.

Root cause

cua-driver >= 0.17 splits the semantic_v2 snapshot payload into two lists:

• refs — action-bearing entries with actions: ["click", "pointer", ...] (the authoritative interactive refs)
• content_refs — every node in the tree, but with empty actions: []

_ref_map() in tools/computer_use/browser_route.py only absorbed content_refs (preferring it over refs), so every ref was registered with an empty action set. _require_ref() then rejected all mutations with browser_ref_stale. Navigation and snapshot reads still worked because they don't need action refs — which made this easy to miss: the browser works until you click.

Observed live on macOS with cua-driver 0.17.0 (upgraded from 0.1.9 via hermes computer-use install --upgrade).

Fix

Merge all three payload shapes instead of picking one:

• refs (0.17 authoritative, action-bearing)
• content_refs (0.17 content nodes; empty actions must not clobber)
• snapshot.refs (legacy fallback)

Actions are combined with set union, so an empty content_refs entry can never drop the actions declared in refs. The old behavior for pre-0.17 drivers (refs-only list, transitional dict shape) is preserved.

Verification

• Unit-tested _ref_map against 4 payload shapes:
  1. 0.17 split format (refs with actions + content_refs with empty actions) → refs keep click/pointer ✅
  2. legacy refs-only list ✅
  3. transitional refs-as-dict ✅
  4. snapshot.refs fallback + empty payload ✅
• End-to-end on a live driver-owned isolated Chromium (cua-driver 0.17.0): bind → snapshot → navigate to github.com search → click a repo link → page navigated to the repo ✅

Test plan for reviewers

import sys
sys.path.insert(0, '.')
from tools.computer_use.browser_route import _ref_map

# 0.17 split format
payload = {
    'refs': [{'ref': 'p1:1', 'actions': ['click', 'pointer']}],
    'content_refs': [{'ref': 'p1:1', 'actions': []}, {'ref': 'p1:2', 'actions': []}],
}
assert _ref_map(payload)['p1:1'] == {'click', 'pointer'}
assert _ref_map(payload)['p1:2'] == set()

Requires cua-driver >= 0.17 for the live end-to-end reproduction; the unit-level check above runs on any version.

[teknium1]

Thanks @weisiwu — this was exactly right, and we proved it live before landing it: on cua-driver 0.19.3 with a real Chrome window, every typed click refused with browser_ref_stale on main, and your merge fix made the click land (verified by the page navigating and a fresh snapshot).

Your commit is merged to main with your authorship preserved via PR #87299 (bbd3462), plus a regression test we added on top pinning the 0.17+ split refs/content_refs shape. Closing this PR since the work has landed. Much appreciated!