fix(computer-use): merge refs+content_refs in _ref_map for cua-driver 0.17+ (salvage #79515) - #87299
Merged
Conversation
… 0.17 cua-driver >= 0.17 splits the semantic_v2 snapshot payload: action-bearing refs live in the `refs` array while `content_refs` carries every node with EMPTY action lists. _ref_map only absorbed content_refs, so every click/pointer/type ref was registered with no declared actions and all typed-browser mutations failed with browser_ref_stale. Merge refs + content_refs + snapshot.refs with set union so action info is never dropped by an empty content entry. Verified against the 0.17 split format, the legacy refs-only format, the transitional dict format, and the snapshot.refs fallback.
Regression test for the _ref_map merge (salvaged from #79515): the live 0.19.3 driver splits action refs into refs[] while content_refs re-lists every node with empty actions; the empty entries must not clobber the action-bearing ones. Caught live: every typed click refused with browser_ref_stale until the merge fix.
Contributor
૮ >ﻌ< ა ci reviewran on 32ec784 — test(computer-use): pin 0.17+ split refs/content_refs merge
|
alt-glitch
added
type/bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Typed browser clicks/typing work again on current cua-driver builds:
_ref_mapnow mergesrefs+content_refsinstead of preferring one exclusively, so the action-bearing refs that cua-driver ≥ 0.17 ships in the split payload are no longer discarded. Salvage of #79515 by @weisiwu, confirmed live on a real desktop before salvaging.Root cause: 0.17 split the semantic_v2 payload — action refs moved to
refs[]whilecontent_refs[]re-lists every node with empty action lists. Our_ref_mapreadcontent_refsfirst and stopped, so every ref registered with zero actions and every typed mutation refused withbrowser_ref_stale.Live proof (the bug and the fix)
Found while live-verifying the just-merged authorization work (#86342) on an unlocked desktop with cua-driver 0.19.3 + real Chrome:
cua_browser_clickon theLearn morelink refbrowser_ref_stalebrowser_verification_required(guard intact)Trusted-input note from the same run: on Linux the driver refuses the trusted CDP route (
browser_input_trust_unavailable, window would activate) and recommends the explicitdom_eventdowngrade — our wrapper surfaced the escalation losslessly and the explicit downgrade path worked as designed.Changes
tools/computer_use/browser_route.py:_ref_mapabsorbsrefs,content_refs, and legacysnapshot.refsshapes, unioning action sets per ref so empty re-listed entries never clobber action-bearing ones (contributor's commit, cherry-picked with authorship).tests/tools/test_computer_use_cua_0_9.py: regression test pinning the 0.17+ split-shape merge (our follow-up; the original PR had no test).contributors/emails/: mapping for @weisiwu.Validation
test_computer_use_cua_0_9.py+test_computer_use_browser_authorization.pyInfographic