fix(browser): add compositor-level coordinate click to browser_click (fixes React onChange/onClick)

[DavidMetcalfe]

Summary

browser_click only supported ref-based clicks, which dispatch synthetic DOM events that React SPAs do not observe for onChange/onClick handlers. Ref-based clicks driven through agent-browser send programmatic .click()/.fill() events that never raise the native low-level input events React's event system listens for — so checkboxes don't toggle and submit buttons don't fire. This is the root cause reported in #55714.

This PR adds an optional (x, y) viewport-coordinate mode that dispatches real compositor-level mouse events via CDP Input.dispatchMouseEvent. Chrome performs its own hit-testing and emits genuine native events, so React state updates exactly as it does for a real user click.

Closes #55714

Prior art: #19189 implemented the same mechanism but was closed without merging (2026-05-10). This re-lands it on current main, minus the supervisor-WS optimization path (that method has since diverged on main; omitted to keep this fix minimal and conflict-free — can follow up as a separate perf PR).

Changes

• browser_click signature now accepts ref OR (x + y), mutually exclusive
• CDP path: single WS connection, cached page session (Target.getTargets + Target.attachToTarget), pipelined mousePressed + mouseReleased, stale-session self-heal
• agent-browser fallback: mouse move/down/up when no CDP endpoint is configured
• Input validation: rejects ref+coords, lopsided coords, or neither
• Schema updated: x/y params documented; required removed (one of ref/coords must be supplied)

Test plan

• 24 new tests in tests/tools/test_browser_coordinate_click.py covering input validation, CDP dispatch via a mock CDP server, and the agent-browser fallback — all pass
• Existing browser tests still green: test_browser_camofox.py, test_browser_private_page_action_guard.py, test_browser_lightpanda.py (83 passed)
• Verified schema exposes x/y and validation returns correct errors

Notes / Open question

This fix is opt-in: the agent must be steered to click by coordinates (via browser_vision with annotate=true or browser_console getBoundingClientRect()) rather than by ref. The default ref-based path remains synthetic and still won't fire React handlers. A fuller fix for the reported flow would make the ref-based click itself resolve the element's bounding box and dispatch a native CDP click — but that's a larger behavioral change (and risks breaking non-React sites that rely on the synthetic path). I scoped this PR to the proven coordinate mechanism so #55714 has a concrete, tested fix. Recommend we decide in review whether to also upgrade the ref path.

[teknium1]

Thanks for the focused reproduction and coverage. The coordinate path is not mergeable as written.

Problems

• tools/browser_tool.py:3337-3346 returns the coordinate dispatch before _blocked_private_page_action. Current main applies that guard to browser_click at tools/browser_tool.py:3035-3038; its helper (tools/browser_tool.py:3253-3267) blocks input on private/internal cloud pages. Coordinate clicks would bypass that boundary.
• The added model-facing x/y schema (tools/browser_tool.py:1851-1865) conflicts with the prior maintainer direction in #19189, which explicitly declined coordinate UX for models. This needs a maintainer design decision before salvage.

Suggested changes

• Put both click modes behind the existing private-page action guard and add a regression test proving a blocked coordinate click sends no CDP or agent-browser input.
• Confirm the intended public API before extending the model schema.

Automated hermes-sweeper review.

[teknium1]

This early coordinate branch bypasses the private-page action guard used by the existing ref path below (_blocked_private_page_action at lines 3360-3363). Move the guard before either dispatch mode and add a test where the private-page guard is active so no CDP or agent-browser input is sent.

[DavidMetcalfe]

@teknium1 — reworked per the review. Both findings addressed:

Finding 1 (private-page guard bypass): Fixed. browser_click now runs _blocked_private_page_action(...) first and returns before any click dispatch. Added tests/tools/test_browser_coordinate_click.py::TestPrivatePageGuard::test_guard_blocks_native_click, which asserts a blocked click issues zero CDP connects / mouse commands / plain clicks.

Finding 2 (coordinate UX declined): Agreed — dropped the model-facing x/y schema entirely. browser_click is ref-only again (schema now has a single ref property). I re-read #19189's closing note and your direction there: port always-CDP-able tools, not a coordinate interface. So this does the opposite of what was declined — it keeps the model API unchanged and fixes the bug at the dispatch layer.

What the rework actually does (fixes #55714): a ref click now dispatches a real compositor-level (native, trusted) click so React onChange/onClick fire:

1. resolve the ref's bounding box via agent-browser get box,
2. if a CDP endpoint exists, Input.dispatchMouseEvent at the element center (trusted events),
3. else fall back to agent-browser mouse move/down/up at the center (also real low-level input),
4. if the box can't be resolved, degrade gracefully to the plain click <ref> (no regression).

The previously-merged supervisor-WS pattern (#23226) isn't required for this — it works with the existing per-call CDP connect (same session-cache + stale-session-reattach logic). If you'd prefer this routed through the supervisor WS once that path is established, happy to follow that up, but it's not a prerequisite for the native-click fix.

Tests: 100 passing across the browser click / camofox / private-page-guard / lightpanda suites. (The 5 failing test_browser_secret_exfil cases are pre-existing on main and unrelated — the file is byte-identical to origin/main.)

I also updated the linked issue #55714's comment to point at this revised approach once it lands. Happy to adjust if you'd rather the native dispatch live behind a different seam.

[DavidMetcalfe]

Update — pushed a follow-up commit addressing two SHOULD-FIXes from a cross-vendor pass (Gemini 3.5 Flash + GPT-OSS 120B), both reviewers agreed:

1. Schema required. browser_click now declares "required": ["ref"] so the model can't call it with no arguments at the API layer. Runtime check inside the function was already present; this just enforces it at schema validation. Schema test updated.

2. _resolve_ref_box zero-size fallback. If width or height is missing or <= 0 (or x/y keys are absent in the dict/string payload), _resolve_ref_box returns None so dispatch falls back to the plain click <ref> path. Without this, a malformed box result would dispatch a click to the top-left corner (or x=0, y=0). Added TestBoxResolutionFailure::test_missing_size_falls_back.

No other findings. The earlier finding that the _native_click docstring duplicates the top-level browser_click docstring (GPT-OSS #11) is a NIT — I'll address in a follow-up rather than expand this PR.

pytest -q across the browser click / camofox / private-page-guard / lightpanda suites: 101 passed.