Skip to content

fix(cua-driver): make macOS browser prepare language independent - #3140

Open
loonghao wants to merge 4 commits into
trycua:mainfrom
loonghao:fix/macos-browser-language-independent
Open

fix(cua-driver): make macOS browser prepare language independent#3140
loonghao wants to merge 4 commits into
trycua:mainfrom
loonghao:fix/macos-browser-language-independent

Conversation

@loonghao

@loonghao loonghao commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make macOS existing-profile Chromium preparation and browser-owned consent independent of UI language
  • remove operational dependence on localized AX titles, headings, labels, tab names, and omnibox suggestion suffixes
  • bind consent discovery, structural matching, prompt-state tracking, and the final AXPress to the one approved PID/CGWindowID
  • preserve native-versus-renderer isolation, exact URL checks, ambiguity refusal, cleanup, and rollback

Refs #3139

Why

Localized browser chrome cannot be covered safely by a finite language list. The macOS adapter now treats AX text as opaque Unicode display data and selects controls from role, supported action, ancestry, focus, geometry, native/renderer ownership, uniqueness, and exact postconditions.

A same-PID sibling window is not part of the authorization grant. Earlier revisions enumerated all visible titled windows owned by the browser process while looking for the consent sheet. That could let a structurally valid sheet in another Chrome window satisfy the matcher. This revision removes that expansion and requires fresh AXWindow ownership both while choosing the action and immediately before AXPress.

Setup contract

  • opens one bounded temporary tab through the exact foreground target using the language-independent Command-T shortcut
  • proves exactly one new native tab appeared and retains its unique descendant close action for cleanup
  • selects the unique native actionable AXTextField rather than matching Address and search bar
  • writes and reads back the fixed internal URL exactly
  • commits through AXConfirm when available; otherwise requires the same exact field to own focus and sends one Return without interpreting suggestion text
  • requires one exact native URL, one renderer AXWebArea, one renderer heading, and exactly one enabled renderer AXCheckBox/AXPress action
  • setup-page names may be empty or missing when exact URL and topology independently prove the control
  • keeps pixel fallback bounded by exact URL, one selected native tab, committed navigation, exact-window capture, and unique checkbox-shaped geometry

Consent contract

  • obtains a tree only for the approved CGWindowID; visible same-PID sibling windows are never added as consent surfaces
  • inspects only AXSheet descendants outside renderer content
  • treats sheet/button title, description, identifier, and help strings as opaque data
  • requires every candidate AXButton to resolve through a fresh AXWindow/AXParent lookup to the approved CGWindowID
  • requires two or three enabled native AXButton/AXPress controls with valid non-overlapping geometry
  • requires exactly one focused default-cancel action; selects its peer in the two-button case or within the uniquely adjacent standard pair in the three-button case
  • re-resolves the selected element's owning window immediately before AXPress and returns browser_binding_stale if it moved
  • mirrored RTL geometry retains the same focus relationship; missing ownership, focus, prompt text, duplicate geometry, unexpected topology, disabled controls, or multiple sheets fail closed

Language-independent and wrong-window coverage

Tests cover composed/decomposed Latin text, Devanagari, Thai, Hebrew, Arabic, Persian, Japanese, Korean, Chinese, Armenian, Georgian, Ethiopic, bidi isolates, emoji/ZWJ, empty setup labels, missing consent text, mirrored RTL geometry, duplicate controls, exact URL/topology checks, same-PID sibling surfaces, and a structurally valid consent sheet owned by the wrong window. No language detection, translation, normalization, or locale allowlist is used.

Validation

HEAD candidate: bc04d830642cb935fcbf154ba44d6407835732b0

  • cargo fmt --all -- --check
  • actionlint .github/workflows/ci-cua-driver-contract-clients.yml
  • git diff --check
  • the existing macOS runtime-parity lane runs the focused platform-macos consent matcher tests
  • exact-SHA macOS runtime-parity job passed, including Prove macOS browser consent stays exact-window and language-independent: https://github.com/loonghao/cua/actions/runs/31717340440
  • Windows parity, generated bindings, and MCP discovery jobs in the same run also passed
  • the first Ubuntu attempt hit an unrelated transient Text file busy (os error 26); the failed-job rerun passed, so the exact-SHA workflow is now fully green
  • no live localized macOS host pass is claimed

Remaining draft evidence

  • run the exact-window setup/consent/cleanup chain in a live localized macOS Chromium host
  • keep this PR Draft until live-host evidence is current for the candidate SHA

Safety invariants

  • exact PID/native-window binding is unchanged and now re-proven at the mutation boundary
  • renderer descendants and same-PID sibling windows cannot satisfy native browser chrome or consent matching
  • missing focus on a symmetric consent sheet fails closed
  • setup requires exact URL and unique role/action/topology proof
  • no label allowlist, fuzzy text, OCR, title-only fallback, or unbounded pixel action is introduced

@loonghao loonghao changed the title fix(cua-driver): remove macOS browser language assumptions fix(cua-driver): make macOS browser prepare language independent Aug 13, 2026
@loonghao
loonghao marked this pull request as ready for review August 13, 2026 12:57
@loonghao
loonghao requested a review from f-trycua as a code owner August 13, 2026 12:57
@loonghao
loonghao marked this pull request as draft August 13, 2026 15:45
@loonghao
loonghao marked this pull request as ready for review August 14, 2026 00:52
injaneity added a commit that referenced this pull request Sep 10, 2026
Refs #3705. Reuse the existing public browser_prepare lifecycle test for native red before changing semantic matching. Related localization contributions #3409 and #3140 remain separate.
injaneity added a commit that referenced this pull request Sep 10, 2026
* chore(cua-driver): begin exact consent-label lifecycle regression

Refs #3705. Reuse the existing public browser_prepare lifecycle test for native red before changing semantic matching. Related localization contributions #3409 and #3140 remain separate.

* fix(cua-driver): deduplicate repeated native consent labels

Normalize identical AX text fields once so an English Allow/Cancel title repeated in the description remains an exact semantic match. Distinct text still combines, and sheet ownership, action capability, uniqueness, and ambiguity checks remain unchanged. Refs #3705.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants