From 1133f18ba7b374a25a168fddfdabf69fc3545a79 Mon Sep 17 00:00:00 2001 From: LaZzyMan Date: Fri, 3 Jul 2026 01:12:58 +0800 Subject: [PATCH 1/4] feat(cua-driver): sync vendored cua-driver from 0.6.8 to 0.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream 0.7.0 brings action-time modality (get_window_state always returns AX tree + screenshot, capture_mode retired), honest verification (every action returns verified/path/effect/escalation), browser page (CDP) tools, desktop-scope (capture_scope=window|desktop, get_desktop_state, screen-absolute click/scroll), and Rust-only consolidation (Swift backend retired). Cherry-pick patch changes: - #2025 (X11 synthetic click): retired — superseded by 0.7.0 honest-verification - #2035 (session revive): retired — superseded by 0.7.0 revive_session/loud-reject/is_session_lifecycle_tool - #2036 (EAGAIN socket write retry): re-injected — merged upstream post-0.7.0 tag, EAGAIN os error 35 is not acceptable - #2021 (null/empty-title windows + resolve_uwp_host_window): retained, adapted to 0.7.0's new is_listable_top_level and window_title helpers Shim extensions: - coord_norm: desktop-scope support — denormalize_args falls back to screen basis when no window cache (desktop-scope click), normalize_result handles get_desktop_state, ingest_screen_size reads get_desktop_state's screen_width/height - serve.rs: rewrite_coord_desc re-injected at both tools/list sites Install scripts simplified to Rust-only (Swift backend retired upstream). Fix platform-linux cross-compilation: cfg-gate wayland::PORTAL_LIBEI_ENABLED reference in health_report.rs for non-linux hosts. --- packages/cua-driver/.vendored-from | 2 +- packages/cua-driver/.vendored-patches.md | 28 +- packages/cua-driver/python/README.md | 33 +- packages/cua-driver/rust/.bumpversion.cfg | 2 +- packages/cua-driver/rust/Cargo.lock | 27 +- packages/cua-driver/rust/Cargo.toml | 3 +- .../rust/Skills/cua-driver/LINUX.md | 181 +- .../rust/Skills/cua-driver/MACOS.md | 183 +- .../rust/Skills/cua-driver/README.md | 11 +- .../rust/Skills/cua-driver/SKILL.md | 339 +- .../rust/Skills/cua-driver/WEB_APPS.md | 176 + .../rust/Skills/cua-driver/WINDOWS.md | 168 +- .../cua-driver-core/src/capture_mode.rs | 52 + .../crates/cua-driver-core/src/coord_norm.rs | 72 +- .../cua-driver-core/src/element_cache.rs | 12 +- .../rust/crates/cua-driver-core/src/lib.rs | 2 + .../rust/crates/cua-driver-core/src/page.rs | 122 +- .../crates/cua-driver-core/src/session.rs | 86 +- .../cua-driver-core/src/session_tools.rs | 27 +- .../rust/crates/cua-driver-core/src/tool.rs | 4 +- .../crates/cua-driver-core/src/tool_schema.rs | 259 ++ .../tests/session_lifecycle.rs | 216 ++ .../rust/crates/cua-driver-testkit/Cargo.toml | 22 + .../rust/crates/cua-driver-testkit/src/ax.rs | 52 + .../rust/crates/cua-driver-testkit/src/cli.rs | 74 + .../crates/cua-driver-testkit/src/driver.rs | 15 + .../rust/crates/cua-driver-testkit/src/lib.rs | 49 + .../rust/crates/cua-driver-testkit/src/mcp.rs | 140 + .../crates/cua-driver-testkit/src/paths.rs | 43 + .../rust/crates/cua-driver-testkit/src/raw.rs | 68 + .../crates/cua-driver-testkit/src/reaper.rs | 148 + .../crates/cua-driver-testkit/src/response.rs | 80 + .../rust/crates/cua-driver/Cargo.toml | 3 + .../rust/crates/cua-driver/src/cli.rs | 55 +- .../rust/crates/cua-driver/src/main.rs | 7 + .../rust/crates/cua-driver/src/proxy.rs | 4 +- .../rust/crates/cua-driver/src/serve.rs | 240 +- .../tests/e2e_windows_bg_input_test.rs | 558 --- .../cua-driver/tests/focus_check_test.rs | 232 -- .../{ux_guard_test.rs => guard_ux_test.rs} | 236 +- .../cua-driver/tests/harness_appkit_test.rs | 270 +- .../tests/harness_bg_modality_test.rs | 498 --- .../cua-driver/tests/harness_gtk3_test.rs | 196 + .../tests/harness_libreoffice_test.rs | 801 ++++ .../cua-driver/tests/harness_lo_vcl_test.rs | 868 ----- .../cua-driver/tests/harness_swiftui_test.rs | 181 +- ...eb_windows_test.rs => harness_web_test.rs} | 222 +- .../cua-driver/tests/harness_winui3_test.rs | 310 +- .../cua-driver/tests/harness_wpf_test.rs | 604 ++- .../cua-driver/tests/mcp_protocol_test.rs | 3412 ----------------- .../tests/modality_background_test.rs | 398 ++ .../tests/modality_capture_mode_test.rs | 289 ++ .../modality_desktop_scope_linux_test.rs | 186 + .../modality_desktop_scope_macos_test.rs | 206 + .../tests/modality_desktop_scope_test.rs | 152 + .../tests/modality_dispatch_linux_test.rs | 61 + .../tests/modality_dispatch_test.rs | 120 + .../cua-driver/tests/modality_focus_test.rs | 172 + .../tests/modality_input_e2e_test.rs | 516 +++ ...test.rs => protocol_element_token_test.rs} | 18 +- .../tests/protocol_handshake_test.rs | 244 ++ .../cua-driver/tests/protocol_media_test.rs | 576 +++ .../cua-driver/tests/protocol_schema_test.rs | 66 + .../cua-driver/tests/protocol_session_test.rs | 349 ++ .../tests/protocol_tools_call_test.rs | 761 ++++ .../tests/schema_consistency_test.rs | 63 + .../transport_config_persistence_test.rs | 88 + .../rust/crates/cursor-overlay/src/lib.rs | 24 +- .../crates/cursor-overlay/src/render_state.rs | 11 +- .../rust/crates/cursor-overlay/src/shape.rs | 125 +- .../rust/crates/pip-preview/src/lib.rs | 21 +- .../crates/platform-linux/src/atspi/mod.rs | 39 +- .../crates/platform-linux/src/atspi/native.rs | 646 +++- .../platform-linux/src/health_report.rs | 36 +- .../platform-linux/src/input/delivery.rs | 173 + .../crates/platform-linux/src/input/mod.rs | 590 ++- .../rust/crates/platform-linux/src/lib.rs | 3 + .../rust/crates/platform-linux/src/overlay.rs | 16 + .../crates/platform-linux/src/session_bus.rs | 202 + .../crates/platform-linux/src/tools/impl_.rs | 1598 ++++++-- .../crates/platform-linux/src/tools/stubs.rs | 33 +- .../crates/platform-linux/src/wayland/mod.rs | 26 +- .../src/wayland/shell_helper.rs | 110 + .../crates/platform-macos/src/ax/bindings.rs | 33 + .../platform-macos/src/browser/cdp_client.rs | 542 ++- .../platform-macos/src/browser/electron_js.rs | 30 +- .../crates/platform-macos/src/browser/mod.rs | 2 +- .../crates/platform-macos/src/focus_guard.rs | 2 +- .../crates/platform-macos/src/input/mouse.rs | 284 +- .../platform-macos/src/input/skylight.rs | 20 + .../src/tools/bring_to_front.rs | 129 +- .../crates/platform-macos/src/tools/click.rs | 340 +- .../platform-macos/src/tools/cursor_tools.rs | 42 +- .../platform-macos/src/tools/double_click.rs | 61 +- .../crates/platform-macos/src/tools/drag.rs | 56 +- .../platform-macos/src/tools/get_config.rs | 9 +- .../src/tools/get_desktop_state.rs | 171 + .../src/tools/get_screen_size.rs | 2 +- .../src/tools/get_window_state.rs | 128 +- .../crates/platform-macos/src/tools/hotkey.rs | 113 +- .../platform-macos/src/tools/launch_app.rs | 12 +- .../crates/platform-macos/src/tools/mod.rs | 149 +- .../crates/platform-macos/src/tools/page.rs | 54 +- .../platform-macos/src/tools/press_key.rs | 101 +- .../platform-macos/src/tools/right_click.rs | 97 +- .../crates/platform-macos/src/tools/scroll.rs | 223 +- .../platform-macos/src/tools/set_config.rs | 96 +- .../platform-macos/src/tools/set_value.rs | 93 +- .../platform-macos/src/tools/type_text.rs | 527 ++- .../platform-windows/src/health_report.rs | 2 +- .../platform-windows/src/input/delivery.rs | 341 ++ .../platform-windows/src/input/dispatch.rs | 280 -- .../platform-windows/src/input/inject.rs | 503 +-- .../platform-windows/src/input/keyboard.rs | 148 +- .../crates/platform-windows/src/input/mod.rs | 8 +- .../platform-windows/src/input/mouse.rs | 265 +- .../crates/platform-windows/src/terminal.rs | 9 +- .../platform-windows/src/tools/impl_.rs | 2123 +++++++--- .../platform-windows/src/tools/stubs.rs | 28 +- .../crates/platform-windows/src/uia/cache.rs | 112 +- .../src/uia/cache_uaf_repro.rs | 301 ++ .../platform-windows/src/uia/fg_bypass.rs | 30 +- .../crates/platform-windows/src/uia/mod.rs | 13 +- .../crates/platform-windows/src/uia/scroll.rs | 182 + .../rust/crates/platform-windows/src/wgc.rs | 10 +- .../crates/platform-windows/src/win32/mod.rs | 2 +- .../platform-windows/src/win32/windows.rs | 112 +- packages/cua-driver/rust/docs/dedup-audit.md | 163 - .../cua-driver/scripts/CuaDriver.entitlements | 12 - .../cua-driver/scripts/_install-common.sh | 18 +- .../cua-driver/scripts/_install-local-rust.sh | 6 +- .../scripts/_install-local-swift.sh | 294 -- packages/cua-driver/scripts/_install-rust.sh | 2 +- .../scripts/build/build-release-notarized.sh | 274 -- packages/cua-driver/scripts/install-local.sh | 67 +- packages/cua-driver/scripts/install.ps1 | 2 +- packages/cua-driver/scripts/install.sh | 470 +-- .../cua-driver/scripts/post-install-hints.txt | 9 + packages/cua-driver/scripts/test.sh | 23 - packages/cua-driver/scripts/uninstall.sh | 27 +- packages/cua-driver/swift/.bumpversion.cfg | 10 - packages/cua-driver/swift/.gitignore | 79 - .../cua-driver/swift/App/CuaDriver/Info.plist | 32 - packages/cua-driver/swift/Package.resolved | 78 - packages/cua-driver/swift/Package.swift | 47 - .../swift/Skills/cua-driver/README.md | 140 - .../swift/Skills/cua-driver/RECORDING.md | 113 - .../swift/Skills/cua-driver/SKILL.md | 887 ----- .../swift/Skills/cua-driver/TESTS.md | 232 -- .../swift/Skills/cua-driver/WEB_APPS.md | 471 --- .../Sources/CuaDriverCLI/BundleHelpers.swift | 34 - .../Sources/CuaDriverCLI/CallCommand.swift | 649 ---- .../Sources/CuaDriverCLI/ConfigCommand.swift | 423 -- .../CuaDriverCLI/CuaDriverCommand.swift | 766 ---- .../CuaDriverCLI/DiagnoseCommand.swift | 234 -- .../CuaDriverCLI/Docs/CLIDocExtractor.swift | 502 --- .../Sources/CuaDriverCLI/DoctorCommand.swift | 262 -- .../CuaDriverCLI/DumpDocsCommand.swift | 86 - .../CuaDriverCLI/RecordingCommand.swift | 277 -- .../CuaDriverCLI/RecordingRenderCommand.swift | 126 - .../Sources/CuaDriverCLI/ServeCommand.swift | 470 --- .../Sources/CuaDriverCLI/VersionCheck.swift | 61 - .../CuaDriverCore/AppState/AppState.swift | 749 ---- .../CuaDriverCore/Apps/AppEnumerator.swift | 146 - .../Sources/CuaDriverCore/Apps/AppInfo.swift | 52 - .../CuaDriverCore/Apps/AppLauncher.swift | 316 -- .../CuaDriverCore/Browser/AXPageReader.swift | 200 - .../CuaDriverCore/Browser/BrowserJS.swift | 336 -- .../CuaDriverCore/Browser/CDPClient.swift | 195 - .../CuaDriverCore/Browser/ElectronJS.swift | 314 -- .../Browser/WebInspectorXPC.swift | 84 - .../CuaDriverCore/Browser/WebKitJS.swift | 48 - .../Capture/DebugCrosshair.swift | 158 - .../CuaDriverCore/Capture/ScreenInfo.swift | 33 - .../CuaDriverCore/Capture/WindowCapture.swift | 467 --- .../CuaDriverCore/Config/ConfigStore.swift | 233 -- .../Config/CuaDriverConfig.swift | 279 -- .../Sources/CuaDriverCore/CuaDriverCore.swift | 5 - .../CuaDriverCore/Cursor/AgentCursor.swift | 662 ---- .../Cursor/AgentCursorOverlayWindow.swift | 72 - .../Cursor/AgentCursorRenderer.swift | 370 -- .../Cursor/AgentCursorView.swift | 117 - .../Sources/CuaDriverCore/Cursor/Bezier.swift | 71 - .../Cursor/CursorMotionPath.swift | 170 - .../Focus/AXEnablementAssertion.swift | 80 - .../CuaDriverCore/Focus/FocusGuard.swift | 192 - .../Focus/SyntheticAppFocusEnforcer.swift | 109 - .../Focus/SystemFocusStealPreventer.swift | 643 ---- .../Sources/CuaDriverCore/Input/AXInput.swift | 297 -- .../CuaDriverCore/Input/CursorControl.swift | 22 - .../CuaDriverCore/Input/CursorPoint.swift | 11 - .../Input/FocusWithoutRaise.swift | 203 - .../CuaDriverCore/Input/KeyboardInput.swift | 268 -- .../CuaDriverCore/Input/MouseInput.swift | 876 ----- .../Input/SkyLightEventPost.swift | 540 --- .../Permissions/Permissions.swift | 67 - .../Permissions/PermissionsGate.swift | 387 -- .../Recording/ClickMarkerRenderer.swift | 164 - .../Recording/CursorSampler.swift | 137 - .../Recording/RecordingSession.swift | 571 --- .../Recording/Render/FrameTransform.swift | 89 - .../Recording/Render/RecordingRenderer.swift | 597 --- .../Recording/Render/TrajectoryLoader.swift | 333 -- .../Recording/VideoRecorder.swift | 412 -- .../Recording/Zoom/ActionSpan.swift | 171 - .../Recording/Zoom/CursorTelemetry.swift | 103 - .../Recording/Zoom/ZoomMath.swift | 84 - .../Recording/Zoom/ZoomRegion.swift | 340 -- .../Telemetry/TelemetryClient.swift | 268 -- .../CuaDriverCore/Windows/SpaceMigrator.swift | 133 - .../Windows/WindowCoordinateSpace.swift | 109 - .../Windows/WindowEnumerator.swift | 107 - .../CuaDriverCore/Windows/WindowInfo.swift | 52 - .../CuaDriverServer/AppStateRegistry.swift | 27 - .../ClaudeCodeComputerUseCompatTools.swift | 161 - .../CuaDriverServer/CuaDriverMCPServer.swift | 209 - .../CuaDriverServer/DaemonClient.swift | 163 - .../CuaDriverServer/DaemonProtocol.swift | 125 - .../CuaDriverServer/DaemonServer.swift | 457 --- .../Docs/MCPDocExtractor.swift | 37 - .../CuaDriverServer/ToolRegistry.swift | 263 -- .../Tools/CheckPermissionsTool.swift | 59 - .../CuaDriverServer/Tools/ClickTool.swift | 595 --- .../Tools/DoubleClickTool.swift | 327 -- .../CuaDriverServer/Tools/DragTool.swift | 327 -- .../Tools/GetAgentCursorStateTool.swift | 68 - .../CuaDriverServer/Tools/GetConfigTool.swift | 75 - .../Tools/GetCursorPositionTool.swift | 37 - .../Tools/GetRecordingStateTool.swift | 67 - .../Tools/GetScreenSizeTool.swift | 46 - .../Tools/GetWindowStateTool.swift | 586 --- .../CuaDriverServer/Tools/HotkeyTool.swift | 142 - .../Tools/ImageResizeRegistry.swift | 54 - .../CuaDriverServer/Tools/LaunchAppTool.swift | 514 --- .../CuaDriverServer/Tools/ListAppsTool.swift | 71 - .../Tools/ListWindowsTool.swift | 245 -- .../Tools/MoveCursorTool.swift | 60 - .../CuaDriverServer/Tools/PageTool.swift | 358 -- .../CuaDriverServer/Tools/PressKeyTool.swift | 202 - .../Tools/ReplayTrajectoryTool.swift | 305 -- .../Tools/RightClickTool.swift | 324 -- .../Tools/ScreenshotTool.swift | 170 - .../CuaDriverServer/Tools/ScrollTool.swift | 211 - .../Tools/SetAgentCursorEnabledTool.swift | 87 - .../Tools/SetAgentCursorMotionTool.swift | 187 - .../Tools/SetAgentCursorStyleTool.swift | 111 - .../CuaDriverServer/Tools/SetConfigTool.swift | 268 -- .../Tools/SetRecordingTool.swift | 146 - .../CuaDriverServer/Tools/SetValueTool.swift | 336 -- .../CuaDriverServer/Tools/TypeTextTool.swift | 225 -- .../Tools/WindowChangeDetector.swift | 260 -- .../CuaDriverServer/Tools/ZoomTool.swift | 211 - .../FocusMonitorApp/FocusMonitorApp.swift | 190 - .../swift/Tests/FocusMonitorApp/build.sh | 35 - .../FocusStealPreventerTests.swift | 320 -- .../Tests/ZoomMathTests/ZoomMathTests.swift | 269 -- .../Tests/integration/assets/test_page.html | 1 - .../swift/Tests/integration/conftest.py | 282 -- .../swift/Tests/integration/driver_client.py | 209 - .../integration/fixtures/form_all_inputs.html | 1 - .../integration/fixtures/interactive.html | 1 - .../Tests/integration/harness/__init__.py | 0 .../swift/Tests/integration/harness/cv.py | 132 - .../swift/Tests/integration/harness/driver.py | 295 -- .../Tests/integration/harness/monitor.py | 224 -- .../swift/Tests/integration/harness/tree.py | 99 - .../test_app_name_locale_fallback.py | 110 - .../integration/test_background_focus.py | 674 ---- .../test_background_menu_shortcut.py | 168 - .../swift/Tests/integration/test_blender.py | 198 - .../Tests/integration/test_browser_js.py | 336 -- .../integration/test_check_permissions_cli.py | 130 - .../swift/Tests/integration/test_chrome.py | 201 - .../integration/test_chrome_minimized_nav.py | 370 -- .../test_click_opens_new_window.py | 275 -- .../Tests/integration/test_click_pixel_ax.py | 175 - .../integration/test_drag_slider_delivery.py | 309 -- .../swift/Tests/integration/test_electron.py | 198 - .../test_hermes_chrome_form_fill.py | 603 --- .../integration/test_hermes_form_fill.py | 396 -- .../integration/test_hermes_form_fill_qwen.py | 223 -- .../test_hermes_launch_safari_bg.py | 270 -- .../integration/test_hidden_app_capture.py | 151 - .../integration/test_launch_app_visible.py | 126 - .../Tests/integration/test_list_windows.py | 112 - .../Tests/integration/test_overlay_z_order.py | 213 - .../swift/Tests/integration/test_safari.py | 333 -- .../swift/Tests/integration/test_tauri.py | 155 - .../swift/Tests/integration/test_webkit_js.py | 220 -- .../test-harness/TEST_HARNESS_STRUCTURE.md | 117 + .../cua-driver/test-harness/TEST_SUITE.md | 128 + .../apps/cross-platform/electron/build.sh | 60 + .../apps/cross-platform/electron/package.json | 13 +- .../test-harness/apps/linux/gtk3/main.py | 203 + .../test-harness/apps/macos/appkit/main.swift | 208 +- .../apps/macos/swiftui/main.swift | 118 +- .../apps/macos/wkwebview/main.swift | 70 + .../winui3/CuaTestHarness.WinUI3.csproj | 2 +- .../apps/windows/winui3/MainWindow.xaml | 67 + .../apps/windows/winui3/MainWindow.xaml.cs | 101 + .../windows/wpf/CuaTestHarness.Wpf.csproj | 2 +- .../cua-driver/test-harness/build/linux.sh | 39 + .../cua-driver/test-harness/build/macos.sh | 33 +- .../test-harness/linux-container/README.md | 148 + .../test-harness/linux-container/calc.sh | 142 + .../test-harness/linux-container/derec.sh | 190 + .../linux-container/modality_matrix.sh | 151 + .../test-harness/linux-container/validate.sh | 32 + .../modality-recordings/FINDINGS.md | 245 ++ .../modality-recordings/README.md | 73 + .../modality-recordings/index.html | 34 + .../linux/lin-all-electron.sh | 11 + .../modality-recordings/linux/lin-all.sh | 6 + .../modality-recordings/linux/lin-dash.py | 26 + .../linux/lin-dashboard-electron.html | 31 + .../linux/lin-dashboard.html | 31 + .../modality-recordings/linux/lin-harness.py | 167 + .../linux/lin-rec-electron.py | 353 ++ .../modality-recordings/linux/lin-rec.py | 307 ++ .../linux/lin-run-electron.sh | 27 + .../modality-recordings/linux/lin-run.sh | 20 + .../modality-recordings/macos/mac-rec.py | 546 +++ .../modality-recordings/windows/run-one.ps1 | 4 + .../windows/wpf-recorder.ps1 | 365 ++ .../test-harness/shared/scenarios.json | 966 +++-- .../test-harness/shared/web/index.html | 26 + .../test-harness/vision-agent-test/README.md | 52 + .../vision-agent-test/vision_agent_test.py | 381 ++ packages/cua-driver/wayland-helper/README.md | 33 + packages/cua-driver/wayland-helper/install.sh | 25 + .../wayland-helper/winrects@cua/extension.js | 87 + .../wayland-helper/winrects@cua/metadata.json | 1 + 332 files changed, 22960 insertions(+), 45685 deletions(-) create mode 100644 packages/cua-driver/rust/crates/cua-driver-core/src/capture_mode.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-core/src/tool_schema.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-core/tests/session_lifecycle.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/Cargo.toml create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/src/ax.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/src/cli.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/src/driver.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/src/lib.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/src/mcp.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/src/paths.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/src/raw.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/src/reaper.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver-testkit/src/response.rs delete mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/e2e_windows_bg_input_test.rs delete mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/focus_check_test.rs rename packages/cua-driver/rust/crates/cua-driver/tests/{ux_guard_test.rs => guard_ux_test.rs} (68%) delete mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/harness_bg_modality_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/harness_gtk3_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/harness_libreoffice_test.rs delete mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/harness_lo_vcl_test.rs rename packages/cua-driver/rust/crates/cua-driver/tests/{harness_web_windows_test.rs => harness_web_test.rs} (51%) delete mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/mcp_protocol_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/modality_background_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/modality_capture_mode_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/modality_desktop_scope_linux_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/modality_desktop_scope_macos_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/modality_desktop_scope_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/modality_dispatch_linux_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/modality_dispatch_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/modality_focus_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/modality_input_e2e_test.rs rename packages/cua-driver/rust/crates/cua-driver/tests/{element_token_test.rs => protocol_element_token_test.rs} (93%) create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/protocol_handshake_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/protocol_media_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/protocol_schema_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/protocol_session_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/protocol_tools_call_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/schema_consistency_test.rs create mode 100644 packages/cua-driver/rust/crates/cua-driver/tests/transport_config_persistence_test.rs create mode 100644 packages/cua-driver/rust/crates/platform-linux/src/input/delivery.rs create mode 100644 packages/cua-driver/rust/crates/platform-linux/src/session_bus.rs create mode 100644 packages/cua-driver/rust/crates/platform-linux/src/wayland/shell_helper.rs create mode 100644 packages/cua-driver/rust/crates/platform-macos/src/tools/get_desktop_state.rs create mode 100644 packages/cua-driver/rust/crates/platform-windows/src/input/delivery.rs delete mode 100644 packages/cua-driver/rust/crates/platform-windows/src/input/dispatch.rs create mode 100644 packages/cua-driver/rust/crates/platform-windows/src/uia/cache_uaf_repro.rs create mode 100644 packages/cua-driver/rust/crates/platform-windows/src/uia/scroll.rs delete mode 100644 packages/cua-driver/rust/docs/dedup-audit.md delete mode 100644 packages/cua-driver/scripts/CuaDriver.entitlements delete mode 100755 packages/cua-driver/scripts/_install-local-swift.sh delete mode 100755 packages/cua-driver/scripts/build/build-release-notarized.sh delete mode 100755 packages/cua-driver/scripts/test.sh delete mode 100644 packages/cua-driver/swift/.bumpversion.cfg delete mode 100644 packages/cua-driver/swift/.gitignore delete mode 100644 packages/cua-driver/swift/App/CuaDriver/Info.plist delete mode 100644 packages/cua-driver/swift/Package.resolved delete mode 100644 packages/cua-driver/swift/Package.swift delete mode 100644 packages/cua-driver/swift/Skills/cua-driver/README.md delete mode 100644 packages/cua-driver/swift/Skills/cua-driver/RECORDING.md delete mode 100644 packages/cua-driver/swift/Skills/cua-driver/SKILL.md delete mode 100644 packages/cua-driver/swift/Skills/cua-driver/TESTS.md delete mode 100644 packages/cua-driver/swift/Skills/cua-driver/WEB_APPS.md delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/BundleHelpers.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/CallCommand.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/ConfigCommand.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/CuaDriverCommand.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/DiagnoseCommand.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/Docs/CLIDocExtractor.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/DoctorCommand.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/DumpDocsCommand.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/RecordingCommand.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/RecordingRenderCommand.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/ServeCommand.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCLI/VersionCheck.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/AppState/AppState.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Apps/AppEnumerator.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Apps/AppInfo.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Apps/AppLauncher.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Browser/AXPageReader.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Browser/BrowserJS.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Browser/CDPClient.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Browser/ElectronJS.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Browser/WebInspectorXPC.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Browser/WebKitJS.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Capture/DebugCrosshair.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Capture/ScreenInfo.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Capture/WindowCapture.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Config/ConfigStore.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Config/CuaDriverConfig.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/CuaDriverCore.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Cursor/AgentCursor.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Cursor/AgentCursorOverlayWindow.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Cursor/AgentCursorRenderer.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Cursor/AgentCursorView.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Cursor/Bezier.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Cursor/CursorMotionPath.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Focus/AXEnablementAssertion.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Focus/FocusGuard.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Focus/SyntheticAppFocusEnforcer.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Focus/SystemFocusStealPreventer.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Input/AXInput.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Input/CursorControl.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Input/CursorPoint.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Input/FocusWithoutRaise.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Input/KeyboardInput.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Input/MouseInput.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Input/SkyLightEventPost.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Permissions/Permissions.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Permissions/PermissionsGate.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/ClickMarkerRenderer.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/CursorSampler.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/RecordingSession.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/Render/FrameTransform.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/Render/RecordingRenderer.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/Render/TrajectoryLoader.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/VideoRecorder.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/Zoom/ActionSpan.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/Zoom/CursorTelemetry.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/Zoom/ZoomMath.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Recording/Zoom/ZoomRegion.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Telemetry/TelemetryClient.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Windows/SpaceMigrator.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Windows/WindowCoordinateSpace.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Windows/WindowEnumerator.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverCore/Windows/WindowInfo.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/AppStateRegistry.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/ClaudeCodeComputerUseCompatTools.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/CuaDriverMCPServer.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/DaemonClient.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/DaemonProtocol.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/DaemonServer.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Docs/MCPDocExtractor.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/ToolRegistry.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/CheckPermissionsTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/ClickTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/DoubleClickTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/DragTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/GetAgentCursorStateTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/GetConfigTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/GetCursorPositionTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/GetRecordingStateTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/GetScreenSizeTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/HotkeyTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/ImageResizeRegistry.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/LaunchAppTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/ListAppsTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/ListWindowsTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/MoveCursorTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/PageTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/PressKeyTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/ReplayTrajectoryTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/RightClickTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/ScreenshotTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/ScrollTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/SetAgentCursorEnabledTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/SetAgentCursorMotionTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/SetAgentCursorStyleTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/SetConfigTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/SetRecordingTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/SetValueTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/TypeTextTool.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/WindowChangeDetector.swift delete mode 100644 packages/cua-driver/swift/Sources/CuaDriverServer/Tools/ZoomTool.swift delete mode 100644 packages/cua-driver/swift/Tests/FocusMonitorApp/FocusMonitorApp.swift delete mode 100755 packages/cua-driver/swift/Tests/FocusMonitorApp/build.sh delete mode 100644 packages/cua-driver/swift/Tests/FocusStealPreventerTests/FocusStealPreventerTests.swift delete mode 100644 packages/cua-driver/swift/Tests/ZoomMathTests/ZoomMathTests.swift delete mode 120000 packages/cua-driver/swift/Tests/integration/assets/test_page.html delete mode 100644 packages/cua-driver/swift/Tests/integration/conftest.py delete mode 100644 packages/cua-driver/swift/Tests/integration/driver_client.py delete mode 120000 packages/cua-driver/swift/Tests/integration/fixtures/form_all_inputs.html delete mode 120000 packages/cua-driver/swift/Tests/integration/fixtures/interactive.html delete mode 100644 packages/cua-driver/swift/Tests/integration/harness/__init__.py delete mode 100644 packages/cua-driver/swift/Tests/integration/harness/cv.py delete mode 100644 packages/cua-driver/swift/Tests/integration/harness/driver.py delete mode 100644 packages/cua-driver/swift/Tests/integration/harness/monitor.py delete mode 100644 packages/cua-driver/swift/Tests/integration/harness/tree.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_app_name_locale_fallback.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_background_focus.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_background_menu_shortcut.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_blender.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_browser_js.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_check_permissions_cli.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_chrome.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_chrome_minimized_nav.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_click_opens_new_window.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_click_pixel_ax.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_drag_slider_delivery.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_electron.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_hermes_chrome_form_fill.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_hermes_form_fill.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_hermes_form_fill_qwen.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_hermes_launch_safari_bg.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_hidden_app_capture.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_launch_app_visible.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_list_windows.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_overlay_z_order.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_safari.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_tauri.py delete mode 100644 packages/cua-driver/swift/Tests/integration/test_webkit_js.py create mode 100644 packages/cua-driver/test-harness/TEST_HARNESS_STRUCTURE.md create mode 100644 packages/cua-driver/test-harness/TEST_SUITE.md create mode 100755 packages/cua-driver/test-harness/apps/cross-platform/electron/build.sh create mode 100755 packages/cua-driver/test-harness/apps/linux/gtk3/main.py create mode 100644 packages/cua-driver/test-harness/apps/macos/wkwebview/main.swift create mode 100755 packages/cua-driver/test-harness/build/linux.sh create mode 100644 packages/cua-driver/test-harness/linux-container/README.md create mode 100644 packages/cua-driver/test-harness/linux-container/calc.sh create mode 100755 packages/cua-driver/test-harness/linux-container/derec.sh create mode 100755 packages/cua-driver/test-harness/linux-container/modality_matrix.sh create mode 100755 packages/cua-driver/test-harness/linux-container/validate.sh create mode 100644 packages/cua-driver/test-harness/modality-recordings/FINDINGS.md create mode 100644 packages/cua-driver/test-harness/modality-recordings/README.md create mode 100644 packages/cua-driver/test-harness/modality-recordings/index.html create mode 100755 packages/cua-driver/test-harness/modality-recordings/linux/lin-all-electron.sh create mode 100644 packages/cua-driver/test-harness/modality-recordings/linux/lin-all.sh create mode 100644 packages/cua-driver/test-harness/modality-recordings/linux/lin-dash.py create mode 100644 packages/cua-driver/test-harness/modality-recordings/linux/lin-dashboard-electron.html create mode 100644 packages/cua-driver/test-harness/modality-recordings/linux/lin-dashboard.html create mode 100644 packages/cua-driver/test-harness/modality-recordings/linux/lin-harness.py create mode 100644 packages/cua-driver/test-harness/modality-recordings/linux/lin-rec-electron.py create mode 100644 packages/cua-driver/test-harness/modality-recordings/linux/lin-rec.py create mode 100755 packages/cua-driver/test-harness/modality-recordings/linux/lin-run-electron.sh create mode 100644 packages/cua-driver/test-harness/modality-recordings/linux/lin-run.sh create mode 100644 packages/cua-driver/test-harness/modality-recordings/macos/mac-rec.py create mode 100644 packages/cua-driver/test-harness/modality-recordings/windows/run-one.ps1 create mode 100644 packages/cua-driver/test-harness/modality-recordings/windows/wpf-recorder.ps1 create mode 100644 packages/cua-driver/test-harness/vision-agent-test/README.md create mode 100644 packages/cua-driver/test-harness/vision-agent-test/vision_agent_test.py create mode 100644 packages/cua-driver/wayland-helper/README.md create mode 100755 packages/cua-driver/wayland-helper/install.sh create mode 100644 packages/cua-driver/wayland-helper/winrects@cua/extension.js create mode 100644 packages/cua-driver/wayland-helper/winrects@cua/metadata.json diff --git a/packages/cua-driver/.vendored-from b/packages/cua-driver/.vendored-from index f7115464c89..c52aeda35ca 100644 --- a/packages/cua-driver/.vendored-from +++ b/packages/cua-driver/.vendored-from @@ -1 +1 @@ -cua-driver-rs-v0.6.8 +cua-driver-rs-v0.7.0 diff --git a/packages/cua-driver/.vendored-patches.md b/packages/cua-driver/.vendored-patches.md index 74305e54147..b249d207039 100644 --- a/packages/cua-driver/.vendored-patches.md +++ b/packages/cua-driver/.vendored-patches.md @@ -1,18 +1,20 @@ -# Cherry-picked upstream PRs (not yet merged) +# Cherry-picked upstream PRs This vendored copy is based on the tag in [`.vendored-from`](./.vendored-from) -(`cua-driver-rs-v0.6.8`), **plus** a small set of upstream trycua/cua pull -requests that were still _open_ when we needed their fixes (all four are still -open as of the 0.6.8 sync — confirmed by the sync applying with no collision on -their lines). They are applied as -ordinary commits on top of the base snapshot. +(`cua-driver-rs-v0.7.0`), **plus** a small set of upstream trycua/cua pull +requests applied as ordinary commits on top of the base snapshot. -| Upstream PR | Fix | Touches | +| Upstream PR | Fix | Status | Touches | +|---|---|---|---| +| [trycua/cua#2036](https://github.com/trycua/cua/pull/2036) | retry daemon socket writes on `EAGAIN`/`EWOULDBLOCK` instead of failing fatally | upstream-merged post-0.7.0 tag; re-injected here, remove when syncing to a ref that includes it | `cua-driver-core` (socket_io), `cua-driver` (serve.rs) | +| [trycua/cua#2021](https://github.com/trycua/cua/pull/2021) | `list_windows` includes empty-/null-title top-level windows; `resolve_uwp_host_window` for UWP apps | open (draft) | `platform-windows` | + +## Retired patches (subsumed by 0.7.0) + +| Upstream PR | Fix | Why retired | |---|---|---| -| [trycua/cua#2035](https://github.com/trycua/cua/pull/2035) | `start_session` revives an idle-reaped session instead of erroring | `cua-driver-core` (session) | -| [trycua/cua#2036](https://github.com/trycua/cua/pull/2036) | retry daemon socket writes on `EAGAIN`/`EWOULDBLOCK` instead of failing fatally | `cua-driver-core` (socket_io) | -| [trycua/cua#2025](https://github.com/trycua/cua/pull/2025) | stop reporting a bare "Clicked" for X11 synthetic (`XSendEvent`) clicks that toolkits ignore | `platform-linux` | -| [trycua/cua#2021](https://github.com/trycua/cua/pull/2021) | `list_windows` includes empty-/null-title top-level windows | `platform-windows` | +| [trycua/cua#2025](https://github.com/trycua/cua/pull/2025) | X11 synthetic click false-success | Superseded by 0.7.0 honest-verification system | +| [trycua/cua#2035](https://github.com/trycua/cua/pull/2035) | `start_session` revive idle-reaped session | Superseded by 0.7.0 `revive_session`/loud-reject/`is_session_lifecycle_tool` | ## When you sync to a newer upstream ref @@ -22,8 +24,8 @@ already in the upstream delta, so: 1. The hunks will collide with the cherry-picked commit (expect `.rej` on the files above), and -2. Once reconciled, **remove the row from this table** — it's no longer a local - patch, it's part of the base. +2. Once reconciled, **remove the row from the active table** — it's no longer a + local patch, it's part of the base. If `` is still *before* a PR merges, that row stays; the cherry-pick keeps carrying the fix. diff --git a/packages/cua-driver/python/README.md b/packages/cua-driver/python/README.md index 5440a2e289b..5a56e183298 100644 --- a/packages/cua-driver/python/README.md +++ b/packages/cua-driver/python/README.md @@ -4,26 +4,15 @@ Python wrapper for [cua-driver](https://github.com/trycua/cua/tree/main/libs/cua ## Installation -```bash -pip install cua-driver -``` +Install and usage docs live at https://cua.ai/docs/how-to-guides/driver/install +and https://cua.ai/docs/reference/cua-driver/mcp-tools. ## Usage -The package provides a `cua-driver` command that wraps the native Rust binary: - -```bash -# Start MCP server over stdio -cua-driver mcp - -# Check version -cua-driver --version - -# View help -cua-driver --help -``` +The package provides a `cua-driver` command that wraps the native Rust binary. +See the canonical tool reference at https://cua.ai/docs/reference/cua-driver/mcp-tools. -### Python API +## Python API You can also use the Python API directly: @@ -44,16 +33,6 @@ binary_path = get_binary_path() - **Stdio passthrough**: Transparent piping for MCP protocol communication - **Bundled binary**: No separate installation required - the Rust binary is included in the wheel -## What is cua-driver? - -`cua-driver` is a cross-platform MCP server that provides 40+ tools for: -- Screen capture and window management -- Mouse and keyboard automation -- Accessibility tree interaction -- Application launching and control - -It enables AI agents to interact with desktop applications through the Model Context Protocol. - ## Platform Support | Platform | Architecture | Status | @@ -70,5 +49,5 @@ MIT License - see [LICENSE](https://github.com/trycua/cua/blob/main/LICENSE.md) ## Links - [GitHub Repository](https://github.com/trycua/cua) -- [Documentation](https://github.com/trycua/cua/tree/main/libs/cua-driver) +- [Documentation](https://cua.ai/docs) - [Issue Tracker](https://github.com/trycua/cua/issues) diff --git a/packages/cua-driver/rust/.bumpversion.cfg b/packages/cua-driver/rust/.bumpversion.cfg index 15123eb60ee..94dff1b4e7a 100644 --- a/packages/cua-driver/rust/.bumpversion.cfg +++ b/packages/cua-driver/rust/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 0.6.8 +current_version = 0.7.0 commit = True tag = True tag_name = cua-driver-rs-v{new_version} diff --git a/packages/cua-driver/rust/Cargo.lock b/packages/cua-driver/rust/Cargo.lock index 86ab7a17f7c..b4fb9ea75c3 100644 --- a/packages/cua-driver/rust/Cargo.lock +++ b/packages/cua-driver/rust/Cargo.lock @@ -602,12 +602,13 @@ dependencies = [ [[package]] name = "cua-driver" -version = "0.6.8" +version = "0.7.0" dependencies = [ "anyhow", "async-trait", "base64", "cua-driver-core", + "cua-driver-testkit", "cursor-overlay", "embed-resource", "flate2", @@ -633,7 +634,7 @@ dependencies = [ [[package]] name = "cua-driver-core" -version = "0.6.8" +version = "0.7.0" dependencies = [ "anyhow", "async-trait", @@ -646,9 +647,17 @@ dependencies = [ "windows 0.58.0", ] +[[package]] +name = "cua-driver-testkit" +version = "0.7.0" +dependencies = [ + "serde_json", + "windows 0.61.3", +] + [[package]] name = "cua-driver-uia" -version = "0.6.8" +version = "0.7.0" dependencies = [ "anyhow", "cua-driver-core", @@ -663,7 +672,7 @@ dependencies = [ [[package]] name = "cursor-overlay" -version = "0.6.8" +version = "0.7.0" dependencies = [ "anyhow", "image", @@ -930,7 +939,7 @@ checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" [[package]] name = "focus-monitor-win" -version = "0.6.8" +version = "0.7.0" dependencies = [ "windows 0.58.0", ] @@ -1883,7 +1892,7 @@ checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" [[package]] name = "pip-preview" -version = "0.6.8" +version = "0.7.0" dependencies = [ "anyhow", "serde_json", @@ -1937,7 +1946,7 @@ checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" [[package]] name = "platform-linux" -version = "0.6.8" +version = "0.7.0" dependencies = [ "anyhow", "ashpd", @@ -1977,7 +1986,7 @@ dependencies = [ [[package]] name = "platform-macos" -version = "0.6.8" +version = "0.7.0" dependencies = [ "anyhow", "async-trait", @@ -2010,7 +2019,7 @@ dependencies = [ [[package]] name = "platform-windows" -version = "0.6.8" +version = "0.7.0" dependencies = [ "anyhow", "async-trait", diff --git a/packages/cua-driver/rust/Cargo.toml b/packages/cua-driver/rust/Cargo.toml index 35fc965f06a..564ca69f51f 100644 --- a/packages/cua-driver/rust/Cargo.toml +++ b/packages/cua-driver/rust/Cargo.toml @@ -2,6 +2,7 @@ resolver = "2" members = [ "crates/cua-driver", + "crates/cua-driver-testkit", "crates/cua-driver-uia", "crates/cua-driver-core", "crates/platform-macos", @@ -13,7 +14,7 @@ members = [ ] [workspace.package] -version = "0.6.8" +version = "0.7.0" edition = "2021" authors = ["trycua"] license = "MIT" diff --git a/packages/cua-driver/rust/Skills/cua-driver/LINUX.md b/packages/cua-driver/rust/Skills/cua-driver/LINUX.md index 1455a44b65f..e73f89d2deb 100644 --- a/packages/cua-driver/rust/Skills/cua-driver/LINUX.md +++ b/packages/cua-driver/rust/Skills/cua-driver/LINUX.md @@ -36,6 +36,150 @@ AT-SPI is talked to natively over D-Bus (the `atspi`/zbus crate) — no It glides on clicks and `move_cursor`; issue `move_cursor` to make it track a field while typing advances focus across cells. +## `delivery_mode` — the background/foreground ladder + +Every input tool (`click`, `type_text`, `press_key`, `hotkey`, +`double_click`, `right_click`, `scroll`) takes an optional **`delivery_mode`** +— the per-call rung of the best-effort-background ladder, matching the macOS +and Windows surface: + +- **`background`** (default) — inject **without activating or raising** the + target. X11: the no-focus-steal paths above (AT-SPI / `XSendEvent` / + XInput2 MPX pointer). This is cua-driver's differentiator and the right + default. +- **`foreground`** — **activate the target first** (X11 EWMH + `_NET_ACTIVE_WINDOW`, proper timestamp handling to beat the WM's + focus-stealing prevention), inject, then **restore the prior active + window**. The explicit escalation when a background inject didn't land — + e.g. a GTK dialog button or a widget that only reads input while focused. + A brief focus swap unless the target was already active. + +**`bring_to_front`** (X11): persistent `_NET_ACTIVE_WINDOW` activation (the +`wmctrl -a` equivalent), kept active. Call it before `delivery_mode:"foreground"` +input to avoid a per-call flash, or to escalate when background didn't land. + +**Read-back / `verified`** — `type_text` reports `{verified}`: the AT-SPI +`EditableText.insertText` path (`path:"ax"`) is the **driver-verifiable** rung +(the a11y layer confirms the insert into the widget model) → `verified:true`; +keystroke / XSendEvent / XTest / foreground rungs are not read-back-confirmed +→ `verified:false` (confirm via screenshot). Mirrors the macOS/Windows verdict. + +**`effect` / `escalation`** — alongside `verified`, action responses carry the +cross-platform `effect` (`confirmed` / `unverifiable` / `suspected_noop`) and, +when you should change rung, `escalation:{recommended, reason}`. See `SKILL.md` +→ behavior matrix. The Linux-specific value of `recommended` is **`foreground` +on Wayland** (background pixel can't target an unfocused window) and +**`px` on X11** (an element px action — background pixel click — lands via +AT-SPI `do_action`-at-point off the screenshot already in the snapshot — the +matrix below). + +## Perception and the ax/px action choice + +`get_window_state` is **perception-mode-agnostic** — by default it returns +**both** the AT-SPI tree **and** a screenshot in one call. You ground on both +and cross-check; the tree **lies** on some surfaces (Electron echo-confirms +`setValue`, virtualized/off-viewport rows report bogus `h:1` frames), so a +grounding screenshot is always present by default. There is no capture mode to +pick. + +**Perf opt-out — `include_screenshot`** (boolean, default `true`). Pass +`include_screenshot:false` to skip the screen grab and return tree-only — the +cheap path when you're re-indexing before an **element ax action** and don't +need fresh pixels. It's a **perf** knob, not a modality choice. + +**`capture_mode` is DEPRECATED and IGNORED.** It's still *accepted* so old +callers don't error, but it has **no effect** — both the tree and the +screenshot come back regardless of what you pass (`ax`/`vision`/`som`). There +is no `ax`/`vision`/`som` capture choice anymore; drop that vocabulary. + +Modality is chosen at **action time**, by how you address the target: + +- **element ax action** — `element_index` / `element_token` → AT-SPI + `do_action`. Backgroundable, driver-verifiable. +- **element px action** — `x,y` → pixel rung, read straight off the screenshot + already in the `get_window_state` response. Best-effort; caller-confirmed. + +`get_window_state` returning `degraded:true` (empty AT-SPI walk) is the cue to +do an **element px action** off that same screenshot (X11) or escalate to +`delivery_mode:"foreground"` (Wayland — background pixel can't target an +unfocused window there). + +## Cross-platform schema residuals (Linux) + +The capture/dispatch/addressing params are a shared cross-platform +contract (see `SKILL.md` → *Cross-platform parameter contract*) — the +same `session`, `delivery_mode`, `capture_mode`, `scope`, `modifier`, +`element_index`/`element_token` *shapes* as macOS and Windows, gated in +CI so the three surfaces can't drift. Linux-relevant notes: + +- **`session` is now accepted on every action/cursor tool.** Earlier + Linux builds rejected it via `additionalProperties:false` (it was + effectively macOS-only); it is now uniformly schema-accepted — Linux + glides a per-session cursor on X11 where the overlay is available. +- **Windowless screen-absolute clicks** are supported on Linux, but Linux + has **no per-call `scope` param** (that form is macOS-only). Linux gates + the windowless path on the persisted `capture_scope` config: opt in once + with `set_config capture_scope=desktop`, then send `x,y` with no + pid/window_id. Under the default `capture_scope=window` a windowless + action is rejected with a structured `desktop_scope_disabled` error. + +## AT-SPI needs the session bus (headless / containers / `runuser`) + +AT-SPI — the accessibility tree behind `get_window_state`, element-indexed +clicks, and focus-free `type_text` — lives **entirely on the desktop +session's D-Bus**. cua-driver reaches it via `DBUS_SESSION_BUS_ADDRESS`. When +the daemon is started *inside* a normal desktop login that variable is already +exported and everything works. When it is started **outside** the session — +a container entrypoint, a headless box, `runuser`/`su` into the desktop user, +a systemd *system* unit, or a VNC session running its own ad-hoc bus — the +variable is unset, the AT-SPI registry walk comes back empty, and +`get_window_state` reports **every** window as having no elements. + +cua-driver now **auto-discovers the session bus at startup** (mirroring the +`XAUTHORITY` recovery): if `DBUS_SESSION_BUS_ADDRESS` is unset it adopts +`/run/user//bus`, or reads the address out of a running desktop-session +process's `/proc//environ` (`xfce4-session`, `gnome-session`, …). So the +common headless cases now "just work". The two things that still must be true: + +1. **An accessibility bus must be running** in that session, and + **`toolkit-accessibility` must be on** — cua-driver advertises a screen + reader at startup to flip it, but a session with no a11y bus at all + (`/usr/libexec/at-spi-bus-launcher`) can't expose a tree. `cua-driver + doctor` now probes `org.a11y.Bus` for real (not just "is there a bus?") + and tells you which of the two is missing. +2. The daemon must run **as the desktop user** (so it can read that user's + session-process environ and the `/run/user//bus` socket). Running the + daemon as root against a user session is the Linux analogue of the Windows + "Session 0" isolation problem. + +An empty AT-SPI walk is now surfaced honestly: `get_window_state` sets +`degraded: true` + a `degraded_reason` (instead of a bare `elements: []`) so a +caller can tell "this window genuinely has no controls" apart from "the a11y +bridge isn't up / the daemon isn't on the session bus". + +## The validated modality matrix (X11 / XFCE) + +Each input rung, and whether the **driver itself** can confirm it (vs. only +the caller agent confirming via screenshot — the same honesty line macOS and +Windows draw): + +| Modality | `delivery_mode` | Path reported | Driver-verifiable? | +|---|---|---|---| +| Element click (`element_index`) | `background` | `x11_atspi` (AT-SPI `do_action`) | ✅ a11y action | +| **element px action (x,y)** | `background` | `x11_atspi` (AT-SPI `do_action`-at-point) for AX apps; else MPX `x11_pixel` | ✅ when AT-SPI-at-point lands; else best-effort | +| Pixel (px) click, escalated | `foreground` | `x11_pixel_fg` (EWMH activate → inject → restore) | ❌ confirm via screenshot | +| `type_text` into editable | `background` | `ax` (AT-SPI `insertText`) | ✅ `verified:true` | +| `type_text`, non-editable focus | `background`/`foreground` | `key_events` / `key_events_fg` | ❌ confirm via screenshot | + +**A background element px action does land on X11** — for an AX-exposing app it +takes the focus-free AT-SPI `do_action`-at-point path (`x11_atspi`), exactly +like the macOS/Windows background pixel click. It falls to the MPX +virtual-pointer path (`x11_pixel`) only for non-AX surfaces, **and that path +needs a real Xorg + `/dev/uinput`** — under Xvnc / minimal containers without +uinput, escalate to `delivery_mode:"foreground"`. (`type_text` in the +`background` rung is focus-dependent for non-editable widgets; that's the one +genuine background limitation, and `foreground` is the documented escalation.) + ## Wayland (opt-in — still preview) Native Wayland support is behind an opt-in flag and not yet at the same @@ -51,20 +195,47 @@ maturity as X11: - Screenshots work via `grim` / wlr-screencopy. **Video recording is not yet available on Wayland** — the recorder is `x11grab`, which is X11-only. +- **`delivery_mode` on Wayland is constrained by the protocol**, honestly: + input goes through libei + xdg-desktop-portal, which injects to the + **compositor's input focus** — there is no per-window background targeting + like X11/macOS/Windows. So `background` can't aim at a specific non-focused + window, and `bring_to_front` has no standalone external activate (the + compositor bundles activation into the virtual-pointer/click path) — it + returns `bring_to_front_wayland_bundled` and points you at + `delivery_mode:"foreground"` on the input call instead. The delivery + contract also defines a structured `background_unavailable` error for the + no-libei-backend case (built without `portal-libei` or a denied portal + session); when input has no actuator the tools surface an error rather + than silently succeeding. +- **Wayland escalation is `foreground`, NOT `px`.** This is the + one place the cross-platform escalation hint flips. Everywhere else + (macOS, X11, most Windows surfaces) a background pixel click can + land on an unfocused window, so `escalation.recommended` is `px`. + On Wayland an unfocused window **cannot** be pixel-targeted in the + background (libei injects only to compositor focus → `background_unavailable`), + so the hint on action responses and on a `degraded` `get_window_state` is + `foreground`: re-call the action with `delivery_mode:"foreground"`, don't + expect an element px action off the screenshot to land in the background. ## Quick triage If a tool call surprises you on Linux: 1. `cua-driver doctor` — reports the display server (X11 / Wayland), - AT-SPI bus reachability, and `ffmpeg` availability (for recording). + **whether `org.a11y.Bus` actually answers on the session bus** (not just + "is there a bus"), the discovered `DBUS_SESSION_BUS_ADDRESS`, and + `ffmpeg` availability (for recording). 2. Check `XDG_SESSION_TYPE` — `x11` is fully supported; `wayland` needs `CUA_DRIVER_RS_ENABLE_WAYLAND=1` for native input (preview), else XWayland. -3. Empty / partial AT-SPI tree — make sure the a11y bus is enabled - (`gsettings set org.gnome.desktop.interface toolkit-accessibility - true`). GTK4 / Qt6 also populate lazily, so re-snapshot after an - interaction. +3. **Empty AT-SPI tree** (`get_window_state` returns `degraded:true`) — in + order of likelihood: (a) the daemon isn't on the desktop session bus + (headless / container / `runuser` / root-against-user-session — see + *AT-SPI needs the session bus* above; doctor will say + `DBUS_SESSION_BUS_ADDRESS unset`); (b) the a11y bridge is off + (`gsettings set org.gnome.desktop.interface toolkit-accessibility true`); + (c) GTK4 / Qt6 / Chromium populate lazily — re-snapshot after an + interaction or an AX-enable settle. ## Forbidden vectors diff --git a/packages/cua-driver/rust/Skills/cua-driver/MACOS.md b/packages/cua-driver/rust/Skills/cua-driver/MACOS.md index 6048621abc8..184c105fac0 100644 --- a/packages/cua-driver/rust/Skills/cua-driver/MACOS.md +++ b/packages/cua-driver/rust/Skills/cua-driver/MACOS.md @@ -148,12 +148,15 @@ failure mode and it steals focus every time. When a cua-driver call surprises you, diagnose cua-driver first: -- **Tiny screenshot / empty `tree_markdown`?** Check - `cua-driver get_config` → `capture_mode`. Default `"som"` returns - both the AX tree and screenshot. `"vision"` omits the AX tree - (PNG only), `"ax"` omits the PNG. If a snapshot lacks a tree, - `capture_mode` is almost certainly `"vision"` — either reason - purely from the PNG or flip to `"som"` / `"ax"` via `set_config`. +- **Empty `tree_markdown`?** `get_window_state` returns **both** the + AX tree and a screenshot by default — there's nothing to configure and + no capture mode to pick. An empty tree means the surface isn't AX (a + non-AX surface: Electron/Chromium/canvas), and the response carries + `degraded: true` — so act by **`px`** off the screenshot that's + already in the same response. `capture_mode` is **deprecated and + ignored** (still accepted so old callers don't error, but it has no + effect — tree + screenshot come back regardless); don't reach for + `get_config` to "switch modes," there is no mode to switch. - **`has_screenshot: false`?** The window capture failed (transient race against a close, or the window has no backing store yet). Re-snapshot; if persistent, pick a different `window_id` via @@ -172,32 +175,48 @@ genuinely needs frontmost state, fall through to the activate fallback. Always name the focus steal in your response ("I'll briefly bring Chrome to the front because …"). -### Verifying actions: prefer `som` over `ax` - -When verifying that an action **landed** (a click registered, a text -input took, a menu pick changed selection), default to `get_window_state({ -capture_mode: "som"})`. `"ax"` is cheaper (no PNG in the response), but -the AX tree **lies** for two common surface types: - -- **Canvas-backed editors** — Monaco (VSCode, Cursor), xterm, Figma, - WebGL apps. The AX tree shows the surrounding chrome but reports - nothing for the canvas content. A snapshot can look unchanged after a - successful text edit. -- **Catalyst / iOS-on-Mac text views** — see "Known text-input limits" - above. The AX `AXValue` can lag the rendered pixels, or report the - placeholder string when the field is actually populated. - -`"som"` returns both the AX tree AND a PNG screenshot in a single call. -Pay the extra tokens; the screenshot is the ground truth for both of -the cases above, and you avoid the "type → AX-check succeeds → think -the action landed → discover the lie three calls later" loop. +### Verifying actions: cross-check the tree against the pixels you already have + +There is no `ax`/`vision` capture toggle. **Every `get_window_state` +returns both the AX tree and a screenshot** (default), so verifying that +an action **landed** never means "go grab a screenshot" — it means +cross-check the tree diff against the pixels you already have in the same +response, and only switch *dispatch rung* on a real signal: + +1. **Re-snapshot and read the tree diff** — a changed `AXValue`, a new + element, a collapsed menu, a disabled button. If the tree shows the + change, you're done. When you only need the tree diff and don't need + fresh pixels, pass `include_screenshot:false` to skip the grab — a + **perf** knob, not a mode flip. +2. **Trust the screenshot and do an element px action** when the tree + **lies** — the action response carried `effect:"suspected_noop"`, the + re-snapshot came back `degraded` (empty tree), or the tree looks + unchanged/unreadable / disagrees with the pixels on a surface where + it's known to lie: + - **Canvas-backed editors** — Monaco (VSCode, Cursor), xterm, Figma, + WebGL. The AX tree shows the chrome but nothing for the canvas + content; a snapshot's tree can look unchanged after a successful + edit while the pixels show it landed. + - **Catalyst / iOS-on-Mac text views** — see "Known text-input + limits" above. `AXValue` can lag the rendered pixels or report the + placeholder while the field is actually populated. + +On these surfaces you read the result off the screenshot already in the +response, then address the target by `x,y` — an **element px action**. +`px` is your **conscious switch to the pixel addressing path**, not a +different capture: the screenshot was always there, you just change *how +you address* the target. The point is to catch the "type → AX-check +succeeds → believe the lie → find out three calls later" trap on exactly +the surfaces that warrant it. Rule of thumb: -- **`som`** — verifying an action landed (default). -- **`ax`** — cheap element lookup before a click (you don't need the - pixels, you need the `[N]` index). -- **`vision`** — pure visual inspection (e.g. asking the model to read - a chart). Skip the AX walk entirely. +- **element ax action** (default) — the element lookup before a click + AND the first verify after it; you address by `[N]` `element_index` + and read the tree diff. +- **element px action** — when the tree is unreadable / `suspected_noop` + / `degraded` / disagrees with the pixels, or for pure visual + inspection (reading a chart). You address by `x,y` off the screenshot + that's already in the snapshot response. ### Self-check pattern @@ -285,6 +304,37 @@ is suppressed. Full mechanics in `Sources/CuaDriverCore/Input/MouseInput.swift` (`clickViaAuthSignedPost`) and the companion `FocusWithoutRaise.swift`. +### `delivery_mode` on the pointer family (macOS) + +`click`, `double_click`, `right_click`, `drag`, and `scroll` accept +`delivery_mode` (`"background"` default / `"foreground"`) — matching the +breadth Windows and Linux already exposed (`type_text` / `press_key` / +`hotkey` carry it too). `"background"` is the SkyLight per-pid path above: +no raise, no focus steal. `"foreground"` briefly fronts the owning app, +acts, then restores the prior frontmost — the explicit last resort for a +surface that only accepts events while frontmost (the canvas/viewport/game +case below). Element-indexed (AX) actions are inherently background and +hold the no-foreground contract without the flag. + +macOS-specific residuals worth knowing (the rest of the capture/dispatch/ +addressing params are a shared cross-platform contract — see `SKILL.md` → +*Cross-platform parameter contract*): + +- **`check_permissions.prompt` is macOS-only.** It raises the TCC + Accessibility / Screen-Recording dialogs; pass `{"prompt":false}` for a + read-only status check. There is no Windows/Linux equivalent — TCC is a + macOS construct, so the param is intentionally absent from the shared + contract. +- **`session` always worked on macOS;** the cross-platform change is that + Windows/Linux stopped *rejecting* it. No macOS-side change to how you + pass it. +- **`scope`** (`window` default / `desktop`) is a **macOS-specific per-call + param** on the pointer tools — pass `scope:"desktop"` with `x,y` and no + pid/window_id for a screen-absolute click. Windows and Linux support the + same windowless capability but gate it on the persisted `capture_scope` + config instead (`set_config capture_scope=desktop`), so they have no + per-call `scope` param. Unifying the knob is a tracked follow-up. + ### Canvases, viewports, games (Blender, Unity, GHOST, Qt, wxWidgets) Apps whose main surface is an OpenGL / Metal / Qt / wxWidgets @@ -322,35 +372,50 @@ There is no backgrounded path that reaches these apps today. ### Known text-input limits (Catalyst + Electron) -`type_text` returns `✅ Inserted N char(s)` against **Catalyst** apps -(WhatsApp, Reminders, Notes-via-Catalyst, anything in `/Applications` -that's actually `iOSAppOnMac.app`) and **Electron** apps (VSCode's -Monaco editor, Slack composer, Discord, Linear) but the characters -don't reach the rendered text view. The AX path (`AXSetAttribute` on -`kAXSelectedText`) reports success because the API call succeeds on -the AX shim, but the UIKit/Chromium text view that owns the actual -input doesn't observe the AX value change. The CGEvent fallback path -runs, but the renderer's focused-process check drops the keystrokes -when the app isn't WindowServer-frontmost. - -Workaround — clipboard + windowed `Cmd+V`: - -1. Put the string on the macOS clipboard externally (e.g. shell - `pbcopy`, or `NSPasteboard.general.setString:forType:` from your - own helper). -2. Dispatch `hotkey({pid, window_id, keys: ["cmd", "v"]})`. **The - `window_id` matters**: the NSMenu key-equivalent path briefly - foregrounds the app via `SLPSSetFrontProcessWithOptions(kCPSNoWindows)` - for ~1 ms — enough for `paste:` to deliver — then restores prior - frontmost. Without `window_id`, the auth-message envelope path - runs and hits the same drop as `type_text`. -3. Verify with a `get_window_state({capture_mode: "som"})` snapshot. - Both the AX value and the rendered pixels should show the pasted - text. (See the verify-with-som pattern below.) - -This costs one round-trip of clipboard pollution. Restore the user's -prior clipboard if your workflow cares — store via `pbpaste` before, -write back via `pbcopy` after. +On **Catalyst** apps (WhatsApp, Reminders, Notes-via-Catalyst, +anything in `/Applications` that's actually `iOSAppOnMac.app`) and +**Electron** apps (VS Code's Monaco editor, Slack composer, Discord, +Linear), an AX `type_text` can't reach the rendered text view: the +`AXSetAttribute(kAXSelectedText)` write succeeds on the AX shim, but +the UIKit/Chromium view that owns the input never observes it — and on +Electron the shim *echoes the value straight back through `AXValue`*, +so a naive read-back "confirms" a value that isn't really there. + +The driver **detects Electron and refuses to trust that echo**: an +AX-path `type_text` on an Electron app returns `effect:"unverifiable"` ++ `escalation:{recommended:"px"}`, **never** a false `verified:true`. +(On Catalyst the AX value reads back unreadable, so it reports +unverified too.) Bottom line: on these surfaces **do not trust the AX +confirm — the screenshot in the same response is the only truth.** + +Fix — **one call**: `type_text({pid, window_id, x, y, text})`. Passing +`x,y` (no `element_index`) is the **element px action** form of +`type_text` — the tool pixel-clicks at `(x,y)` to give the Chromium / +UIKit renderer the real keyboard focus the AX layer can't, then types +into the now-focused field. Read `x,y` straight off the screenshot in +the `get_window_state` response (same convention as `click`). This is +the one-call replacement for the old two-step "pixel-click then +`type_text`" — and you do **not** reach for a clipboard + `Cmd+V` +dance. + +0. **If the control is CLOSED, open it first.** A px focus-click won't + reliably *open and focus* a closed control (a search button, a + collapsed field) — it lands on whatever is already focused (e.g. + the message composer), so your text leaks there. **AX-press to + open/activate the control first** (AX actions work in the + background), then px-type into the now-open field. +1. **`type_text({pid, window_id, x, y, text})`** — focus + type in a + single call. Re-snapshot and read the text off the screenshot to + confirm; the AX value can still lag on Catalyst/Electron. +2. Only if the keystrokes *still* drop (a focus-polling app), escalate + that one `type_text` with `delivery_mode:"foreground"`. + +The `x,y` (px) form is **mutually exclusive** with `element_index` +(ax) — pass one or the other, not both. Why not `Cmd+V` / `hotkey`: a +keyboard combo does **not** focus a text field, and `hotkey` / +`press_key` no longer raise the window on their own (raising is gated +on `delivery_mode:"foreground"`, like every other tool). The reliable +move is the px form of `type_text` — focus and type in one call. ## Navigating native menu bars (AXMenuBar) diff --git a/packages/cua-driver/rust/Skills/cua-driver/README.md b/packages/cua-driver/rust/Skills/cua-driver/README.md index b33afc289ec..5ce2345def9 100644 --- a/packages/cua-driver/rust/Skills/cua-driver/README.md +++ b/packages/cua-driver/rust/Skills/cua-driver/README.md @@ -185,11 +185,12 @@ Use MCP for this Claude Code vision/computer-use-style path. CLI screenshots sti reused across turns, or across different windows of the same app. Call `get_window_state({pid, window_id})` first in the same turn, with the same window_id you're about to act against. -- Empty `tree_markdown` → `capture_mode` is set to `vision`, which - skips the AX walk by design. Flip back to the default `som` - (`cua-driver config set capture_mode som`) to get the tree. - Tiny screenshot → likely a stale window capture. See "Behavior - matrix" in SKILL.md for the full mode table. +- Empty `tree_markdown` (with `degraded:true`) → this is a non-AX + surface (canvas / WebGL / Electron web content), not a mode problem + — `get_window_state` always walks the tree now. Act by pixel off the + screenshot already in the same response (an element px action). + Tiny screenshot → likely a stale window capture. See the perception + + element ax/px action notes in SKILL.md. - System-alert beep when pressing Return on a minimized Chrome omnibox → the keyboard-commit-on-minimized limitation. Use `set_value` on the field instead, or AX-click a Go/Submit button. diff --git a/packages/cua-driver/rust/Skills/cua-driver/SKILL.md b/packages/cua-driver/rust/Skills/cua-driver/SKILL.md index 2932a69982f..23fa3f0f207 100644 --- a/packages/cua-driver/rust/Skills/cua-driver/SKILL.md +++ b/packages/cua-driver/rust/Skills/cua-driver/SKILL.md @@ -190,30 +190,164 @@ returns. ## Behavior matrix -Two orthogonal axes shape what the agent can do. +### Perception is mode-agnostic — `get_window_state` returns BOTH + +`get_window_state(pid, window_id)` **returns both the accessibility +tree AND a screenshot by default.** There is no capture mode to pick +and nothing to configure — you ground on the tree and the screenshot +together, and you cross-check one against the other. This matters +because the tree **lies** on some surfaces: + +- **Electron** echo-confirms a `set_value` / `type_text` against the AX + shim while the rendered text view never changed. +- **Catalyst** (iOSAppOnMac) exposes null / placeholder `AXValue`s. +- **Virtualized / off-viewport list rows** report bogus frames (an + `h:1` height, an off-screen origin) for rows that aren't actually + laid out. + +A grounding screenshot is present by default, so when the tree looks +wrong you look at the pixels **in the same response** — no second +capture, no mode flip. + +> **Perf opt-out — `include_screenshot`.** `include_screenshot` +> (boolean, default `true`) is the one knob, and it is a **perf** knob, +> not a modality choice. Default returns both (grounding-first). Pass +> `include_screenshot:false` to skip the screen grab and get the tree +> only — the cheap path when you're just **re-indexing before an +> element ax action** and don't need to re-ground on pixels. The +> `ax`/`px` decision still lives at action time, not here. + +> **`capture_mode` is DEPRECATED and ignored.** It is still *accepted* +> on `get_window_state` so old callers don't error, but it has **no +> effect** — both the tree and the screenshot come back regardless of +> what you pass (`ax`, `vision`, `som`, anything). There is no +> `ax`/`vision`/`som` capture choice anymore. Drop the word "vision" +> for perception entirely. (The tool named `screenshot` is separate — +> raw PNG, no AX walk — and unrelated.) + +### The modality is chosen at ACTION time — `ax` vs `px` + +You don't pick a capture mode; you pick **how you address the target** +on the action call, and that one choice selects the rung: + +- **element ax action** — pass `element_index` / `element_token`. + Dispatches through the **accessibility rung**: AXPress (macOS) / UIA + Invoke (Windows) / AT-SPI `doAction` (Linux). Backgroundable, + z-order-independent, and the only **driver-verifiable** rung. +- **element px action** — pass `x`, `y`. Dispatches through the **pixel + rung**, reading the coordinate straight off the screenshot that's + already in the `get_window_state` response. Best-effort; the caller + confirms the effect. + +`ax`↔`element_index`, `px`↔pixel `x,y`. We retired the word "vision" +for the *dispatch* path — it conflated perception with dispatch. +Perception is always both; dispatch is `ax` or `px`. + +**The keyboard family has both forms too.** `type_text`, `press_key`, +and `hotkey` take `element_index` (ax) **or** `x,y` (px) — mutually +exclusive, same as the pointer tools. The px form **pixel-clicks at +`(x,y)` to establish real renderer focus, then delivers the +keystroke(s)** to the now-focused element (it reuses `click`'s +coordinate translation + `delivery_mode`). That gives e.g. +`type_text({pid, window_id, x, y, text})` as a one-call focus-then-type +for Chromium/Electron inputs the AX path can't reach, and +`hotkey({pid, x, y, keys:["cmd","v"]})` to paste into a specific field. + +**Typing default (the ladder).** Call `type_text` directly with +`element_index` (ax) — it targets the field, no pre-click. On +Electron/Catalyst the AX layer echoes the write without rendering it, +so the driver returns `effect:"unverifiable"` + `escalation:"px"` +there (never a false `verified:true`) — follow it, and cross-check the +screenshot in the response (the only ground truth). Escalate to the px +form — `type_text({pid, window_id, x, y, text})` — which pixel-clicks +to focus, then types. **If the target control is closed** (a search +button, a collapsed field), AX-press to open it first (AX actions work +in the background): a px focus-click won't reliably open *and* focus a +closed control, so the text leaks into whatever's already focused. +Escalate to `delivery_mode:"foreground"` only if it still drops. + +**`set_value` stays AX-only by design** — it's for **non-text** +controls (dropdown / `AXPopUpButton`, checkbox, slider, stepper). Its +pixel counterpart is a `click`/`drag` on the control, not a "set value +at a pixel." So: text → `type_text` (ax+px); non-text control values → +`set_value`; pixel-manipulate a control → `click`/`drag`. + +**Action responses carry an effect/escalation verdict** + +Every action response keeps `verified` (did the driver read back a +post-condition?) and adds two machine-readable fields so you know +whether — and where — to climb the ladder: + +- `effect`: one of + - `"confirmed"` — the driver read back the effect (`ax` rung only). + - `"unverifiable"` — dispatched, but the driver has no handle to + read back (every `px`/CGEvent path; foreground rung). **You** + confirm it off the screenshot — it is not a failure. + - `"suspected_noop"` — the `ax` action **likely did nothing** (the + element didn't actually advertise the action, or you hit a passive + label). This is the explicit **"cross to `px`"** trigger. +- `escalation`: `{recommended, reason}` when the driver thinks you + should change rung — + - `"px"` — the element isn't really actionable in `ax`; do an + **element px action** off the screenshot you already have. + - `"foreground"` — a background insert/click was *dropped* on + delivery; re-call the same action with `delivery_mode:"foreground"`. + +`get_window_state` itself, when the AX tree comes back empty (a non-AX +surface like Electron/Chromium/canvas), returns `degraded: true` +**plus the same `escalation` hint** — normally pointing at `px` (you +still have the screenshot from the same call to click off). + +**Platform nuance for `escalation`.** On **Wayland** an unfocused +window cannot be pixel-targeted in the background (libei → +`background_unavailable`), so there the recommendation is +**`foreground`, not `px`**. macOS, X11, and most Windows surfaces +*can* pixel-target in the background, so they recommend `px`. See +`LINUX.md` / `WINDOWS.md`. + +## The verify-then-escalate ladder (algorithm) + +Every snapshot already hands you both the tree and the screenshot, so +verifying never means "go take a screenshot" — it means cross-check +the tree against the pixels you already have, and only change +*dispatch rung* on a real signal. Walk the rungs: -**capture_mode → addressing mode** - -| `capture_mode` | `get_window_state` returns | Use for actions | -|---|---|---| -| **`som`** (default) | tree + screenshot | `element_index` preferred; pixel fallback | -| **`ax`** | tree only (no PNG) | `element_index` only | -| **`vision`** | PNG only (no tree) | pixel only — see `SCREENSHOT.md` | - -`vision` was renamed from `screenshot` — the old name still decodes -as a deprecated alias, so an on-disk `"capture_mode": "screenshot"` -keeps working. Default is `som` so element_index clicks work the -first time a user calls `get_window_state`; the other modes are -opt-in when the caller specifically doesn't want one half of the -work. Note the tool named `screenshot` is separate (raw PNG, no AX -walk) and unrelated to the capture mode. - -When a snapshot looks wrong (tiny screenshot / empty tree), check -`cua-driver get_config` for `capture_mode` before anything else. +``` +# Rung 1 — element ax action, backgrounded (the cheap default) +get_window_state(pid, window_id) # tree + screenshot, both, always +resp = click(pid, window_id, element_index) # or type_text / set_value / press_key +get_window_state(pid, window_id) # re-snapshot — did the tree change? + +if resp.effect == "confirmed" and tree changed: + done # driver-verified + +# escalate only on a real signal +if resp.effect == "suspected_noop" + or resp.escalation.recommended == "px" + or get_window_state.degraded # empty tree → non-AX surface + or the tree looks wrong vs the screenshot: # e.g. an h:1 / off-viewport row + + # Rung 2 — element px action off the SAME screenshot + pick the target pixel from the screenshot already in the response + click(pid, x, y) # background pixel — still no foreground + get_window_state(pid, window_id) # re-snapshot, eyeball the result + if it landed: done + +# Rung 3 — background delivery was dropped (insert/click never arrived) +if resp.escalation.recommended == "foreground" + or the px action still did nothing: + re-call the same action with delivery_mode:"foreground" + # on Wayland this is the ONLY escalation — px-bg can't target an + # unfocused window there; see LINUX.md + verify again +``` -Pure-vision mode has its own caveats — Claude Code's vision pipeline -downsamples dense text aggressively, so pixel grounding takes -multiple correction cycles on text-heavy UIs. +The two ideas to hold onto: (1) the AX tree **lies** on canvas / web / +Catalyst / virtualized surfaces, so an unchanged-or-bogus tree plus +`suspected_noop`/`degraded` — or a tree that simply disagrees with the +screenshot — is your cue to do an **element px action** off the +screenshot you already have; (2) `px` is a *conscious* switch to the +pixel addressing path, not a different capture. **Window state → what works** @@ -290,21 +424,30 @@ single-window case you can skip `list_windows` entirely and read the Call `get_window_state({pid, window_id})` with the `window_id` from `launch_app`'s `windows` array (or a fresh `list_windows({pid})` if -you're interacting with a long-lived process). The default `som` -capture_mode returns **both the tree and screenshot**, so the -canonical loop works immediately without any config change. +you're interacting with a long-lived process). It returns **the tree +and the screenshot together** by default, so you can both dispatch by +`element_index` and ground on pixels from one call — no config change, +no mode flip. When you're just re-indexing before an element ax action +and don't need fresh pixels, pass `include_screenshot:false` to skip +the grab (a perf knob, not a modality choice). -In `som` mode (the default) the response carries: +The response carries: - `tree_markdown` — every actionable element tagged `[N]`. That `N` is the `element_index`. The tree can be very large (Finder is ~1600 elements, ~190 KB); when it exceeds token limits the MCP harness saves it to a file and returns the path. Use `Bash` + `jq -r '.tree_markdown'` + `grep` to pull the section you need. -- `screenshot_file_path` — absolute path to the saved screenshot when - `screenshot_out_file` was passed. Absent otherwise. +- `effect` / `escalation` / `degraded` — the verify-then-escalate + signals (see the behavior matrix above): `degraded: true` means the + tree came back empty (non-AX surface), so you act by **`px`** off the + screenshot in the same response. +- `screenshot_file_path` — present when the screenshot was written to + disk instead of inlined (you passed `screenshot_out_file`, or the + context-saving CLI path); otherwise the frame is inlined. - `screenshot_width` / `_height` / `_scale_factor` — dimensions of - the captured image. Present whenever a screenshot was taken. + the captured image. Present whenever a screenshot was taken (i.e. + unless you passed `include_screenshot:false`). **Getting the screenshot as a file (CLI and context-constrained agents):** @@ -312,7 +455,7 @@ In `som` mode (the default) the response carries: # write to file — stdout stays readable (AX/UIA tree / summary only, no base64) cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.jpg"}' -# CLI --screenshot-out-file flag is equivalent and works for all capture modes +# CLI --screenshot-out-file flag is equivalent cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.jpg ``` @@ -323,28 +466,30 @@ content block is omitted from the response when this param is set — the model receives only the tree and `screenshot_file_path`, then reads the image from disk. -**Reason over both the tree AND the screenshot — they're -complementary, not redundant.** In `som` mode every turn's -`get_window_state` gives you both halves and you should pull signal -from each: +**The tree and the screenshot are complementary, not redundant — and +they come from the *same* call.** Each half carries signal the other +can't, which is exactly why you cross-check them: - The **tree** tells you *what's clickable* — roles, labels, `element_index` handles, advertised actions, parent-child - structure. This is the ground truth for dispatching. -- The **screenshot** tells you *which one* — the tree often has - many buttons with similar or empty labels ("Delete", "OK", - anonymous UUID-labeled buttons, repeated static-text), and visual - context disambiguates. Captions, colors, layout relationships - visible in pixels often don't show up in the tree at all - (especially in Chromium / Electron / web content). - -Canonical pattern: look at the screenshot to decide "the blue -Subscribe button on the top-right of the video card", then walk the -tree to find the matching element and dispatch by its -`element_index`. Don't try to do it from just the tree — you'll -pick the wrong element when labels repeat. Don't try to do it from -just the screenshot — you lose the reliable accessibility-action -path and the safe backgrounded-dispatch. + structure. This is the ground truth for an **element ax action**. +- The **screenshot** tells you *which one* — the tree often has many + buttons with similar or empty labels ("Delete", "OK", anonymous + UUID-labeled buttons, repeated static-text), and visual context + disambiguates. Captions, colors, layout relationships visible in + pixels often don't show up in the tree at all (especially in + Chromium / Electron / web content) — and the screenshot is where you + catch the tree *lying* (an `h:1`/off-viewport row, a Catalyst null + value). + +Default to dispatching by `element_index` (the **element ax action**) — +it's the verifiable, backgroundable rung. Do an **element px action** +(`x,y` off the same screenshot) when the tree can't disambiguate +(repeated/empty labels), when it's empty (`degraded` — non-AX +surface), when an action came back `suspected_noop`, or when the tree +disagrees with the pixels. You never re-capture to switch — the +screenshot is already there; you just change *how you address* the +target. Reach for pixel coordinates only when the target is a canvas / video / WebGL / custom-drawn surface that isn't in the tree @@ -370,12 +515,12 @@ against a specific window. | Left click | `click({pid, window_id, element_index})` | default `action: "press"`. Pixel form: `click({pid, x, y})` (window_id optional) — `modifier: ["cmd"\|"ctrl"]` | | Double-click / open | `double_click({pid, window_id, element_index})` | Default action when the element advertises one (Open on Finder items / openable rows), else stamped pixel double-click at the element's center | | Right click / context menu | `right_click({pid, window_id, element_index})` or `click({pid, window_id, element_index, action: "show_menu"})` | Chromium web-content coerces pixel right-click to left on macOS — see `WEB_APPS.md` | -| Type at cursor | `type_text({pid, text, window_id, element_index})` | focuses element first, then writes via the platform's text-set primitive | -| Set whole field value | `set_value({pid, window_id, element_index, value})` | sliders, steppers, text fields; **use for keyboard-commit workarounds on minimized windows** | +| Type at cursor | `type_text({pid, text, window_id, element_index})` (ax) or `type_text({pid, text, window_id, x, y})` (px) | ax focuses the element then writes via the platform's text-set primitive; **px** pixel-clicks `(x,y)` to focus the renderer, then types — the one-call fix for Chromium/Electron inputs the AX path can't reach | +| Set whole non-text control value | `set_value({pid, window_id, element_index, value})` | **AX-only by design** — dropdown/`AXPopUpButton`, checkbox, slider, stepper; **also the keyboard-commit workaround on minimized windows.** For text use `type_text`; to pixel-manipulate a control use `click`/`drag` | | Scroll | `scroll({pid, direction, amount, by, window_id, element_index})` | synthesizes per-pid PageUp/PageDown/arrows | -| Focus + send key | `press_key({pid, key, window_id, element_index, modifiers})` | element_index sets focus, then posts key | +| Focus + send key | `press_key({pid, key, window_id, element_index, modifiers})` (ax) or `press_key({pid, key, x, y})` (px) | ax `element_index` sets focus then posts the key; **px** pixel-clicks `(x,y)` to focus, then sends the key | | Send key to pid | `press_key({pid, key, modifiers})` | no focus change; key goes to pid's current focus | -| Modifier combo | `hotkey({pid, keys})` | e.g. `["cmd","c"]` / `["ctrl","c"]`; posted per-pid, not HID tap | +| Modifier combo | `hotkey({pid, keys})` (no focus) or `hotkey({pid, x, y, keys})` (px) | e.g. `["cmd","c"]` / `["ctrl","c"]`; posted per-pid, not HID tap. **px** pixel-clicks `(x,y)` to focus a field first, e.g. `["cmd","v"]` to paste into it | **All keyboard/text primitives require `pid`.** There is no frontmost-routed variant — every key goes to the named target via @@ -387,6 +532,62 @@ occluded / off-desktop windows, no focus steal, stable across rebuilds, labels tell you what you're clicking. Reach for pixel coordinates only when the accessibility tree can't. +## Cross-platform parameter contract + +The capture, dispatch, and addressing params — `session`, +`delivery_mode`, `capture_mode` (deprecated/ignored — see the behavior +matrix; still in the schema only so old callers don't error), `scope`, +`modifier`, `button`, `element_index`, `element_token` — are a **shared +schema contract**: identical *shape* (`type`/`enum`/`items`) on macOS, +Windows, and Linux. +They compose from canonical fragments in +`cua-driver-core::tool_schema` (+ `capture_mode`), and a CI gate +(`schema_consistency_test`) runs every tool's live `tools/list` through a +structural checker on each platform, so the three surfaces can't +silently drift. *Contributor note:* when you add or edit one of these +shared params on a tool, pull from the fragment — don't re-hand-write the +JSON, or the gate fails. (Descriptions may legitimately vary per tool; +the gate compares shape, not prose.) + +Two consequences for callers: + +- **`session` is accepted on every action and cursor tool, on all three + platforms.** It's cursor-wired where the platform glides a cursor and + schema-accepted everywhere else — so the same `session` you pass on + macOS is no longer *rejected* by Windows/Linux, which previously + refused unknown keys via `additionalProperties:false`. +- **`delivery_mode` (`"background"` default / `"foreground"`) is on the + whole input family** — `click`, `double_click`, `right_click`, `drag`, + `scroll`, `type_text`, `press_key`, `hotkey` — uniformly. The + `foreground` rung briefly fronts the target, acts, then restores the + prior frontmost: the explicit last resort when a background attempt + didn't land. **`foreground` is a reaction, never a prediction.** Always + fire the `background` default first and let the driver tell you it + can't (a `background_unavailable` error or `escalation.recommended == + "foreground"`) — or observe a verified no-op — *before* you escalate. + Do **not** reason "it's a GTK/Chromium/Electron app, so background will + drop, so I'll front up-front": the toolkit lists in the tool schemas + are the *driver's* internal detectors, not a checklist for you to front + on a guess. (Concretely: GIMP's GTK toolbox accepts background pixel + clicks fine — a preemptive foreground click there just steals the + user's focus for nothing.) What each platform's *background* rung can + actually carry differs (e.g. a Windows background click can't carry + `modifier` state — see `WINDOWS.md`); the schema is uniform, the + residual limits are per-OS. + +**Required-set contract.** `click` requires nothing (`required:[]`), +`scroll` requires `["direction"]`, `zoom` requires +`["window_id","x1","y1","x2","y2"]` — same on every platform. `pid` is +**conditionally** required (needed unless a windowless desktop-scope +call) and validated in code with a clear error, NOT pinned in the schema +— so omitting `pid` for a desktop-scope action is no longer +schema-rejected. + +Genuinely platform-specific params stay OUT of the shared contract by +design (launch-app identifiers, the Windows-only `debug_window_info`, the +macOS-only `check_permissions.prompt`). The per-OS files list the +residuals that matter when you drive on that platform. + ## Pixel-coordinate clicks The pixel path (`click({pid, x, y})`) is for surfaces the @@ -482,10 +683,22 @@ the snapshot invariant. Check the tree diff: a changed value, a new element, a new window, or the disappearance of the thing you just clicked (menus collapse -after selection, buttons may become disabled, etc.). If nothing -changed, the action likely failed silently — **tell the user what -you attempted and what you observed**, don't paper over with "done" -language. Agents that skip this step report success on +after selection, buttons may become disabled, etc.). The re-snapshot +gives you both the tree and the screenshot, so you cross-check the tree +diff against the pixels in one call — and when you're only confirming a +tree change, `include_screenshot:false` skips the grab. + +Switch to an **element px action** only on a real signal: the action +response carried `effect:"suspected_noop"`, the re-snapshot came back +`degraded` (empty tree → non-AX surface), the tree looks +unchanged/unreadable or disagrees with the screenshot, or +`escalation.recommended` points you there (`px`). That's the +verify-then-escalate ladder in the behavior-matrix section. If the tree +is unchanged AND the screenshot confirms nothing moved, the action +likely failed silently — **tell the user what you attempted and what +you observed**, don't paper over with "done" language (and consider +`delivery_mode:"foreground"` when `escalation.recommended == +"foreground"`). Agents that skip this step report success on silently-dropped actions — the single most common failure mode. ## Recording trajectories @@ -518,9 +731,15 @@ respective companion files. ## Things to avoid - **Never** reuse an `element_index` across a re-snapshot of the same pid. -- **Never** translate screenshot pixels into a click — the screenshot - is for visual disambiguation, not coordinates. Use the - `element_index`. +- **Don't conflate the two addressing modes.** The tree gives you + `element_index` handles; the screenshot (same call) gives you the + pixel frame. An **element ax action** addresses by index, an + **element px action** by `x,y`. Default to `element_index` and only + do a px action on a real signal (`suspected_noop` / `degraded` / + repeated labels / tree-disagrees-with-pixels). Don't pass an + `element_index` you read off the screenshot, and don't pixel-click a + coordinate you computed from the tree's (possibly lying) frame + without checking it against the image. - **Prefer accessibility actions over pixels.** `click({pid, x, y})` works for canvas / WebView regions, but it lands blindly on raw coordinates. Exhaust accessibility paths (menu bars, cmd-k palettes, diff --git a/packages/cua-driver/rust/Skills/cua-driver/WEB_APPS.md b/packages/cua-driver/rust/Skills/cua-driver/WEB_APPS.md index fd19a68de8e..05aa52c28e1 100644 --- a/packages/cua-driver/rust/Skills/cua-driver/WEB_APPS.md +++ b/packages/cua-driver/rust/Skills/cua-driver/WEB_APPS.md @@ -483,3 +483,179 @@ If it silently drops (some web inputs don't implement synthesis — pure CGEvent keystrokes delivered to the pid, reaching any focused keyboard receiver. You can also click the field first to ensure focus before typing. + +### `escalation.recommended == "page"` — stop typing, switch tools + +On a rich-text contenteditable driven by React/Draft.js (X's tweet +composer is the canonical example), CGEvent keystroke synthesis is +fundamentally racy: a combined px-click-then-type call can drop or +reorder the first character, and `hotkey(["cmd","a"])` / `press_key +delete` frequently no-op against the editor's own selection model. +`type_text` knows this and says so — when its response includes +`escalation: {recommended: "page", reason: "...AX type_text on a +contenteditable is racy..."}`, that is your cue to **stop iterating +on the keyboard ladder immediately.** Do not retry with +`delivery_mode:"foreground"`, do not triple-click to select, do not +try `cmd+a`+`delete` first — every one of those is the wrong rung. +Go straight to the `page` tool instead: + +``` +page({pid, window_id, action: "execute_javascript", javascript: ` + (function() { + var el = document.querySelector('[data-testid="tweetTextarea_0"]'); + el.focus(); + document.execCommand('selectAll', false, null); + document.execCommand('delete', false, null); + document.execCommand('insertText', false, ${JSON.stringify(text)}); + return el.innerText; + })() +`}) +``` + +`document.execCommand('insertText', ...)` dispatches the real +`beforeinput`/`input` events Draft.js/React listen for, so it lands +in the framework's own state — not just a raw DOM mutation the +framework will stomp on next render. Swap the selector for whatever +`[data-testid]`/`role="textbox"` the target editor uses; find it via +`page(query_dom)` first if unfamiliar with the site. + +**Verification gotcha:** if the window is occluded or off-screen +when you re-snapshot, `get_window_state`'s screenshot can return a +stale cached frame while the DOM has actually already updated — +`list_windows` showing `is_on_screen: false` for the target is the +tell. Don't conclude the insert failed off a stale screenshot; verify +with a follow-up `page(execute_javascript)` reading `el.innerText` +directly, which reads the live DOM regardless of on-screen state. + +**`execCommand` is not durable on editors that reconcile their own +state.** Confirmed on X's composer: `insertText` lands in the DOM +immediately (verified via `el.innerText` right after), but a later +background re-render of the surrounding page (e.g. the timeline +polling in new posts) can silently wipe it back to empty — because +it was one synthetic event, not something the editor's own input +pipeline ever actually observed as a keystroke. If you see the text +vanish on a follow-up read with no error anywhere, that's this, +not a transient bug — reach for `insert_text` or `type_keystrokes` +(below) instead of retrying `execCommand` in a loop. + +### `insert_text` — a cheaper, more durable middle rung + +**macOS-only for now.** Windows and Linux don't yet implement the CDP +`Input.insertText`/`Input.dispatchKeyEvent` calls this and +`type_keystrokes` need — both return a clean "not implemented" error +there, not a silent no-op. Tracked in +[trycua/cua#2084](https://github.com/trycua/cua/issues/2084). Every +other `page` action (`get_text`, `query_dom`, `click_element`, +`execute_javascript`) works cross-platform. + +``` +page({pid, window_id, action: "insert_text", text: "…"}) +``` + +Inserts `text` at the focused element in a single CDP +`Input.insertText` call — no synthesized keydown/keyup sequence, so +it's one round-trip instead of three-per-character. It's still more +durable than `execCommand`, though, because Chrome's renderer treats +it like an IME composition commit rather than a JS-triggered DOM +mutation, and rich editors already have to handle real IME commits +correctly (CJK input, etc.) — so they're less likely to treat it as +untrusted and discard it. It does skip real key events, so an +editor's own keydown/keyup handlers (shortcut bindings, per-key +validation) never fire for it. + +Try this before `type_keystrokes` when `execCommand` got discarded. +Escalate to `type_keystrokes` only if `insert_text` also gets wiped, +or the editor specifically needs to observe real keystrokes. + +Same preconditions as `type_keystrokes` below (focus + live CDP +port on a non-default profile). + +### `type_keystrokes` — the slowest, most durable rung + +``` +page({pid, window_id, action: "type_keystrokes", text: "…"}) +``` + +Types `text` as a stream of real per-character keyboard events +(`Input.dispatchKeyEvent`: keyDown → char → keyUp per character) via +the Chrome DevTools Protocol, rather than one synthetic DOM write. +This is what makes it durable against React/Draft.js/Lexical/Slate +editors that reconcile their own internal model on every render — +they only trust input their own keydown/keypress/input pipeline +actually saw, and `execCommand`/`innerText =` never touches that +pipeline. Slower than `insert_text` (3 CDP round-trips per +character, with a small inter-character delay) — reach for it only +after `insert_text` didn't stick, or the editor needs real keydown +events. + +Preconditions (shared with `insert_text`): +- The target element must already have DOM focus — click it first + (`page(click_element)` or `execute_javascript` with `el.click()` / + `el.focus()`). +- The browser needs a live CDP port. Two ways to get one: + - **Dedicated automation profile** — `launch_app`'s + `cdp_debugging_port` appends `--remote-debugging-port=N`, but + Chrome refuses to open that port on what it considers its + default data directory — confirmed this refusal happens even + when you pass `--user-data-dir` explicitly pointing at that same + default path; only a genuinely different path satisfies the + check. So this route needs `additional_arguments: + ["--user-data-dir="]` too — a separate profile + without the user's existing logins/session. + - **The user's real, already-logged-in Chrome** — have them open + `chrome://inspect/#remote-debugging` and check "Allow remote + debugging". Confirmed this opens a CDP port on the *default* + profile without relaunching anything, but it doesn't serve the + classic `/json` HTTP discovery (confirmed `/json/version` returns + 404) — `insert_text`/`type_keystrokes` fall back automatically to + connecting straight to the browser-level `ws://host:port/devtools/browser` + endpoint and `Target.attachToTarget{flatten:true}` in that case, + but since that path can't be auto-discovered via `pid` + `lsof` + (no working `/json` to validate against), pass the port explicitly + via `cdp_port` (commonly 9222, but whatever the chrome://inspect + page shows) — auto-discovery only works for the dedicated-profile + route above. + **This can't be made fully unattended, and the popup fires more + often than you'd think.** Chrome shows a live "Allow remote + debugging?" confirmation on every *new* WebSocket connection to the + browser endpoint — not once per Chrome process, and NOT something a + saved preference can skip. Confirmed by testing live, twice, with + different results depending on what actually opened a new socket: + - Naively calling `insert_text`/`type_keystrokes` fresh each time + (each one connecting from scratch) got prompted on **every single + call, 3 for 3** — worse than "once per session." + - `CdpSessionCache` (this platform's backend) fixes that: it caches + one open connection per port and reuses it across calls, + re-attaching to a different tab on the *same* connection via + `Target.getTargets`/`Target.attachToTarget` when + `target_url_contains` points somewhere new — flattened-mode + re-attach doesn't open a new socket, so it doesn't re-prompt. + Confirmed live: after the first approval, two more calls (typing + into the same tab) went through silently in well under a second. + - The popup *does* come back the moment the cache has to open a + genuinely new connection — confirmed by closing the tab the + cached session was attached to: the next call took ~2s (evict + + reconnect) and needed a fresh click. Also expect this after a + full Chrome restart, or a daemon restart (the cache is in-memory, + not persisted). + Net: one click to start, silent after that for as long as the + browser and the daemon both stay up and you're re-attaching rather + than reconnecting — but don't promise a caller "no more prompts, + ever" for a long session that outlives either of those. + If no CDP port is found (and none was given explicitly), both + actions fail fast with an actionable error rather than silently + no-op'ing. +- Multi-tab ambiguity: with more than one tab open, both actions act + on whichever tab is found first — not necessarily the one you + mean — unless you pass `target_url_contains` (a substring to match + against the tab's URL, e.g. `"x.com/compose"`). Always pass this when + the browser might have more than one tab open, which in practice is + every real, non-disposable-profile session. Be as specific as the + URL allows — `"x.com"` alone matched BOTH a compose tab and a + notifications tab in testing, and `pick_target` just took whichever + came first; `"x.com/compose"` was needed to disambiguate. + +Prefer `execCommand` first for plain ``/`