feat(computer_use): add native Windows UIA backend - #46667
Closed
teknium1 wants to merge 13 commits into
Closed
Conversation
Brings desktop control to Windows hosts: UI Automation element discovery with SOM overlays, SendInput mouse/keyboard (virtual- desktop-normalized absolute coords, Unicode typing), and focus-free set_value via UIA value/selection/range patterns. Backend selection is platform-aware (HERMES_COMPUTER_USE_BACKEND still overrides) and check_computer_use_requirements() now gates per platform. Windows session-killing key combos (win+l, ctrl+alt+del, alt+f4) are hard-blocked alongside the macOS list. Unlike cua-driver on macOS there is no background input injection on Windows: pointer/keyboard actions briefly foreground the target window, and the platform-aware tool schema tells the model so. Requires uiautomation (+comtypes) in the venv; windows_backend degrades to unavailable when imports fail. 118 computer_use tests pass incl. 21 new dependency-free Windows tests; verified live against Notepad (capture/SOM/type/set_value/key). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The injected guidance block hardcoded macOS background-control rules (do-not-steal-focus, do-not-raise-windows). On Windows that is backwards: pointer and keyboard actions foreground the target window. Select Windows-specific guidance on win32 - foreground behavior, set_value as the focus-free path, cmd to ctrl and win mapping, and the Windows blocked combos - so the model is told the truth about how its actions behave on this host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Visible "PC use mode": a persistent banner pill while desktop control is active, the numbered SOM element boxes mirrored onto the real screen after each capture, click ripples / drag arrows where actions land, and short action flashes (typing, key combos, scroll). overlay.py runs as a subprocess: a fullscreen transparent click-through topmost tkinter window spanning the virtual desktop, driven over localhost UDP, excluded from screen capture via SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE) so the model's own screenshots never contain it (verified by pixel-sampling a capture taken while a box was on screen). The overlay returns foreground focus after spawning, and the backend never targets the overlay process as a capture subject. All overlay traffic is fire-and-forget: any failure disables the overlay without affecting actions. Disable with HERMES_COMPUTER_USE_OVERLAY=0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full-resolution desktop captures (1920x1032+) tokenize to thousands of vision tokens and overflow small local vision models' context windows - the aux call came back "the vision API rejected the image" and the model got no description at all. Cap the long side at 1456px before writing the temp image for vision_analyze: SOM badges stay legible, the request fits comfortably, and per-capture vision latency drops roughly in half. Also drop the hardcoded "macOS" from the describe prompt now that captures come from Windows hosts too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Win32-only marker with a <3 ceiling per dependency policy; comtypes arrives transitively. Non-Windows installs are unaffected - the backend availability check degrades gracefully when the import is absent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract the capture downscale into _shrink_capture_for_vision so it is unit-testable without the aux-vision plumbing, and add dependency-free tests for it (oversize shrinks with aspect preserved, small and non-image bytes pass through untouched) plus the overlay client's fail-safe contract (env kill switch spawns nothing, sends before start or after death are silent no-ops). Also update the one capture-routing assertion that pinned the literal "macOS application screenshot" prompt wording, which became platform-neutral when Windows hosts started producing captures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three failure modes found preparing for real daytime use on a shared desktop: 1. Vision-node outage broke captures outright. When aux-vision routing is requested (the main model cannot consume images) and the aux call fails, the old fallthrough returned the multimodal envelope - putting a screenshot in front of a text-only model and erroring the capture. Degrade to the AX/SOM text payload instead (vision_unavailable flag set): element-index actions keep working blind until vision returns. 2. Stale coordinates after a window move. Element bounds are absolute screen coords frozen at capture time; dragging the window between capture and click landed clicks on whatever sat at the old position. Track the captured window rect and translate element centers by the origin delta; a resize (interior layout changed) fails with an explicit re-capture message instead of guessing. 3. Input collisions with an active user. Synthetic input lands in whatever has focus; injecting mid-keystroke sprays input across both parties' targets. All input actions now wait for a short user-idle window (HERMES_COMPUTER_USE_IDLE_WAIT, default 1.5s, 0 disables), capped at 8s so the agent yields but never deadlocks. Routing tests updated for the new degradation contract; new tests cover all three behaviors and run dependency-free off-Windows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses review feedback on the Windows computer_use backend (#43927). 1. A failed click/drag/scroll left modifier keys - and, for drag, the mouse button - synthetically held down: the release ran after the action inside the same try block, so an injection error skipped it. Move the release into a finally so Ctrl/Alt/Shift and the button are always released and the cursor restored even when an injection raises. 2. capture (_walk_elements) and set_value (_control_at_index) each reimplemented the same BFS + interactability filter; if the two ever diverged, set_value would resolve an index to a different control than the capture advertised. Both now consume one _iter_interactable generator, so element #N is the same control in both paths. 3. That shared walk uses collections.deque.popleft() instead of the O(n) list.pop(0). 7 new dependency-free tests (modifier/button release on failure, capture/set_value index agreement, BFS order, Text-pattern filter); they pass off-Windows. Full computer_use suites: 137 passed on Windows 11. Thanks to @Icather for spotting all three and proposing the fixes (originally raised in #45976). Suggested-by: ChengLong Han <97326386+Icather@users.noreply.github.com> Co-authored-by: ChengLong Han <97326386+Icather@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stop the overlay subprocess before switching virtual desktops, then restart it on the new desktop — avoids the tkinter display context teardown that kills the overlay during SendInput-based Ctrl+Win+Left/Right. - _switch_desktop_via_keybd(): stop overlay → two-phase SendInput → restart overlay - switch_desktop() method on WindowsUIABackend - Schema and dispatch updated with 'switch_desktop' action and 'direction' parameter Co-authored-by: lEWFkRAD
Two-batch SendInput (press → sleep → release) crashes the embedded gateway (Dashboard Chat tab) because its single-channel event loop cannot handle multi-batch keyboard injection without disrupting the PTY pipeline. The full system gateway is unaffected because its multi-client dispatch loop handles concurrent channels. Switch to single-batch SendInput matching _press_combo semantics (hold modifiers → tap arrow → release). This works in both embedded and full gateway modes.
- Validate direction before key dispatch (unknown values return False) - Log exceptions instead of silently swallowing them - Add 150ms delay before overlay restart to avoid DWM race - Route switch_desktop through _maybe_follow_capture for consistency - Add JSON Schema if/then to require direction for switch_desktop
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
unresolved-import |
9 |
unresolved-attribute |
8 |
unsupported-operator |
2 |
invalid-argument-type |
1 |
First entries
tools/computer_use/windows_backend.py:46: [unresolved-import] unresolved-import: Cannot resolve imported module `win32gui`
tools/computer_use/windows_backend.py:44: [unresolved-import] unresolved-import: Cannot resolve imported module `win32api`
tools/computer_use/windows_backend.py:503: [unresolved-attribute] unresolved-attribute: Module `ctypes` has no member `windll`
tools/computer_use/windows_backend.py:146: [unresolved-attribute] unresolved-attribute: Module `ctypes` has no member `get_last_error`
tools/computer_use/windows_backend.py:47: [unresolved-import] unresolved-import: Cannot resolve imported module `win32process`
tools/computer_use/windows_backend.py:43: [unresolved-import] unresolved-import: Cannot resolve imported module `uiautomation`
tools/computer_use/windows_backend.py:428: [unresolved-attribute] unresolved-attribute: Attribute `readline` is not defined on `None` in union `IO[Any] | None`
hermes_cli/config.py:4769: [unresolved-attribute] unresolved-attribute: Attribute `items` is not defined on `str`, `int`, `float`, `list[Unknown]`, `None` in union `Unknown | str | int | ... omitted 15 union elements`
tests/agent/test_auxiliary_config_bridge.py:284: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["provider"]` and `Unknown | str | int | ... omitted 15 union elements`
tools/computer_use/windows_backend.py:45: [unresolved-import] unresolved-import: Cannot resolve imported module `win32con`
tests/tools/test_computer_use_windows.py:142: [unresolved-attribute] unresolved-attribute: Unresolved attribute `WindowsUIABackend` on type `ModuleType`
gateway/restart.py:10: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `Unknown | str | int | ... omitted 15 union elements`
tools/computer_use/windows_backend.py:465: [unresolved-attribute] unresolved-attribute: Attribute `close` is not defined on `None` in union `IO[Any] | None`
tools/computer_use/tool.py:717: [unresolved-import] unresolved-import: Cannot resolve imported module `PIL`
tests/tools/test_computer_use_windows.py:143: [unresolved-attribute] unresolved-attribute: Unresolved attribute `windows_backend_available` on type `ModuleType`
tests/tools/test_computer_use_windows.py:295: [unresolved-import] unresolved-import: Cannot resolve imported module `PIL`
tools/computer_use/overlay.py:241: [unresolved-attribute] unresolved-attribute: Module `ctypes` has no member `windll`
tests/tools/test_computer_use_windows.py:21: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tools/computer_use/windows_backend.py:48: [unresolved-import] unresolved-import: Cannot resolve imported module `PIL`
tests/agent/test_auxiliary_config_bridge.py:285: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["model"]` and `Unknown | str | int | ... omitted 15 union elements`
✅ Fixed issues (4):
| Rule | Count |
|---|---|
unsupported-operator |
2 |
invalid-argument-type |
1 |
unresolved-attribute |
1 |
First entries
tests/agent/test_auxiliary_config_bridge.py:285: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["model"]` and `Unknown | int | str | ... omitted 15 union elements`
gateway/restart.py:10: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `Unknown | int | str | ... omitted 15 union elements`
tests/agent/test_auxiliary_config_bridge.py:284: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["provider"]` and `Unknown | int | str | ... omitted 15 union elements`
hermes_cli/config.py:4758: [unresolved-attribute] unresolved-attribute: Attribute `items` is not defined on `int`, `str`, `list[Unknown]`, `float`, `None` in union `Unknown | int | str | ... omitted 15 union elements`
Unchanged: 5737 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
alt-glitch
added
type/feature
13 tasks
lEWFkRAD
reviewed
Jun 15, 2026
lEWFkRAD
left a comment
lEWFkRAD
left a comment
Contributor
There was a problem hiding this comment.
Thanks for pulling this together, @teknium1 — and for the credit. The config.yaml consolidation, the backend aliases + noop fallback, and the added Windows modifier keys are all improvements over what we had, and the salvage carried the backend over faithfully. I diffed all four touched files against #43927; one behavior change is worth fixing before this merges:
_switch_desktop_via_keybd now force-clears overlay_client._dead and calls start() unconditionally on every switch. That re-introduces the kill-switch bug ea294afb5 fixed — a user who disabled the overlay (computer_use.overlay: false, or HERMES_COMPUTER_USE_OVERLAY=0 → _dead=True, _proc=None) gets it resurrected on screen on every virtual-desktop switch, against their opt-out. CI is green only because the regression test that guarded this didn't survive the salvage (_dead appears once in the test file; the one overlay-disable test checks construction, not the switch path).
I verified it against this branch's own backend: test_disabled_overlay_is_not_resurrected fails on the current code (assert 'start' not in ['stop', 'start']) and passes once the overlay_was_live guard is restored.
Fix is up as #46678 against this branch — restores the guard, drops the _dead force-clear, and re-adds the three overlay-safety tests. 242 passed under per-file isolation (the way run_tests.sh/CI runs it); footgun + ruff clean. Merge it in or just cherry-pick the one-function change, whichever's easier.
Contributor
Author
|
Closing this one — I split the native Python UIA backend out from the trycua path, but the intended Windows computer-use direction is the trycua I’m moving the salvage work back onto the existing trycua Windows CUA PR (#44221 / original #30660), and will only carry over backend-agnostic pieces from the duplicate native UIA PRs where they make sense: prompt/docs wording, tests, setup polish, and safety semantics above the driver layer. The separate Python |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Windows Computer Use now has a native UIA backend while preserving the existing single
computer_usetool surface.Changes
set_value, and virtual desktop switching.config.yamldefaults:computer_use.backend,idle_wait_seconds, andoverlay.hermes toolslabeling, dependencies, docs, and targeted tests.Credit
Salvages the best parts from:
switch_desktopand single-batch SendInput fix.Validation
python3 -m py_compile tools/computer_use/tool.py tools/computer_use/backend.py tools/computer_use/windows_backend.py tools/computer_use/schema.py tests/tools/test_computer_use_windows.py tests/tools/test_computer_use_capture_routing.pyscripts/run_tests.sh tests/tools/test_computer_use.py tests/tools/test_computer_use_capture_routing.py tests/tools/test_computer_use_windows.py tests/hermes_cli/test_tools_config.pyswitch_desktopfallback, schema shapeuv lock --checkgit diff --check origin/main..HEADNotes
I intentionally did not merge the cua-driver-rs Windows/Linux path into this PR. That has different installer/update/binary lifecycle concerns and should stay separate from the native UIA salvage.
Infographic