Feat(tools): add Windows UIA backend for computer_use

[lEWFkRAD]

What does this PR do?

Adds a Windows backend for the computer_use toolset, which is currently macOS-only (cua-driver). Windows hosts get the same model-agnostic desktop control through the existing ComputerUseBackend ABC: SOM captures with numbered element overlays, element-index clicking, drag/scroll, Unicode typing, key combos, and focus-free set_value — driven by any tool-capable model through the existing universal schema, unchanged.

The approach mirrors the repo's own architecture (backend.py explicitly anticipates "future Linux/Windows" implementations): a sibling backend behind the same ABC, selected per-platform with HERMES_COMPUTER_USE_BACKEND still overriding. Element discovery and set_value go through UI Automation (the Windows analogue of the AX tree, via the pure-python uiautomation package); screenshots through Pillow (already a core dep); input synthesis through ctypes SendInput (no new dep). All coordinates are physical pixels with per-monitor-v2 DPI awareness.

One honest platform difference, reflected in the platform-selected schema description and system-prompt guidance: Windows has no supported background input injection, so pointer/keyboard actions briefly foreground the target window. set_value is the exception (UIA patterns work unfocused). cmd maps to Ctrl for cross-platform model habits; Windows session-killers (win+l, ctrl+alt+del, alt+f4) join the hard-blocked combo list.

Also included:

• an optional on-screen overlay (banner + live element boxes + click ripples) so the user can see what the agent sees and does; excluded from the agent's own captures via SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE), strictly fire-and-forget (overlay failure can never affect actions), kill switch HERMES_COMPUTER_USE_OVERLAY=0;
• a capture downscale (≤1456px) before aux-vision routing — full-resolution desktop captures overflowed small local vision models' context windows ("the vision API rejected the image") and roughly double per-capture latency.

Related Issue

No existing issue; this implements the Windows direction anticipated in tools/computer_use/backend.py's module docstring. Happy to open a tracking issue if preferred.

Type of Change

• ✨ New feature (non-breaking change that adds functionality)

Changes Made

• tools/computer_use/windows_backend.py — new: WindowsUIABackend (UIA element walk with node/depth/time budgets; PIL screenshot + SOM overlay; ctypes SendInput layer with virtual-desktop-normalized absolute coords, UTF-16-aware Unicode typing, batched key combos; window enum/focus with foreground-lock workaround; UIA Value/SelectionItem/RangeValue set_value with COM-safe re-find — live COM pointers are never reused across tool calls), plus the _OverlayClient and availability check.
• tools/computer_use/overlay.py — new: optional overlay subprocess (tkinter, transparent/click-through/topmost, UDP-driven, capture-excluded).
• tools/computer_use/tool.py — platform-aware backend selection; check_computer_use_requirements() gates per platform; Windows blocked key combos + win aliases; _shrink_capture_for_vision() before aux-vision routing.
• tools/computer_use/schema.py, tools/computer_use_tool.py, toolsets.py — platform-selected descriptions (previously told every model "macOS only").
• agent/prompt_builder.py — COMPUTER_USE_GUIDANCE platform-selected; the macOS text instructs the model it has background control and must not raise windows, which is exactly backwards on Windows.
• pyproject.toml + uv.lock — uiautomation>=2.0.29,<3; sys_platform == 'win32' (comtypes transitive; +23 lock lines).
• tests/tools/test_computer_use_windows.py — new: 27 tests, dependency-free (win32 modules stubbed; collects and passes on Linux).
• tests/tools/test_computer_use.py, tests/tools/test_computer_use_capture_routing.py — two assertions updated where they pinned macOS-only behavior (check_fn gate, prompt wording).

How to Test

1. On Windows: uv sync, then hermes -z "Use computer_use: list_apps, then capture mode='som' and describe the foreground window and three elements by index." — requires no config; the toolset gates on via check_computer_use_requirements().
2. Interactive loop: open Notepad, then hermes -z "focus_app 'notepad' raise_window=true, capture som, click the editor element, type a line, re-capture and quote the text back."
3. On Linux/macOS: pytest tests/tools/test_computer_use_windows.py -q — all tests collect and pass off-Windows (win32 imports are stubbed/guarded); macOS behavior is untouched (cua remains the darwin default).

Verified on Windows 11 (10.0.26200), Python 3.12.10:

• pytest tests/tools/test_computer_use_windows.py tests/tools/test_computer_use.py tests/tools/test_computer_use_capture_routing.py -q → 124 passed.
• scripts/check-windows-footguns.py --diff origin/main → clean (9 files).
• Live end-to-end (agent-driven via hermes -z, qwen-class model on local vLLM): focus → SOM capture (26 elements) → element click → type → re-capture verified the typed text from the window title, tab label, and status bar. Capture exclusion of the overlay verified by pixel-sampling a capture taken while overlay boxes were on screen (0 overlay pixels found).

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass — full tests/tools/ run on Windows; one pre-existing, unrelated collection error in tests/tools/test_search_hidden_dirs.py (missing external binary on this host, fails identically on origin/main) excluded
• I've added tests for my changes (27 new, dependency-free)
• I've tested on my platform: Windows 11 (10.0.26200)

Documentation & Housekeeping

• I've updated relevant documentation (module docstrings document the architecture and platform difference) — schema/tool/toolset descriptions updated
• I've updated cli-config.yaml.example if I added/changed config keys — N/A (no new config keys; env vars HERMES_COMPUTER_USE_BACKEND pre-existing, HERMES_COMPUTER_USE_OVERLAY documented in code)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — footgun scan clean; all new code win32-gated with graceful degradation; macOS path untouched
• I've updated tool descriptions/schemas if I changed tool behavior — yes (platform-selected descriptions)

Screenshots / Logs

124 passed, 1 warning in 5.00s          # computer_use suites, Windows 11
✓ No Windows footguns found (9 file(s) scanned).

Agent-driven live verification (one-shot hermes -z, local vLLM model):

Step 1 — focus_app 'notepad' raise_window=true: SUCCESS (window raised)
Step 2 — capture mode='som': SUCCESS (25 elements; vision described title bar + toolbar)
Step 3 — click element #1 (Document 'Text editor'): SUCCESS at (1039, 307)
Step 4 — type 81-char text: SUCCESS
Step 5 — re-capture: window title, tab label and status bar (line 3, col 60,
         81 characters) all confirm the typed text landed

Capture-exclusion proof (overlay): screenshot taken while overlay boxes were drawn on screen; sampling the box edge found 0 overlay-colored pixels in the captured image while the boxes were visibly on the desktop.

[tonydwb]

Code Review Summary

Verdict: Comment (high-surface-area, full review deferred)

Feature: Add Windows UIA backend for computer_use

This is a large PR (1,801 additions / 40 deletions, 2,137 diff lines) touching a new platform backend. In cron batch mode I cannot do a thorough deep review of this scope within available resources.

Surface area concerns (require human review):

• Windows UIA automation spans many files — inspect the integration with existing computer_use tool abstractions for regressions
• Error handling in the UIA interaction layer
• Whether existing tests cover the new platform path

This is a feature PR, not a security or critical fix. Please treat this as an informational comment — the PR may be mergeable but warrants a focused human review before merge.

---

Reviewed by Hermes Agent (batch mode — high-surface-area, full review deferred)

[lEWFkRAD]

Pushed one more commit (569c4af) — hardening from the first day of real daily-driver use on a shared desktop, before this gets reviewed:

1. Vision-node outage no longer breaks captures. When aux-vision routing is requested (text-only main model) and the aux call fails, the old fallthrough returned a multimodal envelope to a model that can't consume images, erroring the whole capture. It now degrades to the AX/SOM text payload with vision_unavailable: true — element-index actions keep working blind until vision comes back.

2. Window-move coordinate translation. Element bounds are frozen at capture time; dragging the window between capture and click landed clicks on stale pixels. The backend now tracks the captured window rect and translates click points by the origin delta; a resize fails with an explicit re-capture message instead of guessing.

3. User-collision idle guard. Synthetic input lands in whatever has focus, so injecting mid-keystroke sprays input across both parties' targets. Input actions now wait for a short user-idle window (HERMES_COMPUTER_USE_IDLE_WAIT, default 1.5s, 0 disables, 8s cap) — the agent yields to an actively-working human but never deadlocks.

Routing tests updated for the new degradation contract (aux-failure fallback for non-vision mains is text, not multimodal) + 8 new tests, all dependency-free off-Windows. Full computer_use suites: 130 passed on Windows 11.

Found all three the honest way: running it for real on the machine I work at. Still waiting on a workflow approval whenever a maintainer gets a chance 🙏

[Icather]

Nice work — I was building the same thing at #45976 (closed now as duplicate). During review I spotted a few things that might help here:

1) Modifier key leak (click/drag/scroll, ~L817-825)
_send_inputs(mods_down) runs before the action but _send_inputs(mods_up) is after it, outside a finally block. If _mouse_button or _mouse_move raises, the except catches it and returns without releasing modifiers — user gets stuck with Ctrl/Alt held. Wrapping the body in try/finally fixes it.

2) BFS queue.pop(0) is O(n) (_walk_elements L683, _control_at_index L1018)
With max_nodes=1500, each pop shifts the whole list. Switching to collections.deque + popleft() is a one-line fix and avoids ~1M element moves per full walk.

3) _control_at_index duplicates _walk_elements tree-walk (~L1008-1044 vs ~L659-725)
Both methods do the same BFS + interactability filter; the only difference is one collects all elements and the other stops at a target index. Could share a generator-based walk to keep discovery order consistent between capture and set_value.

Happy to help with the try/finally fix or the deque change if you'd like.

[lEWFkRAD]

Thanks @Icather — all three confirmed and fixed in eba52edf5:

1. Modifier leak — real. mods_up ran after the action inside the same try, so any injection failure skipped it. Moved the release into a finally in click/drag/scroll; drag also releases the mouse button there, since a stuck primary button turns every later move into a drag-select.
2. _control_at_index duplicating _walk_elements — agreed, and I treated it as a latent correctness bug rather than just DRY: if the two BFS+filter copies ever drifted, set_value would resolve an index to a different control than the capture advertised. Both now consume one _iter_interactable generator, so element #N means the same control in both paths.
3. pop(0) is O(n) — the shared walk now uses collections.deque.popleft().

Added 7 dependency-free tests, including one that asserts capture and set_value resolve the same control for every advertised index. Sorry your #45976 got closed as the dup — appreciate you reviewing here regardless.

[Icather]

Been running this daily on Windows 10 — the overlay (numbered element boxes, click ripples, action flashes) is the best part. The visual feedback makes computer_use actually usable for a human sharing the desktop, not just the agent. Everything works: screenshots, click/drag/scroll, Unicode typing, set_value. Tested against Notepad, Explorer, and the Windows shell. Happy to help test on different Windows configs if needed.

[kent-ai-dev]

yoooo lets get this in i need

[Icather]

Found one more issue while daily-driving this: switch_desktop("left"/"right") via key("ctrl+win+left") kills the overlay subprocess — the full-screen tkinter display context tears down on virtual-desktop switch regardless of whether you use SendInput or keybd_event.

Fix: stop the overlay before switching → switch desktop → restart overlay on the new desktop. The overlay client already has stop() / start() with a clean restart path (stop unsets _proc, start spawns a fresh subprocess). 35 lines added to a switch_desktop() method, zero new dependencies. Tested on Windows 10 — switching both directions works, Hermes survives, subsequent captures work on the new desktop.

Happy to share the code if you want to fold it in.

[lEWFkRAD]

Yes please — let's fold it in. You've been daily-driving this on configs I can't easily reach, and that switch_desktop teardown is exactly the kind of thing that only shows up in real use, so I'd rather have it as your commit than reimplement it from the description.

Two easy paths, whichever's less friction for you:

• Open a PR against lEWFkRAD:feat/computer-use-windows (this PR's branch) and I'll merge it in, or
• Drop the diff here and I'll apply it with you as Co-authored-by:.

Either way the stop → switch → restart approach sounds right — the overlay client's stop()/start() was built for exactly that clean-restart case. If you can include a quick test that asserts the overlay _proc is respawned after a simulated desktop switch (mirroring the dependency-free pattern the rest of the suite uses), that keeps it consistent with the off-Windows CI. Appreciate you sticking with this thread instead of walking after #45976 got closed.

[Icather]

Submitted as a PR against your branch: lEWFkRAD#1

Single-commit addition — overlay-safe virtual desktop switching via SendInput. Went through several rounds of cross-gateway-mode testing (embedded Chat tab vs full system gateway) and one Copilot review pass. Should be ready to merge on your end whenever you're comfortable.

[lEWFkRAD]

Update — folded in a contributed action and a follow-up fix since the last review pass:

• switch_desktop action (Ctrl+Win+Left/Right) — contributed by @Icather via Terminal tool #1, overlay-safe across the virtual-desktop transition (the full-screen tkinter overlay is torn down by any desktop switch, so the action stops it, switches with a single-batch SendInput, and restarts it). Came with cross-mode testing on the embedded Chat-tab gateway vs the full system gateway.
• Overlay kill-switch fix (ea294af) — the restart had force-cleared _OverlayClient._dead before start(), which would revive an overlay the user had disabled with HERMES_COMPUTER_USE_OVERLAY=0. Now gated on whether the overlay was actually running before the switch; dependency-free regression tests added (disabled-stays-down fails on the old behavior).

Totals now: 12 commits, computer_use suites 140 passed on Windows 11 (10.0.26200), check-windows-footguns.py --diff origin/main clean (10 files). Attribution: @Icather's commits auto-resolve via the noreply pattern; no AUTHOR_MAP change needed.

[teknium1]

Superseded by #50552 — a comprehensive cross-platform cua-driver implementation (macOS/Windows/Linux) now merged to main. It covers the platform this PR targeted. Thanks for the contribution; closing as superseded.