feat(computer_use): enable Windows via cua-driver-rs (salvage #30660) by teknium1 · Pull Request #44221 · NousResearch/hermes-agent

teknium1 · 2026-06-11T11:08:07Z

Summary

computer_use now works on Windows — the cua-driver backend's macOS-only gate is opened to darwin + win32, with the install path, prompts/UI, version handling, and the optional mcp dependency made cross-platform to match.

Salvage of #30660 by @f-trycua (cua-driver upstream author), cherry-picked onto current main with authorship preserved. Empirically verified by the author on Windows 11 24H2 (UWP UIA at Medium and High IL).

Changes

tools/computer_use/tool.py: gate flip to darwin/win32 (Linux stays off — cua-driver-rs Linux is alpha); failed start() no longer caches a half-initialised backend
hermes_cli/tools_config.py: per-OS installer dispatch (Windows → install.ps1 via PowerShell), OS+arch release-asset pre-check, de-macOS'd labels
agent/prompt_builder.py + agent/system_prompt.py: computer_use_guidance() renders platform-aware guidance (no "Mac / Space / cmd+s" on Windows); deterministic per host, byte-stable per conversation
tools/computer_use/cua_backend.py: removed the phantom HERMES_CUA_DRIVER_VERSION "pin" (never gated anything); native cua-driver check-update --json powers a once-per-process update nudge and hermes computer-use status
tools/lazy_deps.py: tool.computer_use → mcp==1.26.0 lazy install — no more dead-end No module named 'mcp'
hermes_cli/main.py: hermes update refresh + computer-use CLI help made cross-platform
Docs (EN + zh-Hans): cross-platform usage, local-build testing section

Follow-up fix (ours): the PR branch's merge of main dropped import subprocess from cua_backend.py (our unused-import prune raced their new cua_driver_update_check() use); the bare try/except silently disabled the update check. Restored.

Validation

	Result
`tests/tools/test_computer_use*.py` + `test_install_cua_driver.py` + `test_post_setup_gating.py`	134 passed
`tests/agent/ -k prompt`	196 passed
`tests/tools/test_lazy_deps.py`	61 passed
E2E (real imports, linux host)	guidance variants byte-stable; darwin constant matches; linux gate still off; update_check → None without binary

Infographic

github-actions · 2026-06-11T11:08:56Z

🔎 Lint report: `hermes/hermes-d13b1ab7` vs `origin/main`

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 10915 on HEAD, 10914 on base (🆕 +1)

🆕 New issues (3):

Rule	Count
`unsupported-operator`	1
`invalid-argument-type`	1
`unresolved-import`	1

First entries

hermes_cli/tools_config.py:3367: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `dict[Unknown, Unknown]` and `str | list[dict[str, str | list[Unknown]] | dict[str, str | list[Unknown] | bool | list[str]] | dict[str, str | list[dict[str, str]]]] | list[dict[str, str | list[Unknown] | bool | list[str]] | dict[str, str | list[dict[str, str]]]] | ... omitted 5 union elements`
tests/hermes_cli/test_tools_config.py:745: [invalid-argument-type] invalid-argument-type: Argument to function `_detect_active_provider_index` is incorrect: Expected `list[Unknown]`, found `str | list[dict[str, str | list[Unknown]] | dict[str, str | list[Unknown] | bool | list[str]] | dict[str, str | list[dict[str, str]]]] | list[dict[str, str | list[Unknown] | bool | list[str]] | dict[str, str | list[dict[str, str]]]] | ... omitted 5 union elements`
tools/computer_use/tool.py:680: [unresolved-import] unresolved-import: Cannot resolve imported module `PIL`

✅ Fixed issues (2):

Rule	Count
`unsupported-operator`	1
`invalid-argument-type`	1

First entries

hermes_cli/tools_config.py:3278: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `dict[Unknown, Unknown]` and `str | list[dict[str, str | list[Unknown]] | dict[str, str | list[Unknown] | bool | list[str]] | dict[str, str | list[dict[str, str]]]] | list[dict[str, str | list[Unknown] | bool | list[str]] | dict[str, str | list[dict[str, str]]]] | ... omitted 4 union elements`
tests/hermes_cli/test_tools_config.py:745: [invalid-argument-type] invalid-argument-type: Argument to function `_detect_active_provider_index` is incorrect: Expected `list[Unknown]`, found `str | list[dict[str, str | list[Unknown]] | dict[str, str | list[Unknown] | bool | list[str]] | dict[str, str | list[dict[str, str]]]] | list[dict[str, str | list[Unknown] | bool | list[str]] | dict[str, str | list[dict[str, str]]]] | ... omitted 4 union elements`

Unchanged: 5745 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@teknium1

The cua-driver backend was gated to macOS only: # tools/computer_use/tool.py def check_computer_use_requirements() -> bool: if sys.platform != "darwin": return False ... But cua-driver itself has been Windows-feature-complete since cua-driver-rs (the cross-platform Rust port) shipped its Windows backend. Every action tool — click, type_text, hotkey, drag, scroll, screenshot, launch_app, list_apps, list_windows, get_window_state, move_cursor, wait — is marked VERIFIED on Windows in the cross-platform PARITY matrix: https://github.com/trycua/cua/blob/main/libs/cua-driver-rs/PARITY.md This PR widens the gate to `sys.platform in ("darwin", "win32")`. No new code paths — the existing MCP stdio integration in cua_backend.py works identically against cua-driver on Windows because cua-driver's tool surface is uniform across OSes. Linux is not in scope. cua-driver-rs Linux support exists in tree but is alpha (most Linux rows in PARITY are OPEN, not VERIFIED) — keeping it gated off here until upstream flips those to VERIFIED. The plumbing is OS-agnostic so flipping the gate later is one-line. Empirical verification on Windows 11 24H2 (2026-05-22 dogfood): - Built-in Administrator (RID 500) at High IL via cua-driver-rs RunLevel=Highest autostart task: `cua-driver call get_window_state` for Calculator UWP → element_count: 41 - Regular admin (UAC-split, Medium IL primary token) running `cua-driver call` directly from PowerShell: `cua-driver call get_window_state` for Calculator UWP → element_count: 41 UWP / AppContainer UIA works at any IL for any user. No EV cert, no uiAccess="true" manifest, no Program Files install requirement. ## Changes - tools/computer_use/tool.py: replace `sys.platform != "darwin"` early-return with `sys.platform not in ("darwin", "win32")`. Update top-of-file docstring + vision-prompt phrasing ("macOS application" → "desktop application") so the model isn't told to expect a Mac UI when it's looking at a Windows screen. - tools/computer_use/cua_backend.py: rewrite top-of-file docstring to cover macOS + Windows + the Linux-alpha caveat. `is_available()` matches the same `darwin/win32` allowlist. `cua_driver_install_hint()` returns the Windows installer (irm | iex) on Windows, the bash installer on macOS. - tools/computer_use_tool.py: update registry description from "macOS desktop control" to "desktop control (macOS, Windows; Linux alpha)". The macOS-specific bits in `cua_backend.py` (the `_is_arm_mac` helper, the "macOS reports localized app names" warning) stay as-is — they're macOS runtime details that are conditionally taken when running on macOS, not gates that block other OSes. ## Install Same one-liner story, OS-specific installer: macOS: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" Windows (PowerShell): irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex After install, `cua-driver` is on $PATH and Hermes's check_fn sees it. ## Related Replies to @teknium1's question on #20660 about whether cua-driver-rs ships Windows + Linux backends and whether @Abd0r's per-OS Python work should be absorbed into cua-driver as a starting point. Short answer: the cua-driver-rs Rust impl is months ahead of a fresh Python port on Windows. Linux is alpha and will get there. Several pieces of #20660 (kill-switch, JSONL audit log, screenshot redact_regions, the per-OS SKILL.md docs) are worth absorbing into cua-driver as follow-up work — separate from this PR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

… version warning, lazy mcp) Enable the computer_use toolset beyond macOS, matching cua-driver's cross-platform runtime support. - install: install_cua_driver() dispatches per-OS (Windows install.ps1 via PowerShell, macOS/Linux install.sh); arch pre-check recognizes Windows (AMD64/ARM64) and Linux (x86_64/aarch64); `hermes update` and `hermes computer-use install --upgrade` run cross-platform. - prompt/UI: COMPUTER_USE_GUIDANCE is now platform-aware (no macOS-only wording on Windows/Linux; Windows gets the dispatch:"foreground" note); de-macOS'd toolset labels, descriptions, and CLI help. - version: removed the non-functional HERMES_CUA_DRIVER_VERSION "pin" (it never gated anything); added a per-OS MIN_CUA_DRIVER_VERSION soft warning (macOS 0.5.0, the Rust build 0.2.16), surfaced at startup and in `computer-use status`. Local 0.0.0-* builds are exempt. - deps: lazy-install the optional `mcp` SDK via tools/lazy_deps.py on first use (tool.computer_use -> mcp==1.26.0) instead of dead-ending on "No module named 'mcp'"; clearer backend-unavailable hint; don't cache a backend whose start() failed. - tests: cross-platform install, version-warning, lazy-install, and corrected platform-gating tests (Linux gated off, Windows supported). - docs: computer-use.md (EN + zh-Hans) updated for cross-platform use, local-build testing, and the removed version pin. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… hardcoded version floor Replace the per-OS MIN_CUA_DRIVER_VERSION soft-warning (hardcoded, rot-prone) with cua-driver's native check-update — the source-of-truth freshness check shipped in trycua/cua#1734 (`check-update --json` CLI verb / `check_for_update` MCP tool). - cua_backend: cua_driver_update_check() shells `check-update --json` (stdin=DEVNULL so a pre-#1734 driver that falls through to a stdin read fails fast instead of blocking; timeout-guarded). Returns None when indeterminate (verb absent / offline / error payload / unparseable) so callers stay quiet. cua_driver_update_nudge() formats the one-liner; the startup nudge runs off-thread so the (cached, ~20h) GitHub poll never blocks the first computer_use action. - status: `hermes computer-use status` reports current/latest and an "update available" line via the native check. - update: install_cua_driver(upgrade=True) skips the network re-install when the driver reports it's already on the latest release. - Graceful on pre-#1734 drivers (e.g. 0.2.18 on the dev host): verb absent → stay quiet (verified: returns None in ~0.4s). - tests: TestUpdateCheck replaces TestVersionWarning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cua_driver_update_check() calls subprocess.run but the import was dropped when the PR branch merged main's unused-import prune (66827f8). The try/except Exception swallowed the NameError, silently disabling the update check.

Icather · 2026-06-15T16:25:35Z

Great to see Windows computer_use landing via cua-driver-rs. A few design notes from the Python UIA backend work that may be useful when adding switch_desktop (virtual desktop switching) to the Rust path — these were validated end-to-end in the now-closed salvage PR (#46667) and the original UIA fork (lEWFkRAD#1).

From lEWFkRAD's Windows UIA backend (#43927) — reusable in Rust

Overlay subprocess lifecycle — A full-screen transparent overlay for SOM capture needs stop/restart around any desktop switch, since the compositor tears it down during the transition. This pattern is language-agnostic.
Auxiliary vision model degradation — When the overlay capture is stale after a desktop switch, falling back to a pre-analysis pass through the auxiliary vision model before feeding the agent keeps results reliable.
Unified computer_use tool surface — Keeping a single tool entry point across platforms rather than splitting into per-OS tools avoids agent confusion and keeps the schema clean. Already the direction of this PR.

From Icather's `switch_desktop` (lEWFkRAD#1) — SendInput safety

Single-batch SendInput only — Multi-batch keyboard injection (press, sleep, release as separate SendInput calls) crashes the embedded gateway (Dashboard Chat tab) because its single-channel event loop cannot handle interleaved PTY disruption. Single-batch delivery (all key events in one SendInput call) worked in both embedded and full gateway modes. The same constraint applies regardless of whether SendInput is called from Python or Rust.

Prior art reference

teknium1's salvage PR #46667 included both lEWFkRAD's overlay framework and the single-batch switch_desktop fix with credits — worth referencing when these features are ported to Rust.

Happy to elaborate on any of these if useful.

Make the computer_use toolset platform-agnostic by driving cua-driver on macOS, Windows, and Linux. Consumes the 8 cua-driver decoupling surfaces (capability discovery, structuredContent AX tree, opaque element_token, click button enum, explicit mimeType, machine-readable manifest, structured list_windows, structured health_report), each degrading gracefully on older drivers. Adds `hermes computer-use doctor` (drives cua-driver health_report with a per-OS check matrix and an exit 0/1/2 ok/degraded/blocked contract), full typed wrappers for the previously-uncovered cua-driver tools plus a generic call_tool escape hatch, per-session agent-cursor lifecycle, platform-aware system-prompt guidance (host-deterministic, cache-safe), and honors HERMES_CUA_DRIVER_CMD end-to-end. Replaces the macOS-only skills/apple/macos-computer-use skill with a cross-platform skills/computer-use skill, and refreshes the EN + zh-Hans docs. Supersedes #44221 (Windows-enablement salvage of #30660). Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>

teknium1 · 2026-06-22T13:42:49Z

Superseded by #50552, which builds on this Windows-enablement work (your commits from here were folded in) and extends it to full cross-platform (Linux + Windows), 8 cua-driver decoupling surfaces, the doctor subcommand, full tool coverage, and a fix to the install probe. Merged to main.

Make the computer_use toolset platform-agnostic by driving cua-driver on macOS, Windows, and Linux. Consumes the 8 cua-driver decoupling surfaces (capability discovery, structuredContent AX tree, opaque element_token, click button enum, explicit mimeType, machine-readable manifest, structured list_windows, structured health_report), each degrading gracefully on older drivers. Adds `hermes computer-use doctor` (drives cua-driver health_report with a per-OS check matrix and an exit 0/1/2 ok/degraded/blocked contract), full typed wrappers for the previously-uncovered cua-driver tools plus a generic call_tool escape hatch, per-session agent-cursor lifecycle, platform-aware system-prompt guidance (host-deterministic, cache-safe), and honors HERMES_CUA_DRIVER_CMD end-to-end. Replaces the macOS-only skills/apple/macos-computer-use skill with a cross-platform skills/computer-use skill, and refreshes the EN + zh-Hans docs. Supersedes NousResearch#44221 (Windows-enablement salvage of NousResearch#30660). Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com> # Conflicts: # scripts/release.py

Make the computer_use toolset platform-agnostic by driving cua-driver on macOS, Windows, and Linux. Consumes the 8 cua-driver decoupling surfaces (capability discovery, structuredContent AX tree, opaque element_token, click button enum, explicit mimeType, machine-readable manifest, structured list_windows, structured health_report), each degrading gracefully on older drivers. Adds `hermes computer-use doctor` (drives cua-driver health_report with a per-OS check matrix and an exit 0/1/2 ok/degraded/blocked contract), full typed wrappers for the previously-uncovered cua-driver tools plus a generic call_tool escape hatch, per-session agent-cursor lifecycle, platform-aware system-prompt guidance (host-deterministic, cache-safe), and honors HERMES_CUA_DRIVER_CMD end-to-end. Replaces the macOS-only skills/apple/macos-computer-use skill with a cross-platform skills/computer-use skill, and refreshes the EN + zh-Hans docs. Supersedes NousResearch#44221 (Windows-enablement salvage of NousResearch#30660). Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>

alt-glitch mentioned this pull request Jun 14, 2026

feat(computer_use): add Windows backend via pyautogui + UIA #45976

Closed

9 tasks

teknium1 mentioned this pull request Jun 15, 2026

feat(computer_use): add native Windows UIA backend #46667

Closed

f-trycua and others added 5 commits June 15, 2026 07:28

chore(release): add AUTHOR_MAP entry for f-trycua

6c5a5ad

lEWFkRAD mentioned this pull request Jun 15, 2026

fix(tools): keep switch_desktop from resurrecting a disabled overlay #46678

Closed

13 tasks

fix(computer-use): fold duplicate PR polish into cua-driver salvage

2e80a38

teknium1 force-pushed the hermes/hermes-d13b1ab7 branch from 7579f81 to 2e80a38 Compare June 15, 2026 14:40

f-trycua mentioned this pull request Jun 16, 2026

Track: decouple Hermes' computer_use wrapper from cua-driver internals #47072

Open

8 tasks

Icather mentioned this pull request Jun 19, 2026

[Bug]: Windows computer_use: UIPI blocks UIA enumeration and input injection on elevated (admin) windows #49067

Closed

f-trycua mentioned this pull request Jun 22, 2026

feat(computer_use): cross-platform cua-driver — supersede #44221, consume 8 decoupling surfaces, new doctor, full tool coverage, refreshed docs + skill #50507

Closed

8 tasks

teknium1 mentioned this pull request Jun 22, 2026

feat(computer_use): cross-platform cua-driver (macOS/Windows/Linux) — salvage #50507 #50552

Merged

teknium1 closed this Jun 22, 2026

teknium1 mentioned this pull request Jun 22, 2026

feat(computer_use): enable Windows via cua-driver-rs #30660

Closed

6 tasks

Navigation Menu

Uh oh!

feat(computer_use): enable Windows via cua-driver-rs (salvage #30660) - #44221

feat(computer_use): enable Windows via cua-driver-rs (salvage #30660)#44221
teknium1 wants to merge 6 commits into
mainfrom
hermes/hermes-d13b1ab7

teknium1 commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026 •

edited

Loading

Uh oh!

Icather commented Jun 15, 2026

Uh oh!

teknium1 commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

teknium1 commented Jun 11, 2026

Summary

Changes

Validation

Related

Infographic

Uh oh!

github-actions Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔎 Lint report: hermes/hermes-d13b1ab7 vs origin/main

ruff

ty (type checker)

Uh oh!

Icather commented Jun 15, 2026

From lEWFkRAD's Windows UIA backend (#43927) — reusable in Rust

From Icather's switch_desktop (lEWFkRAD#1) — SendInput safety

Prior art reference

Uh oh!

teknium1 commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Jun 11, 2026 •

edited

Loading

🔎 Lint report: `hermes/hermes-d13b1ab7` vs `origin/main`

From Icather's `switch_desktop` (lEWFkRAD#1) — SendInput safety