feat(tools/computer_use): native per-OS desktop control (macOS / Linux / Windows)

[Abd0r]

Summary

Adds three OS-specific desktop-control tools — computer_use_macos, computer_use_linux, computer_use_windows — sharing one JSON schema and one set of action semantics, but with native backends per platform. Per-OS skills teach the model platform-specific shortcuts and idioms.

This PR is complementary to the three existing computer-use efforts in flight; it owns a different deployment shape:

Effort	Deployment shape
#15876	Linux inside Docker (containerised body, host-OS-agnostic, sandboxed)
#4562	macOS native, Anthropic computer-use protocol
#13308	macOS native, Hermes-protocol with approval gating
This PR	Native host desktop on macOS + Linux + Windows — same agent-facing abstraction, native backends per OS

There's no overlap with #15876 (we're outside Docker, native to the host). There's surface overlap with #4562 / #13308 on macOS — but the value here is the consistent 3-OS surface using one schema and one grammar; the operator can run Hermes on any of the three majors and the model sees the same action vocabulary. Maintainers can land any combination of the four — they target genuinely different operator setups.

Problems this solves

Concretely, what does an operator gain by having computer_use_* available?

1. Drive native desktop apps that have no CLI or web surface — Adobe apps, Office desktop, native installers, system settings panels, finance/CAD/DAW apps. Today the agent has to give up or ask the user to do it manually. With this PR, the agent screenshots, clicks, types, and gets the job done.

2. Consistent abstraction across the three majors — same JSON schema, same action vocabulary, same grammar (Cmd+Tab parses on every host; the parser routes to the correct OS primitive). An operator running Hermes on a personal Mac, a Linux workstation, and a Windows VM at work doesn't have to retrain prompts or learn three different tool surfaces.

3. Native host control without a container — the containerised path in Proposal: Optional desktop computer-use module (noVNC + screenshot + mouse/keyboard control) #15876 is excellent for VPS / always-on / sandboxed deployments, but it's the wrong answer when the operator wants Hermes to drive their actual desktop: their logged-in browser sessions, their installed apps, their multi-monitor setup. This PR fills that gap.

4. Foundation for higher-level "GUI teaching" / record-and-replay efforts — issue [Feature]: GUI Teaching for Hermes Agent – Record mouse/keyboard workflows as executable skills (macOS) #19802 (Linka — record mouse/keyboard workflows on macOS, generate skills from demonstrations, replay them) needs a runtime primitive to execute recorded events. This PR provides exactly that primitive across all three OSes, so a Linka-style record-and-replay layer can sit on top without re-implementing per-OS native input.

5. Keeps Hermes provider-neutral — by emitting a Hermes-internal tool spec rather than the Anthropic computer-use-tool block, this works across every provider Hermes supports (OpenRouter, OpenAI, Anthropic, custom local) without protocol-specific plumbing in the tool layer.

Issues referenced

This PR doesn't auto-close any single issue (computer-use is a new capability, not a bug fix). Issues it relates to:

• Refs Proposal: Optional desktop computer-use module (noVNC + screenshot + mouse/keyboard control) #15876 — same problem space, different deployment shape (native host vs containerised). Authors of Proposal: Optional desktop computer-use module (noVNC + screenshot + mouse/keyboard control) #15876 explicitly framed their work as the "containerised desktop body" path; this PR is the "native host desktop body" path. They can land alongside each other.
• Refs feat: Computer Use Tool — macOS desktop control via Anthropic native API #4562 — same problem space on macOS, different protocol (Anthropic computer-use vs Hermes-internal tool spec).
• Refs feat: stage macOS computer-use progress #13308 — same problem space on macOS, different scope (this PR keeps approval gating to the existing tool-call surface; feat: stage macOS computer-use progress #13308 adds a new approval flow).
• Refs [Feature]: GUI Teaching for Hermes Agent – Record mouse/keyboard workflows as executable skills (macOS) #19802 — Linka GUI-teaching layer needs a runtime primitive on each OS to execute recorded workflows; this PR provides that primitive on macOS, Linux, and Windows.

Architecture

• tools/computer_use_common.py — ActionRequest / ActionResult dataclasses, shared JSON schema (mirrors Anthropic's computer_20251124 action set with practical additions: get_active_window, screen_size, cursor_position), parameter validation, screen-bounds enforcement.
• tools/computer_use_safety.py — HERMES_COMPUTER_USE_ENABLED env gate (default off; tool refuses every action when unset), process-global kill-switch flag, append-only JSONL action log under \$HERMES_HOME/logs/computer_use.jsonl, screenshot redaction via PIL.
• tools/computer_use_grammar.py — one parser, four targets. Cmd+Shift+T produces Quartz CGEvent flag mask + Carbon keycode on macOS, lowercase xdotool key argument on X11, Linux input event-code sequence on Wayland, Win32 VK codes for SendInput on Windows. Modifier aliases collapse cross-platform (Cmd/Win/Super/Meta → same canonical token), so a model trained against one OS's idiom routes correctly on the others.
• tools/computer_use_macos.py — Quartz CGEventCreateMouseEvent / CGEventCreateKeyboardEvent for input, screencapture CLI for capture (always present on macOS), CGWindowListCopyWindowInfo for active-window queries. Only new dependency: pyobjc-framework-Quartz.
• tools/computer_use_linux.py — runtime detection of X11 vs Wayland from \$WAYLAND_DISPLAY / \$XDG_SESSION_TYPE. X11 path uses xdotool + scrot / ImageMagick import. Wayland path uses ydotool + grim (wlroots) / gnome-screenshot / spectacle. Active-window queries: Sway IPC / hyprctl on Wayland, xdotool getactivewindow on X11.
• tools/computer_use_windows.py — ctypes wrapper over user32.SendInput (modern path; legacy keybd_event / mouse_event avoided). Per-monitor v2 DPI awareness set on import so click coordinates aren't scaled. Screenshot via mss if installed, falls back to a built-in ctypes BitBlt path so the tool works on a default Python install with no extra deps.

All three OS tools call registry.register() at module top so the AST tool-discovery picks them up. Each tool's check_fn returns False off the matching platform / when HERMES_COMPUTER_USE_ENABLED is unset / when required deps are missing — so on any given host the model only sees the one tool it can actually use, never the other two.

Skills

Per-OS skill teaches the model what's actually different on each host:

• skills/computer-use/common/SKILL.md — when to reach for computer_use_* at all (vs browser_tool / terminal), screenshot-first discipline, cost discipline (token budget on base64 PNGs), redact_regions usage.
• skills/computer-use/macos/SKILL.md — Cmd-vs-Ctrl, Spotlight (Cmd+Space) idiom, Mission Control, Accessibility + Screen Recording permission setup, multi-monitor caveats, things that won't work (Touch ID prompts, system password dialogs).
• skills/computer-use/linux/SKILL.md — X11 vs Wayland detection, ydotoold + uinput setup, DE-specific shortcuts (GNOME / KDE / wlroots / Xfce), polkit / pkexec lockout warnings.
• skills/computer-use/windows/SKILL.md — Win+S as Spotlight analogue, UAC / UIPI restrictions on synthetic input to elevated windows, DPI awareness, mss vs BitBlt screenshot tradeoff.

Validation

Unit tests: 56 / 56 passing (tests/tools/test_computer_use.py). Mocked Quartz, subprocess, and ctypes user32. Coverage:

• Common: schema parse, validation rejection paths, screen-bounds enforcement.
• Grammar: round-trip parsing, all four backend targets, modifier aliases, F1-F24 mapping, unknown-token rejection.
• Safety: env-gate truthy/falsy values, kill-switch lifecycle, redact_image (PIL pixel-level assertion), JSONL log writes.
• macOS: every action path against a stubbed Quartz, including screenshot-b64 round-trip and CGEvent flag emission.
• Linux: X11 path (xdotool + scrot, xdpyinfo screen-size parsing) and Wayland path (ydotool + grim, wlr-randr screen-size).
• Windows: SendInput call counts, screen-size, off-screen click rejection.

macOS integration on the author's MacBook (live, real Quartz, real screencapture): 11 / 11 cases pass —
```
✓ screen_size → {1470, 956}
✓ cursor_position → real cursor
✓ get_active_window → frontmost app correctly identified
✓ screenshot full → 575,589-byte real PNG round-trips through base64
✓ screenshot region → 6,171-byte cropped PNG (200×200)
✓ screenshot redact → redacted PNG (PIL fills rectangles black)
✓ wait → 50ms sleep returns clean result
✗ off-screen click (rejected) → validation error with screen bounds
✗ type empty (rejected) → validation error with field name
✗ unknown action (rejected) → validation error listing valid actions
✗ env-off refusal → 'refused: HERMES_COMPUTER_USE_ENABLED is not set'
```
Action log JSONL writes verified.

Linux + Windows: unit-test-only. Author has no Linux or Windows host immediately available for live integration validation. Mocked-subprocess and mocked-ctypes coverage exercises every code path, but I want to be honest that real-host validation hasn't happened yet. Happy to run live tests once a tester is available, or for maintainers to gate landing the Linux/Windows files on community validation.

Safety posture

• Default-off env gate. `HERMES_COMPUTER_USE_ENABLED=true` required. Default behaviour: every action returns `{success: false, error: 'refused: …'}`. The operator opts in explicitly.
• Action allowlist + per-action validation. Off-screen coordinates rejected before reaching the OS. Type strings capped at 10K chars. Wait values capped at 30s. Scroll amounts capped at 50 ticks. Unknown actions rejected with a list of valid actions in the error.
• Process-global kill-switch. `set_kill_switch()` engages a flag checked before every action. Once engaged, subsequent actions refuse until `clear_kill_switch()` or gateway restart. OS backends that can register a global hotkey can wire it to set the flag; that's an obvious follow-up.
• JSONL audit log. Every action attempt (including refusals) is logged to `$HERMES_HOME/logs/computer_use.jsonl` with timestamp, action name, params (screenshot bytes stripped), success bit, error if any. Browsable post-incident.
• Screenshot redaction. `screenshot` action accepts `redact_regions=[[x1, y1, x2, y2], …]` to blank rectangles before the image reaches the model. Useful for hiding password fields, MFA codes, or sensitive UI zones.

Test plan

• Reviewer runs the unit suite locally: `python -m unittest tests.tools.test_computer_use`
• On macOS with `pyobjc-framework-Quartz` installed and `HERMES_COMPUTER_USE_ENABLED=true`, confirm tool registers and `screen_size` returns the host's resolution
• On Linux with `xdotool` + `scrot` installed (X11 session), confirm registration and `screen_size`
• On Linux Wayland with `ydotool` + `grim` installed and ydotoold running, confirm registration and `screen_size`
• On Windows confirm registration and `screen_size` with default Python install (BitBlt fallback)
• Verify the env gate refuses every action when unset
• Verify the off-screen coordinate rejection on at least one host

🤖 Generated with Claude Code

[teknium1]

Tagging @f-trycua @ddupont808 @jamesmurdza for context — this PR proposes native per-OS desktop backends (macOS / Linux / Windows) inside Hermes, parallel to the cua-driver path we already integrate with (issue #24015, recently expanded for auxiliary.vision routing in #30126, and the bug-bundle cleanup in #24170).

Two questions where your perspective would be useful before we decide direction here:

1. Windows + Linux roadmap on cua-driver. Is upstream cua-driver planning Windows/Linux backends, and on what timeline? If yes, we'd rather hold for those and keep one backend abstraction in Hermes rather than ship two parallel desktop-control surfaces.

2. The Hermes-internal native Windows backend in this PR (tools/computer_use_windows.py — ctypes over user32.SendInput, per-monitor v2 DPI, mss + BitBlt fallback). If cua-driver is going that direction independently, would the trycua team be open to absorbing this work as a cua-driver-Windows starting point so the abstraction stays unified? Or do you see independent value in a non-cua-driver Windows path?

No urgency — just want to make sure we don't fork the desktop-control surface unnecessarily. Happy to wait on your read before this lands.

[f-trycua]

Thanks for the tag, @teknium1.

Windows cua-driver has just shipped, pending some further tests in Hermes. Linux is still alpha.

cua-driver-rs v0.2.18 (cross-platform Rust port) is Windows-feature-complete — every action tool marked VERIFIED in PARITY.md, including UWP apps (Calculator, Settings, etc.). Linux is in tree but most rows are still OPEN; we'd suggest holding for VERIFIED there before Hermes flips it on.

Drafted what unified-backend looks like: #30660 - ~43 LOC across 3 files, widens the sys.platform != "darwin" gate to allow Windows. The existing cua_backend.py MCP stdio integration works identically on Windows because cua-driver's tool surface is uniform. Linux stays gated until cua-driver-rs Linux goes VERIFIED upstream (one-line flip when that happens).

Re: absorbing the Python computer_use_windows.py - probably does not make sense as a starting point; the Rust impl in cua-driver-rs/crates/platform-windows is weeks ahead (Chromium/Electron click routing, UWP CoreWindow fallback, UIA cache batching, autostart at RunLevel=Highest). But @Abd0r's safety primitives are great and worth absorbing into cua-driver as follow-up: kill-switch, JSONL audit log, modifier-aliasing grammar (Cmd/Win/Super/Meta → canonical), screenshot redact_regions for password/MFA blanking.

Thanks @Abd0r for the careful per-OS work!

cc @ddupont808

[teknium1]

Superseded by #50552 — a comprehensive cross-platform cua-driver implementation (macOS/Windows/Linux) now merged to main. It covers the platform this PR targeted. Thanks for the contribution; closing as superseded.