NousResearch · Abd0r · May 15, 2026
diff --git a/skills/computer-use/common/SKILL.md b/skills/computer-use/common/SKILL.md
@@ -0,0 +1,90 @@
+---
+name: computer-use
+description: When and how to use the native desktop computer_use_* tool — screenshot first, click by absolute pixel, never reach for it when browser_tool or terminal will do.
+metadata:
+  hermes:
+    tags: [desktop, mouse, keyboard, click, screenshot, gui, automation, computer-use]
+---
+
+# Native desktop control (`computer_use_*`)
+
+You have a desktop-control tool that takes screenshots, clicks, types, and sends key combinations on the host machine. The exact tool name depends on the OS the agent is running on:
+
+* `computer_use_macos` — when the host is macOS
+* `computer_use_linux` — when the host is Linux (X11 or Wayland)
+* `computer_use_windows` — when the host is Windows
+
+Only one of these is registered per session — whichever matches the host. Don't try to call a different one. The tool surface (parameters, action names, return shape) is identical across the three; only the OS-specific shortcuts and idioms differ — see the per-OS skill for that.
+
+## When to use it
+
+Reach for `computer_use_*` only when **simpler tools genuinely can't do the job**:
+
+- `browser_tool` / `browser_camofox` — already covers any web workflow. Don't drive a browser through screenshots when CDP gives you the DOM.
+- `terminal` — covers anything a CLI can do. Don't click through a GUI installer when a `brew install` / `apt install` / `winget install` line exists.
+- `file_*` tools — for reading/writing files on disk.
+
+`computer_use_*` is the right tool when:
+
+- The target is a **native desktop app** with no usable CLI or web surface (Adobe apps, Office desktop, native installers, system settings panels).
+- You need to **interact with a popup, modal, or system dialog** that lives outside any controllable surface.
+- The user explicitly asks you to "click", "open this app", "drag", "use the GUI".
+- A vision-on-dense-UI workflow benefits from screenshot grounding (proofreading a slide layout, validating a render in a 3D app).
+
+## When NOT to use it
+
+- Web tasks → use `browser_tool` instead. Faster, more reliable, no screenshot ambiguity.
+- Anything scriptable via shell → use `terminal`.
+- File reads / edits → use `file_*`.
+- Anything touching credentials, password fields, MFA codes, or banking UIs unless the user has explicitly asked. Even then: prefer not to. Logs of synthetic clicks near sensitive UI elements are a footgun.
+
+## The screenshot-first discipline
+
+Every desktop-control session starts the same way:
+
+1. **Screenshot first.** Always. You don't know what's on the screen until you look. A click at coordinates you guessed from "well the button is usually in the top-right" lands somewhere wrong about half the time.
+2. **Identify the target visually** in the screenshot. Note its approximate pixel position.
+3. **Take the action** at those absolute coordinates.
+4. **Screenshot again** after any non-trivial action (window opened, dialog appeared, focus changed) to confirm the world looks like you expected.
+5. **Adjust or recover** if it doesn't.
+
+This is slow. That's the price. Skipping it is how computer-use agents go off the rails.
+
+## Coordinates
+
+Coordinates are **absolute screen pixels**, origin at top-left. On HiDPI / Retina displays, the tool already runs in DPI-aware mode; the numbers you see in screenshots are the numbers you click. No scaling math.
+
+## Cost discipline
+
+- A screenshot costs an action and adds a base64 PNG (often >100KB) to the next turn's context. Don't take six in a row.
+- A `wait` between actions is sometimes necessary (window opens, network roundtrip, animation completes) but each `wait` costs latency too. Use `ms: 200` or `500`, not `5000`.
+- If a target app needs many clicks to do something a CLI command would do in one line, **switch to terminal** mid-task. There's no shame.
+
+## Safety
+
+The tool is **opt-in by env var**: `HERMES_COMPUTER_USE_ENABLED=true` must be set on the host. If it isn't, every call returns a refusal — that's by design, not a bug. Tell the user how to enable it; don't try to enable it yourself.
+
+A `redact_regions` parameter on `screenshot` lets you blank rectangles (e.g. password manager popup, MFA code) before the image reaches the model. Use it when you can identify a sensitive zone in advance.
+
+Every action attempt is logged to `$HERMES_HOME/logs/computer_use.jsonl`. If you do something the user doesn't expect, the log is the audit trail.
+
+## Reading the result
+
+Every action returns a JSON dict. Useful fields:
+
+- `success` — bool. Always check this; non-zero `error` means the action didn't happen.
+- `screenshot_b64` — base64 PNG (only on `screenshot`).
+- `cursor` — `{x, y}` after the action.
+- `screen` — `{width, height}` of the primary display.
+- `active_window` — `{app, title}` (best-effort; some Wayland compositors return empty).
+- `error` — a string if the action failed; surface it to the user, don't silently retry.
+
+## Per-OS skills
+
+For platform-specific shortcuts, screenshot tooling, and gotchas (Cmd-vs-Ctrl, X11-vs-Wayland, UAC dialogs, accessibility permissions), load the skill that matches the host:
+
+- `computer-use-macos`
+- `computer-use-linux`
+- `computer-use-windows`
+
+You typically load the per-OS skill once at the start of a desktop-control task and follow this common skill for the discipline.
diff --git a/skills/computer-use/linux/SKILL.md b/skills/computer-use/linux/SKILL.md
@@ -0,0 +1,123 @@
+---
+name: computer-use-linux
+description: Linux X11 vs Wayland gotchas, xdotool/ydotool/grim setup, and DE-specific shortcuts for computer_use_linux.
+metadata:
+  hermes:
+    tags: [linux, x11, wayland, xdotool, ydotool, grim, gnome, kde, computer-use]
+---
+
+# Linux desktop control — what's different
+
+You have access to `computer_use_linux`. Linux is the awkward one because it's two display servers in a trench coat. Same agent-facing API, two completely different toolchains under the hood.
+
+## Detecting the active session
+
+The tool detects the display server automatically each call from `WAYLAND_DISPLAY` and `XDG_SESSION_TYPE`. You don't choose. You can confirm what the host is using with a short terminal call:
+
+```
+echo "$XDG_SESSION_TYPE"  # x11 or wayland
+```
+
+If the user can choose, **X11 is more capable** for scripted automation right now. Wayland's app-isolation model deliberately makes synthetic input and full-screen capture harder. If you have the choice and the workflow is automation-heavy, suggest the user log in to an X11 session.
+
+## One-time host setup
+
+### X11 path (preferred when available)
+
+The host needs `xdotool` and a screenshot tool — usually `scrot`, sometimes `imagemagick` (which provides `import`). On Debian/Ubuntu/Mint:
+
+```
+sudo apt install xdotool scrot xdpyinfo xprop
+```
+
+On Arch/Manjaro:
+
+```
+sudo pacman -S xdotool scrot xorg-xdpyinfo xorg-xprop
+```
+
+After install everything works without further setup; xdotool synthesises events through XTEST.
+
+### Wayland path (when X11 isn't available)
+
+Wayland needs `ydotool` (input via `/dev/uinput`) and one of `grim` (wlroots compositors: Sway, Hyprland, labwc, river), `gnome-screenshot` (GNOME), or `spectacle` (KDE).
+
+```
+# Sway / Hyprland / labwc
+sudo apt install ydotool grim
+sudo usermod -aG input "$USER"  # for /dev/uinput access
+sudo systemctl enable --now ydotoold
+
+# GNOME on Wayland
+sudo apt install ydotool gnome-screenshot
+sudo systemctl enable --now ydotoold
+
+# KDE on Wayland
+sudo apt install ydotool kde-spectacle
+sudo systemctl enable --now ydotoold
+```
+
+The `usermod` change requires re-login to take effect. If the operator hasn't done this, every input action will fail with a permission error on `/dev/uinput`.
+
+## Modifier keys
+
+Linux uses **Ctrl** for the same shortcuts macOS uses **Cmd** for. The grammar parser accepts `Ctrl+T`, `ctrl+t`, `control+t` — all equivalent. The macOS `Cmd` token is reinterpreted as the **Super** (Windows) key on Linux, which is what the user wants in practice when porting muscle-memory.
+
+| Action | Combo |
+|---|---|
+| New tab / new window / save / close | `Ctrl+T` / `Ctrl+N` / `Ctrl+S` / `Ctrl+W` |
+| Cut / copy / paste / undo | `Ctrl+X` / `Ctrl+C` / `Ctrl+V` / `Ctrl+Z` |
+| Find | `Ctrl+F` |
+| Switch window (most DEs) | `Alt+Tab` |
+| Open terminal in many DEs | `Ctrl+Alt+T` |
+| Lock screen (GNOME / KDE) | `Super+L` / `Ctrl+Alt+L` |
+| Activities / overview (GNOME) | `Super` |
+| App launcher (KDE) | `Alt+F1` |
+
+In a terminal **Ctrl+C is interrupt**, not copy. Use `Ctrl+Shift+C` / `Ctrl+Shift+V` for clipboard inside terminal emulators.
+
+## Active-window queries
+
+- **X11**: `get_active_window` returns `{id, title, app}` derived from `xdotool getactivewindow` and `xprop WM_CLASS`. Reliable.
+- **Wayland (Sway)**: returns the `app_id` and window name from the i3 IPC tree. Reliable.
+- **Wayland (Hyprland)**: returns from `hyprctl -j activewindow`. Reliable.
+- **Wayland (GNOME)**: there is no public IPC for this. Returns empty `{}`. Don't depend on it.
+- **Wayland (KDE)**: best-effort via KWin scripting; often empty.
+
+## Screenshot quirks
+
+- **X11**: `scrot` returns the full root window — works on multi-monitor setups, captures everything.
+- **Wayland (wlroots / `grim`)**: full virtual desktop including all outputs.
+- **Wayland (GNOME / `gnome-screenshot`)**: full screen of the focused monitor; multi-monitor capture is a known gap.
+- **Wayland (KDE / `spectacle`)**: full screen of all outputs.
+
+`gnome-screenshot` produces a flash + shutter sound by default. There's no reliable way to suppress it from the API; warn the user once if it bothers them.
+
+## DE-specific things to know
+
+| DE | Distinctive feature | Watch out for |
+|---|---|---|
+| GNOME (Wayland) | Activities overview opens with `Super` | No window-position queries; `move`/`resize` programmatic control is limited |
+| KDE Plasma | Most flexible; rich KWin scripting | Spectacle screenshot is async — add `wait 300` after triggering |
+| Sway / Hyprland / labwc (wlroots) | Best Wayland tooling | Tiling — clicks at fixed coordinates may target wrong window if user resizes |
+| Xfce / MATE / Cinnamon (X11) | Just works with xdotool | None significant |
+| Unity / Pantheon | X11 — works but DE-specific shortcuts vary | Some shortcuts are DE-overridden |
+
+## Don't try to do these
+
+- **Sudo password prompts** in graphical password dialogs (polkit / pkexec) — the focus locks out synthetic input as a security measure. Use `sudo` over terminal instead, or ask the user.
+- **Wayland security keyrings** (gnome-keyring unlock prompt) — same restriction.
+- **VirtualKeyboard / OSK input** — these run in compositor-privileged space.
+
+## Example — open a terminal, run a command, screenshot
+
+```
+{"action": "key", "keys": "Ctrl+Alt+T"}
+{"action": "wait", "ms": 500}
+{"action": "type", "text": "uname -a"}
+{"action": "key", "keys": "Return"}
+{"action": "wait", "ms": 200}
+{"action": "screenshot"}
+```
+
+For most CLI workflows just use the `terminal` tool — it's faster than driving a GUI terminal through screenshots.
diff --git a/skills/computer-use/macos/SKILL.md b/skills/computer-use/macos/SKILL.md
@@ -0,0 +1,88 @@
+---
+name: computer-use-macos
+description: macOS-specific shortcuts, screenshot tool, and accessibility/screen-recording permission setup for computer_use_macos.
+metadata:
+  hermes:
+    tags: [macos, desktop, mouse, keyboard, accessibility, screen-recording, computer-use]
+---
+
+# macOS desktop control — what's different
+
+You have access to `computer_use_macos`. The action set and parameter shape are documented in the parent `computer-use` skill — load that first if you haven't. This skill covers macOS-only things you must know.
+
+## One-time host setup (the user does this, not you)
+
+Before `computer_use_macos` works at all, the operator needs to grant **two** permissions in *System Settings → Privacy & Security*:
+
+1. **Accessibility** — required for synthetic mouse and keyboard events to reach other apps. Without this, `CGEventPost` returns success but nothing actually happens — the events are silently dropped at the WindowServer.
+2. **Screen Recording** — required for `screencapture` to include other apps' windows. Without this, screenshots show the desktop background and your own app's windows only — every other window is rendered as wallpaper, which is misleading rather than blank.
+
+Both prompts appear automatically the first time the tool runs. Tell the user this once at the start of a session if it looks like the permissions aren't granted (you can detect this by an action that "succeeds" but the screenshot doesn't reflect the click).
+
+The Hermes process needs both permissions — toggling them on is a per-binary grant, so if the user runs Hermes from a virtualenv vs system Python they'll need to grant it for whichever they're using.
+
+## Modifier keys
+
+macOS uses **Cmd** where Linux/Windows use **Ctrl** for almost every shortcut. The grammar parser accepts both `Cmd+...` (canonical for macOS) and `cmd+...`. Common combinations:
+
+| Action | Combo |
+|---|---|
+| New / open / save / close window | `Cmd+N` / `Cmd+O` / `Cmd+S` / `Cmd+W` |
+| Cut / copy / paste / undo | `Cmd+X` / `Cmd+C` / `Cmd+V` / `Cmd+Z` |
+| Find / find-next | `Cmd+F` / `Cmd+G` |
+| Quit app | `Cmd+Q` |
+| Switch app | `Cmd+Tab` |
+| Switch window within app | `Cmd+~` |
+| Spotlight | `Cmd+Space` |
+| Mission Control | `Ctrl+Up` |
+| Force quit | `Cmd+Option+Esc` |
+| Full screenshot to clipboard | `Cmd+Shift+Ctrl+3` (rarely needed; use the tool's `screenshot` action) |
+
+Don't use `Ctrl+...` for app shortcuts on macOS unless the app is a Linux/Windows port that documented the Ctrl form (some IDEs do this, e.g. Cursor).
+
+## Spotlight is your friend
+
+To open any app reliably:
+
+1. `key Cmd+Space` — opens Spotlight.
+2. `wait 200` — let it focus.
+3. `type <app name>` — narrow the result.
+4. `wait 100` — let Spotlight resolve.
+5. `key Return` — launch the top hit.
+
+This works whether or not the app is in the Dock and is much more reliable than clicking the Dock or hunting in Finder.
+
+## Active-window queries
+
+`get_active_window` returns `{app, title}` on macOS — the frontmost on-screen application name and the window title (when available). Some apps (especially Electron) don't expose a window title; expect empty strings sometimes.
+
+## Screenshot quirks
+
+- `screencapture` captures the full primary display. On a multi-monitor Mac you'll see only the primary; we don't currently expose multi-display capture.
+- HiDPI / Retina screens return native-pixel screenshots (e.g. 3024×1964 on a 14" MacBook Pro). The pixel coordinates you click are in this same native space — no scaling.
+- The first call shows a system permission prompt; subsequent calls are silent.
+
+## Don't try to do these
+
+- **Mission Control swipes** with `mouse_drag` — three-finger swipe is a trackpad-gesture-only interaction, not synthesisable through CGEvent.
+- **Touch ID / Apple Watch unlock** — system-modal prompts that synthetic clicks can't pass through.
+- **Quartz screen rotation / display arrangement** — those panes in System Settings have a UIPI-like restriction; ask the user instead.
+- **Anything that requires admin privilege escalation** — the standard sudo/Authorization Services prompt won't accept synthetic password input.
+
+## Example — open Safari, navigate to a URL, screenshot
+
+```
+{"action": "key", "keys": "Cmd+Space"}
+{"action": "wait", "ms": 200}
+{"action": "type", "text": "Safari"}
+{"action": "wait", "ms": 150}
+{"action": "key", "keys": "Return"}
+{"action": "wait", "ms": 800}
+{"action": "key", "keys": "Cmd+L"}
+{"action": "type", "text": "https://example.com"}
+{"action": "key", "keys": "Return"}
+{"action": "wait", "ms": 1500}
+{"action": "screenshot"}
+```
+
+(For web tasks `browser_tool` is faster and more reliable — this is just an illustration of the macOS idiom.)