Add Desktop-managed Computer Use bridge

[DanBennettUK]

Summary

• Add an authenticated Computer Use bridge backend and CLI so remote Hermes backends can route Computer Use calls to a local machine safely.
• Add a Desktop-managed reverse WebSocket path: Desktop starts a loopback-only local Computer Use bridge sidecar, authenticates it with an in-process token, and connects outbound to the remote backend.
• Add Desktop remote connection config/UI for the Local Computer Use bridge, enabled by default for remote backends unless explicitly disabled.
• Update toolset/status handling so Desktop remote sessions expose computer_use when the bridge is connected.
• Document Desktop-managed remote Computer Use routing and keep the manual HTTP bridge as the advanced/server path.

Safety / behaviour

• Local bridge binds to 127.0.0.1 by default.
• Bridge token is generated and kept in process state; it is not logged or displayed.
• Non-loopback HTTP bridge binds still require explicit opt-in.
• Direct dangerous key/type patterns remain blocked by the bridge layer.
• Desktop cleanup tears down the sidecar/reverse WebSocket on reset/quit.

Verification

Server/local checks:

• python -m py_compile tools/computer_use/desktop_bridge.py tools/computer_use/bridge.py tools/computer_use/tool.py hermes_cli/web_server.py tui_gateway/server.py tests/test_web_server.py tests/computer_use/test_desktop_bridge_backend.py
• python -m pytest tests/computer_use/test_bridge_backend.py tests/computer_use/test_desktop_bridge_backend.py tests/tools/test_computer_use.py tests/test_web_server.py -o 'addopts=' -q → 200 passed
• python -m ruff check tests/test_web_server.py tests/computer_use/test_desktop_bridge_backend.py tools/computer_use/desktop_bridge.py tools/computer_use/bridge.py tools/computer_use/tool.py hermes_cli/web_server.py tui_gateway/server.py → pass
• git diff --check → pass
• cd apps/desktop && npm run typecheck -- --noEmit → pass
• cd apps/desktop && npx tsx --test electron/connection-config.test.ts → 53 passed
• cd apps/desktop && npm run lint → 0 errors with existing warnings
• cd apps/desktop && npm run build → pass

Mac / live E2E:

• Synced the branch to /tmp/hermes-desktop-remote-tool-locality on DBENNETT-O3.
• Ran focused Mac pytest for the bridge/status tests → 5 passed.
• Started a temporary remote backend on 127.0.0.1:9139 with HERMES_COMPUTER_USE_BACKEND=desktop-bridge.
• Launched Desktop from the Mac temp checkout with isolated temp HERMES_HOME / user-data.
• Verified Desktop started its loopback Computer Use sidecar and opened the reverse WebSocket to the remote backend.
• Verified /api/tools/computer-use/status on the remote backend reported the local Mac path:
  • platform: darwin
  • ready: true
  • bridge.connected: true
  • cua-driver 0.5.1
  • accessibility/screen recording/capturable all true
• No clicks, typing, scrolling, focus, or window changes were performed.
• Cleaned temporary 9139 backend, Mac Electron process, bridge sidecar, and SSH tunnel; normal 9119 returned HTTP 200 on both server and Mac.

Notes

The live E2E found and fixed an event-loop deadlock in the status route: FastAPI status handlers must use the async broker request path, while off-loop tool calls keep the synchronous path.

[tonydwb]

Code Review Summary

Verdict: Comment (high surface area)

Overview

• Multiple files, +2396/-41 lines
• Adds Desktop-managed Computer Use bridge
• Substantial new feature

Note

• Large diff — recommend human deep-dive
• Security surface area: computer use bridge handles external commands

---

Reviewed by Hermes Agent

[teknium1]

Thanks for addressing the real remote-tool-locality gap; current main still runs Computer Use on the backend host (tools/computer_use/tool.py:151-154).

Problems

• tools/computer_use/desktop_bridge.py:50-54,75-80,146-178 maintains one global Desktop WebSocket and replaces it on a later connection. tools/computer_use/tool.py:151-169 then selects that connected bridge globally. The bridge route authenticates a connector but does not retain its principal, although ticket consumption has one (hermes_cli/dashboard_auth/ws_tickets.py:62-99). This can route another session's Computer Use calls to the newest Desktop.
• website/docs/reference/environment-variables.md:517-521 makes non-secret behavior user-configurable through new HERMES_* variables, contrary to the config.yaml policy.

Suggested changes

• Scope bridge registration and lookup to the authenticated principal plus the agent/session/profile context; reject unmatched calls. Add a two-principal isolation test.
• Move backend mode, bridge URL, timeout, and enablement to config/UI; keep only the bearer token in .env.

Automated hermes-sweeper review.

[DanBennettUK]

Re-scoped this PR to the two review concerns and updated it against current main in 0ee631fa2:

• Desktop Computer Use bridge selection is now fail-closed and scoped by verified provider/principal/profile through bridge registration, live execution context, delegation/background/branch paths, and connector lifecycle.
• Behavioural Computer Use settings moved to structured config/Desktop state; only the bridge bearer token remains environment-backed.

I deliberately removed the broader public-OAuth/session policy, generic command-policy work, and unrelated primary-startup lifecycle hardening from this revision. Those are preserved separately for follow-up review rather than smuggled into this feature fix.

Validation: targeted Python bridge/auth/propagation suites, Electron config/lifecycle suites, ESLint/Prettier, focused TS/esbuild checks, and git diff --check passed. Full Desktop typecheck remains dependency-blocked in this isolated worktree.

[fleps]

Now with the HUD feature, this PR is a must.
+1

[hwangsyh]

+1 — this is exactly the missing piece for the common “Hermes Desktop on a Mac, agent brain on a remote Docker/NAS backend” setup.

Live repro (today, without this PR)

• Desktop: macOS M4, Hermes Desktop connected to a remote backend
• Backend: Linux Docker Hermes Agent v0.20.0 (2026.8.3) (HERMES_SESSION_SOURCE=desktop)
• computer_use is enabled in platform_toolsets.cli, but check_computer_use_requirements() is False on the container (no host display / no cua-driver on the agent host)
• Tool never appears in the session; enabling tools + restarting Desktop doesn’t help
• read_window_below reaches the Mac client (platform: darwin) but still fails window enumeration separately
• Workaround today: SSH from the NAS container to the Mac and AppleScript-list foreground apps — works, but is not real Computer Use

Why this PR matters

A Desktop-managed reverse bridge (loopback sidecar + outbound WS to remote backend) is the right shape: keep TCC/cua-driver on the Mac where the GUI lives, keep the agent runtime on the always-on host. Manual full-Hermes-on-Mac or SSH hacks shouldn’t be required for the default remote-Desktop topology.

Happy to re-test against a build that includes this once it lands.

[OutThisLife]

Leaving this open — the gap it identifies is real and now confirmed at line level, but the way in has changed.

Two PRs went up today that move it:

#90380 adds a ComputerUseProvider ABC and registry (superseding #61311). The Desktop bridge fits it as a provider registered through ctx.register_computer_use_provider() rather than as an edit to the dispatcher, which answers the sweeper's first finding structurally: a provider is built per session with the session id in hand, so there is no global "newest Desktop wins" registration to route another principal's calls to. Selection is explicit config — the bridge never activates for anyone who did not name it.

#90374 does the part that shouldn't wait for a bridge at all: on a remote gateway the agent is now told that the window it just read is on a machine it cannot click, and the Computer Use settings card names which machine it is describing. That is the reported symptom ("computer_use never appears, enabling it and restarting doesn't help") turned into an explanation instead of silence.

What still needs an answer before the bridge lands, and it is the one teknium raised directly rather than the sweeper: the lite client. The desktop app ships in a build with no agent venv, and a provider that spawns a local Python sidecar has nothing to spawn there. A provider's is_available() is the right place for that — it has to answer honestly on a lite client rather than registering and failing at call time.

The second sweeper finding is a straight fix under #90380: backend mode, bridge URL, timeout, and enablement move to config.yaml (computer_use.provider selects the provider; the rest go under its own section), leaving only the bearer token in .env.

Rebasing 5k lines onto a branch that far ahead is real work, so if you would rather hand it off than carry it, say so and I will re-land it as a provider with your authorship preserved. Either way the core edits should drop out of the diff.

[OutThisLife]

Correcting my last comment: I offered you the choice of carrying this or handing it off. Scratch that — we're taking it forward ourselves on top of #90380, rebuilt as a provider. Your authorship stays on the commits.

Nothing for you to do, and no reflection on the work — the locality gap you identified is real and the line-level confirmation in #90374 came straight out of this PR. It's that the rebase, the principal scoping, the config move, and the lite-client question are ours to answer rather than yours to chase across a branch this far behind.

I'll link the follow-up here when it's up.

[OutThisLife]

Carried this forward myself rather than asking you to rebase — the branch was far enough from current main that a rebase wasn't tractable, and I'd rather own the integration than leave it sitting.

It's now #90423, stacked on the provider seam in #90380. Your commits are in it with authorship intact; the docs, the bridge backend, the reverse-WebSocket broker, and the principal/profile scoping are all yours.

Two things changed in the carry, both because the original was global where the product isn't:

• Session-scoped, not process-scoped. One gateway serves a Desktop client, a phone on Telegram, and a cron tick in the same process, and only the first may drive the user's Mac. The backend files each bridge under the identity its credential carried and computer_use resolves it per call from the connection's own verified scope, failing closed on no match.
• Per remote+profile sockets. The single global connection couldn't serve a profile pool, so the Electron side moved out of main.ts into its own module with per-scope ownership and cancellation generations over a shared sidecar.

Also added: a Desktop with no local agent runtime now says the bridge can't run here instead of leaving the toggle looking on.

Closing this one in favor of #90423 — thanks for building it.