feat(computer-use): launch_app action, urls param, hidden-app capture fix (requires cua-driver ≥ 0.1.0) - #18519
Open
ddupont808 wants to merge 5 commits into
Open
Conversation
Background macOS desktop control via cua-driver MCP — does NOT steal the user's cursor or keyboard focus, works with any tool-capable model. Replaces the Anthropic-native `computer_20251124` approach from the abandoned NousResearch#4562 with a generic OpenAI function-calling schema plus SOM (set-of-mark) captures so Claude, GPT, Gemini, and open models can all drive the desktop via numbered element indices. - `tools/computer_use/` package — swappable ComputerUseBackend ABC + CuaDriverBackend (stdio MCP client to trycua/cua's cua-driver binary). - Universal `computer_use` tool with one schema for all providers. Actions: capture (som/vision/ax), click, double_click, right_click, middle_click, drag, scroll, type, key, wait, list_apps, focus_app. - Multimodal tool-result envelope (`_multimodal=True`, OpenAI-style `content: [text, image_url]` parts) that flows through handle_function_call into the tool message. Anthropic adapter converts into native `tool_result` image blocks; OpenAI-compatible providers get the parts list directly. - Image eviction in convert_messages_to_anthropic: only the 3 most recent screenshots carry real image data; older ones become text placeholders to cap per-turn token cost. - Context compressor image pruning: old multimodal tool results have their image parts stripped instead of being skipped. - Image-aware token estimation: each image counts as a flat 1500 tokens instead of its base64 char length (~1MB would have registered as ~250K tokens before). - COMPUTER_USE_GUIDANCE system-prompt block — injected when the toolset is active. - Session DB persistence strips base64 from multimodal tool messages. - Trajectory saver normalises multimodal messages to text-only. - `hermes tools` post-setup installs cua-driver via the upstream script and prints permission-grant instructions. - CLI approval callback wired so destructive computer_use actions go through the same prompt_toolkit approval dialog as terminal commands. - Hard safety guards at the tool level: blocked type patterns (curl|bash, sudo rm -rf, fork bomb), blocked key combos (empty trash, force delete, lock screen, log out). - Skill `apple/macos-computer-use/SKILL.md` — universal (model-agnostic) workflow guide. - Docs: `user-guide/features/computer-use.md` plus reference catalog entries. 44 new tests in tests/tools/test_computer_use.py covering schema shape (universal, not Anthropic-native), dispatch routing, safety guards, multimodal envelope, Anthropic adapter conversion, screenshot eviction, context compressor pruning, image-aware token estimation, run_agent helpers, and universality guarantees. 469/469 pass across tests/tools/test_computer_use.py + the affected agent/ test suites. - `model_tools.py` provider-gating: the tool is available to every provider. Providers without multi-part tool message support will see text-only tool results (graceful degradation via `text_summary`). - Anthropic server-side `clear_tool_uses_20250919` — deferred; client-side eviction + compressor pruning cover the same cost ceiling without a beta header. - macOS only. cua-driver uses private SkyLight SPIs (SLEventPostToPid, SLPSPostEventRecordTo, _AXObserverAddNotificationAndCheckRemote) that can break on any macOS update. Pin with HERMES_CUA_DRIVER_VERSION. - Requires Accessibility + Screen Recording permissions — the post-setup prints the Settings path. Supersedes PR NousResearch#4562 (pyautogui/Quartz foreground backend, Anthropic- native schema). Credit @0xbyt4 for the original NousResearch#3816 groundwork whose context/eviction/token design is preserved here in generic form.
…ured windows, MIME detection Extends the cua-driver computer-use backend to drive backgrounded macOS windows without stealing keyboard or mouse focus from the foreground app. All changes target the cua-driver MCP backend and the shared dispatcher. ## cua_backend.py **Window-aware capture**: capture() now calls list_windows + get_window_state instead of the removed capture tool. Prefers structuredContent.windows (MCP 2024-11-05+ cua-driver) for zero-parse window enumeration; falls back to regex-parsed text for older builds. Stores the selected (pid, window_id) as sticky context so subsequent action calls do not need a redundant round-trip. **Action routing**: click/scroll/type_text/key all carry the sticky pid (and window_id for element-indexed clicks). type_text routes through type_text_chars (individual key events) rather than AX attribute write -- WebKit AXTextFields reject attribute writes from backgrounded processes. **Key parsing**: _parse_key_combo splits cmd+s-style strings into (key, [modifiers]) and routes to hotkey (modifier present) or press_key (bare key) -- cua-driver actual tool names. **set_value method**: new set_value(value, element) calls the cua-driver set_value MCP tool. For AXPopUpButton / HTML select in a backgrounded Safari, AXPress opens the native macOS popup which closes immediately when the app is non-frontmost; set_value AX-presses the matching child option directly (no menu required, no focus steal). **focus_app**: reimplemented as a pure window-selector (enumerates list_windows, sets sticky pid/window_id) without ever raising the window or stealing focus. **list_apps**: fixed tool name from listApps to list_apps; handles plain-text response via regex when structured data is absent. **Structured-content extraction**: _extract_tool_result now surfaces structuredContent from MCP results, enabling the list_windows window array without text parsing. **Helpers**: _parse_windows_from_text, _parse_elements_from_tree, _split_tree_text, _parse_key_combo extracted as module-level functions. ## schema.py Added set_value to the action enum with a description explaining when to prefer it over click (select/popup elements, sliders, no focus steal). Added value field for set_value payloads. ## tool.py Routed set_value action through _dispatch to backend.set_value. Added set_value to _DESTRUCTIVE_ACTIONS (approval-gated). Fixed MIME-type detection in _capture_response: cua-driver may return JPEG; detect from base64 magic bytes (/9j/ -> image/jpeg, else image/png) rather than hardcoding image/png. ## agent/display.py + run_agent.py Guard _detect_tool_failure and result-preview logic against non-string function_result values: multimodal tool results (dicts with _multimodal=True) are not string-sliceable; treat them as successes and fall back to str() for length/preview.
…r non-Anthropic providers
Tool handlers (e.g. computer_use capture) return a _multimodal envelope
dict when a screenshot is attached. The tool-message builder was passing
this raw dict as the `content` field of role:tool messages, which is an
illegal format — OpenAI-compatible APIs expect a string or a content-parts
list, not a plain Python dict, and would reject it with a 400/422 error.
Fix: unwrap _multimodal results to their `content` list
([{type:text,...},{type:image_url,...}]) in both the parallel and
sequential tool-call paths. The Anthropic adapter already handles content
lists natively; vision-capable OpenAI-compatible servers (mlx-vlm,
GPT-4o, etc.) accept image_url parts in tool messages directly.
Also add a _vision_supported adaptive fallback: on first image-rejection
error ("Only 'text' content type is supported." etc.) the agent strips all
image parts from the message history and retries with text only, so
text-only endpoints degrade gracefully without crashing the session.
Follow-up to NousResearch#15328's vision-unsupported retry branch in run_agent.py. _strip_images_from_messages() previously deleted any message whose content was entirely images. That's fine for synthetic user messages injected for attachment delivery, but it breaks providers for tool-role messages — the paired tool_call_id on the preceding assistant message ends up unmatched, which OpenAI-compatible APIs reject with HTTP 400. Fix: tool-role messages whose content becomes empty are replaced with a plaintext placeholder that preserves the tool_call_id linkage. Only non-tool messages are dropped. Added 10 tests covering the role-alternation invariants + image-type coverage. Image-rejection detector: expanded phrase list (image content not supported / multimodal input / vision input / model does not support image) and gated on 4xx status so transient 5xx errors never get misinterpreted as 'server said no to images'. Detection is documented as best-effort English phrase matching. AUTHOR_MAP: mapped 3820588+ddupont808@users.noreply.github.com to ddupont808 so release notes attribute the salvage correctly.
… fix (cua-driver ≥ 0.1.0)
Requires cua-driver ≥ 0.1.0. The launch_app tool and the consolidated
type_text (AX + CGEvent fallback in one tool, replacing type_text_chars)
both shipped in that release.
**launch_app action** (schema.py, tool.py, backend.py, cua_backend.py)
- New `action='launch_app'` in the computer_use schema with `bundle_id`
and `urls` parameters.
- `urls` is explicitly documented as required for browsers (Safari, Chrome,
Firefox): without a URL the browser process starts but never creates a
window, making subsequent capture/click impossible.
- Added to _DESTRUCTIVE_ACTIONS (goes through approval gate).
- Abstract method on ComputerUseBackend; implemented in CuaDriverBackend
by forwarding to cua-driver's launch_app MCP tool.
- _NoopBackend stub for tests.
**capture() fix for hidden/background apps** (cua_backend.py)
- capture(app='X') was calling list_windows(on_screen_only=True), which
skips apps launched hidden by launch_app. Now calls list_windows({})
(all windows) when an app filter is specified, so hidden apps are
reachable. Falls back to the hidden window when no on-screen window
exists for the target app.
**type_text_chars → type_text** (cua_backend.py)
- type_text_chars was removed in cua-driver v0.1.0 and merged into
type_text (which now tries AX bulk-insert first and falls back to
CGEvent character synthesis automatically). Updated the backend to
call type_text instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
alt-glitch
added
type/feature
Contributor
|
Thanks for the focused launch/background-capture work. The premise still exists on current main: Problems
Suggested changes
Automated hermes-sweeper review. |
teknium1
added
sweeper:risk-compatibility
GottZ
reviewed
Jul 27, 2026
GottZ
left a comment
GottZ
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
Summary
Three PRs reference this complex: #16919 and #16936 carry the same full computer-use foundation but do not address app launching or hidden-window capture, while #18519 adds the targeted launch_app/urls wiring, hidden-app window lookup, and cua-driver type_text update on top of that foundation.
Related pull requests
- #16919 [merged]
related— (+3538/-37) — merged baseline reference: Introduced the universal cua-driver computer_use backend, multimodal result plumbing, approvals, tests, skills, and documentation, but its model-facing schema has no launch_app action and capture(app=...) searches only on-screen windows. - #16936 [closed]
related— (+3538/-37) — superseded duplicate: Its diff is effectively identical to #16919 and remains relevant as the stale source branch whose work was later re-salvaged onto current main via #21967, with authorship preserved; it does not contain the launch/hidden-window fix. - #18519
related— (+3607/-37) — keep open for a current-main-aware port: Relative to the duplicated foundation, the meaningful delta adds launch_app with bundle_id/urls dispatch, queries all windows for explicit app captures, and switches to cua-driver's consolidated type_text action. As noted by the keep_open review on #18519, the premise remains valid, but the branch must reconcile current main's dictionary-returning CuaDriverBackend.launch_app() with the generic ActionResult contract and add model-facing normal/capture_after dispatch plus hidden-window regression coverage before merge.
Duplicates
#16919 and #16936 are effectively identical foundation changes; most of #18519 also duplicates that foundation, but #18519 has a distinct launch_app, URLs, hidden-window capture, and type_text delta.
Suggested consolidation
Merge #18519 only after reducing it to a current-main-aware focused port and satisfying the keep_open review's ActionResult and regression-test requirements. Treat merged #16919 as the historical baseline reference and closed #16936 as superseded by #21967; no additional duplicate PR needs closing.
Cross-PR triage: Reviewed 3 pull requests and 0 issues in this complex. Each diff was read against this issue; Assessment working set: 515 kB of PR diffs, 13 kB of issue/PR text, 2 kB of discussion (2 comments), 1 verify verdict. verdicts reflect diff content, not PR titles. Part of an automated triage batch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends #16936 by bumping to
cua-driver ≥ 0.1.0and adding thelaunch_apptool for launching an app without stealing focusSummary
launch_appaction to thecomputer_usetool, allowing agents to launch macOS apps (including browsers with a URL) in the background without stealing focusurlsparameter — required for browsers (Safari, Chrome, Firefox): without a URL the browser process starts but NSWorkspace never creates a window, so subsequentcaptureandclickcalls will failcapture(app=...)for apps that were launched hidden: now queries all windows (not juston_screen_only) and falls back to the hidden window when no on-screen window matchestype_textcall to use the consolidated tool name from cua-driver ≥ 0.1.0 (the oldtype_text_charsaction was merged intotype_text)launch_appin themacos-computer-useskillRequires cua-driver ≥ 0.1.0
This PR depends on features that shipped in cua-driver v0.1.0:
launch_apptool (new in v0.1.0)type_texttool — the AX path and the CGEvent fallback path were unified in v0.1.0; the oldtype_text_charsaction no longer existsIf cua-driver < 0.1.0 is installed,
launch_appcalls will return an unknown-tool error andtype_textwith the AX path will silently fall through to CGEvent input.Changes
tools/computer_use/schema.pylaunch_appto action enum; addedbundle_idandurlsparameterstools/computer_use/backend.pylaunch_appmethodtools/computer_use/cua_backend.pylaunch_app; fixedcapturehidden-app window lookup; updatedtype_text_chars→type_texttools/computer_use/tool.pylaunch_appto destructive actions; added dispatch case; added_NoopBackendstubskills/apple/macos-computer-use/skill.mdlaunch_appaction and browser URL requirementTest plan
hermes chat --toolsets computer_use -q "launch Safari in the background with urls=['https://example.com'], then capture its window"— Safari window should appear without stealing focuspython3 -m pytest libs/cua-driver/Tests/integration/test_hermes_launch_safari_bg.py -v(in the cua repo, requires cua-driver ≥ 0.1.0 installed andANTHROPIC_API_KEYset)capture(app=...)calls should still work for foreground apps