feat(cua-driver): cursor overhaul + Hermes-decoupling MCP surface - #1961
Merged
Conversation
- Fix `mcp-config --client claude` command shape across tutorials and recipes so it matches what the CLI actually emits - Add `Using cua-driver,` preamble to example prompts (export-contacts, build-a-report, calculator) - New how-to: personalize the agent cursor — runtime style override, bringing your own SVG/PNG/ICO via `--cursor-icon`, the default render details (gradient body, white outline, 2× rasterize for retina), plus what's intentionally not personalizable today (multi-palette, motion curve shape, the dot-style `cursor_size` field)
Two built-in cursor silhouettes, selectable via `--cursor-shape`:
- `arrow` (default): the procedural gradient diamond drawn from vector
primitives each frame. Sharp at any backing scale because nothing
rasterises. This is the same silhouette as before this PR.
- `teardrop`: the embedded `cursor-up` SVG (upward teardrop with notched
bottom, gradient body, white outline). Rasterised once into a 52 px
RGBA buffer at startup and blitted with a runtime transform.
`arrow` stays the default until the teardrop's retina rasterisation is
fully sorted; users can opt into the SVG with `--cursor-shape teardrop`.
`--cursor-icon <path>` (custom asset) always wins over `--cursor-shape`.
Plumbing in this commit:
- New `BuiltinShape` enum in `cursor-overlay::shape` with parse + default
Arrow; re-exported alongside `CursorShape`.
- `CursorConfig` gains `builtin_shape: BuiltinShape`; `--cursor-shape`
parsed in `CursorConfig::parse`; unknown names warn and fall back.
- `paint_cursor` branches on `(custom shape, builtin)`: custom file →
blit it; else `Arrow` → call `draw_default_arrow`; else `Teardrop` →
blit `CursorShape::teardrop()` (renamed from `cua_brand()`).
- Backing-scale plumbed end-to-end so the teardrop renders at physical
pixels on retina: `paint_cursor` takes `backing_scale: f32`,
platform-macos sources it from `NSScreen.backingScaleFactor` and sets
`CALayer.contentsScale`, platform-{linux,windows} pass `1.0`.
- CLI surface: `--cursor-shape` added to `VALUE_FLAGS` and to the
cursor-overlay help block.
- Docs: `personalize-cursor.mdx` describes both built-ins, the
default-arrow choice, and the runtime knobs.
Runtime switching between `arrow` and `teardrop` via
`set_agent_cursor_style` is intentionally deferred — restart with a
different `--cursor-shape` to swap today.
Lands the MCP-layer changes Hermes' `computer_use` wrapper needs to
decouple from cua-driver internals (Surfaces 4, 6, 7, 8 from
hermes#47072). The integer `element_index` surface is preserved —
`element_token` is purely additive.
- `mcp-config --client claude` emits `--scope user` and drops the dead
`--claude-code-computer-use-compat` flag — fixes the per-project-scope
footgun where tools registered into one repo didn't surface elsewhere
- Machine-readable CLI manifest via `--help --machine-readable`:
subcommand list + arg shapes so consumers don't parse free-form text
- Per-tool `capabilities[]` array on `tools/list` + top-level
`capability_version` on `initialize` — consumers detect support
before calling instead of trying tool names blindly (Surface 4)
- Image-part responses gain explicit `mimeType` so consumers don't
sniff base64 magic bytes (Surface 7)
- `click`: `button` arg accepts `left` / `right` / `middle`; unknown
buttons rejected instead of silently falling through to left
- `get_window_state` emits `structuredContent.elements[]` and a
top-level `snapshot_id` — single canonical shape with a bounded walk
- Opaque `element_token` alongside `element_index` on 7 token-accepting
tools (`click`, `double_click`, `right_click`, `scroll`, `type_text`,
`press_key`, `set_value`): format `s{snapshot_hex}:{index}`,
per-pid LRU cap 8, stale tokens return an explicit "stale" error
(Surface 6)
- `accessibility.element_tokens` capability claimed by all 7
token-accepting tools + `get_window_state` (which emits the tokens)
Includes Linux fixups for cfg-gated paths Agent D couldn't reach from
a macOS host build: `pid: u32 → i32` cast at `register_snapshot` and
`element_index` + `element_token` added to the Linux `press_key`
schema + invoke (resolves the token's `window_id` before XSendEvent).
`type_text` now detects when the focused window is a terminal emulator
(bundle id on macOS, WM_CLASS + process name on Linux, window class on
Windows) and routes past the accessibility-text channel to direct
key-event synthesis. Fixes the silent-drop that otherwise affects
Ghostty / iTerm2 / Terminal.app / Windows Terminal / mintty / GVim and
other consoles where AX-text injection is broken or no-op.
- macOS: `TERMINAL_BUNDLE_IDS` list (10 entries) → CGEvent
`type_text_with_delay`
- Linux: `TERMINAL_WM_CLASS_SUBSTRINGS` + existing process-name list →
tty master injection (`crate::tty::inject_via_master`) or
`send_type_text_xtest` on X11 / virtual-keyboard on Wayland
- Windows: `TERMINAL_CLASS_PREFIXES` (Cascadia hosting class,
`ConsoleWindowClass`, `mintty`, `nvim`, `Vim`) → SendInput +
`KEYEVENTF_UNICODE` via `inject_text_cloaked`
- Structured response: `path: "ax" | "key_events"` + `characters: <n>`
- New capability `input.keyboard.type.terminal_safe`; claimed by
`type_text` only — `type_text_chars` does NOT claim it (Linux
per-char XSendEvent path has no terminal short-circuit; contract is
intentionally narrower)
- Per-platform terminal constants live in
`platform-{macos,linux,windows}/src/terminal.rs` for easy extension
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughWalkthroughIntroduces "Surface 6" opaque per-snapshot element tokens for stable MCP tool addressing, adds configurable bounded tree walking with Changescua-driver Surface 6 feature set
Sequence Diagram(s)sequenceDiagram
participant Agent as MCP Agent
participant driver as cua-driver MCP server
participant gws as get_window_state
participant reg as TokenRegistry
participant tool as click / type_text / etc.
Agent->>driver: tools/call get_window_state {pid, max_elements, max_depth}
driver->>gws: invoke
gws->>gws: walk_tree_bounded(pid, window_id, max_elements, max_depth)
gws->>reg: register_snapshot(pid, window_id, element_count)
reg-->>gws: snapshot_id
gws-->>Agent: structuredContent{elements[{element_token, element_index, ...}], snapshot_id}
Agent->>driver: tools/call click {pid, element_token: "s001a:3", button: "right"}
driver->>tool: invoke
tool->>reg: resolve_element_args(pid, None, "s001a:3", None)
reg-->>tool: ResolvedElement::Element{window_id, element_index:3, via_token:true}
tool->>tool: AX element path → effective_action = show_menu
tool-->>Agent: success "right-click element 3"
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Linux visual regression artifactsMatrix jobs now run independently. Download visual artifacts from this workflow run.
|
This was referenced Jun 22, 2026
dominofeng-maker
added a commit
to dominofeng-maker/hermes-agent
that referenced
this pull request
Jul 30, 2026
…token availability (P0-1, P0-2) P0-1: cua-driver's drag tool is coordinates-only — verified against the local v0.8.3 schema (drag is additionalProperties:false with no element/token parameters) and against upstream trycua/cua, where from_element has zero source references in any version. The element-drag path used to send from_element/to_element straight into the driver and die on an obscure schema rejection. It is now schema-gated before dispatch (CuaDriverBackend._session_supports_drag_elements, cached from tools/list at startup): element drags return ok=False with code='drag_elements_unsupported' and an actionable message (re-capture, read bounds, use from_coordinate/to_coordinate, or upgrade cua-driver). from/to_element token attachment stays capability-gated for future drivers that add element-drag support. P0-2: capture now reports tokens_available — on the CaptureResult, as a summary note for tokenless snapshots, and in every response payload (multimodal meta, vision_unavailable and AX text paths) — so the model can tell whether element actions carry stale-element protection. An action carrying element_index without a matching snapshot token is now marked degraded with meta.bare_element_index and a re-capture hint (trycua/cua#1961), but only on token-capable drivers; on older tokenless drivers the capture-level notice covers it without per-action noise. Verified: all computer_use tests green (289/289 incl. 5 new regression tests) plus a live cua-driver 0.8.3 capture (tokens_available=True over a 491-element window) and a blocked element-drag (code=drag_elements_unsupported, zero input events).
x7peeps
added a commit
to x7peeps/hermes-agent
that referenced
this pull request
Aug 9, 2026
…t frames cua-driver builds older than 0.10.0 lack the structuredContent.elements frame (trycua/cua#1961) that Hermes' _parse_elements_from_structured reads (NousResearch#47072 Surface 2). Those builds still 'work' — element-index AXPress clicks resolve through the driver — but bounds are always (0,0,0,0), so pixel-coordinate clicks (UIElement.center()) silently degrade. Users see no error and no path forward. Add a version_gate check to the doctor composite report: * _parse_version_tuple — X.Y.Z / X.Y into a comparable tuple. * _version_gate_check — fail when the parsed version < 0.10.0 with an upgrade hint; pass at/above; skip on unparseable strings (don't mislead on exotic version output). * Wired into _compose_fallback_report after binary_version, so the silent bounds=0 degradation becomes a diagnosable failure. Minimum pinned to 0.10.0 — the contract Hermes' computer-use backend aligns to (0.9/0.10 permission-mode alignment); 0.19.x verified working. 9 new tests: pass/fail at threshold, below, unparseable skip, prerelease parse, version tuple parsing. 31 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four logical groups in this branch:
docs(driver):Fix themcp-config --client claudecommand shape in tutorials + recipes; add a newpersonalize-cursorhow-to.feat(cursor-overlay):Retina-aware rendering +--cursor-shape arrow|teardropselection.arrow(the procedural gradient diamond — the cursor shape before this PR) stays the default until the embeddedteardropSVG's retina rasterisation is fully sorted; users can opt into the SVG with--cursor-shape teardrop.--cursor-icon <path>(custom file) still wins over--cursor-shape. Backing-scale plumbed end-to-end so macOS renders at physical pixels viaCALayer.contentsScale.feat(cua-driver):MCP surface needed by Hermes'computer_usewrapper to decouple from cua-driver internals (hermes#47072 Surfaces 4, 6, 7, 8) —mcp-config --client claudeemits--scope user, machine-readable CLI manifest via--help --machine-readable, per-toolcapabilities[]ontools/list,capability_versiononinitialize, explicitmimeTypeon image-part responses,click.buttonenum (left/right/middle),get_window_state.structuredContent.elements[]with bounded walk, opaqueelement_tokenalongsideelement_indexon 7 token-accepting tools (click, double_click, right_click, scroll, type_text, press_key, set_value),accessibility.element_tokenscapability claim.feat(cua-driver):type_textfalls back to direct key-event synthesis when the target is a terminal emulator (Ghostty / iTerm2 / Terminal.app / Windows Terminal / mintty / GVim, etc.) — bypasses the silent-drop affecting AX-text injection on those consoles. New capabilityinput.keyboard.type.terminal_safe;type_text_charsdeliberately doesn't claim it.The integer
element_indexsurface is preserved —element_tokenis purely additive. NoCAPABILITY_VERSIONbump (additive only).Test plan
cargo test --workspace --lib --exclude platform-macos— all lib tests green (cua-driver-core, cursor-overlay, platform-linux, platform-windows).cargo test -p cua-driver --tests— only the pre-existingmcp_protocol_testfailures present on the base commit (verified on base by checking out and rerunning).cargo build --workspace+cargo test --workspace --lib --exclude platform-macos --exclude platform-windows+cargo test -p cua-driver --tests— green.cargo test -p cua-driver-core -p platform-windows --lib— 56 platform-windows lib tests + cua-driver-core green.cargo test -p cua-driver --tests— 5 pre-existing fails on base, 0 new.--cursor-shape arrow(default) and--cursor-shape teardrop. Cross-platform cursor parity confirmed.cua-driver mcp-config --client claudeand confirmtools/listcarries the new fields (capabilities[],element_token,mimeType,buttonenum).Notes for reviewers
main.element_tokenformat:s{snapshot_hex}:{index}(8–12 chars), per-pid LRU cap 8. Stale tokens return an explicitstaleerror rather than silently mis-resolving.--cursor-shapedefault is intentionallyarrowand notteardrop— the SVG path still has retina-rasterisation work pending. See thepersonalize-cursor.mdxpage for the rationale.arrowandteardropviaset_agent_cursor_styleis NOT in this PR — CLI-only today. Can be added in a follow-up.Summary by CodeRabbit
New Features
Documentation
Tests