feat(cua-driver): Dubins-path cursor, window-bbox zoom, variable-speed recording render - #1360
Merged
Conversation
…e-speed render
**Cursor**
- AgentCursorView/AgentCursorRenderer: rewrite agent cursor rendering with
SwiftUI Canvas + Dubins-path motion engine (arc → straight → arc)
- AgentCursor.animate(): always arrive at 45° (upper-left tip), approaching
targets from the lower-right — consistent visual signature on every click
**Recording**
- ToolRegistry: capture CLOCK_UPTIME_RAW before handler.invoke() so the
recorded span brackets the full animation time (t_start_ms_from_session_start)
- RecordingSession: add lastAutoRenderURL + auto-render to recording_rendered.mp4
on stop; embed display_scale_factor in session.json
- TrajectoryLoader: add loadActionSpans() — walks all turn-*/action.json and
extracts ActionSpan{startMs, endMs, windowBounds, clickPoint}; read
displayScaleFactor from session.json
- ActionSpan (new): ClickPoint, FocusWaypoint; ActionSpanGenerator with
padMs=500, fastSpeed=8×, mergeGapMs=5000; per-span focus waypoints for
smooth camera pan on merged spans
**Video post-processing**
- RecordingRenderer: window-bbox zoom (ZoomRegion from windowBounds, letterbox
aspect ratio, 400ms eased); variable-speed PTS remapping (1× inside spans,
8× outside); Metal-backed CIContext for GPU frame processing; falls back to
legacy click-zoom when no action spans present
- SetRecordingTool: report rendered path in stop confirmation message
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
f-trycua
approved these changes
Apr 23, 2026
f-trycua
added a commit
that referenced
this pull request
Apr 23, 2026
* cua-driver v0.1 — initial public release
macOS computer-use driver that speaks the Model Context Protocol
over stdio. Drop-in backend for MCP clients (Claude Code, Cursor) or
as part of the cua-computer-server stack; also usable standalone via
a CLI where every MCP tool is a top-level subcommand.
Designed around a strict no-foreground contract: the user's frontmost
app never changes, the real cursor never warps, and the target never
raises or switches Space. You drive a backgrounded macOS app in one
window while your foreground editor keeps typing in another.
Highlights:
- Element-indexed AX actions that work on hidden, off-Space, or
occluded targets — `get_window_state(pid, window_id)` returns a
per-window AX tree (filtered correctly on multi-window apps),
`click({pid, window_id, element_index})` fires the AX action
without cursor move or focus steal.
- Backgrounded pixel clicks via auth-signed SLEventPostToPid with a
yabai-style focus-without-raise primer — same (x, y) addressing
space as the returned screenshot, supports modifiers and count.
- Three capture modes — `vision` (PNG only; default), `ax` (tree
only, no screen-capture hit), `som` (both).
- Chromium / Electron AX support via the private
`_AXObserverAddNotificationAndCheckRemote` SPI so the tree stays
populated without activating the target.
- Pid-mandatory keyboard — every `press_key` / `type_text` routes
through `CGEvent.postToPid` so keys can't leak into the user's
foreground app.
- Agent-cursor overlay that glides to each target before dispatch,
press-in/ripple on landing, idle-hides. Uniform across AX clicks
and pixel clicks.
- Trajectory recording + replay — per-turn folders with app state,
screenshot, action, click marker. Optional video capture with
zoom-on-click render for demos.
- ScreenCaptureKit screenshots defaulting to a 1568-long-side cap
that matches Anthropic's multimodal input limit, so model-picked
pixel coords match the tool's coordinate space.
See libs/cua-driver/README.md for the full feature list plus
comparison against Codex Computer Use and Claude Computer Use, and
libs/cua-driver/Skills/cua-driver/SKILL.md for the canonical action
loop.
Co-Authored-By: Sarina Li <sarinajin.li@gmail.com>
Co-Authored-By: Dillon DuPont <ddupont@mit.edu>
Co-Authored-By: Claude <noreply@anthropic.com>
* cua-driver: bounds-checked narrowing across tool entry points
Wrap every `Int32(raw...)` / `UInt32(raw...)` conversion in
`Int32(exactly:)` / `UInt32(exactly:)` so malformed MCP input
returns a structured error instead of trapping the daemon.
Addresses CodeRabbit #5 + #6 on PR #1359.
Co-Authored-By: Claude <noreply@anthropic.com>
* cua-driver: daemon signal handling via DispatchSourceSignal
Replaces the POSIX signal handler that called Task.detached (not
async-signal-safe per Apple's Swift concurrency docs) with a
DispatchSource-based pattern. SIGINT/SIGTERM routed through
dispatch sources on a user-initiated queue; SIGPIPE continues to
be SIG_IGN'd so write() returns EPIPE cleanly.
Addresses CodeRabbit #4 on PR #1359.
Co-Authored-By: Claude <noreply@anthropic.com>
* cua-driver: make som the default capture_mode
som returns tree + screenshot, so element_index clicks — the
driver's primary addressing mode — work on the first
get_window_state call without any configuration. vision (PNG only)
stays available as opt-in for vision-first VLM pipelines that
specifically don't want the AX walk.
The previous vision default meant a user who installed the driver
and ran the canonical snapshot-then-click loop ended up on the
pixel-click fallback path instead of the element-indexed primary
path, which made the driver's distinguishing feature invisible out
of the box.
Co-Authored-By: Claude <noreply@anthropic.com>
* cua-driver: bounds-check pid narrowing in ListWindowsTool
The bounds-checked-narrowing sweep caught the 11 tools CodeRabbit
called out but missed `ListWindowsTool.swift`, which had the same
`Int32($0)` trap in its `pidFilter` path (filters the window list
against a caller-supplied pid). A malformed `pid` over Int32.max
would crash the daemon instead of returning a structured error.
Adds a `private static func errorResult` alongside the existing
`summary` helper so the same pattern the other tools use is
available here.
Co-Authored-By: Claude <noreply@anthropic.com>
* cua-driver: v0.1 docs
Adds the cua-driver docs subtree to docs/content/docs/cua-driver/,
matching the lume structure (guide + reference sections, no
examples for v0.1). Registers cua-driver in the root docs
meta.json alongside cua/cuabench/cuabot/lume.
- guide/getting-started/{introduction,installation,quickstart,comparison,faq}.mdx
- reference/{cli-reference,mcp-tools,limits}.mdx
- Fumadocs format: frontmatter + Callout imports matching the
lume pages; no VersionHeader yet (only one shipped version).
Co-Authored-By: Claude <noreply@anthropic.com>
* docs: wire cua-driver section-root redirects
Matches the cuabench / lume pattern: bare `/cua-driver`, the
`/guide` section root, and the `/reference` section root each
redirect to their first content page. Before this, hitting
`/docs/cua-driver` returned a 404 — the docs tree existed but
the section landing redirects in middleware.ts hadn't been
added alongside the new content.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(cursor+recording): Dubins-path cursor, window-bbox zoom, variable-speed render (#1360)
**Cursor**
- AgentCursorView/AgentCursorRenderer: rewrite agent cursor rendering with
SwiftUI Canvas + Dubins-path motion engine (arc → straight → arc)
- AgentCursor.animate(): always arrive at 45° (upper-left tip), approaching
targets from the lower-right — consistent visual signature on every click
**Recording**
- ToolRegistry: capture CLOCK_UPTIME_RAW before handler.invoke() so the
recorded span brackets the full animation time (t_start_ms_from_session_start)
- RecordingSession: add lastAutoRenderURL + auto-render to recording_rendered.mp4
on stop; embed display_scale_factor in session.json
- TrajectoryLoader: add loadActionSpans() — walks all turn-*/action.json and
extracts ActionSpan{startMs, endMs, windowBounds, clickPoint}; read
displayScaleFactor from session.json
- ActionSpan (new): ClickPoint, FocusWaypoint; ActionSpanGenerator with
padMs=500, fastSpeed=8×, mergeGapMs=5000; per-span focus waypoints for
smooth camera pan on merged spans
**Video post-processing**
- RecordingRenderer: window-bbox zoom (ZoomRegion from windowBounds, letterbox
aspect ratio, 400ms eased); variable-speed PTS remapping (1× inside spans,
8× outside); Metal-backed CIContext for GPU frame processing; falls back to
legacy click-zoom when no action spans present
- SetRecordingTool: report rendered path in stop confirmation message
Co-authored-by: cua <cua@cua.localdomain>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs: add cua-driver to the header nav with its icon
Adds cua-driver alongside Cua / Cua Bench / Lume / Cua-Bot in the
docs header nav. Ships Guide + Reference tabs (no Examples in
v0.1). Icon is a koala-astronaut mascot provided by Francesco,
tinted into black and white PNG variants at build time so both
light and dark themes have a readable contrast.
SVG equivalents would be nicer (crisper at high DPI, smaller
payload) — left as a design follow-up.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs: tighten cua-driver header description
"Background computer-use" reads punchier and matches the length
of the other product descriptions in the dropdown (Benchmarking
toolkit, macOS VM CLI and Framework, etc.).
Co-Authored-By: Claude <noreply@anthropic.com>
* cua-driver: fix Swift 6.1 CI build
Restore `nonisolated(unsafe)` on `capturedWriter` in RecordingRenderer;
PR #1360 dropped it and Swift 6.1 (Xcode 16.3) rejects implicit capture
of non-Sendable AVAssetWriter in the @sendable finishWriting closure.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs: fix broken links flagged by link-check CI
Add two entries to .lycheeignore to unbreak the Check External Links
(lychee) job on PR #1359: `https://openai.com/*` (openai.com returns
403 to automated checkers on URLs like /codex but loads fine in a
browser — flagged from docs/content/docs/cua-driver/guide/getting-started/comparison.mdx)
and the self-referential install.sh raw-content URL used by the CLI
reference page (docs/content/docs/cua-driver/reference/cli-reference.mdx),
which 404s only because the install script doesn't exist on main yet
and will resolve once this PR merges. The internal link check failure
(`/cua/guide/get-started/self-hosted-sandboxes` in
docs/content/docs/cua/guide/sandbox/lifecycle.mdx, introduced by
PR #1228) is pre-existing on main and out of scope for this PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* cua-driver: main-actor-isolate AppKitBootstrap.runBlockingAppKitWith
Swift 6.1 on CI flags `NSApplication.shared`, `setActivationPolicy`,
and `.run()` as main-actor-isolated calls from a nonisolated
context. Our local Swift 6.3 toolchain was more permissive here, so
the errors only surfaced on CI. Compiler-suggested fix: annotate the
helper itself with `@MainActor`. All callers are already on the
main actor (this is the AppKit bootstrap path), so isolation is
tightened without behavior change.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Sarina Li <sarinajin.li@gmail.com>
Co-authored-by: Dillon DuPont <ddupont@mit.edu>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: ddupont <3820588+ddupont808@users.noreply.github.com>
Co-authored-by: cua <cua@cua.localdomain>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
display_scale_factorinsession.json; captures pre-action monotonic timestamp for accurate span bracketing; auto-rendersrecording_rendered.mp4on stop and reports path in tool resultZoomRegionfromwindow_boundsinaction.json(letterbox fit, 400ms eased transitions); variable-speed PTS remapping (1× inside action spans, 8× outside); Metal-backedCIContextfor GPU frame processing; graceful fallback to legacy click-zoom for older recordingsNew files
AgentCursorRenderer.swift— SwiftUI Canvas renderer for the Dubins-path cursorActionSpan.swift—ActionSpan,ClickPoint,FocusWaypoint,ActionSpanGenerator(pad/merge logic)Test plan
--video-experimental, confirmrecording_rendered.mp4is emitted automatically on stop and path appears in tool resultwindow_boundsis present inaction.jsonrecording render --no-zoomstill produces a valid re-encoded video (reader/writer baseline)🤖 Generated with Claude Code