feat(cua-driver-rs)(recording): rename set_recording→start/stop + cross-platform video + zoom-on-click renderer + rename crate to cua-driver-core - #1718
Merged
Conversation
… cross-platform video capture
Three changes landed together because they're a single user-visible
contract change (autonomous 2-hour task, see JOURNAL_VIDEO.md for the
full log + design rationale, this is local-only — no PR).
## 1. Tool rename: set_recording → start_recording + stop_recording
The old toggle-style `set_recording({enabled: bool, ...})` mixed two
distinct verbs ("start" vs "stop") into one config-write-shaped tool.
That made it hard to discover in MCP tool listings ("set_recording"
reads like a config write, not a session action) and put the MCP surface
out of sync with the CLI's verb-first `cua-driver recording start|stop|
status` subcommand group. Split into two tools so:
- `start_recording({output_dir, record_video?})` — explicit, intent-
revealing. Required arg dropped to `output_dir` (the `enabled` boolean
was always `true` here anyway).
- `stop_recording({})` — no args, idempotent.
`RecordingSession` gained `start()`/`stop()` methods; the legacy
`configure(enabled, output_dir)` is kept as a thin shim so the rest of
the rename window doesn't break. The tool registry's recording-exclusion
list (`tool.rs::ToolRegistry::invoke`) was updated to exclude both new
names — caught by `test_recording_session_windows` failing loud when
`start_recording` itself was being recorded as turn-00001.
## 2. video_experimental → record_video, default-on
Promoted the experimental video flag. New shape:
`start_recording({output_dir, record_video: true})` — `record_video`
defaults to true, callers opt out with false. The `_experimental`
suffix was a code smell; flags named like that either get promoted or
languish, and the user asked for default-on.
## 3. Cross-platform video capture (mcp-server/src/video.rs)
New `VideoRecorder` struct that spawns an ffmpeg subprocess writing the
main display to `<output_dir>/recording.mp4` (H.264 / yuv420p / 30 fps).
Platform-specific input devices:
- Windows: `-f gdigrab -framerate 30 -draw_mouse 1 -i desktop`
- macOS: `-f avfoundation -framerate 30 -pix_fmt uyvy422 -i 1:`
- Linux: `-f x11grab -framerate 30 -i $DISPLAY`
Shared encoder flags: `-c:v libx264 -preset ultrafast -pix_fmt yuv420p
-movflags +faststart -g 30 -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2"`.
The pad filter is non-obvious — libx264 with yuv420p subsampling
requires even dimensions; many desktop heights aren't (the dev host
ran 1512×949, which made the first test produce a 0-byte mp4 with
ffmpeg complaining "height not divisible by 2"). Padding by ≤1 px on
the bottom/right beats cropping (full display stays in frame).
Lifecycle:
- `start(path)` spawns ffmpeg, returns the handle.
- `stop()` sends `q\n` on ffmpeg's stdin (its clean-shutdown trigger
that finalizes the moov atom), polls for exit up to 3 s, falls back
to `kill()`. Returns `VideoMetadata { duration_ms, finalized }` so
the caller distinguishes a clean stop from a forced one.
A background thread drains ffmpeg's stderr into a 4 KB ring buffer for
two reasons: (1) ffmpeg's stderr pipe can fill up and block the
encoder; (2) when the process exits non-zero we log the tail at `warn`
level — that's how I caught the height-divisible-by-2 issue inside
2 minutes.
ffmpeg discovery: `find_ffmpeg()` probes PATH first, then well-known
install paths per platform (winget Gyan.FFmpeg / Homebrew / apt). Lets
a freshly-installed ffmpeg work without a shell restart — needed
because the cua-driver process inherits PATH from its parent shell,
and a winget install only updates the current shell's PATH (subsequent
shells get the new entry, but the already-running parent doesn't).
When ffmpeg isn't available, `start()` returns a structured error
pointing at the right package manager; `start_recording` continues
without video and surfaces the install hint via `last_error` in the
state response. Per-turn capture (screenshots + action.json) is
independent of video and keeps running.
## Verified end-to-end
- `flash-repro/test_video_recording.py` — 5 s desktop capture →
1.94 MB mp4, H.264 1512×950, finalized:true, ffprobe-valid.
- `flash-repro/test_video_calc_demo.py` — full agent flow: start →
launch Calc → click 5+7=12 via UIA → stop → 6.6 MB mp4, 6 turn
folders, session.json carries full video metadata.
- `cargo test --test mcp_protocol_test recording` — 2/2 passing
(renamed test_recording_session_windows + new
test_start_recording_record_video_flag_accepted_windows).
## Known not-done
- macOS / Linux ffmpeg branches are coded but unverified from this
Windows host. avfoundation device-index "1" may need per-host
resolution (Swift impl did this via SCShareableContent).
- Wayland Linux: x11grab only, no PipeWire path.
- get_window_state is not tagged read_only despite walking the UIA
tree — it's being recorded as a turn. Pre-existing; separate ticket.
- No audio capture; video-only.
- No per-window video; main-display only.
## Files touched
- mcp-server/src/{video.rs (new), recording.rs, recording_tools.rs,
tool.rs, lib.rs}
- cua-driver/src/cli.rs (CLI subcommand dispatch)
- cua-driver/tests/mcp_protocol_test.rs (renamed test + recording calls)
- platform-windows/examples/{list_tools_parity.rs, recording_parity.rs}
- Skills/cua-driver/{SKILL.md, RECORDING.md}
- JOURNAL_VIDEO.md (new, autonomous-task log)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… — cross-platform via ffmpeg sendcmd
The previous commit shipped the capture half of the recording pipeline
(start/stop/video). It didn't carry the post-process zoom-on-click
renderer the Swift impl has — the user immediately noticed and asked
for it next. This adds it as a separate, optional pass while keeping
the layering consistent with the Swift source tree.
## Layering
Mirrors Swift 1:1:
- `mcp-server` = library (math, types, loader, renderer)
- `cua-driver` = binary (CLI subcommand wiring)
The renderer lives in mcp-server because it reads what `RecordingSession`
writes (so schema refactors stay in one place), it could be called by
a future `render_recording` MCP tool without a new dep, and it keeps
`cua-driver/src/cli.rs` a thin arg-parser → library-call wrapper.
## What landed
- `recording_zoom.rs` — pure-Rust port of Swift `Zoom/`:
- `clamp01`, `lerp`, `cubic_bezier` (Newton-Raphson + binary-search),
`easeOutExpo` math primitives.
- `ZoomRegion` + `FocusWaypoint` + `ClickEvent` + `CursorSample` types.
- `generate_zoom_regions(clicks)` — turns clicks into a zoom timeline,
merging adjacent clicks within `CHAINED_ZOOM_PAN_GAP_MS` into a
single pan-between-foci region.
- `sample_curve(t_ms, regions, cursors)` — instantaneous
`(scale, focus_x, focus_y)` at playback time. Cubic-bezier eased
zoom-in / hold / zoom-out, with waypoint pan inside merged regions.
- `generate_action_spans` + `map_pts` — span/PTS math for the
variable-speed feature (1× in spans, 8× between). Not yet wired
into the renderer; ready for the speed-zone pass.
- 9 unit tests.
- `cursor_sampler.rs` — cross-platform cursor poll thread (30 Hz),
writes `cursor.jsonl` lines for the lifetime of a recording session.
Per-platform: `GetCursorPos` (Windows), `CGEventCreate +
CGEventGetLocation` (macOS via inline extern "C"), Linux stub (no
portable API; renderer falls back to click-point-only zoom).
- `recording_loader.rs` — port of `Render/TrajectoryLoader.swift`:
reads `session.json`, `cursor.jsonl`, every `turn-*/action.json`.
Falls back to parsing screen coords out of `result_summary`
("...(screen (X,Y))...") for element-indexed clicks, since the
current `RecordingSession` doesn't write `click_point` for that
path (follow-up tracked in JOURNAL_VIDEO.md).
- `recording_render.rs` — the actual renderer:
- Pre-computes per-frame `(scale, focus_x, focus_y)` at 30 Hz in Rust.
- Writes a `<input-dir>/render.sendcmd` file with timed `crop@c`
parameter updates.
- Builds ffmpeg filter chain
`sendcmd=f=<file>,crop@c=…,scale=W:H,pad=…,format=yuv420p` and
shells out to ffmpeg for the actual encode.
- The sendcmd-driven approach avoids the giant `if(between(t,t1,t2),…)`
expression a one-shot crop expression would need — ffmpeg's
parser has depth limits and the sendcmd file is inspectable.
- `cua-driver/src/cli.rs::run_recording_render` — new subcommand:
`cua-driver recording render <input-dir> <out.mp4> [--no-zoom] [--scale N]`.
Does NOT require the daemon (pure file→file work). The output path
is the second positional because the global CLI parser strips
`--output` flags before subcommand dispatch sees them.
- `RecordingSession.start/stop` now spawns / stops the cursor sampler
alongside the video recorder, sharing a single monotonic anchor so
cursor samples, action timestamps, and video frames line up.
`session.json` carries `cursor.sample_count` after stop.
## Verified end-to-end on Windows
Fresh capture (`5 clicks, 20.8 s, 621 cursor samples`):
- `recording.mp4` 4.6 MB, `cursor.jsonl` 621 lines, 5 turn folders
- Render: `cua-driver recording render … rendered.mp4`
→ `zoom_region_count: 2` (4 clicks within proximity merged into 2
chained regions per the Swift algorithm)
- Rendered mp4: 1.26 MB, plays cleanly
- Frame at t=1s = full desktop (no zoom)
- Frame at t=9s = clearly zoomed in on the calc "5" button area,
agent cursor visible at click point
- `render.sendcmd` shows crop shrinking 1512×950 → 784×492 (≈2×) at
click moments, easing back
## Cross-platform shape
| Concern | Strategy | Status |
|---|---|---|
| Zoom math | pure-Rust, no deps | ✓ everywhere |
| Trajectory loading | std::fs + serde_json | ✓ everywhere |
| Renderer (ffmpeg filter chain) | ffmpeg subprocess | ✓ everywhere ffmpeg is |
| Cursor poll: Windows | `GetCursorPos` | ✓ |
| Cursor poll: macOS | `CGEventGetLocation` | coded, unverified from Win host |
| Cursor poll: Linux X11 | stub (needs `x11` crate) | renderer copes |
| Cursor poll: Linux Wayland | not feasible portably | renderer copes |
## Local commit only — no PR.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…e-smooth zoom
The first cut of the render pipeline used `sendcmd` to drive the `crop`
filter at 30 Hz with integer pixel values. That produced visible
stair-stepping during zoom-in / zoom-out — the user spotted it
immediately ("the zoom in and out is not very fluent").
Two compounding causes:
1. `sendcmd` issues **discrete state changes**; between updates the
crop params are held constant. Even at 30 Hz the snap-to-next-value
was visible.
2. `crop` filter's `w` and `h` are evaluated **only once at init** in
ffmpeg (only `x` and `y` re-evaluate per frame). So even when I
tried switching from sendcmd to a time-varying `crop` expression,
the width/height were locked to their t=0 values and no zoom
appeared at all.
Fix: switch to ffmpeg's `zoompan` filter — built exactly for this use
case (Ken Burns slideshow zoom adapted for video). Its `z` (zoom),
`x` (top-left in input coords), and `y` expressions ALL evaluate per
output frame.
New filter chain:
```
zoompan=z='<zoom_expr>':x='<x_expr>':y='<y_expr>':d=1:s=WxH:fps=30,
pad=ceil(iw/2)*2:ceil(ih/2)*2,format=yuv420p
```
The `d=1` (one output frame per input frame) is critical — zoompan
defaults to `d=90` which is a slideshow assumption that would freeze
the video.
Per-region math is now expressed in zoompan's `time` variable:
- `phase` = piecewise zoom-in / hold / zoom-out
- `ease` = `1 - (1-phase)^5` (quintic ease-out — approximates the
Swift `cubic-bezier(0.16,1,0.3,1)` shape closely)
- `z` = `1 + ease * (SCALE - 1)`
- `x` = `clip(focus_x - iw/(2z), 0, iw - iw/z)`
- `y` = `clip(focus_y - ih/(2z), 0, ih - ih/z)`
Multiple regions chain via `if(between(time, START, END), region_expr,
prev_expr)`, falling through to `z=1, x=0, y=0` (no zoom) outside.
Verified: rendered output now shows smooth zoom-in over ~1.5 s lead-in
window. Frames sampled at 200ms intervals during the ramp (8.5, 8.7,
8.9, 9.1, 9.3, 9.5s) show monotonic magnification with no perceptible
steps. PNG file sizes scale smoothly (398 KB at the un-zoomed end,
226 KB at peak zoom) — the encoder's per-frame entropy reduction is
the smoking gun that the zoom is genuinely continuous.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…` to match its actual scope
The crate started life as the MCP JSON-RPC server. Over time it grew
all of cua-driver's general-purpose library code (recording session,
video recorder, cursor sampler, zoom math, trajectory loader, renderer,
element cache, CDP client, image utilities, text sanitizers, tool args
parsing). Only `protocol.rs`, `server.rs`, `tool.rs`, and
`recording_tools.rs` are actually MCP-specific now — the rest is used
by the CLI binary too.
The user flagged it directly: "still not a fan that is in an mcp folder
while the logic is also used by the cli." Mirrors the Swift layout
(`CuaDriverCore` is the library, `CuaDriverCLI` is the binary).
Mechanical rename:
- `crates/mcp-server/` → `crates/cua-driver-core/`
- Package name in `Cargo.toml` ditto.
- 6 sibling Cargo.tomls updated to point at `../cua-driver-core`.
- 207 `use mcp_server::…` import sites swept to `use cua_driver_core::…`
across 51 source files in 5 crates (`cua-driver`, `cua-driver-uia`,
`platform-{windows,macos,linux}`, `cursor-overlay`, `focus-monitor-win`).
- User-facing strings about the MCP protocol concept (Hermes config
keys, parity test text) are unchanged — those are about the MCP wire
protocol, not the Rust crate.
Verified:
- `cargo build --workspace` clean (only pre-existing platform-linux
warnings)
- 45 cua-driver-core unit tests pass
- `cargo test --test mcp_protocol_test recording` passes
- End-to-end render still works: `cua-driver recording render <dir>
<out.mp4>` produces the same smooth zoom output
No behavior change; pure rename for naming honesty.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stray `C:/Users/cuademo/Desktop/...` from a test-run transcript that the audit caught before opening the PR. Pure doc hygiene. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughWalkthroughThis PR renames the ChangesCrate restructure and recording refactor
Platform-specific tool and protocol migration
Integration tests and documentation updates
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
|
f-trycua
added a commit
that referenced
this pull request
May 26, 2026
…p_state.json/click.png regressions (#1720) * fix(cua-driver-rs)(recording): app_state.json + click.png-for-element_index + ffmpeg TCC fast-fail Three regressions introduced by the cross-platform Rust recording refactor (#1718, since the rename to cua-driver-core). User reproduced all three live on macOS Calculator (5+3) — zero app_state.json files, zero click.png files (all 5 turns used element_index), and a 0-byte recording.mp4 with `finalized: false` while ffmpeg silently blocked forever on a TCC prompt the daemon couldn't surface. 1. **app_state.json never written.** `recording::write_turn` had no AX snapshot path. Added `AX_SNAPSHOT_FN` callback mirroring the existing `SCREENSHOT_FN`/`CLICK_MARKER_FN` pattern, wired to `platform_macos::recording_hooks::app_state_json_for` (walks the AX tree fresh for the target pid + window_id and emits the same shape `get_window_state` returns minus the screenshot fields) and the same on Windows via UIA. Linux is intentionally a no-op — ATSPI has no cheap whole-tree snapshot and the file is omitted rather than faked. 2. **click.png never written for element_index clicks.** The check at recording.rs:335 only set `click_point` when `x` AND `y` were present in args, so AX-indexed clicks (the dominant path on Calculator AX buttons — all 5 user-reproed turns) fell through and click.png was never produced. Added `ELEMENT_BOUNDS_FN` resolving `element_index` → window-local screenshot-pixel coords via the live AX cache (macOS: `element_screen_center` minus window origin, multiplied by Retina scale derived from PNG width / logical width; Windows: cached center minus window origin, no scale). The element cache is shared with the recording-hook layer at `register_all` startup so the callback is wired without a registry detour. 3. **recording.mp4 never finalized on macOS.** ffmpeg+avfoundation needs its own per-binary Screen Recording grant; TCC doesn't propagate from the parent process. When spawned by a daemon there's no UI thread to display the consent dialog, so ffmpeg blocks forever on `[ScreenCaptureKit] requesting consent...`. Two fast-fail probes added to `VideoRecorder::start`: (a) `try_wait` polling for the first 1500 ms catches immediate exits with stderr tail, (b) on macOS, after another ~500 ms, if the output file is still 0 bytes we treat it as a TCC hang, kill the subprocess, and return an actionable error pointing the user at Privacy & Security → Screen & System Audio Recording with the resolved ffmpeg path. The error ends up in `session.json` `video.error` AND the `start_recording` response text so callers can't miss it. Deferred (separate follow-up issue): replace ffmpeg+avfoundation on macOS with a native ScreenCaptureKit binding so video works zero-config. Multi-week native-binding work, out of scope for a regression fix. Docs: RECORDING.md updated with the per-turn `app_state.json` shape + when it's omitted (Linux), the click.png element_index path, and the macOS ffmpeg TCC requirement with the planned migration. The `start_recording` MCP tool description picks up the same TCC note. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(cua-driver-rs)(recording): native ScreenCaptureKit video on macOS, ffmpeg only on Win/Linux Replaces the macOS ffmpeg+avfoundation subprocess pipeline with an in-process SCStream + SCRecordingOutput. ScreenCaptureKit runs in the cua-driver process, so it inherits the daemon's Screen Recording TCC grant — no separate subprocess prompt, no fast-fail hang heuristic. video.rs now exposes `VideoBackend` + `VideoBackendFactory` traits; each platform crate registers its concrete factory at startup (`set_video_backend_factory`), mirroring how `SCREENSHOT_FN` / `AX_SNAPSHOT_FN` are wired. macOS registers `SckitVideoBackendFactory` (platform-macos/src/video_sckit.rs); Windows + Linux register `FfmpegVideoBackendFactory` (cua-driver-core/src/video_ffmpeg.rs, carrying the existing stderr-tail fast-fail). Requires macOS 15.0+ (SCRecordingOutput introduced in macOS 15). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cua-driver-rs)(recording): address CodeRabbit findings on PR #1720 7 CodeRabbit nits across the recording refactor: 1. recording_tools.rs — `record_video` schema description now says macOS uses ScreenCaptureKit, only Win/Linux need ffmpeg (was contradicting the description text just above) 2. recording.rs — `element_index` u64→u32 narrowing now uses `u32::try_from(...).ok().and_then(...)` instead of silent `as` cast 3. video_ffmpeg.rs — `create_dir_all` error now propagates with context instead of being swallowed 4. platform-macos/recording_hooks.rs — i64→i32 and u64→u32 conversions for pid + window_id now use checked `try_from` 5. video_sckit.rs — fs::create_dir_all error propagates; fs::remove_file ignores only NotFound, propagates other IO errors 6. platform-windows/recording_hooks.rs — same checked conversions for pid + window_id casts 7. RECORDING.md — header no longer claims "macOS-only" since recording now ships cross-platform via SCKit/ffmpeg Build: `cargo build --release -p cua-driver` green on macOS; `cargo check -p platform-windows --target=x86_64-pc-windows-msvc` green. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four-part change to the recording / video pipeline, plus a crate
rename. Full design log in
JOURNAL_VIDEO.md(autonomous-task journalkept while implementing).
1. Tool rename:
set_recording→start_recording+stop_recordingThe old
set_recording({enabled: bool, …})was toggle-shaped andread like a config write in MCP tool listings. Split into two intent-
revealing tools, matching the CLI's existing
cua-driver recording start|stop|statusverbiage.start_recording({output_dir, record_video?})— required argdropped to
output_dir;record_videodefaults to true (wasthe experimental opt-in
video_experimental: false).stop_recording({})— no args, idempotent.2. Video promoted out of experimental + default-on
Dropped the
video_experimentalflag entirely. Video capture is nowdefault behavior on
start_recording; passrecord_video: falsetoopt out.
3. Cross-platform video capture (
cua-driver-core::video)New
VideoRecorderthat spawns ffmpeg writing the main display to<output_dir>/recording.mp4(H.264 / yuv420p / 30 fps). Platform-specific input device:
gdigrab -i desktopavfoundation -i "1:"x11grab -i $DISPLAYffmpeg is the only runtime dep; discovery probes PATH first then
well-known package-manager install locations (winget Gyan.FFmpeg /
Homebrew / apt) so a fresh install works without a shell restart.
When ffmpeg isn't found, the per-turn capture (action.json +
screenshots) keeps running and
last_errorcarries the install hint.4. Zoom-on-click renderer (Phase 2 of the recording pipeline)
The Swift codebase had a separate
cua-driver recording rendersubcommand that post-processes a captured directory into a zoomed
MP4. Ported it to Rust as a cross-platform pipeline:
recording_zoom.rs— pure-Rust math: cubic-bezier eased zoomcurves, click → region generation with chained-region merging,
variable-speed PTS remap. 9 unit tests.
cursor_sampler.rs— cross-platform 30 Hz mouse-positionsampler thread (
GetCursorPoson Windows,CGEventGetLocationonmacOS, X11 stub on Linux). Writes
cursor.jsonlalongside therecording.
recording_loader.rs— parsessession.json+cursor.jsonlturn-*/action.json. Falls back to scanningresult_summaryfor screen coords on element-indexed clicks.recording_render.rs— builds a per-frame ffmpegzoompanfilter expression and shells out to ffmpeg for the encode.
Smoothness gotcha worth remembering: the initial implementation
used the
cropfilter, butcrop'sw/hevaluate only once atinit — only
x/yre-evaluate per frame. Switched tozoompan,whose
z/x/yall evaluate per output frame. Withd=1and anif(between(time, START, END), region_expr, prev_expr)chain perregion the output is continuously smooth at frame granularity.
5. Crate rename:
mcp-server→cua-driver-coreThe crate started life as the MCP JSON-RPC server but grew all of
cua-driver's general-purpose library code. Only
protocol.rs,server.rs,tool.rs, andrecording_tools.rsare actually MCP-specific now — the rest (RecordingSession, VideoRecorder,
CursorSampler, zoom math, trajectory loader, renderer, element
cache, CDP client, image utilities, text sanitizers, tool args
parsing) is used by both the MCP server and the CLI binary.
Mechanical rename: directory + package name + 6 sibling
Cargo.tomlpath references + 207
use mcp_server::…import sites across 51source files. Mirrors the Swift
CuaDriverCorelibrary layout.User-facing strings about the MCP protocol concept (Hermes config
keys, parity test text) are unchanged.
Verified end-to-end
cargo build --workspaceclean (only pre-existing platform-linux /focus-monitor-win warnings)
cua-driver-coreunit tests: 45/45 passcargo test --test mcp_protocol_test recording: 2/2 passstart_recording/stop_recordingtools: 6.6 MB raw mp4 + 5 turn folders +
cursor.jsonl(621samples over 20.8s).
cua-driver recording render <dir> <out.mp4>→ smooth-zoom output:1.08 MB, H.264, finalized, 2 zoom regions (4 clicks within
CHAINED_ZOOM_PAN_GAP_MS=1500of each other merged into chainedpan-between regions per the Swift algorithm). Sampled frames at
200 ms intervals across the zoom-in ramp show monotonic
magnification with no perceptible steps.
Cross-platform shape
GetCursorPosCGEventCreate+CGEventGetLocation(inlineextern "C")x11crateKnown not-done (documented in JOURNAL_VIDEO.md)
RecordingSessionshould writeclick_pointfor element-indexedclicks (currently only pixel-clicks get it). Workaround: loader
parses screen coords out of
result_summary. Real fix is a smalledit in
recording.rs::write_turn.Math is cross-platform pure-Rust so it should work; ffmpeg
subprocess args are platform-keyed and validated for input device
only.
to a single focus in the current
build_zoompan_expressions(usesregion.focus_x/y from the first waypoint). Would require expressing
piecewise lerps in ffmpeg's expression language — a separate change.
cua-driver doctorshould grow anffmpegcheck.Test plan
cargo build --workspacecargo test -p cua-driver-core --lib(45 pass)cargo test --test mcp_protocol_test recording(2 pass)🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
set_recordingwith separatestart_recordingandstop_recordingtools for clearer session control.record_videoflag).Documentation