feat(cua-driver-rs)(recording): native ScreenCaptureKit on macOS + app_state.json/click.png regressions

libs/cua-driver/rust/Skills/cua-driver/RECORDING.md

@@ -1,11 +1,9 @@
 # Recording & replaying trajectories
 
-> **Platform: macOS-only today.** Trajectory recording / replay is
-> currently implemented on the macOS backend only. On Windows, `cua-driver
-> recording {start,stop,status}` is registered but returns "Recording is
-> currently macOS-only". On Linux (BETA): not supported. See `WINDOWS.md`
-> / `LINUX.md` for capture-state alternatives via `screenshot` and
-> `get_window_state`.
+> **Cross-platform.** Recording is available on macOS (native
+> ScreenCaptureKit), Windows (ffmpeg + `gdigrab`), and Linux (ffmpeg +
+> `x11grab`). Replay is cross-platform as long as the recorded artifacts
+> are present.
 
 Session-scoped capture of action sequences + pre/post state, suitable
 for demos, regression diffs, and training data. Invoked only when the
@@ -20,12 +18,22 @@ permission probes, agent-cursor getters / setters, and the recording
 controls themselves) are not recorded.
 
 **Video on by default.** `start_recording` also captures the main
-display to `<output_dir>/recording.mp4` (H.264 / yuv420p / 30 fps) via
-an ffmpeg subprocess for the lifetime of the session. The mp4 is
-finalized on `stop_recording`. Opt out with `record_video: false` when
-you don't want video. Requires ffmpeg on PATH; when missing, the per-
-turn capture continues without video and `last_error` carries the
-install hint.
+display to `<output_dir>/recording.mp4` (H.264 / 30 fps) for the
+lifetime of the session. The mp4 is finalized on `stop_recording`. Opt
+out with `record_video: false` when you don't want video.
+
+**macOS — native ScreenCaptureKit, zero-config.** On macOS the
+recorder uses an in-process `SCStream` + `SCRecordingOutput`, so it
+inherits cua-driver's own Screen Recording grant — no separate
+subprocess prompt, no fast-fail, no second TCC dance. Requires macOS
+15.0+ (SCRecordingOutput introduced in macOS 15). No ffmpeg needed.
+
+**Windows / Linux — ffmpeg subprocess.** Outside macOS the recorder
+shells to ffmpeg with `gdigrab` (Windows) or `x11grab` (Linux). The
+binary needs to be on PATH (`winget install Gyan.FFmpeg` /
+`apt install ffmpeg`); when missing, the per-turn capture continues
+without video and `last_error` carries the install hint. ffmpeg
+startup failures fast-fail with a stderr tail in the error.
 
 ## Start / stop
 
@@ -59,22 +67,26 @@ daemon restart resets to disabled.
 
 Each action writes to `turn-NNNNN/` (five-digit zero-padded counter):
 
-- `app_state.json` — post-action AX snapshot for the target pid, same
-  shape `get_window_state` returns (tree_markdown, element_count,
-  turn_id, etc.) minus the screenshot fields. The recorder resolves a
-  frontmost window internally (visible + on-current-Space preferred,
-  max-area fallback) since individual action tools carry a
-  window_id but the recorder has no caller-supplied anchor.
-- `screenshot.png` — post-action capture of the same window the
-  recorder just snapshotted. Omitted when the pid has no visible
-  window.
+- `app_state.json` — post-action AX/UIA snapshot for the target
+  `(pid, window_id)` carrying the same `tree_markdown` +
+  `element_count` shape `get_window_state` returns (minus the
+  screenshot fields — those live in `screenshot.png`). On macOS the
+  recorder resolves a frontmost window internally when the action's
+  args don't carry one; on Windows it uses the first window of the
+  target pid. **Omitted on Linux** — ATSPI doesn't expose a cheap
+  whole-tree snapshot, and the file is left out rather than faked.
+- `screenshot.png` — post-action capture of the target window.
+  Omitted when the pid has no visible window.
 - `action.json` — the tool name, full input arguments, result
   summary, pid, click point (when applicable), ISO-8601 timestamp.
-- `click.png` — only for click-family actions (`click`,
+- `click.png` — for click-family actions (`click`, `double_click`,
   `right_click`): a copy of `screenshot.png` with a red dot drawn at
-  the click point (screen-absolute point → window-local pixels via
-  the screenshot's `scale_factor`). Absent for other tools and for
-  clicks whose point falls outside the captured window.
+  the click point. **Both addressing modes are covered:** explicit
+  `x, y` clicks use the supplied coordinates directly, and
+  `element_index`-addressed clicks resolve to the element's center
+  via the live AX/UIA cache, then convert to window-local screenshot
+  pixels. Absent for non-click tools and for clicks whose resolved
+  point falls outside the captured window.
 
 ## When to use it
 

libs/cua-driver/rust/crates/cua-driver-core/src/lib.rs

@@ -26,5 +26,6 @@ pub mod text_sanitize;
 pub mod tool;
 pub mod tool_args;
 pub mod video;
+pub mod video_ffmpeg;
 
 pub use recording::RecordingSession;

libs/cua-driver/rust/crates/cua-driver-core/src/recording.rs

@@ -18,7 +18,7 @@ use std::time::Instant;
 use serde_json::Value;
 
 use crate::cursor_sampler::CursorSampler;
-use crate::video::{VideoMetadata, VideoRecorder};
+use crate::video::{self, VideoBackend, VideoMetadata};
 
 // ── Platform screenshot callback ─────────────────────────────────────────────
 //
@@ -49,6 +49,35 @@ pub fn set_click_marker_fn(f: impl Fn(&[u8], f64, f64) -> Option<Vec<u8>> + Send
     let _ = CLICK_MARKER_FN.set(Box::new(f));
 }
 
+// ── Platform AX-snapshot callback ────────────────────────────────────────────
+//
+// Takes (window_id, pid) and returns JSON bytes for `app_state.json` (the
+// post-action AX/UIA snapshot), or None if no snapshot is available on this
+// platform.
+
+type AxSnapshotFnBox = Box<dyn Fn(Option<u64>, Option<i64>) -> Option<Vec<u8>> + Send + Sync>;
+static AX_SNAPSHOT_FN: OnceLock<AxSnapshotFnBox> = OnceLock::new();
+
+/// Register the platform-specific AX/UIA snapshot callback. Call once at startup.
+pub fn set_ax_snapshot_fn(f: impl Fn(Option<u64>, Option<i64>) -> Option<Vec<u8>> + Send + Sync + 'static) {
+    let _ = AX_SNAPSHOT_FN.set(Box::new(f));
+}
+
+// ── Platform element-bounds callback ─────────────────────────────────────────
+//
+// Resolves an element_index to its center point in window-local screenshot
+// pixels (the same coordinate space as the existing `(cx, cy)` arg to
+// `CLICK_MARKER_FN`). Used so click.png is also written on element-indexed
+// clicks, not just pixel-addressed ones.
+
+type ElementBoundsFnBox = Box<dyn Fn(u64, i64, u32) -> Option<(f64, f64)> + Send + Sync>;
+static ELEMENT_BOUNDS_FN: OnceLock<ElementBoundsFnBox> = OnceLock::new();
+
+/// Register the platform-specific element-bounds resolver. Args: (window_id, pid, element_index).
+pub fn set_element_bounds_fn(f: impl Fn(u64, i64, u32) -> Option<(f64, f64)> + Send + Sync + 'static) {
+    let _ = ELEMENT_BOUNDS_FN.set(Box::new(f));
+}
+
 /// Persistent recording session state (singleton per process).
 pub struct RecordingSession {
     inner: Mutex<RecordingInner>,
@@ -63,9 +92,10 @@ struct RecordingInner {
     /// matches the action-timeline anchor in `action.json`.
     session_monotonic_start: Option<Instant>,
     last_error: Option<String>,
-    /// Live ffmpeg subprocess when video capture is active. Recreated
-    /// per session.
-    video: Option<VideoRecorder>,
+    /// Live video backend when capture is active. Recreated per
+    /// session. The concrete type is platform-determined (SCKit on
+    /// macOS, ffmpeg subprocess elsewhere).
+    video: Option<Box<dyn VideoBackend>>,
     /// Recorded after `stop()` until the next start — exposed in
     /// `current_state()` so callers can read the finalized video info
     /// after stopping.
@@ -144,7 +174,7 @@ impl RecordingSession {
         let mut video_error: Option<String> = None;
         if record_video {
             let path = dir.join("recording.mp4");
-            match VideoRecorder::start(&path) {
+            match video::start_video(&path) {
                 Ok(rec) => {
                     inner.video = Some(rec);
                     video_present = true;
@@ -330,14 +360,23 @@ fn write_turn(
     // Extract window_id and pid from args for screenshot capture.
     let window_id = args.opt_u64("window_id");
     let pid       = args.opt_i64("pid");
+    let element_index = args.opt_u64("element_index");
 
-    // Extract click point for click-family tools.
+    // Extract click point for click-family tools. Falls back to the
+    // platform element_index → window-local-pixels resolver when the call
+    // used `element_index` instead of explicit `x, y`, so click.png is
+    // written for AX-indexed clicks too.
     let click_point: Option<(f64, f64)> = if matches!(
         tool_name, "click" | "double_click" | "right_click"
     ) {
         match (args.opt_f64("x"), args.opt_f64("y")) {
             (Some(x), Some(y)) => Some((x, y)),
-            _ => None,
+            _ => match (window_id, pid, element_index, ELEMENT_BOUNDS_FN.get()) {
+                (Some(wid), Some(p), Some(idx), Some(f)) => {
+                    u32::try_from(idx).ok().and_then(|idx32| f(wid, p, idx32))
+                }
+                _ => None,
+            },
         }
     } else {
         None
@@ -356,6 +395,14 @@ fn write_turn(
     }
     write_json_atomic(&turn_dir.join("action.json"), &payload)?;
 
+    // Post-action AX/UIA snapshot — omitted on platforms that don't expose
+    // a cheap snapshot helper (today: Linux ATSPI).
+    if let Some(ax_fn) = AX_SNAPSHOT_FN.get() {
+        if let Some(json_bytes) = ax_fn(window_id, pid) {
+            let _ = std::fs::write(turn_dir.join("app_state.json"), &json_bytes);
+        }
+    }
+
     // Capture screenshot if a callback is registered.
     if let Some(screenshot_fn) = SCREENSHOT_FN.get() {
         if let Some(png_bytes) = screenshot_fn(window_id, pid) {

libs/cua-driver/rust/crates/cua-driver-core/src/recording_loader.rs

@@ -125,7 +125,7 @@ fn load_session_metadata(
 /// absolute last resort.
 fn probe_video_dimensions(video_path: &Path) -> Option<(u32, u32)> {
     use std::process::Command;
-    let ffprobe = crate::video::find_ffprobe()?;
+    let ffprobe = crate::video_ffmpeg::find_ffprobe()?;
     let out = Command::new(ffprobe)
         .args(["-v", "error",
                "-select_streams", "v:0",

libs/cua-driver/rust/crates/cua-driver-core/src/recording_render.rs

@@ -36,7 +36,7 @@ use std::process::{Command, Stdio};
 
 use crate::recording_loader::{load, LoadError};
 use crate::recording_zoom::{generate_zoom_regions, ZoomRegion};
-use crate::video::{find_ffmpeg, find_ffprobe};
+use crate::video_ffmpeg::{find_ffmpeg, find_ffprobe};
 
 /// Default zoom magnification.
 const DEFAULT_ZOOM_SCALE: f64 = 2.0;

libs/cua-driver/rust/crates/cua-driver-core/src/recording_tools.rs

@@ -69,12 +69,18 @@ impl Tool for StartRecordingTool {
                 Turn folders are named `turn-00001/`, `turn-00002/`, etc.  Turn \
                 numbering restarts at 1 each time recording is (re-)started.\n\n\
                 **Video is on by default** — the main display is captured to \
-                `<output_dir>/recording.mp4` (H.264 / yuv420p / 30 fps) via an ffmpeg \
-                subprocess for the lifetime of the session. Pass `record_video: false` \
-                to opt out. Requires ffmpeg on PATH (winget install Gyan.FFmpeg / \
-                brew install ffmpeg / apt install ffmpeg); when ffmpeg is missing the \
-                per-turn capture (screenshots + action.json) still runs and the \
-                session's `last_error` field carries the ffmpeg-not-found message.\n\n\
+                `<output_dir>/recording.mp4` (H.264 / 30 fps) for the lifetime of \
+                the session. Pass `record_video: false` to opt out.\n\n\
+                **macOS uses native ScreenCaptureKit** (in-process SCStream + \
+                SCRecordingOutput) so video inherits cua-driver's own Screen \
+                Recording grant — no extra TCC prompt, no ffmpeg subprocess. \
+                Requires macOS 15.0+.\n\n\
+                **Windows + Linux use an ffmpeg subprocess** (`gdigrab` / \
+                `x11grab` + libx264). Requires ffmpeg on PATH (winget install \
+                Gyan.FFmpeg / apt install ffmpeg); when ffmpeg is missing or \
+                fails on startup the per-turn capture (screenshots + \
+                action.json) still runs and the session's `last_error` field \
+                carries the diagnostic.\n\n\
                 State persists for the life of the daemon / MCP session; a restart \
                 resets to disabled with no on-disk state. Call `stop_recording` to \
                 disable + finalize the mp4.".into(),
@@ -91,7 +97,9 @@ impl Tool for StartRecordingTool {
                         "type": "boolean",
                         "description": "Capture the main display to <output_dir>/recording.mp4. \
                             Default: true. Set to false to record only the per-turn \
-                            screenshots + JSON. Requires ffmpeg on PATH."
+                            screenshots + JSON. On macOS this uses native \
+                            ScreenCaptureKit (no extra TCC prompt, macOS 15.0+); on \
+                            Windows + Linux it requires ffmpeg on PATH."
                     }
                 },
                 "additionalProperties": false
@@ -114,11 +122,17 @@ impl Tool for StartRecordingTool {
         match self.session.start(output_dir.as_deref().unwrap(), record_video) {
             Ok(()) => {
                 let state = self.session.current_state();
+                // When the caller asked for video and it failed (e.g. macOS
+                // ffmpeg TCC prompt deadlock), surface the actual error
+                // prominently — the per-turn capture still runs, but the
+                // caller deserves to know the mp4 won't materialize.
+                let video_failed = record_video && !state.video_active;
                 let video_note = if record_video && state.video_active {
-                    " (video → recording.mp4)"
-                } else if record_video && !state.video_active {
-                    " (video requested but ffmpeg not available — see last_error)"
-                } else { "" };
+                    " (video → recording.mp4)".to_string()
+                } else if video_failed {
+                    let err = state.last_error.clone().unwrap_or_else(|| "unknown".into());
+                    format!("\n\n⚠️ Video capture failed (per-turn JSON+screenshot still running):\n{err}")
+                } else { String::new() };
                 let msg = format!("✅ Recording started -> {}{}",
                     state.output_dir.as_deref().unwrap_or("?"),
                     video_note);