feat(cua-driver-rs)(linux): hyprland background capture + full recording

libs/cua-driver/rust/Cargo.toml

@@ -13,7 +13,7 @@ members = [
 ]
 
 [workspace.package]
-version = "0.5.1"
+version = "0.5.2"
 edition = "2021"
 authors = ["trycua"]
 license = "MIT"
@@ -30,3 +30,12 @@ tracing-subscriber = { version = "0.3", features = ["env-filter"] }
 base64 = "0.22"
 uuid = { version = "1", features = ["v4"] }
 image = { version = "0.25", default-features = false, features = ["png", "jpeg"] }
+# Wayland capture (Linux): hyprland-toplevel-export per-window capture +
+# wlr-screencopy video frames. wayland-backend and bitflags must be direct
+# deps of any crate running wayland-scanner macros (generated code references
+# both by crate-root path).
+wayland-client = "0.31"
+wayland-backend = "0.3"
+wayland-scanner = "0.31"
+wayland-protocols-wlr = { version = "0.3", features = ["client"] }
+bitflags = "2"

libs/cua-driver/rust/Skills/cua-driver/LINUX.md

@@ -26,14 +26,18 @@ behaviors that the macOS / Windows skills consider table-stakes are
   KDE-KWin with `org.freedesktop.portal.RemoteDesktop` enabled, some
   click and key paths work. Under most other compositors, input
   synthesis is denied by the security model and the tool surface
-  degrades to "passive" (snapshot, screenshot) only.
+  degrades to "passive" (snapshot, screenshot) only. Hyprland is the
+  exception: it is fully supported for background element-index
+  workflows — see the Hyprland section below.
 - **UIA / AX-tree equivalent**: AT-SPI when available, otherwise
   empty. Many GTK4 / Qt6 apps populate AT-SPI lazily; agents should
   expect partial trees and re-snapshot.
 - **launch_app**: backed by `xdg-open` / `gtk-launch` / `dbus-send`
   with display-environment scrubbing to avoid stealing the user's
   workspace. Not yet equivalent to macOS `FocusRestoreGuard`.
-- **Recording**: not supported.
+- **Recording**: supported. Per-turn screenshots + `app_state.json`
+  (AT-SPI tree) + video. Video uses wlr-screencopy on Wayland and
+  `x11grab` on X11; requires ffmpeg on PATH.
 
 See `SKILL.md` (macOS) and `WINDOWS.md` (Windows) for the full
 patterns. This file will grow as the Linux backend reaches GA. For
@@ -55,6 +59,36 @@ If you're agent-driving on Linux and a tool call surprises you:
    /org/freedesktop/portal/desktop`). Without it, input synthesis
    is denied.
 
+## Hyprland
+
+Hyprland is the exception to the "passive-only under Wayland" rule —
+background element-index workflows are fully supported. What's
+specific to it:
+
+- **Window ids**: `window_id` values come from hyprctl window
+  addresses and exceed `u32::MAX`. That's expected — pass them
+  through verbatim, don't truncate.
+- **Per-window screenshots**: captured via the
+  `hyprland-toplevel-export-v1` protocol — true surface capture, so
+  the screenshot shows the correct content even for occluded /
+  background windows and windows on other workspaces. This is what
+  makes background computer use verifiable on Hyprland. grim
+  region-crop is the fallback when the protocol is unavailable.
+- **Input**: native-Wayland windows accept `element_index` actions
+  (AT-SPI) but not pixel input.
+- **launch_app**: if a newly launched window steals focus, the driver
+  restores the previously active window. Best-effort, watches for
+  ~2 s after launch.
+- **Recording**: video captures the focused monitor via
+  wlr-screencopy frames piped to ffmpeg. `cursor.jsonl` sampling
+  works via the Hyprland IPC `cursorpos` query (global logical
+  coords; empty on other Linux sessions).
+- **Permission caveat**: if `ecosystem:enforce_permissions` is
+  enabled in the Hyprland config and screencopy is denied, captures
+  silently return black "permission denied" frames — no error is
+  raised. Add an allow rule for the cua-driver binary to the
+  Hyprland permission config.
+
 ## Forbidden vectors
 
 Same idea as macOS / Windows — don't shell out to anything that
@@ -78,10 +112,10 @@ ask the user.
 | Element-indexed click | ⚠️ AT-SPI `accDoDefaultAction` when supported |
 | Type text | ⚠️ XTest, focus-sensitive |
 | Hotkey | ⚠️ XTest, focus-sensitive |
-| Screenshot full-display | ✅ X11 (xshm); ⚠️ Wayland (portal-gated) |
-| Screenshot per-window | ⚠️ X11 with composite extension; Wayland TBD |
-| launch_app | ⚠️ xdg-open / gtk-launch; no FocusRestoreGuard yet |
-| Recording | ❌ not implemented |
+| Screenshot full-display | ✅ X11 (xshm); ✅ Wayland via grim (no portal) |
+| Screenshot per-window | ✅ X11; ✅ Hyprland via toplevel-export (correct even when occluded); other Wayland TBD |
+| launch_app | ✅ direct exec / xdg-open; focus-restore guard on Hyprland (see Hyprland section) |
+| Recording | ✅ wlr-screencopy on Wayland / `x11grab` on X11; ffmpeg required |
 
 Until Linux reaches GA, treat this doc as a planning placeholder
 rather than a contract.

libs/cua-driver/rust/Skills/cua-driver/README.md

@@ -36,7 +36,8 @@ platform: no focus steal, no cursor warp.
   `set_value` workaround). Windows web-apps coverage lives in
   `WINDOWS.md`'s "Web apps on Windows" section.
 - Trajectory recording (`RECORDING.md`) — optional per-session
-  recording + replay for demos and regressions. macOS-only today.
+  recording + replay for demos and regressions. Cross-platform
+  (macOS / Windows / Linux).
 - Canvas/viewport apps (Blender, Unity, GHOST, Qt, wxWidgets) —
   fallback paths when the AX/UIA/AT-SPI tree is empty.
 
@@ -169,8 +170,8 @@ Use MCP for this Claude Code vision/computer-use-style path. CLI screenshots sti
   Electron, Tauri, minimized-Chrome keyboard-commit caveat).
   Loaded on demand from `SKILL.md`. **Note**: Windows web-apps
   coverage lives in `WINDOWS.md`'s "Web apps on Windows" section.
-- `RECORDING.md` — trajectory recording / replay (macOS-only
-  today; Windows / Linux not yet supported).
+- `RECORDING.md` — trajectory recording / replay (cross-platform:
+  macOS / Windows / Linux).
 - `TESTS.md` — manual test scripts for end-to-end skill verification.
 
 ## Troubleshooting

libs/cua-driver/rust/Skills/cua-driver/RECORDING.md

@@ -1,9 +1,9 @@
 # Recording & replaying trajectories
 
 > **Cross-platform.** Recording is available on macOS (native
-> ScreenCaptureKit), Windows (ffmpeg + `gdigrab`), and Linux (ffmpeg +
-> `x11grab`). Replay is cross-platform as long as the recorded artifacts
-> are present.
+> ScreenCaptureKit), Windows (ffmpeg + `gdigrab`), and Linux (ffmpeg;
+> wlr-screencopy frames on Wayland, `x11grab` on X11). Replay is
+> cross-platform as long as the recorded artifacts are present.
 
 Session-scoped capture of action sequences + pre/post state, suitable
 for demos, regression diffs, and training data. Invoked only when the
@@ -17,10 +17,10 @@ turn folder under a caller-chosen output directory. Read-only tools
 permission probes, agent-cursor getters / setters, and the recording
 controls themselves) are not recorded.
 
-**Video on by default.** `start_recording` also captures the main
-display to `<output_dir>/recording.mp4` (H.264 / 30 fps) for the
-lifetime of the session. The mp4 is finalized on `stop_recording`. Opt
-out with `record_video: false` when you don't want video.
+**Video off by default.** Pass `record_video: true` to also capture
+the main display to `<output_dir>/recording.mp4` (H.264 / 30 fps) for
+the lifetime of the session. The mp4 is finalized on
+`stop_recording`.
 
 **macOS — native ScreenCaptureKit, zero-config.** On macOS the
 recorder uses an in-process `SCStream` + `SCRecordingOutput`, so it
@@ -29,7 +29,8 @@ subprocess prompt, no fast-fail, no second TCC dance. Requires macOS
 15.0+ (SCRecordingOutput introduced in macOS 15). No ffmpeg needed.
 
 **Windows / Linux — ffmpeg subprocess.** Outside macOS the recorder
-shells to ffmpeg with `gdigrab` (Windows) or `x11grab` (Linux). The
+shells to ffmpeg with `gdigrab` (Windows), wlr-screencopy frames
+piped to ffmpeg (Linux Wayland), or `x11grab` (Linux X11). The
 binary needs to be on PATH (`winget install Gyan.FFmpeg` /
 `apt install ffmpeg`); when missing, the per-turn capture continues
 without video and `last_error` carries the install hint. ffmpeg
@@ -73,8 +74,10 @@ Each action writes to `turn-NNNNN/` (five-digit zero-padded counter):
   screenshot fields — those live in `screenshot.png`). On macOS the
   recorder resolves a frontmost window internally when the action's
   args don't carry one; on Windows it uses the first window of the
-  target pid. **Omitted on Linux** — ATSPI doesn't expose a cheap
-  whole-tree snapshot, and the file is left out rather than faked.
+  target pid. On Linux the snapshot is the AT-SPI tree (same shape as
+  macOS); editable / text widgets surface their Text-interface
+  content as `value="…"` even when the widget has a name, so typed /
+  set text is verifiable from the tree.
 - `screenshot.png` — post-action capture of the target window.
   Omitted when the pid has no visible window.
 - `action.json` — the tool name, full input arguments, result
@@ -102,8 +105,8 @@ Each action writes to `turn-NNNNN/` (five-digit zero-padded counter):
 This skill does **not** auto-enable recording. The client invokes
 `start_recording` explicitly when the user asks to capture a session.
 If the user says "record this session" or similar, call
-`start_recording({output_dir:…})` before the first action (video on
-by default; pass `record_video: false` to opt out), and
+`start_recording({output_dir:…})` before the first action (video off
+by default; pass `record_video: true` to capture it), and
 `stop_recording({})` when done.
 
 ## Replaying a recorded trajectory

libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md

@@ -449,8 +449,8 @@ subcommands:
   registers it (idempotent — replaces existing). `kick` runs it
   immediately without waiting for a fresh logon.
 - **`cua-driver recording start|stop|status`** — see `RECORDING.md`.
-  **Note: recording is currently macOS-only on the Rust port. The
-  command is registered but returns "not yet supported" on Windows.**
+  Recording works on Windows via ffmpeg + `gdigrab`; ffmpeg must be
+  on PATH (`winget install Gyan.FFmpeg`).
 
 Canonical multi-step workflow:
 
@@ -895,11 +895,8 @@ registered to auto-start at logon AND whether it's currently running:
 
 ## Recording
 
-Screen recording is **not yet supported on Windows** in
-cua-driver-rs. The `recording start|stop|status` subcommands are
-registered but return "Recording is currently macOS-only" on
-Windows. Tracking: see the cua-driver-rs roadmap in the main repo.
-
-For now, capture state via `screenshot` (per-window or full-desktop)
-or `get_window_state` (returns a screenshot embedded alongside the
-UIA tree).
+Screen recording **works on Windows** in cua-driver-rs via an ffmpeg
+subprocess (`gdigrab`). ffmpeg must be on PATH — install with
+`winget install Gyan.FFmpeg`. When ffmpeg is missing, per-turn
+capture continues without video and `last_error` carries the install
+hint. See `RECORDING.md` for the full recording / replay surface.