feat(cua-driver-rs)(linux): hyprland background capture + full recording by shuv1337 · Pull Request #1876 · trycua/cua

shuv1337 · 2026-06-10T02:33:04Z

Summary by CodeRabbit

Release Notes

New Features
- Added video recording support for Linux across Wayland and X11 platforms
- Added per-window screenshot capability for Hyprland compositor
- Enhanced element bounds detection and window discovery on Linux
- Improved native text input handling on Wayland sessions
- Added focus-restoration support for app launches on Hyprland
Chores
- Bumped version to 0.5.2

Per-window screenshots on Hyprland now use hyprland-toplevel-export-v1 (vendored XML, wayland-scanner bindings): the toplevel's own buffer is copied, so occluded/background/other-workspace windows capture their real content instead of the grim screen-region crop (which silently returned whatever covered the rectangle). grim remains the fallback; display screenshots prefer grim full-output under Wayland. Recording reaches parity with macOS/Windows on Linux: - video: wlr-screencopy frames piped to ffmpeg rawvideo stdin (30fps wall-clock-paced with frame duplication; bounded stop with encoder kill on back-pressure). Replaces x11grab on Wayland sessions, which only saw the XWayland root; x11grab remains for pure-X11. No silent fallback on Wayland - screencopy failures land in session.json. - per-turn app_state.json (native AT-SPI tree) and element-index click.png markers (element bounds -> window-local screenshot pixels, with native-Wayland window-local extents disambiguation and monitor render-scale handling for fractional scaling). - cursor.jsonl via Hyprland IPC cursorpos at 30Hz, mapped into the recorded monitor's physical pixel space. - hooks hop to deadline-bounded scratch threads (AT-SPI block_on panics on tokio workers; wedged walks must not stall tool calls). Background-use hardening: - launch_app restores the previously focused window when a newly launched window steals focus on Hyprland (Lua dispatch syntax for Hyprland >=0.55 with legacy fallback; only reverts focus grabs by windows that did not exist pre-launch). - AT-SPI snapshots surface Text content as value="..." on editable widgets whose name hides the typed text (GTK entries), escaped for the line-oriented tree markdown. recording_loader no longer feeds click_point into render zoom: it is window-local marker space, not video-frame coordinates (latent mismatch on macOS, activated by the new Linux hooks). Proven live on Hyprland 0.55 at 1.5x fractional scaling: occluded- window capture, 4K30 finalized video, click markers, render + replay. Skill docs updated; Linux recording integration test added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

vercel · 2026-06-10T02:33:10Z

@shuv1337 is attempting to deploy a commit to the Cua Team on Vercel.

A member of the Team first needs to authorize it.

coderabbitai · 2026-06-10T02:33:22Z

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR adds comprehensive Linux recording support by introducing Wayland-native video capture, Hyprland window manager integration, per-turn app state JSON and element bounds via AT-SPI, and unified input handling for native Wayland windows alongside X11 compatibility.

Changes

Linux Recording Support

Layer / File(s)	Summary
Documentation and Build Configuration `libs/cua-driver/rust/Cargo.toml`, `libs/cua-driver/rust/Skills/cua-driver/LINUX.md`, `README.md`, `RECORDING.md`, `WINDOWS.md`, `libs/cua-driver/rust/crates/platform-linux/Cargo.toml`	Version bumped to 0.5.2, Wayland capture dependencies added (wayland-client, wayland-backend, wayland-scanner, wayland-protocols-wlr, bitflags), and documentation updated to reflect Recording as supported on Linux via wlr-screencopy (Wayland) / x11grab (X11), per-window captures with Hyprland protocol, element-index support, app state JSON, and Windows recording via gdigrab.
Wayland Protocol & Core Capture Library `libs/cua-driver/rust/crates/platform-linux/protocols/hyprland-toplevel-export-v1.xml`, `libs/cua-driver/rust/crates/platform-linux/src/wayland_capture.rs`	Hyprland toplevel-export-v1 protocol XML specification added (manager/frame interfaces, buffer negotiation, copy, damage events, Y-invert flags). wayland_capture.rs module implements deadline-bounded Wayland event dispatch, shared-memory pool/buffer management, frame format conversion (RGBA/ffmpeg rawvideo), and APIs for per-window Hyprland protocol capture and repeated full-output screencopy sessions.
Hyprland Window Manager Backend `libs/cua-driver/rust/crates/platform-linux/src/hyprland.rs`	Hyprland session detection via HYPRLAND_INSTANCE_SIGNATURE, window discovery/filtering from `hyprctl clients -j`, per-window screenshots via wayland_capture protocol with grim fallback, full-display grim screenshots, monitor scale/focus queries, active window address polling, and focus-restore guard that refocuses the previous window when focus moves to newly-created windows after launch attempts.
Recording Hooks and AT-SPI Bounds `libs/cua-driver/rust/crates/platform-linux/src/recording_hooks.rs`, `libs/cua-driver/rust/crates/platform-linux/src/atspi/native.rs`	Recording hooks app_state_json_for and element_window_local_xy use timeout-bounded scratch threads to run AT-SPI operations. app_state_json_for validates PID, fetches AT-SPI tree markdown, and returns pretty JSON with element count. element_window_local_xy computes window-local element center coordinates, choosing between screen-relative and window-local interpretations for Wayland vs X11. AT-SPI bounds improved with component extent helpers, XWayland fallback translation, plausibility checks, and markdown-safe value escaping.
Video Recording Backend `libs/cua-driver/rust/crates/platform-linux/src/video.rs`	LinuxVideoBackendFactory selects WaylandVideoBackend on Wayland sessions (WAYLAND_DISPLAY set) and falls back to x11grab. WaylandVideoBackend opens ScreencopyCapturer, captures initial frame for pixel format/geometry lock, spawns ffmpeg with rawvideo stdin encoding, maintains wall-clock-aligned FPS by duplicating last frame on slow capture, counts consecutive failures, and stops after threshold with bounded ffmpeg finalization and stderr logging.
Capture Routing and Window Discovery `libs/cua-driver/rust/crates/platform-linux/src/capture.rs`, `libs/cua-driver/rust/crates/platform-linux/src/x11/mod.rs`, `libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs` (window state validation)	screenshot_window/screenshot_display detect large window IDs (>u32::MAX) and WAYLAND_DISPLAY to route to Hyprland/grim paths. X11 list_windows merges Hyprland results and deduplicates by XID or PID+title. get_window_state validates window visibility via list_windows before returning state.
Tool Input Handling and Cursor Sampling `libs/cua-driver/rust/crates/platform-linux/src/tools/impl_.rs`, `libs/cua-driver/rust/crates/cua-driver-core/src/cursor_sampler.rs`	Tools detect native Wayland window IDs and reject pixel-based input (click, key, hotkey, scroll, etc.) with standardized error; type_text attempts AT-SPI insert_text for Wayland. launch_app forces GDK_BACKEND=x11 and QT_QPA_PLATFORM=xcb for XWayland and installs focus-restore guard. Cursor sampler connects to Hyprland IPC socket, caches focused monitor snapshot, translates logical cursor coordinates to physical video pixel coordinates, and drops samples outside monitor bounds.
Registry Integration and Core Support `libs/cua-driver/rust/crates/cua-driver/src/main.rs`, `libs/cua-driver/rust/crates/platform-linux/src/lib.rs`, `libs/cua-driver/rust/crates/cua-driver-core/src/video_ffmpeg.rs`, `libs/cua-driver/rust/crates/cua-driver-core/src/recording_loader.rs`	Linux video backend set to LinuxVideoBackendFactory in build_registry paths (both with/without cursor). Platform-linux exports hyprland, wayland_capture, video, recording_hooks modules. find_ffmpeg made public for platform reuse. Click coordinate recovery order prefers arguments.x/y before result_summary screen scan, removing click_point fallback.
Testing and Python Auth `libs/cua-driver/rust/crates/cua-driver/tests/mcp_protocol_test.rs`, `libs/python/cua-cli/pyproject.toml`, `libs/python/cua-cli/tests/commands/test_auth.py`	Linux recording integration test launches cua-driver, initializes JSON-RPC, starts recording without video, launches a trivial app, advances turn, stops recording, validates turn-00001 artifact. Python auth tests updated to expect workspace/org persistence via save_workspace/set_active_workspace instead of legacy save_api_key behavior. uv dependency sources configured for local editable paths.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

trycua/cua#1720: Shares recording infrastructure changes including per-turn app_state_json hook wiring, element-index coordinate conversion, and platform video backend factory selection and registration.
trycua/cua#1718: Further modifies cursor sampling (cursor_sampler.rs), click coordinate recovery order (recording_loader.rs), and ffmpeg discovery visibility (video_ffmpeg.rs) already changed in this PR.

Suggested reviewers

ddupont808

Poem

🐰 Wayland whispers grim frames to our lens,
Hyprland heights yield their window friends,
Screencopy spins while ffmpeg encodes,
AT-SPI tells tales of keyboard codes.
From Wayland to X, let recording be free—
Cross-platform at last, from the North Pole to sea! 🎬

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the primary change: adding Hyprland background window capture and full recording support for Linux.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

shuv1337

Opened in error. Inline review comments were deleted and this PR was closed.

shuv1337 · 2026-06-10T02:40:09Z

sorry, stupid clanker got lost

Resolve one Hyprland client from compositor title/app-id plus the verified target PID, then capture its own surface through hyprland-toplevel-export-v1. Ambiguous or unavailable identities retain the typed fail-closed result. Preserve existing capture behavior on other Wayland compositors. Salvaged from trycua#1876. Co-authored-by: shuv <shuv@shuv.dev>

RodriMora · 2026-08-10T19:03:51Z

Material parts of this Hyprland toplevel-export implementation have been salvaged into draft stacked PR #3052, which targets the active fix in #2964. The adaptation credits @shuv1337 with a Co-authored-by trailer and explicitly records Salvaged from #1876; repository attribution CI passes. It has now been validated against a real Hyprland 0.56 XWayland window on an inactive workspace. I’ll keep the landing location linked through #2964 rather than opening a competing main-branch workstream.

shuv1337 and others added 3 commits June 9, 2026 00:54

fix: support cua-driver on hyprland

5ab4e91

test: update cua-cli auth expectations

b400206

shuv1337 commented Jun 10, 2026

View reviewed changes

shuv1337 closed this Jun 10, 2026

AksharP5 mentioned this pull request Aug 7, 2026

Linux/Hyprland: get_window_state returns current-workspace pixels for an off-workspace XWayland window #2962

Closed

This was referenced Aug 10, 2026

fix(cua-driver): fail closed on unproven Wayland window capture #2964

Closed

fix(cua-driver): support Hyprland capture and input #3052

Draft

Navigation Menu

Uh oh!

Uh oh!

feat(cua-driver-rs)(linux): hyprland background capture + full recording - #1876

feat(cua-driver-rs)(linux): hyprland background capture + full recording#1876
shuv1337 wants to merge 3 commits into
trycua:mainfrom
shuv1337:codex/adversarial-review-149f84c2

shuv1337 commented Jun 10, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

vercel Bot commented Jun 10, 2026

Uh oh!

coderabbitai Bot commented Jun 10, 2026 •

edited

Loading

Review failed

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

Uh oh!

shuv1337 left a comment •

edited

Loading

Uh oh!

shuv1337 commented Jun 10, 2026

Uh oh!

RodriMora commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

shuv1337 commented Jun 10, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Release Notes

Uh oh!

vercel Bot commented Jun 10, 2026

Uh oh!

coderabbitai Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

Uh oh!

shuv1337 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shuv1337 commented Jun 10, 2026

Uh oh!

RodriMora commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shuv1337 commented Jun 10, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 10, 2026 •

edited

Loading

shuv1337 left a comment •

edited

Loading