Skip to content

fix(cua-driver): support Hyprland capture and input - #3052

Draft
RodriMora wants to merge 6 commits into
trycua:codex/2962-wayland-fail-closedfrom
RodriMora:fix/2964-hyprland-toplevel-capture
Draft

fix(cua-driver): support Hyprland capture and input#3052
RodriMora wants to merge 6 commits into
trycua:codex/2962-wayland-fail-closedfrom
RodriMora:fix/2964-hyprland-toplevel-capture

Conversation

@RodriMora

@RodriMora RodriMora commented Aug 10, 2026

Copy link
Copy Markdown

Purpose

This is a stacked contribution to #2964, not a competing landing pull request. Its base is #2964's branch. Merge or cherry-pick it into #2964; do not land it independently on main.

The original #2964 candidate correctly failed closed for #2962, but did so for every Wayland window and therefore removed screenshots required by the accepted Sway/capture matrix. This adaptation now covers the complete tested Hyprland desktop contract:

  • correlate one mapped Hyprland client using compositor-owned PID, title, app-id, geometry, workspace, and stable address evidence;
  • capture that exact surface through hyprland-toplevel-export-v1, including occluded and off-workspace XWayland windows;
  • return the existing typed surface_identity_unproven result for missing, ambiguous, timed-out, or failed identity/capture;
  • preserve existing behavior for non-Hyprland Wayland compositors;
  • normalize mixed-scale output, screenshot, cursor, scroll, and renderer accessibility coordinates;
  • position the Hyprland cursor through compositor-owned logical coordinates while retaining standard button/axis protocols;
  • bind embedded Chromium/Electron page routes through compositor-attested native-window cardinality;
  • propagate the interactive Wayland/Hyprland display scope into private workers;
  • expose unsupported WebKitGTK pointer and background shapes as typed refusals instead of false success; and
  • add a representative hyprland desktop harness mode, exact-source evidence, and strict result validation.

Material protocol/capture design is salvaged from #1876 and credited in the commit trailer. The cursor-positioning design is adapted from hypruse and credits Ilyas Khallouki. The attribution config maps the source contributor's verified email to @shuv1337 as required by repository CI.

Validation

Final stack head: 2d6d88e58c2aa163e44c8abb41fc3f89f62fe225

Certified executable candidate: 1bcb4b59148d1bbe82f41717010ee45c4c749039. The final head differs only by carrying forward a trusted main-branch attribution identity override alongside the credited contributor mapping; no executable or test source changed.

Representative Hyprland desktop E2E

Canonical command:

CUA_E2E_SOURCE_SHA=1bcb4b59148d1bbe82f41717010ee45c4c749039 \
  scripts/ci/linux/run-rust-e2e-desktop.sh hyprland --no-build

Result on Arch/Omarchy, Hyprland 0.56, mixed native Wayland/XWayland, and 1.25x/1.5x outputs:

  • 129/129 contracts passed;
  • 79 delivered;
  • 50 expected typed refusals;
  • 0 failed;
  • 0 skipped;
  • exact source SHA recorded in environment.jsonl and the generated report;
  • all required behavioral videos passed; and
  • fixture processes and isolated workspaces were cleaned up, with the personal desktop restored.

This includes Electron, Tauri/WebKitGTK, GTK3, GTK4, private-worker display inheritance, embedded-browser binding, mixed-scale desktop actions, agent cursor pixels, strict click evidence, and off-workspace XWayland target capture.

Focused and package checks

  • cargo fmt --all -- --check
  • git diff --check
  • desktop-wrapper Python tests: 3 passed
  • cargo test -p platform-linux --features portal-input --locked: 285 passed, 4 environment-dependent ignored
  • cargo test -p cua-driver-core --locked: 503 passed, 1 doc test ignored
  • cargo test -p cua-driver-sdk --locked: 45 passed
  • XDG_SESSION_TYPE=x11 cargo test -p cua-driver-testkit --locked: 75 passed
  • release build: cargo build --release -p cua-driver --features portal-input --locked

A separate read-only off-workspace native-window capture smoke preserved the active workspace and focused window while returning the target's current pixels and accessibility state.

Known boundaries and integration gates

  • Standard Wayland cannot deliver arbitrary raw input to an unfocused surface. Semantic background actions remain available; unsupported shapes return exact typed refusals.
  • WebKitGTK ignores Hyprland virtual-pointer button events. Element-addressed left clicks use AT-SPI; unsupported right-click, double-click, drag, and background pixel shapes fail truthfully. The fixture uses WebKitGTK's supported SHM renderer because this host rejects its DMA-BUF explicit-sync path.
  • A manual off-workspace Brave browser-chrome smoke required foreground escalation and exposed that Hyprland did not restore the prior workspace after each native chrome action. The run restored it explicitly. This does not affect target-bound read-only capture, but universal cross-workspace foreground restoration remains a follow-up limitation.
  • portal-capture/all-features builds are blocked on this Arch host by PipeWire 1.6 versus libspa 0.8; the tested portal-input feature builds and passes.
  • Nix/Sway tooling is unavailable on this host, so accepted non-Hyprland behavior is preserved in code and unit coverage but was not replayed through the canonical Sway lane here.
  • The stack remains based on fix(cua-driver): fail closed on unproven Wayland window capture #2964's branch. After it is folded into fix(cua-driver): fail closed on unproven Wayland window capture #2964, that branch still needs rebasing onto current main and ordinary PR CI.

Contributes to #2964. Refs #2962.

Resolve one Hyprland client from compositor title/app-id plus the verified target PID, then capture its own surface through hyprland-toplevel-export-v1. Ambiguous or unavailable identities retain the typed fail-closed result. Preserve existing capture behavior on other Wayland compositors.

Salvaged from trycua#1876.

Co-authored-by: shuv <shuv@shuv.dev>
RodriMora and others added 4 commits August 11, 2026 08:48
Use compositor geometry for stable window ids, mixed-scale pointer positioning, screenshot mapping, and renderer AT-SPI bounds. Exercise WebKitGTK through its valid SHM path and expose unsupported Hyprland button semantics as a typed limitation instead of false success.

Adapted the Hyprland cursor-positioning design from https://github.com/IlyasKhallouki/hypruse.

Co-authored-by: Ilyas Khallouki <IlyasKhallouki@users.noreply.github.com>
Bind embedded browser routes through compositor-attested native cardinality, normalize desktop and scroll coordinates under fractional scaling, keep the foreground sentinel out of the tiling tree, and refuse WebKitGTK background pixel clicks rather than reporting false delivery.
Propagate native Wayland and Hyprland session identity into private workers, preserve explicit 64-bit target ids during recording, classify geometry-free semantic clicks without invented pixel markers, and make two wheel ticks clear a full WebKit viewport.
@RodriMora RodriMora changed the title fix(cua-driver): capture identified Hyprland toplevels fix(cua-driver): support Hyprland capture and input Aug 11, 2026
Carry forward the trusted main-branch identity override while adding the credited Hyprland protocol contributor mapping.
@RodriMora

Copy link
Copy Markdown
Author

The following is human written:
All this work has been obviously AI assisted, but I have tested everything locally and now the hyprland experience is way better and feels like everything is working.
Foreground tasks controlling the browser and windows.
Background tasks checking what the window has.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant