Skip to content

Start Linux multi-cursor plumbing - #1872

Merged
r33drichards merged 26 commits into
r33d/linux-background-x11-drag-toolsfrom
r33d/linux-multi-cursor-plumbing
Jun 10, 2026
Merged

Start Linux multi-cursor plumbing#1872
r33drichards merged 26 commits into
r33d/linux-background-x11-drag-toolsfrom
r33d/linux-multi-cursor-plumbing

Conversation

@r33drichards

Copy link
Copy Markdown
Collaborator

Summary

  • key Linux held-button state by session/cursor_id instead of a single global hold slot
  • thread the same cursor-key resolution into Linux agent-cursor state/config tools
  • allow querying a single Linux cursor instance by session or cursor_id

Why

Windows already treats cursor instances as keyed state. Linux still had a single held-button lane, which blocks any attempt to orchestrate independent concurrent drag streams. This change does not claim true X11 multi-pointer support yet; it starts by giving Linux the per-cursor state model needed for that next step.

Follow-ups

  • move Linux overlay command routing off the single global cursor
  • thread cursor keys through more pointer actions beyond held-button gestures
  • evaluate whether X11 event delivery needs a stronger per-pointer abstraction than keyed background events

Testing

  • not run locally: this shell does not have cargo installed

@vercel

vercel Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Ignored Ignored Preview Jun 10, 2026 5:44pm

Request Review

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 88eba3d5-d49c-4f5d-b9e2-13e1294bb33d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch r33d/linux-multi-cursor-plumbing

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Linux visual regression artifacts

Matrix jobs now run independently. Download visual artifacts from this workflow run.
Each background-GUI job uploads a .gif of the interaction plus two annotated PNGs (<app>.png raw, <app>-atspi.png with AT-SPI element boxes); the cua-driver-linux-som-overlays artifact adds <app>-som.png cua Set-of-Marks overlays:

  • cua-driver-linux-cursor-click-gif
  • cua-driver-linux-background-terminal-gif
  • cua-driver-linux-background-gui-chromium
  • cua-driver-linux-background-gui-tk
  • cua-driver-linux-background-gui-gtk3-gedit
  • cua-driver-linux-background-gui-gtk3-mousepad
  • cua-driver-linux-background-gui-gtk3-scite
  • cua-driver-linux-background-gui-gtk4-characters
  • cua-driver-linux-background-gui-qt5-manuskript
  • cua-driver-linux-background-gui-qt5-klog
  • cua-driver-linux-background-gui-qt5-openambit
  • cua-driver-linux-background-gui-qt6-kate
  • cua-driver-linux-background-gui-qt6-kcalc
  • cua-driver-linux-background-gui-qt6-okular
  • cua-driver-linux-background-gui-qt6-qownnotes
  • cua-driver-linux-background-gui-electron-zettlr
  • cua-driver-linux-background-gui-electron-joplin
  • cua-driver-linux-background-gui-electron-logseq
  • cua-driver-linux-som-overlays

Open workflow run and download artifacts

@r33drichards

Copy link
Copy Markdown
Collaborator Author

Validated this branch in the live Linux test env after the latest guest rebuild.

What I verified:

  • parallel two-session background drag works through the JS runtime + cua-driver
  • held-button state is scoped per session/cursor
  • bad held-target reuse now fails correctly:
    • mismatched pid returns Cursor '<session>' is holding a button for pid <actual>, not pid <requested>.
    • mismatched window_id returns Cursor '<session>' is holding a button for window_id <actual>, not <requested>.
  • service came back healthy after install and restart

I also recorded a short demo from the test env showing the Linux multi-cursor paint flow.
Artifact in the local workspace:

  • /home/node/cua-multicursor-rec/multicursor-demo-cropped.gif

Constraint from this environment: I can post PR comments, but I cannot directly upload that local GIF binary into the GitHub comment from this shell session. If we want it embedded on the PR, the next step is to push the artifact somewhere addressable or attach it from a browser session.

@r33drichards

Copy link
Copy Markdown
Collaborator Author

Added an inline demo for the current Linux multi-cursor branch.

Inline renderable asset:

Linux multi-cursor demo

Fresh shortlink instance:

  • https://shlink-gif.tail204509.ts.net/multicursor-demo

Notes:

  • I tried the S3 signed-URL path first, but this workspace role can list buckets and cannot PutObject, so there was no writable bucket available from this environment.
  • To get a URL GitHub can actually render inline, I pushed the GIF to a dedicated artifact branch and pointed the new shlink instance at the raw GitHub asset.

@r33drichards

Copy link
Copy Markdown
Collaborator Author

Recorded a cleaner in-flight Linux drag demo from the live gvisor-dev test workspace after fixing the JS orchestration to await mcp.callTool(...) calls.

Embedded GIF:

linux drag motion demo

Raw asset:
https://raw.githubusercontent.com/trycua/cua/artifacts-linux-drag-demo-20260609/artifacts/linux-drag-motion-demo.gif

@r33drichards

Copy link
Copy Markdown
Collaborator Author

Two-cursor drawing GIF from the live Linux test workspace. This is a fresh capture of two independent cua-driver sessions drawing onto the same Tk canvas.

two cursors drawing

Raw asset:
https://raw.githubusercontent.com/trycua/cua/artifacts-linux-drag-demo-20260609/artifacts/two-cursors-drawing.gif

@r33drichards

Copy link
Copy Markdown
Collaborator Author

Lockstep-concurrency GIF from the single-runtime Promise.all(...) path. This capture shows both line draws dispatched concurrently from one JS execution rather than two overlapping executions.

two cursors lockstep

Raw asset:
https://raw.githubusercontent.com/trycua/cua/artifacts-linux-drag-demo-20260609/artifacts/two-cursors-lockstep.gif

@r33drichards

Copy link
Copy Markdown
Collaborator Author

Lockstep-concurrency GIF from single-runtime Promise.all(...) dispatch.

two cursors lockstep

Raw asset:
https://raw.githubusercontent.com/trycua/cua/artifacts-linux-drag-demo-20260609/artifacts/two-cursors-lockstep.gif

codex and others added 8 commits June 9, 2026 06:42
Three fixes that turn the Linux parallel_mouse_drag uinput/MPX pipeline
from "raw events only" into real app-visible same-window drags:

- The cursor overlay's click-through was a no-op: ShapeMask with a None
  pixmap *resets* the input shape to the full window, so the fullscreen
  overlay swallowed every pointer event before it reached any app. Set
  an empty input region via ShapeRectangles with no rects instead.

- XIWarpPointer was only XFlush'd, so the uinput button press (separate
  kernel pipeline) raced ahead of the queued warp and landed at the
  master's spawn position (screen center), starting an implicit grab on
  the desktop. XSync the warp so the press lands at the warped spot.

- Relative uinput deltas went through libinput's adaptive accel profile
  and drag endpoints drifted a few pixels off-target. Pin the slave's
  accel profile to flat after attaching it to its master.

Also moves the drag path off the fake single-pointer/XTEST approach to
per-session MPX master pointers fed by per-session uinput slaves with
concurrent step scheduling (carried over from earlier session work).

Verified in the Xorg+x11vnc desktop-workspace guest: an XI2 test app
receives interleaved cooked XI_ButtonPress/XI_Motion/XI_ButtonRelease
for both masters with pixel-exact endpoints across repeated calls and
fresh MCP sessions, and paints all concurrent strokes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@r33drichards

r33drichards commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

Live demo of parallel_mouse_drag as of 0ac9a51 — two MPX master pointers drawing concurrently in the same window, agent cursors tracking each gesture, and the window manager never sees the press at all: the focused terminal keeps its active titlebar throughout — no mid-drag flicker, nothing to restore afterwards.

two agent cursors drawing concurrently with zero focus steal

How it works (the "shield grab"): before each drag's press, the driver installs a device-specific XI2 synchronous passive button grab for that master pointer on the target window. Per the X server's grab rules (verified against dix/events.c/dix/grabs.c): this grab is newer than the WM's click-to-focus grab on the same window so it's matched first, and being device-specific it never conflicts with the WM's core/all-master grabs. The press freezes on our grab and is delivered to us; XIAllowEvents(XIReplayDevice) then replays it, which re-checks grabs only below the grab window and delivers the press normally to the app. So the app gets the full drag (press → implicit-grab motion → release) while the WM is completely blind to it — focus is never touched.

Each press is replayed before the next is emitted, so only one device is ever frozen at a time (the server drops replayed presses if several are frozen on one window and replayed together). The earlier EWMH/core focus save-restore remains only as a safety net for non-conforming setups.

Verified on xfwm4 (the wedge-prone WM): _NET_ACTIVE_WINDOW polled at 50 ms across consecutive two-cursor calls never left the terminal; 40/40 presses+releases delivered at pixel-exact coordinates across 20 stress calls; masters cleaned up; WM stayed healthy.

🤖 Generated with Claude Code

The drag loop now snaps each session's overlay cursor to the
interpolated step position (heading along the motion vector), so the
gesture is visible on screen instead of the cursor teleporting from
start to end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
r33drichards pushed a commit that referenced this pull request Jun 9, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
codex and others added 3 commits June 10, 2026 00:41
Click-to-focus WMs grab buttons for XIAllMasterDevices, so an MPX
drag's press activates the dragged window exactly like a user click —
parallel agent drags were stealing focus from whatever the user had
active.

Snapshot the focus state before the drags and hand it back afterwards,
WM-agnostically:

- EWMH path: re-activate the previous _NET_ACTIVE_WINDOW via client
  message (source=pager) so the WM's own bookkeeping stays consistent.
  The WM finishes its click handling on the release replay, which can
  land after a one-shot request, so settle briefly then verify/retry.
- Fallback for bare X / non-EWMH WMs: save and restore the core input
  focus directly, under a scoped ignore-errors handler since the saved
  window may be gone by then.

Also drops the XISetClientPointer/XISetFocus calls before drags — they
were debugging leftovers, not needed for cooked event delivery, and
pointing master keyboard focus at the target fights the restore.

Verified under openbox: delivery unchanged, active window returns to
the previously-focused terminal after every call. xfwm4 4.18 has a
broken MPX path — one foreign-master press and it permanently ignores
_NET_ACTIVE_WINDOW requests (ours and xdotool's alike, until restarted)
— so the desktop-workspace image should ship a different WM; the
driver logs a warning when the WM refuses the re-activation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_NET_ACTIVE_WINDOW requests carrying CurrentTime(0) lose to the WM's
focus-stealing prevention whenever newer input exists, so fetch the
server time via the standard PropertyNotify round-trip and stamp the
re-activation request with it. With concurrent live user input the WM
may still legitimately refuse (that IS focus-stealing prevention); the
driver retries briefly and logs a warning if the WM declines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two changes that keep non-MPX-aware WMs healthy across parallel drags:

- Remove the per-session master pairs as soon as the gesture completes
  instead of leaving them attached until session end. One-shot MCP
  sessions never end, so masters used to accumulate without bound, and
  lingering foreign master keyboards multiply the core focus-event
  noise WMs have to digest.

- The focus restore now requires the active window to hold stable for
  consecutive checks, and escalates to a core-focus bounce when plain
  re-activation doesn't take. Verified mechanism: after an MPX click, a
  core-protocol WM (xfwm4, openbox) can record the dragged window as
  active while the core focus never actually moved there; its
  XSetInputFocus for our re-activation request is then a no-op, no
  FocusIn ever arrives, and its bookkeeping stays wedged — even real
  user clicks stop updating the titlebar. Setting the core focus onto
  the window the WM believes active and then re-activating produces the
  focus transition its state machine is waiting for.

With this, two consecutive parallel_mouse_drag calls against an
unfocused window under openbox leave the user's terminal focused
(titlebar, _NET_ACTIVE_WINDOW, and core focus all agree), with no
leftover master devices.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
r33drichards pushed a commit that referenced this pull request Jun 10, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
codex and others added 2 commits June 10, 2026 02:05
Prevents the focus steal at the source instead of restoring focus after
the fact: the window manager now never sees the drag's button press.

Before each press, install a device-specific XI2 synchronous passive
button grab (XIGrabButton) for the drag's master pointer on the target
window. Per the X server's grab semantics (dix/events.c, dix/grabs.c):
this grab is newer than the WM's click-to-focus grab on the same window
so it's matched first, and being device-specific it never BadAccess-
conflicts with the WM's core/all-master grabs. The press freezes the
device and is delivered to us; XIAllowEvents(XIReplayDevice) then replays
it, which re-checks passive grabs only *below* the grab window and
delivers the event normally to the app — so the app gets the full drag
(press, implicit-grab motion, release) while the WM is blind to it.

Shields are removed after release. Failures degrade gracefully: if a
shield can't install or a frozen press isn't seen before a 1s timeout,
the drag still runs and the existing focus save/restore catches any
leak.

Verified on xfwm4 (the wedge-prone WM): _NET_ACTIVE_WINDOW polled at
50ms across two consecutive two-cursor drag calls never left the
focused terminal — no mid-drag flicker — the app received all presses
and releases at pixel-exact coordinates, masters were cleaned up, and
the WM stayed healthy (activation still toggles both ways afterwards).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first shield cut focus-stealing but dropped ~1 in 8 presses under
concurrent drags: with both masters' presses frozen on their grabs and
replayed back to back, the X server delivered only one. Diagnostics
confirmed both XIReplayDevice calls returned success yet only one press
reached the app — a server-side race when multiple devices are frozen on
the same window and replayed together (XSync between replays narrowed it
but didn't close it).

Fix: replay each frozen press immediately after emitting it, before the
next press, so only one device is ever frozen at a time. The few-ms
stagger this adds to the presses is invisible — the concurrency that
matters is the motion phase, which is unchanged.

Verified on xfwm4: 40/40 presses and releases delivered across 20
consecutive two-cursor calls, focus never left the terminal, WM stayed
healthy, no leftover masters.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
r33drichards pushed a commit that referenced this pull request Jun 10, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
r33drichards and others added 2 commits June 10, 2026 10:32
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The MPX multi-cursor plumbing pulls in the raw-Xlib x11 crate, whose
build.rs locates libX11/libXi/libXtst via pkg-config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@r33drichards

Copy link
Copy Markdown
Collaborator Author

Hillclimb summary — CI is green ✅

All 19 failing checks (NixOS integration test, cursor click GIF, terminal GIF, and the 15 background GUI matrix jobs) were dying on the same step: building the cua-driver Rust crate in Nix. Two fixes:

  1. 97d5b26dcargoHash in nix/cua-driver/package.nix was stale after this PR changed the Cargo dependency set. Recomputed the vendor hash (sha256-3oz8KeW8a6ak8uOLqPCmb4Sf59f2c4NXr6PTti8eS/Q=).
  2. 33707962 — the new raw-Xlib x11 crate (MPX multi-cursor plumbing) has a build.rs that locates libX11/libXi/libXtst via pkg-config, but the derivation declared empty build inputs from its pure-Rust days. Added pkg-config to nativeBuildInputs and libx11/libxi/libxtst/libxext to buildInputs.

After the build was unblocked, all VM tests passed on the first try — 29/29 checks green on run 27294778618.

🤖 Generated with Claude Code

@r33drichards
r33drichards marked this pull request as ready for review June 10, 2026 21:32
@r33drichards
r33drichards merged commit 45debc1 into r33d/linux-background-x11-drag-tools Jun 10, 2026
28 of 29 checks passed
r33drichards added a commit that referenced this pull request Jun 11, 2026
* Add Linux background drag hold tools

* Start Linux multi-cursor plumbing (#1872)

* Start Linux multi-cursor plumbing

* Key Linux overlay state by cursor

* Fix Linux keyed overlay build

* Restore Linux click tool arg helpers

* Restore Linux double-click arg helpers

* Restore Linux right-click arg helpers

* Import Linux tool arg helpers once

* Tighten Linux multi-cursor tool validation

* Show Linux cursor state during drags

* Animate Linux drag overlay in motion engine

* Add Linux MPX parallel drag tool

* Fix x11 feature flags for MPX tool

* Fix MPX tool FFI usage

* Use XTEST slave devices for Linux MPX drags

* Initialize Xlib threading for Linux MPX drags

* Fail MPX drags on Xtigervnc and clean up masters

* Detect Xtigervnc by process for MPX guard

* Fix MPX drag delivery: overlay click-through, warp sync, flat accel

Three fixes that turn the Linux parallel_mouse_drag uinput/MPX pipeline
from "raw events only" into real app-visible same-window drags:

- The cursor overlay's click-through was a no-op: ShapeMask with a None
  pixmap *resets* the input shape to the full window, so the fullscreen
  overlay swallowed every pointer event before it reached any app. Set
  an empty input region via ShapeRectangles with no rects instead.

- XIWarpPointer was only XFlush'd, so the uinput button press (separate
  kernel pipeline) raced ahead of the queued warp and landed at the
  master's spawn position (screen center), starting an implicit grab on
  the desktop. XSync the warp so the press lands at the warped spot.

- Relative uinput deltas went through libinput's adaptive accel profile
  and drag endpoints drifted a few pixels off-target. Pin the slave's
  accel profile to flat after attaching it to its master.

Also moves the drag path off the fake single-pointer/XTEST approach to
per-session MPX master pointers fed by per-session uinput slaves with
concurrent step scheduling (carried over from earlier session work).

Verified in the Xorg+x11vnc desktop-workspace guest: an XI2 test app
receives interleaved cooked XI_ButtonPress/XI_Motion/XI_ButtonRelease
for both masters with pixel-exact endpoints across repeated calls and
fresh MCP sessions, and paints all concurrent strokes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Track parallel drags with the agent cursor overlay

The drag loop now snaps each session's overlay cursor to the
interpolated step position (heading along the motion vector), so the
gesture is visible on screen instead of the cursor teleporting from
start to end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Restore the active window after parallel drags

Click-to-focus WMs grab buttons for XIAllMasterDevices, so an MPX
drag's press activates the dragged window exactly like a user click —
parallel agent drags were stealing focus from whatever the user had
active.

Snapshot the focus state before the drags and hand it back afterwards,
WM-agnostically:

- EWMH path: re-activate the previous _NET_ACTIVE_WINDOW via client
  message (source=pager) so the WM's own bookkeeping stays consistent.
  The WM finishes its click handling on the release replay, which can
  land after a one-shot request, so settle briefly then verify/retry.
- Fallback for bare X / non-EWMH WMs: save and restore the core input
  focus directly, under a scoped ignore-errors handler since the saved
  window may be gone by then.

Also drops the XISetClientPointer/XISetFocus calls before drags — they
were debugging leftovers, not needed for cooked event delivery, and
pointing master keyboard focus at the target fights the restore.

Verified under openbox: delivery unchanged, active window returns to
the previously-focused terminal after every call. xfwm4 4.18 has a
broken MPX path — one foreign-master press and it permanently ignores
_NET_ACTIVE_WINDOW requests (ours and xdotool's alike, until restarted)
— so the desktop-workspace image should ship a different WM; the
driver logs a warning when the WM refuses the re-activation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Stamp focus-restore activation with real server time

_NET_ACTIVE_WINDOW requests carrying CurrentTime(0) lose to the WM's
focus-stealing prevention whenever newer input exists, so fetch the
server time via the standard PropertyNotify round-trip and stamp the
re-activation request with it. With concurrent live user input the WM
may still legitimately refuse (that IS focus-stealing prevention); the
driver retries briefly and logs a warning if the WM declines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Clean up drag masters per call and resync wedged WM focus state

Two changes that keep non-MPX-aware WMs healthy across parallel drags:

- Remove the per-session master pairs as soon as the gesture completes
  instead of leaving them attached until session end. One-shot MCP
  sessions never end, so masters used to accumulate without bound, and
  lingering foreign master keyboards multiply the core focus-event
  noise WMs have to digest.

- The focus restore now requires the active window to hold stable for
  consecutive checks, and escalates to a core-focus bounce when plain
  re-activation doesn't take. Verified mechanism: after an MPX click, a
  core-protocol WM (xfwm4, openbox) can record the dragged window as
  active while the core focus never actually moved there; its
  XSetInputFocus for our re-activation request is then a no-op, no
  FocusIn ever arrives, and its bookkeeping stays wedged — even real
  user clicks stop updating the titlebar. Setting the core focus onto
  the window the WM believes active and then re-activating produces the
  focus transition its state machine is waiting for.

With this, two consecutive parallel_mouse_drag calls against an
unfocused window under openbox leave the user's terminal focused
(titlebar, _NET_ACTIVE_WINDOW, and core focus all agree), with no
leftover master devices.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Shield parallel drags from the WM with a device-specific grab + replay

Prevents the focus steal at the source instead of restoring focus after
the fact: the window manager now never sees the drag's button press.

Before each press, install a device-specific XI2 synchronous passive
button grab (XIGrabButton) for the drag's master pointer on the target
window. Per the X server's grab semantics (dix/events.c, dix/grabs.c):
this grab is newer than the WM's click-to-focus grab on the same window
so it's matched first, and being device-specific it never BadAccess-
conflicts with the WM's core/all-master grabs. The press freezes the
device and is delivered to us; XIAllowEvents(XIReplayDevice) then replays
it, which re-checks passive grabs only *below* the grab window and
delivers the event normally to the app — so the app gets the full drag
(press, implicit-grab motion, release) while the WM is blind to it.

Shields are removed after release. Failures degrade gracefully: if a
shield can't install or a frozen press isn't seen before a 1s timeout,
the drag still runs and the existing focus save/restore catches any
leak.

Verified on xfwm4 (the wedge-prone WM): _NET_ACTIVE_WINDOW polled at
50ms across two consecutive two-cursor drag calls never left the
focused terminal — no mid-drag flicker — the app received all presses
and releases at pixel-exact coordinates, masters were cleaned up, and
the WM stayed healthy (activation still toggles both ways afterwards).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Replay shield presses per-device to fix dropped concurrent presses

The first shield cut focus-stealing but dropped ~1 in 8 presses under
concurrent drags: with both masters' presses frozen on their grabs and
replayed back to back, the X server delivered only one. Diagnostics
confirmed both XIReplayDevice calls returned success yet only one press
reached the app — a server-side race when multiple devices are frozen on
the same window and replayed together (XSync between replays narrowed it
but didn't close it).

Fix: replay each frozen press immediately after emitting it, before the
next press, so only one device is ever frozen at a time. The few-ms
stagger this adds to the presses is invisible — the concurrency that
matters is the motion phase, which is unchanged.

Verified on xfwm4: 40/40 presses and releases delivered across 20
consecutive two-cursor calls, focus never left the terminal, WM stayed
healthy, no leftover masters.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(nix): update cua-driver cargoHash for new dependency set

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(nix): add pkg-config and X11 libs for the x11 crate build

The MPX multi-cursor plumbing pulls in the raw-Xlib x11 crate, whose
build.rs locates libX11/libXi/libXtst via pkg-config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* Replace /proc cmdline scan with direct X lock file + exe check for TigerVNC detection

* Add NixOS integration test piloting parallel_mouse_drag (MPX)

A scenario that drives cua-driver through its Linux multi-cursor
parallel_mouse_drag path end to end, in the style of the other
linux-*-gif NixOS tests, and records a GIF artifact.

The scenario (mcp-parallel-drag-test.py) pilots the driver over MCP
through two parallel_mouse_drag calls — two per-session master pointers
drawing concurrent crossing strokes into one window that does NOT hold
focus — and the test asserts the three guarantees the feature rests on:

- concurrent cooked delivery: an XI2 paint app (compiled in-tree) logs
  every window-targeted XI2 event with its device id; we assert four
  presses from two distinct master devices plus motion;
- no focus steal: the shield grab keeps the WM blind to the presses, so
  the separate control window stays active throughout;
- a GIF of the two crossing strokes is produced and copied out.

Unlike the other Linux visual tests this one needs a REAL Xorg (dummy
video driver + libinput) rather than Xvfb, because the MPX path attaches
uinput slave devices to per-session masters and only a real Xorg with
libinput enumerates uinput devices as X input devices — the same reason
the old Xvfb/Xtigervnc setups couldn't host it. The test loads the
uinput kernel module and launches Xorg with an explicit module path.

Wired into flake checks and the nix-build CI matrix (+ the visual
artifact comment list).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(platform-linux): fail parallel drag when shield protection is unavailable

* fix(platform-linux): import anyhow::Context for mandatory-shield drag

The mandatory-shield change uses .with_context() on the shield-grab
install, but anyhow::Context wasn't in scope, so the crate didn't
compile. Add it to the use list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add confirmation-gated install_ffmpeg tool for the video backend

start_recording's video capture shells out to the ffmpeg binary on
Linux/Windows (macOS records natively). When ffmpeg is absent, video
silently fails. Add an install_ffmpeg MCP tool that resolves it:

- Detects the platform package manager (apt-get/dnf/yum/zypper/pacman/
  apk/snap on Linux, brew on macOS, winget/choco on Windows) and, on
  Linux, prefixes sudo -n when not root.
- Two-step / confirmed: without `confirm` it only REPORTS the exact
  command it would run (read-only preview); `confirm: true` runs it.
  Marked destructive + open_world so conforming MCP clients also gate it
  behind a human approval. No-op if ffmpeg is already on PATH.
- start_recording's video-failed message now points at install_ffmpeg
  when find_ffmpeg() is None.

ffmpeg is still only ever invoked as a separate process, never linked —
this just installs the same user-provided binary find_ffmpeg looks for.

Verified end to end on Linux: with ffmpeg removed, start_recording
reports the hint; install_ffmpeg (no confirm) previews
`apt-get install -y ffmpeg`; confirm:true installs it; video capture
then works.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(platform-linux): function (y=f(x)) paths + held glide for parallel_mouse_drag

parallel_mouse_drag pressed and released on every call, so drawing a curve
as many short calls looked like a chain of clicks rather than a drag.

Now each drag item carries a full waypoint PATH: the master presses once at
path[0], glides through every point (arc-length interpolated over the whole
duration), and releases once at the end — a single continuous held drag, so
a curved stroke is smooth instead of stamped.

The path is given either as a straight from→to segment (back-compatible) or
as a function `fn` = y(x) sampled over [x_from, x_to] in window-local pixels
(e.g. fn:"x", fn:"300+150*sin(x/40)", fn:"560-0.0011*(x-400)^2"). Expressions
are evaluated with the meval crate (+ - * / ^, sin/cos/tan, sqrt, abs, exp,
ln, pi, e). Each cursor can follow its own function, all concurrently.

- VirtualPointerDrag now holds `path: Vec<(i32,i32)>` (screen coords) instead
  of from/to; the scheduler interpolates position by arc-length fraction.
- The tool builds the path once per item, translating window-local → screen
  via a single origin lookup (translate is a pure offset).
- steps defaults to a path-length-scaled count for a smooth glide.

Verified on Linux: three cursors drew a line, a sine, and a parabola in one
call as smooth continuous strokes, focus held on another window.

NOTE: adds the `meval` dependency — nix package.nix cargoHash needs regen.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(platform-linux): unit tests for held-path + fn sampling; reset cargoHash

- Add path_tests: path_cumulative / point_on_path (arc-length glide) and
  sample_function (y=f(x) sampling: linear, affine, trig, invalid-expr,
  non-finite drop). Factor the fn sampling out of the tool into
  input::sample_function so it's unit-testable without X.
- Reset nix cargoHash to "" so CI prints the correct got: hash for the
  new meval/nom/fnv dependency set (will be filled in next commit).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(nix): update cargoHash for the meval/nom/fnv dependency set

The function-path drag feature added the meval crate (+ nom, fnv) to
Cargo.lock, which changed the vendored dependency set; the nix cua-driver
build (and every nix integration job that depends on it) failed with a
cargoHash mismatch. Set the new fetchCargoVendor hash.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ci: de-list parallel-drag GIF test (needs real Xorg, not viable in emulated VM)

The cargoHash fix turned the nix suite green except this one test, which
fails because a hand-launched real Xorg (dummy+libinput) — required so the
MPX uinput slaves enumerate as X input devices — does not start within the
timeout in the emulated GHA nixos-test VM (uinput loads fine; the 'Start a
real Xorg' subtest times out).

The parallel_mouse_drag / held-path / fn feature is covered by unit tests
(platform-linux/src/input/mod.rs). Remove this scenario from the flake
checks + CI matrix + artifact list; keep the .nix file (documented) for
local / real-X manual runs. Can be brought back to CI via services.xserver
if a reliable real-X env is wired up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(nix): CI-viable parallel-drag test on real Xorg via services.xserver

Replaces the hand-launched-Xorg linux-parallel-drag-gif.nix (which timed
out because a self-launched Xorg can't get a VT/seat in the emulated
nixos-test VM) with a version that lets NixOS bring Xorg up properly:

  - services.xserver with the `dummy` video driver + a 1280x1024 virtual
    screen, libinput input backend, and an icewm window manager
  - lightdm + services.displayManager.autoLogin a normal user into the
    icewm session; the session runs `xhost +local:` so the root-run
    cua-driver / test clients can connect to :0
  - boot.kernelModules uinput + udev rule so MPX slaves enumerate as X
    input devices under libinput

Same proof as before: two per-session master pointers draw concurrent,
window-targeted XI2 events into one unfocused window (assert 2 distinct
devices + >=4 motions), the shield grab keeps focus on a separate control
window, and a GIF artifact is produced. First-run diagnostics dump the
display-manager journal / Xorg log if X fails to come up.

Wired into flake checks and the nix-build matrix (+ visual-artifact list).
The old GIF scenario file is kept for local/real-X manual runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(nix): start X with -ac so the root-run driver can reach :0

The first CI run booted fine — wait_for_x() confirmed /tmp/.X11-unix/X0 and
the graphical target, lightdm autologged-in the user, and the session's
`xhost +local:` ran — but root's xdpyinfo on :0 still timed out for 60s. The
server's auth ACL only lists the autologin user, and xhost +local: did not
reliably grant the other uid. Disable access control outright with the X
`-ac` flag (fine for a throwaway single-user test VM); keep xhost as a
fallback. Also extend the bring-up diagnostics to cover the xdpyinfo step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(nix): install xdpyinfo (the :0 connectivity probe)

Both prior CI runs failed at the same line — the X bring-up subtest's
`DISPLAY=:0 xdpyinfo` probe timed out — not because root couldn't reach :0
but because xdpyinfo was never in systemPackages, so the command was
not-found and thus always nonzero. The dumped Xorg log shows X itself is
healthy (XINPUT enumerates devices). Add xorg.xdpyinfo so the probe runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants