fix(cua-driver-rs)(windows): per-session overlay windows, visible distinct cursors, parallel-transport discoverability by f-trycua · Pull Request #1804 · trycua/cua

f-trycua · 2026-06-01T18:27:49Z

Summary

Four related improvements to the Rust driver's agent-cursor + parallel-agent story, developed and runtime-verified live on Windows:

1. Per-session overlay windows — stable multi-cursor z-order

The overlay was a single layered window compositing every session cursor, pinned z+1 of only the last-active cursor's target — any action by one cursor dragged all others out of their own window's z-band. Each cursor key now owns its own small layered window (lazy-created, destroyed on session_end, supervisor cleans up all on shutdown), each pinned z+1 of its own target, painted into a 128×128 box that follows the cursor instead of a ~14MB full-screen buffer per cursor. Also: WM_DISPLAYCHANGE re-captures the virtual screen (the VM resolution flip 1512×949↔2560×1440 previously left cursors rendering outside a stale overlay), and a disabled cursor hides its window instead of freezing a ghost frame. macOS parity: #1807. Long-lived-daemon degradation: #1806.

2. Visible, distinctly-colored cursors

First accessibility action glides in (seed + glide) instead of an easy-to-miss snap, matching macOS.
set_agent_cursor_motion {cursor_color} now actually renders (it was stored but never reached the painter) via SetGradient.
Collision-free palette assignment: ids sharing a numeric suffix all hashed to one palette (stable_index suffix rule) → identical colors; for_instance_distinct probes past in-use palettes, and the palette grows by six hue-gap colors.
Display capture uses SRCCOPY|CAPTUREBLT so the layered overlay appears in full-screen screenshots.

3. Parallel HTTP transport discoverability

serve --http-port <port> first-class flag (env var still honored).
cua-driver status reports the live mcp-http endpoint (self-correcting via port probe).
The daemon self-advertises the live URL in its MCP initialize instructions only when the listener genuinely bound.

4. Docs: the concurrency contract

SKILL.md + MCP instructions now state it explicitly: one session = one connection = sequential (FIFO); parallelism is across sessions, never within one (per-session cursor + recording assume an ordered stream) — with a worked curl example and the same-daemon element-cache caveat.

Test plan

cargo test: cursor-overlay 7/7, cua-driver-core 51/51, platform-windows 49/49, cua-driver bin 62/62; release build clean.
Runtime-verified on Windows 11 (2560×1440 VM): per-session 128×128 overlay windows tracking their cursors independently; end_session destroys exactly its window; four concurrent sessions filled four windows simultaneously (staggered, sequential-within-session) with distinct colors and stable z; native recording captures it end-to-end.

🤖 Generated with Claude Code

…on HWND A brand-new session cursor stays invisible until the first sibling tool runs because move_cursor doesn't trigger PinAbove. The overlay HWND lands at an unpinned z-position — usually behind the target window — so the visible cursor is hidden under the app even though the daemon reports the move succeeded. Repro: start_session("demo") + move_cursor(x, y) → invisible. Add a click(...) first → cursor appears (click's pin_overlay_above brings the overlay to z+1 of the target HWND). Sibling tools (click, type_text, drag, …) explicitly pin against their declared target window. move_cursor only carries (x, y), so this patch derives the target via WindowFromPoint(x, y) at the destination and feeds it to the existing pin_overlay_above helper. The overlay is WS_EX_TRANSPARENT (overlay.rs:482), so WindowFromPoint walks past it and returns the app behind. Over empty desktop the desktop HWND comes back — pin_overlay_above resolves to GA_ROOT and PinAbove no-ops harmlessly. macOS handles the same bug differently: its move_cursor routes through `animate_cursor_to`, which seeds the off-screen sentinel and lets the render-loop ZOrderEnforcer handle pinning per-tick. The Windows overlay uses explicit per-tool pins instead — hence the per-tool fix here. See project_agent_cursor_zorder.md for cross-platform context. Verified live: fresh session + move_cursor → cursor visible at the destination without any prior click on the target window. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

vercel · 2026-06-01T18:27:56Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Ignored	Preview	Jun 2, 2026 4:07am

coderabbitai · 2026-06-01T18:28:14Z

📝 Walkthrough

Walkthrough

MoveCursorTool::invoke now pins the agent-cursor overlay above the window at the destination coordinates. When cursor placement occurs, the code uses WindowFromPoint to find the HWND and calls pin_overlay_above to ensure z-visibility immediately without waiting for other tools.

Changes

Cursor overlay pinning

Layer / File(s)	Summary
Overlay pinning on cursor move `libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs`	MoveCursorTool::invoke uses `WindowFromPoint` at rounded destination coordinates to locate the target HWND and conditionally pins the overlay above it via `pin_overlay_above`, making the cursor z-visible immediately for new sessions.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

trycua/cua#1801: Builds directly on the per-session cursor model for cursor-key-aware move_cursor behavior.
trycua/cua#1688: Coordinates with the pinned HWND z-order maintenance logic through reapply_z_order.

Poem

🐰 A cursor floats through windows wide,
Now pinned above with z-order pride!
No waiting games, it's bright and fast—
Fresh sessions soar, no ghosting past. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The title mentions per-session overlay windows and visible distinct cursors, but the actual change only handles pinning the overlay z-order above the destination HWND in move_cursor—a specific z-order fix rather than a comprehensive per-session overlay system overhaul.	Consider a more precise title like 'fix(cua-driver-rs)(windows): pin overlay z-order above destination HWND in move_cursor' to accurately reflect the specific z-order bug fix.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/cua-driver-move-cursor-pin-overlay

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

The Streamable-HTTP MCP transport (one connection per agent = true parallelism) was enabled only via an env var and invisible to agents. - serve --http-port <port>: first-class flag (env var still honored, flag wins), documented in --help. - cua-driver status reports 'mcp-http: listening on <url>' / disabled, re-probing the port so a stale state file after a crash reads as off. - The daemon self-advertises the LIVE endpoint URL in its MCP initialize instructions only when the listener actually bound. - Instructions + SKILL.md now spell out the concurrency contract: ONE session = one connection = sequential (FIFO); parallelism is ACROSS sessions, never within one (per-session cursor + recording assume an ordered stream). SKILL.md gains a worked curl example and the same-daemon element-cache caveat. - Fixed stale 'snaps on first action' cursor wording (instructions + --help) to match the seed-and-glide behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Three visibility/color fixes for per-session cursors: - First action glides in instead of snapping: overlay_glide_to dropped its ClickPulse-snap shortcut and always defers to animate_cursor_to (seed_start_if_sentinel gives the seeded glide), matching macOS. - cursor_color now actually renders: set_agent_cursor_motion stored it in the registry but never reached the painter (which reads only gradient_override/palette). It now emits SetGradient with a solid one-stop gradient + lightened bloom. - Collision-free auto palettes: ids sharing a numeric suffix (e.g. mil-005253/rom-005253) all hashed to the SAME palette via stable_index's suffix rule, so concurrent cursors rendered one color. Palette::for_instance_distinct linear-probes past palettes held by live cursors (hash stays the preferred pick), and PALETTE_DATA gains six hue-gap fillers (emerald, gold, indigo, tangerine, hot_pink, teal). Regression-tested. - Display capture uses SRCCOPY|CAPTUREBLT so layered windows (the agent cursor overlay) appear in full-screen screenshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…z-order The overlay was ONE layered window compositing every session cursor, pinned z+1 of only the LAST-ACTIVE cursor's target — so when cursor B acted, the shared window re-pinned to B's z-band and cursors over other windows lost z+1 of their own target (occluded by their own window). Each cursor key now owns its OWN layered window: - RenderState gains hwnd_isize + per-key WinZOrderEnforcer + pixmap; RenderMap.last_active and the shared Z_ORDER band are gone. - Single STA thread + single WM_TIMER still drives everything: lazily creates a key's window on its first observed tick, ticks, paints, and reasserts each window z+1 of ITS OWN pinned_wid. - Each cursor paints into a tiny 128x128 box that FOLLOWS it (UpdateLayeredWindow pt_dst), not a full-virtual-screen RGBA buffer (~14MB/cursor at 2560x1440) — the cursor's drawn extent is fixed (arrow <=14px, bloom +-22px, ring <=~26px). - session_end DestroyWindow's that key's window; supervisor WM_DESTROY tears down every per-key window (no HWND leak when a session never ends); a disabled cursor hides its window once (no frozen ghost frame — found in adversarial review). - WM_DISPLAYCHANGE re-captures the virtual screen and resizes all windows: the VM's resolution flip (1512x949 <-> 2560x1440) left the overlay covering a stale region, so cursors outside it never painted. - Per-key palette assignment threads palettes_in_use into render_state_for_key for the collision-free picker. Runtime-verified: per-session 128x128 overlay windows tracking their cursors independently, end_session destroying exactly its window, and four sessions filling four windows simultaneously with stable z. macOS parity tracked in #1807; long-lived-daemon degradation in #1806. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

f-trycua · 2026-08-09T17:45:08Z

Closing this broad, conflicting prototype because its concerns have since evolved through narrower work: HTTP transport shipped in #1798; cursor badge and presentation work continued in #2616, #2677, and #2842; and the macOS overlay z-order question is tracked in #1800. Any still-desired Windows per-window overlay semantics should be proposed as a focused issue against current main rather than reviving this multi-topic PR.

f-trycua and others added 3 commits June 2, 2026 04:06

f-trycua changed the title ~~fix(cua-driver-rs)(windows): move_cursor pins overlay z-order above destination HWND~~ fix(cua-driver-rs)(windows): per-session overlay windows, visible distinct cursors, parallel-transport discoverability Jun 2, 2026

ddupont808 mentioned this pull request Jun 4, 2026

Windows Improvements: Multi-cursor Overlays, Background Input & DPI Awareness #1836

Closed

ddupont808 added a commit that referenced this pull request Jun 4, 2026

merge: PR #1804 - per-session overlay windows + distinct cursors

ec30e23

f-trycua requested review from ddupont808 and r33drichards as code owners July 16, 2026 11:21

f-trycua closed this Aug 9, 2026

Navigation Menu

Uh oh!

Uh oh!

fix(cua-driver-rs)(windows): per-session overlay windows, visible distinct cursors, parallel-transport discoverability - #1804

fix(cua-driver-rs)(windows): per-session overlay windows, visible distinct cursors, parallel-transport discoverability#1804
f-trycua wants to merge 4 commits into
mainfrom
fix/cua-driver-move-cursor-pin-overlay

f-trycua commented Jun 1, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jun 1, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jun 1, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Poem

❌ Failed checks (1 warning)

Uh oh!

f-trycua commented Aug 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

f-trycua commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

1. Per-session overlay windows — stable multi-cursor z-order

2. Visible, distinctly-colored cursors

3. Parallel HTTP transport discoverability

4. Docs: the concurrency contract

Test plan

Uh oh!

vercel Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Poem

❌ Failed checks (1 warning)

Uh oh!

f-trycua commented Aug 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

f-trycua commented Jun 1, 2026 •

edited

Loading

vercel Bot commented Jun 1, 2026 •

edited

Loading

coderabbitai Bot commented Jun 1, 2026 •

edited

Loading