feat(windows): background input without z-raise + multi-cursor demos + overlay improvements - #1809
Merged
Merged
Conversation
…ithout z-raise or cursor movement Unifies the Windows input path so a caller just targets an app and plays click/type actions — no `dispatch` knob, no app-internals knowledge — and the target window is never raised to the foreground. Mechanism (input/inject.rs): - NoActivateGuard: per-window WS_EX_NOACTIVATE for the duration of a background click/type. Windows then refuses to make the target foreground at all (click-activation, WM_MOUSEACTIVATE, and a WPF/XAML/Tauri handler's own SetForegroundWindow(self) are all denied) while the window still RECEIVES the input. Per-window, reverted on drop — no session side effects. - Coordinate-routed, cursor-free delivery, layered by the default background path: UIA Invoke (Chromium DOM / WebView2 / UWP / controls) -> PostMessage (plain Win32 + WM_CHAR text) -> touch injection (canvas left-click) / pen injection (right-click). ZorderGuard (DWM cloak + SetWindowPos SWP_NOACTIVATE) hides any residual z movement. - Keyboard accelerators are capability-first: inject_key_cloaked cloaks the target, takes focus via the AttachThreadInput trick (beats the foreground-lock without UIAccess), SendInputs the combo so GetKeyState/TranslateAccelerator fire, then restores the user's foreground. The keystroke is always delivered; the brief focus is hidden and reverted. Falls back to PostMessage if focus can't be obtained — never drops the action. The global SPI_SETFOREGROUNDLOCKTIMEOUT freeze was tried and rejected: it is a session-wide setting that leaks if the daemon dies mid-action, and is ineffective anyway (our own injected input legitimizes the target's foreground claim even under a maxed lock). Verified end-to-end in crates/cua-driver/tests/e2e_windows_bg_input_test.rs (6/6, #[ignore], serial): Electron/Chromium, Tauri/WebView2, and Win32 — left click, right click, text type, and Ctrl+A — all stay background (oracle: target pid never becomes foreground) with the keystroke/click delivered. The suite auto-downloads the trycua test apps, runs every child under a JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE Job Object plus a ChildBag/broker-pid teardown so nothing orphans (even on panic or Ctrl-C), discovers multi-process app windows via a new-window diff, and bounds every JSON-RPC call with a 25s timeout. Also adds examples/zdrop_probe.rs (z-order/foreground probe) and docs/windows-background-input-re-plan.md (RE methodology, findings, and the implemented solution). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ross 5 UI frameworks
A live demo of background actuation: one action in a foreground "master" window
is replayed onto FOUR background windows simultaneously, each driven by its own
cua-driver session = its own uniquely-coloured agent cursor — with no window
raised and the user's mouse never moved.
Layout: 2x2 corners + a center master (the foreground window the human drives).
Five UI frameworks, spanning the with/without-accessibility-tree axis:
- Win32 + GDI custom-drawn — NO a11y tree -> pixel/injection path (crimson)
- .NET WinForms — MSAA/UIA -> UIA Invoke (amber)
- .NET WPF (XAML) — UIA -> UIA Invoke (aqua)
- Electron (Chromium) — UIA -> UIA Invoke (mint_lime)
- Win32 standard controls — center/master (instrumented, emits user actions)
The Rust orchestrator starts the cua-driver daemon, launches + positions the
five windows, reads the master's emitted click/type events, and fans each one
out to the four corners over four concurrent `cua-driver call ... {session:<color>}`
invocations. Every spawned process is assigned to a KILL_ON_JOB_CLOSE Job Object
so closing the master (or killing the orchestrator) tears the whole tree down
with no orphans.
Verified (--auto self-play, 3 cycles): all four corners driven concurrently
(TYPE + CLICK) — including the no-a11y GDI corner via the pixel path — with the
foreground staying on the center master throughout (no corner ever z-raised),
and a clean teardown (no orphaned processes).
Components: legacy-app (Rust Win32, gdi + master modes), orchestrator (Rust),
dotnet/{winforms,wpf}, electron/, README.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…overlay perf cursor-overlay: glide_duration_ms now drives a fixed-duration glide in the shared render path (tick_motion + tick_swift_constants), so a move takes that exact time regardless of distance on macOS/Windows/Linux alike; 0 (the default) keeps the original speed-based timing — no platform drift. Adds regression tests. platform-windows/overlay: repaint-gate — skip the full-virtual-screen composite + UpdateLayeredWindow on frames where nothing changed (no command, no cursor mid-glide/spring/click), so a resting overlay costs ~nothing instead of blitting at the timer rate. Adds an env-gated overlay-FPS probe (CUA_DRIVER_RS_OVERLAY_FPS_FILE) for diagnosing render throughput. platform-macos + platform-windows: correct the glide_duration_ms tool doc (0 = speed-based default), matched across both surfaces. Also carries this branch's in-progress windows background-input refinements (input/inject, dispatch, mouse, mod) and the RE plan doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…computer-use A grid of miniwob-style minigame windows, one per MIDI track, each actuated by its own coloured cua-driver agent cursor in the background — the click is what triggers the sound. One cursor per part, one colour per agent, off a single clock: coordinated multi-cursor computer-use as music. - jukebox-app: one Win32/GDI binary, two modes. Controller = a 600x80 transport bar (white "CUA JUKEBOX", a single ▶/⏸ icon toggle, track-colour swatches, playhead; accepts a .mid drop). Instrument = a 200x160 borderless tile: a pitch strip (click-X = semitone, fitted per-track so wide ranges aren't clamped), a kick/snare/hat drum pad (MIDI ch.10 → zones), or a brightness pad. Each owns a rodio synth voice (sine/square/saw/triangle + drum waves) so notes mix at the OS mixer. Double-buffered, region-clipped repaints. - orchestrator: parses a .mid (midly) or a built-in demo loop, infers each track's voice from its name, lays out the grid, and pins each track to a fixed-duration glide. Per-track cursor POOLS grow to the same colour when notes fall within the glide window (chords / fast lines fan out). Drives every note over a persistent per-voice named-pipe connection to the daemon (no per-note process spawn), with a per-voice adaptive lead (EMA) that tracks the measured actuation error to ~0 median; reports the timing diff at song end. PLAY/PAUSE/STOP; dropping a .mid re-execs the demo onto that track. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…liding The arrow heading was rate-limited toward the motion tangent (rotate_toward at 14 rad/s), so on fast/short glides — especially the fixed-duration ones — it couldn't turn quickly enough and lagged behind (or pointed the wrong way) during the move. Assign the heading directly to the path tangent each frame instead, so the tip tracks the trajectory. Applied to both shared render paths (tick_motion for Windows/Linux, tick_swift_constants for macOS) — no drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…+ click latency decouple cursor-overlay/motion: Added `turn_radius` field to MotionConfig (default 80, clamped 1–1000) so callers can tighten glide curves. Exposed on both Windows and macOS `set_agent_cursor_motion` schemas (parity). The shared path planner reads it; no platform drift. platform-windows/overlay: Multi-window z-order fix — when >1 distinct window is driven, pin the overlay above the TOPMOST driven window (walks z-order top→down via GetTopWindow/GW_HWNDNEXT) rather than flipping to the last-active one. This keeps the overlay above all driven windows but below anything stacked above them (the user's foreground), fixing the cursor blink-out when many windows are actuated concurrently (e.g. jukebox's 8 tiles). Removed the earlier repaint-gate (skip composite when idle) — starting/stopping the full-screen UpdateLayeredWindow as activity changed made resting cursors flicker when any one clicked; steady per-frame blit is flicker-free and costs little during playback since something's almost always active. platform-windows/tools/impl_: Click latency decoupling — when `glide_duration_ms > 0` (fixed-duration glide), `overlay_glide_to` now sleeps the known duration instead of awaiting the render thread's arrival oneshot, so click dispatch is deterministic (glide_duration_ms wall-clock) and independent of overlay FPS. Under many concurrent cursors (10–12 fps debug, ~60 fps release) the arrival-await coupled click latency to render-thread scheduling; the sleep decouples it. Speed-based glides (glide_duration_ms == 0, the default) keep the precise arrival-await since their duration depends on distance — non-breaking. Per-cursor fade in/out already respected (each cursor's idle_alpha is applied in paint_cursor), and now animates correctly every frame with the repaint-gate removed. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…iles Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR adds two complete demo applications ( ChangesDemo Applications: Jukebox and Multi-Cursor Showcases
Windows Background Input Infrastructure and Hardening
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related issues
Possibly related PRs
Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 13
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs (1)
4058-4118:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winReturn
turn_radiusin the motion payloads too.Line 4094 applies the new
turn_radius, but Lines 4098-4118 still serialize the old motion shape. That makes the field look like it was ignored, andget_agent_cursor_statelater in this file has the same gap. Please addturn_radiusto the setter response and getter payload so callers can verify what was applied.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs` around lines 4058 - 4118, The motion's turn_radius is applied but not returned: update the invoke() response and the get_agent_cursor_state getter to include the turn_radius field; specifically, add updated.turn_radius into the formatted summary (similar to glideDurationMs/dwellAfterClickMs) and include "turn_radius": updated.turn_radius in the structured JSON returned by ToolResult::text(...).with_structured(...) and likewise add turn_radius to the payload produced by get_agent_cursor_state so callers can verify the applied value.
🟡 Minor comments (8)
demo/jukebox/README.md-15-23 (1)
15-23:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winAdd fence languages to satisfy markdownlint.
Lines 15 and 96 start unlabeled fenced blocks, which is why markdownlint is already flagging MD040 here. Adding a language like
textkeeps the README lint-clean without changing the rendered output.Also applies to: 96-98
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@demo/jukebox/README.md` around lines 15 - 23, Add a language label to the unlabeled fenced code blocks to satisfy markdownlint MD040: replace the opening ``` fences for the ASCII-art block that begins with "┌───────────────────────────────────────────────────────────────┐" and the other unlabeled fenced block further down with a labeled fence such as ```text (and keep the corresponding closing ``` unchanged) so the rendered output is identical but markdownlint no longer flags the blocks.libs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rs-94-105 (1)
94-105:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winTighten the shared
dispatchdescription.This string is now out of sync with the shipped behavior:
inject_key_cloaked/inject_text_cloakedtemporarily foreground the target, andinject_click_screensupports right-click. Leaving the schema text as “background never swaps foreground” and “right/middle click” unavailable will misstate the tool contract to callers.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rs` around lines 94 - 105, Update the shared "dispatch" description string in dispatch.rs to reflect current behavior: remove the absolute claim that "background" never swaps foreground (acknowledge that inject_key_cloaked and inject_text_cloaked may temporarily foreground targets), remove the statement that right/middle clicks are unavailable (inject_click_screen supports right-click), and clarify the fallback behavior and error conditions (e.g., background_unavailable for inputs that cannot be expressed). Edit the description text associated with the dispatch schema so it mentions these exceptions and accurately describes modes 'background', 'foreground', and 'auto'; locate and update the string literal named or used for the shared dispatch description in this file.libs/cua-driver/rust/docs/windows-background-input-re-plan.md-18-27 (1)
18-27:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winUpdate the implementation notes to match the code that landed.
This section still says left-clicks use
InjectTouchInputand thatZorderGuardhides residual z movement via cloaking, but the current implementation uses synthetic pen taps for clicks and the guard only toggles topmost z-order. Keeping this stale will send the next investigation down the wrong path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libs/cua-driver/rust/docs/windows-background-input-re-plan.md` around lines 18 - 27, The docs are stale: update the implementation notes so they match the landed code by replacing references to touch-based left-clicks (InjectTouchInput) with the current synthetic-pen tap approach (InjectSyntheticPointerInput / PT_PEN used for clicks), and change the description of ZorderGuard from a cloak that hides residual z movement to the actual behavior where it only toggles topmost z-order. Also note that dispatch:"background" still transparently selects the appropriate delivery method (UIA Invoke, PostMessage, or synthetic pen injection) so callers don't pass a dispatch knob.libs/cua-driver/rust/crates/cua-driver/tests/e2e_windows_bg_input_test.rs-67-81 (1)
67-81:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winJob object configuration failure is silently ignored.
If
SetInformationJobObjectfails, the job won't haveKILL_ON_JOB_CLOSEand child processes could orphan on test crash. Consider logging the failure for diagnostic visibility.Suggested fix
- let _ = SetInformationJobObject( + if SetInformationJobObject( h, JobObjectExtendedLimitInformation, &info as *const _ as *const c_void, std::mem::size_of::<JOBOBJECT_EXTENDED_LIMIT_INFORMATION>() as u32, - ); + ).is_err() { + eprintln!("[e2e] WARN: SetInformationJobObject failed — orphan protection may not work"); + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libs/cua-driver/rust/crates/cua-driver/tests/e2e_windows_bg_input_test.rs` around lines 67 - 81, The job() helper currently ignores the return value of SetInformationJobObject so the JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE flag may not be applied; update job() to check the result of SetInformationJobObject (called after CreateJobObjectW and creating JOBOBJECT_EXTENDED_LIMIT_INFORMATION with BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE) and on failure either log the error details (using the test logger) or panic with a descriptive message including the Windows error code so test runs surface the configuration failure and avoid orphaned child processes.demo/multi-cursor/README.md-15-28 (1)
15-28:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winAdd a language tag to this fenced block.
This trips the documented markdownlint rule and also drops syntax-highlighting/hinting for the layout snippet.
textwould be enough here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@demo/multi-cursor/README.md` around lines 15 - 28, The fenced code block containing the ASCII diagram (the triple backticks wrapping the layout with lines like "MASTER", "Win32 GDI", "WinForms", etc.) needs a language tag to satisfy markdownlint and enable highlighting; update the opening fence from ``` to ```text so the block is treated as plain text and the lint rule/syntax highlighting are applied.demo/multi-cursor/README.md-46-58 (1)
46-58:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winDocument the .NET 10 prerequisite in the build steps.
Both
dotnetdemo projects targetnet10.0-windows, but the README only saysdotnet build. Calling out the required SDK version here will save readers from a generic “target framework not supported” failure.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@demo/multi-cursor/README.md` around lines 46 - 58, Update the Build section to call out that both dotnet demo projects target net10.0-windows and therefore require the .NET 10 SDK to be installed; mention the exact target "net10.0-windows" and add a short note/example command to check/install the SDK (for example `dotnet --list-sdks` or a link to Microsoft .NET 10 download) so users know to install .NET 10 before running the existing `dotnet build` commands for the WinForms and WPF projects.demo/multi-cursor/orchestrator/src/main.rs-305-311 (1)
305-311:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winThe DWM fallback returns screen-space coordinates, not window-local coordinates.
When
DwmGetWindowAttributefails, this returns(pt.x, pt.y)directly afterClientToScreen, so the fallback path feeds absolute screen coordinates into a helper that otherwise returns window-local screenshot pixels. Any window on that path will click the wrong location.Proposed fix
- if DwmGetWindowAttribute(hwnd, DWMWA_EXTENDED_FRAME_BOUNDS, - &mut dwm as *mut _ as *mut core::ffi::c_void, - std::mem::size_of::<RECT>() as u32).is_err() - { - return Some((pt.x, pt.y)); - } + if DwmGetWindowAttribute(hwnd, DWMWA_EXTENDED_FRAME_BOUNDS, + &mut dwm as *mut _ as *mut core::ffi::c_void, + std::mem::size_of::<RECT>() as u32).is_err() + { + return Some((pt.x - cr.left, pt.y - cr.top)); + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@demo/multi-cursor/orchestrator/src/main.rs` around lines 305 - 311, The fallback branch after DwmGetWindowAttribute currently returns screen-space coords (pt.x, pt.y); instead convert those screen coords to window-local coords like the success path does by querying the window rectangle and subtracting its origin. Replace the early return with a GetWindowRect (or equivalent) call for hwnd, store into a RECT (e.g., rect), and return Some((pt.x - rect.left - 1, pt.y - rect.top - 1)); if GetWindowRect itself fails you can fall back to returning the original pt but prefer using the rect subtraction to keep coordinates consistent with the DwmGetWindowAttribute path (symbols: DwmGetWindowAttribute, ClientToScreen, hwnd, pt, RECT).demo/multi-cursor/legacy-app/src/main.rs-232-243 (1)
232-243:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winPreserve the full
WM_CHARcode unit here.
wparam.0 as u8 as chardrops everything above0xFF, so any non-ASCII text typed into the GDI target is corrupted before it reachesst.text.Proposed fix
- let ch = wparam.0 as u8 as char; + let Some(ch) = char::from_u32(wparam.0 as u32) else { + return LRESULT(0); + };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@demo/multi-cursor/legacy-app/src/main.rs` around lines 232 - 243, The WM_CHAR handler currently truncates characters with `wparam.0 as u8 as char`, losing codepoints above 0xFF; change how `ch` is derived so the full Unicode code unit is preserved (e.g., convert `wparam.0` to u32 and use char::from_u32 or equivalent to get an Option<char>`), then handle the Option (ignore invalid codepoints) and keep the existing logic that treats backspace, newline, and non-control characters when updating `st.text` in the STATE/Mode::Gdi branch.
🧹 Nitpick comments (2)
libs/cua-driver/rust/crates/platform-windows/src/overlay.rs (1)
604-608: 💤 Low valueUnused
drainedcounter.The
drainedvariable is incremented on each message processed but only consumed bylet _ = drained;which discards it. Either remove the tracking or use it (e.g., in the FPS diagnostic output).Option A: Remove dead code
- let mut drained = 0u32; if let Ok(rx_guard) = CMD_RX_WIN.try_lock() { if let Some(ref rx) = *rx_guard { while let Ok(m) = rx.try_recv() { - drained += 1; if let Some(k) = apply_msg(map, m) { map.last_active = Some(k); } } } } ... - let _ = drained;Option B: Include in FPS diagnostic
- let _ = writeln!(f, "overlay fps={fps:.1} avg_dt_ms={:.1} cursors={cursors}", secs * 1000.0 / n as f64); + let _ = writeln!(f, "overlay fps={fps:.1} avg_dt_ms={:.1} cursors={cursors} drained={drained}", secs * 1000.0 / n as f64);Also applies to: 663-663
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libs/cua-driver/rust/crates/platform-windows/src/overlay.rs` around lines 604 - 608, The local counter `drained` is unused—remove the dead variable and its increments in the CMD_RX_WIN try_lock / try_recv loop (the `drained` declaration, the `drained += 1;` line and the later `let _ = drained;` discard) so the loop simply drains messages without tracking, or alternatively, if you prefer to keep the metric, wire `drained` into the FPS/diagnostic output where FPS is computed; locate the try_recv loop guarded by CMD_RX_WIN and either delete the `drained` handling or replace the discard with emitting the value into the existing FPS diagnostic path.libs/cua-driver/rust/crates/platform-windows/examples/zdrop_probe.rs (1)
213-226: 💤 Low valueDead code:
x, yparameters are captured but explicitly discarded.The
xandyparameters are passed torun_trialbut thenlet _ = (x, y);discards them. They're used by the closure, but this explicit discard suggests incomplete refactoring. Either remove the line or the parameters if they're truly unused in this scope.Remove the dead binding
fn run_trial(name: &str, baseline: HWND, target: HWND, x: i32, y: i32, actuate: impl FnOnce()) { let b = baseline.0 as usize; let t = target.0 as usize; let poller = thread::spawn(move || poll(b, t, Duration::from_millis(1200))); thread::sleep(Duration::from_millis(120)); // let poller establish "baseline on top" - let _ = (x, y); actuate();🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@libs/cua-driver/rust/crates/platform-windows/examples/zdrop_probe.rs` around lines 213 - 226, run_trial currently takes x and y but then immediately discards them with let _ = (x, y); which prevents any closure from capturing those parameters — remove that dead binding so the closure passed as actuate can capture and use x and y, or if no caller/closure actually needs them, remove x and y from run_trial's signature and update all callers accordingly; locate the function run_trial and either delete the let _ = (x, y); line or remove the parameters and fix call sites.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@demo/jukebox/app/src/main.rs`:
- Around line 354-356: The WM_KEYDOWN branch that calls PostQuitMessage when
wp.0 == 0x1B should only run for the controller's main window; change the
condition to also check the message's hwnd against the stored controller window
handle (e.g., compare the wndproc's hwnd param to your controller window
variable like controller_hwnd or MAIN_WINDOW_HANDLE) before calling
PostQuitMessage. Locate the WM_KEYDOWN arm containing wp.0 == 0x1B in main.rs
and modify it to require hwnd == controller_hwnd (or an equivalent identity
check) so Esc quits only the controller window.
In `@demo/jukebox/orchestrator/src/main.rs`:
- Around line 217-224: The cleanup loop currently calls
Command::new("taskkill").args(["/F", "/IM", img]) which indiscriminately kills
every process by image name; instead record and target only the PIDs/process
handles you spawned for this run. Change the launcher code that spawns
cua-driver.exe and jukebox-app.exe to store their Child objects or PIDs (e.g., a
Vec of Child or pid_t) and replace the taskkill-by-name loop with termination
logic that iterates only over those stored children (use taskkill /PID or
Child::kill on Windows), or create and use a Job Object / unique tag so you can
reap processes created by this process only. Ensure the code that currently uses
Command::new("taskkill") and the for loop over
["cua-driver.exe","jukebox-app.exe"] is updated to reference the stored
PIDs/Child handles instead of image names.
- Around line 635-652: The active-note map `on` is keyed only by pitch so
overlapping notes or same pitch on different channels collide; change the key to
include channel (e.g., use a tuple (channel.as_int(), key.as_int()) as the
HashMap key) and update all insert/remove accesses in the event loop (the
TrackEventKind::Midi NoteOn and NoteOff arms that call on.insert and on.remove)
to use that composite key, ensuring you still push Note entries using the
original pitch and velocity and preserve drum_notes logic for channel 9.
In `@demo/multi-cursor/dotnet/winforms/Program.cs`:
- Around line 12-13: Program.Main currently calls
Application.EnableVisualStyles() and
Application.SetCompatibleTextRenderingDefault(false) but does not initialize the
WinForms bootstrap; call ApplicationConfiguration.Initialize() at the start of
Program.Main (before EnableVisualStyles/SetCompatibleTextRenderingDefault) so
the project-wide <ApplicationHighDpiMode>PerMonitorV2</ApplicationHighDpiMode>
is applied; update Program.Main to invoke ApplicationConfiguration.Initialize()
(or alternatively call Application.SetHighDpiMode(...) using the same
PerMonitorV2 mode) to ensure PerMonitorV2 DPI behavior for the app.
In `@demo/multi-cursor/orchestrator/src/main.rs`:
- Around line 338-346: find_window_by_title currently only matches window
captions and can return windows from unrelated processes; change its signature
to accept the expected child PID and filter EnumWindows by that PID. Pass the
expected pid into the Finder struct (or add an expected_pid field), then in
enum_cb call GetWindowThreadProcessId for each HWND and only consider matches
where the process id equals expected_pid and the caption contains needle; keep
the existing timeout/sleep behavior and return the (HWND, pid) when both pid and
title match. Ensure callers pass the spawned child's PID to the updated
find_window_by_title.
- Around line 190-203: Remove the unsupported "session" property from the JSON
payloads sent by run_call in the click and type_text paths (e.g., the click call
in the Action::Type branch and the subsequent type_text call) so the payloads
match the tool schemas; keep pid, window_id (hwnd_addr), x/y or text and ensure
json_escape is used for text. Update window selection logic in
find_window_by_title/EnumWindows to prefer the spawned window by verifying the
window's process id (GetWindowThreadProcessId) matches the spawn pid (or use the
exact HWND returned at spawn) instead of relying solely on
title.contains(&needle). Fix client_rel_to_local_px so that when
DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS) fails you still compute and
subtract the window origin and the 1px inset (e.g., fall back to GetWindowRect
and subtract left/top +1) before returning (pt.x - origin.x - 1, pt.y - origin.y
- 1) to ensure coordinates map correctly.
In `@libs/cua-driver/rust/crates/platform-windows/src/input/inject.rs`:
- Around line 304-312: The guard created in arm() currently always forces
HWND_TOPMOST and drop() always sets HWND_NOTOPMOST, which loses the window’s
original topmost state; modify arm() (in the same impl that defines arm and
Drop) to detect and store the target’s original topmost status (e.g., call the
existing set_topmost/inspection helper or use GetWindowLongPtr/WS_EX_TOPMOST
check) as a new field like original_topmost on the returned Self, and then
change drop() to restore the recorded original_topmost (call set_topmost(target,
original_topmost)) instead of unconditionally using HWND_NOTOPMOST so the
window’s prior z-order class is preserved.
- Around line 304-312: ZorderGuard::arm currently sets set_topmost for any
non-foreground target, causing visible z-raises; change arm to only promote the
target when it is actually occluded by other windows (not merely
not-foreground). Replace the raised condition (!target.0.is_null() && target !=
prev_fg) with a call to a helper like is_obscured(target, prev_fg) that returns
true only if the target's client/target area is covered (e.g., check
IsWindowVisible, GetWindowRect and whether WindowFromPoint(s) over the target
return a different window, or test several sample points including center and
edges), then call set_topmost(target, true) only when that helper reports
occlusion and set raised accordingly; ensure prev_fg, target, and raised reflect
the actual action taken so no unnecessary z-raise occurs.
- Around line 390-407: inject_click_screen (and the related coordinate-routed
entry point around lines 585-629, e.g., inject_drag_screen) currently proceeds
when target == 0 or the HWND is stale, letting coordinate-based injection affect
whatever window is under (sx,sy); reject null/invalid targets early by
validating the target HWND before any guards or coordinate routing.
Specifically, in inject_click_screen and the counterpart function, check if
target == 0 and/or call a Windows validity check on the constructed HWND (e.g.,
IsWindow) immediately after computing target_h and before calling
NoActivateGuard::arm or ZorderGuard::arm; if the HWND is null or not a valid
window, return an Err/bail! with a clear message (e.g., "invalid or null target
HWND") so injection is aborted early.
- Around line 224-232: The guard incorrectly skips applying WS_EX_NOACTIVATE
when prev == 0; update NoActivateGuard so it doesn't gate on prev != 0: compute
want = WS_EX_NOACTIVATE.0 as isize and set applied = (prev & want) == 0 && {
SetWindowLongPtrW(root, GWL_EXSTYLE, prev | want); (GetWindowLongPtrW(root,
GWL_EXSTYLE) & want) != 0 }; keep prev_exstyle = prev and root_addr as before so
we correctly record the original style and whether the change was applied even
when prev was zero.
In `@libs/cua-driver/rust/crates/platform-windows/src/input/mouse.rs`:
- Around line 204-206: The first PostMessageW call sends WM_MOUSEMOVE before the
down_msg but uses the existing wparam (mk_flag); change it to send the pre-drag
WM_MOUSEMOVE with a wParam of 0 (no button state) instead of wparam so the
sequence is: PostMessageW(target, WM_MOUSEMOVE, 0, make_lparam(c_from.x,
c_from.y)) followed by PostMessageW(target, down_msg, wparam,
make_lparam(c_from.x, c_from.y)); keep the rest of the parameters (target,
down_msg, make_lparam) unchanged.
In `@libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs`:
- Around line 1850-1862: The NoActivateGuard is currently armed only when an
explicit window_id (hwnd_opt) is present, leaving pid-resolved clicks
unprotected; change the logic so you defer creating _noact until after the
actual target HWND is resolved and then arm crate::input::NoActivateGuard::arm
with that resolved HWND (converted to windows::Win32::Foundation::HWND).
Specifically, remove or relocate the current match that sets _noact around
hwnd_opt and instead construct _noact after the code path that resolves hwnd
(the later resolution logic around the places that reference hwnd or resolve by
pid), guarding both explicit hwnd_opt and pid-resolved HWNDs while preserving
the DispatchMode::Foreground no-op behavior.
- Around line 3736-3772: The injected drag path is missing duration_ms: when
switching to synthetic injection in the background branch you must thread the
existing duration_ms into the injection helper so the wall-clock drag duration
remains consistent. Update the call to
crate::input::inject::inject_drag_screen(...) to accept and forward duration_ms
(and update that function signature/implementations where defined), and ensure
any spawn_blocking closure and callers (e.g., the branch in impl_.rs around
dispatch == DispatchMode::Background, overlay_glide_to, and the
inject::inject_drag_screen function) propagate the duration_ms parameter so both
PostMessage and injected paths honor the same duration contract.
---
Outside diff comments:
In `@libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs`:
- Around line 4058-4118: The motion's turn_radius is applied but not returned:
update the invoke() response and the get_agent_cursor_state getter to include
the turn_radius field; specifically, add updated.turn_radius into the formatted
summary (similar to glideDurationMs/dwellAfterClickMs) and include
"turn_radius": updated.turn_radius in the structured JSON returned by
ToolResult::text(...).with_structured(...) and likewise add turn_radius to the
payload produced by get_agent_cursor_state so callers can verify the applied
value.
---
Minor comments:
In `@demo/jukebox/README.md`:
- Around line 15-23: Add a language label to the unlabeled fenced code blocks to
satisfy markdownlint MD040: replace the opening ``` fences for the ASCII-art
block that begins with
"┌───────────────────────────────────────────────────────────────┐" and the
other unlabeled fenced block further down with a labeled fence such as ```text
(and keep the corresponding closing ``` unchanged) so the rendered output is
identical but markdownlint no longer flags the blocks.
In `@demo/multi-cursor/legacy-app/src/main.rs`:
- Around line 232-243: The WM_CHAR handler currently truncates characters with
`wparam.0 as u8 as char`, losing codepoints above 0xFF; change how `ch` is
derived so the full Unicode code unit is preserved (e.g., convert `wparam.0` to
u32 and use char::from_u32 or equivalent to get an Option<char>`), then handle
the Option (ignore invalid codepoints) and keep the existing logic that treats
backspace, newline, and non-control characters when updating `st.text` in the
STATE/Mode::Gdi branch.
In `@demo/multi-cursor/orchestrator/src/main.rs`:
- Around line 305-311: The fallback branch after DwmGetWindowAttribute currently
returns screen-space coords (pt.x, pt.y); instead convert those screen coords to
window-local coords like the success path does by querying the window rectangle
and subtracting its origin. Replace the early return with a GetWindowRect (or
equivalent) call for hwnd, store into a RECT (e.g., rect), and return Some((pt.x
- rect.left - 1, pt.y - rect.top - 1)); if GetWindowRect itself fails you can
fall back to returning the original pt but prefer using the rect subtraction to
keep coordinates consistent with the DwmGetWindowAttribute path (symbols:
DwmGetWindowAttribute, ClientToScreen, hwnd, pt, RECT).
In `@demo/multi-cursor/README.md`:
- Around line 15-28: The fenced code block containing the ASCII diagram (the
triple backticks wrapping the layout with lines like "MASTER", "Win32 GDI",
"WinForms", etc.) needs a language tag to satisfy markdownlint and enable
highlighting; update the opening fence from ``` to ```text so the block is
treated as plain text and the lint rule/syntax highlighting are applied.
- Around line 46-58: Update the Build section to call out that both dotnet demo
projects target net10.0-windows and therefore require the .NET 10 SDK to be
installed; mention the exact target "net10.0-windows" and add a short
note/example command to check/install the SDK (for example `dotnet --list-sdks`
or a link to Microsoft .NET 10 download) so users know to install .NET 10 before
running the existing `dotnet build` commands for the WinForms and WPF projects.
In `@libs/cua-driver/rust/crates/cua-driver/tests/e2e_windows_bg_input_test.rs`:
- Around line 67-81: The job() helper currently ignores the return value of
SetInformationJobObject so the JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE flag may not
be applied; update job() to check the result of SetInformationJobObject (called
after CreateJobObjectW and creating JOBOBJECT_EXTENDED_LIMIT_INFORMATION with
BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE) and on
failure either log the error details (using the test logger) or panic with a
descriptive message including the Windows error code so test runs surface the
configuration failure and avoid orphaned child processes.
In `@libs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rs`:
- Around line 94-105: Update the shared "dispatch" description string in
dispatch.rs to reflect current behavior: remove the absolute claim that
"background" never swaps foreground (acknowledge that inject_key_cloaked and
inject_text_cloaked may temporarily foreground targets), remove the statement
that right/middle clicks are unavailable (inject_click_screen supports
right-click), and clarify the fallback behavior and error conditions (e.g.,
background_unavailable for inputs that cannot be expressed). Edit the
description text associated with the dispatch schema so it mentions these
exceptions and accurately describes modes 'background', 'foreground', and
'auto'; locate and update the string literal named or used for the shared
dispatch description in this file.
In `@libs/cua-driver/rust/docs/windows-background-input-re-plan.md`:
- Around line 18-27: The docs are stale: update the implementation notes so they
match the landed code by replacing references to touch-based left-clicks
(InjectTouchInput) with the current synthetic-pen tap approach
(InjectSyntheticPointerInput / PT_PEN used for clicks), and change the
description of ZorderGuard from a cloak that hides residual z movement to the
actual behavior where it only toggles topmost z-order. Also note that
dispatch:"background" still transparently selects the appropriate delivery
method (UIA Invoke, PostMessage, or synthetic pen injection) so callers don't
pass a dispatch knob.
---
Nitpick comments:
In `@libs/cua-driver/rust/crates/platform-windows/examples/zdrop_probe.rs`:
- Around line 213-226: run_trial currently takes x and y but then immediately
discards them with let _ = (x, y); which prevents any closure from capturing
those parameters — remove that dead binding so the closure passed as actuate can
capture and use x and y, or if no caller/closure actually needs them, remove x
and y from run_trial's signature and update all callers accordingly; locate the
function run_trial and either delete the let _ = (x, y); line or remove the
parameters and fix call sites.
In `@libs/cua-driver/rust/crates/platform-windows/src/overlay.rs`:
- Around line 604-608: The local counter `drained` is unused—remove the dead
variable and its increments in the CMD_RX_WIN try_lock / try_recv loop (the
`drained` declaration, the `drained += 1;` line and the later `let _ = drained;`
discard) so the loop simply drains messages without tracking, or alternatively,
if you prefer to keep the metric, wire `drained` into the FPS/diagnostic output
where FPS is computed; locate the try_recv loop guarded by CMD_RX_WIN and either
delete the `drained` handling or replace the discard with emitting the value
into the existing FPS diagnostic path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4d7724d9-9f5d-4074-9b7b-3057f86d2d8f
⛔ Files ignored due to path filters (2)
demo/jukebox/Cargo.lockis excluded by!**/*.lockdemo/multi-cursor/Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (35)
.gitignoredemo/jukebox/Cargo.tomldemo/jukebox/README.mddemo/jukebox/app/Cargo.tomldemo/jukebox/app/src/main.rsdemo/jukebox/orchestrator/Cargo.tomldemo/jukebox/orchestrator/src/main.rsdemo/multi-cursor/.gitignoredemo/multi-cursor/Cargo.tomldemo/multi-cursor/README.mddemo/multi-cursor/dotnet/winforms/Program.csdemo/multi-cursor/dotnet/winforms/winforms.csprojdemo/multi-cursor/dotnet/wpf/Program.csdemo/multi-cursor/dotnet/wpf/wpf.csprojdemo/multi-cursor/electron/index.htmldemo/multi-cursor/electron/main.jsdemo/multi-cursor/electron/package.jsondemo/multi-cursor/legacy-app/Cargo.tomldemo/multi-cursor/legacy-app/src/main.rsdemo/multi-cursor/orchestrator/Cargo.tomldemo/multi-cursor/orchestrator/src/main.rslibs/cua-driver/rust/crates/cua-driver/Cargo.tomllibs/cua-driver/rust/crates/cua-driver/tests/e2e_windows_bg_input_test.rslibs/cua-driver/rust/crates/cursor-overlay/src/motion.rslibs/cua-driver/rust/crates/cursor-overlay/src/render_state.rslibs/cua-driver/rust/crates/platform-macos/src/tools/cursor_tools.rslibs/cua-driver/rust/crates/platform-windows/Cargo.tomllibs/cua-driver/rust/crates/platform-windows/examples/zdrop_probe.rslibs/cua-driver/rust/crates/platform-windows/src/input/dispatch.rslibs/cua-driver/rust/crates/platform-windows/src/input/inject.rslibs/cua-driver/rust/crates/platform-windows/src/input/mod.rslibs/cua-driver/rust/crates/platform-windows/src/input/mouse.rslibs/cua-driver/rust/crates/platform-windows/src/overlay.rslibs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rslibs/cua-driver/rust/docs/windows-background-input-re-plan.md
Comment on lines
+354
to
+356
| // Esc quits (borderless windows have no close button); closing the | ||
| // transport EOFs its stdout, which tears the whole demo down. | ||
| WM_KEYDOWN if wp.0 == 0x1B => { PostQuitMessage(0); LRESULT(0) } |
Contributor
There was a problem hiding this comment.
Restrict Esc-to-quit to the controller window.
Line 356 currently exits any jukebox-app process, not just the transport. If an instrument tile ever gets focus, pressing Esc drops that part out of the performance while the orchestrator keeps running.
Proposed fix
- WM_KEYDOWN if wp.0 == 0x1B => { PostQuitMessage(0); LRESULT(0) }
+ WM_KEYDOWN if wp.0 == 0x1B => {
+ if STATE.with(|s| s.borrow().as_ref().map(|st| st.mode)) == Some(Mode::Controller) {
+ PostQuitMessage(0);
+ }
+ LRESULT(0)
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| // Esc quits (borderless windows have no close button); closing the | |
| // transport EOFs its stdout, which tears the whole demo down. | |
| WM_KEYDOWN if wp.0 == 0x1B => { PostQuitMessage(0); LRESULT(0) } | |
| // Esc quits (borderless windows have no close button); closing the | |
| // transport EOFs its stdout, which tears the whole demo down. | |
| WM_KEYDOWN if wp.0 == 0x1B => { | |
| if STATE.with(|s| s.borrow().as_ref().map(|st| st.mode)) == Some(Mode::Controller) { | |
| PostQuitMessage(0); | |
| } | |
| LRESULT(0) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@demo/jukebox/app/src/main.rs` around lines 354 - 356, The WM_KEYDOWN branch
that calls PostQuitMessage when wp.0 == 0x1B should only run for the
controller's main window; change the condition to also check the message's hwnd
against the stored controller window handle (e.g., compare the wndproc's hwnd
param to your controller window variable like controller_hwnd or
MAIN_WINDOW_HANDLE) before calling PostQuitMessage. Locate the WM_KEYDOWN arm
containing wp.0 == 0x1B in main.rs and modify it to require hwnd ==
controller_hwnd (or an equivalent identity check) so Esc quits only the
controller window.
Comment on lines
+217
to
+224
| for img in ["cua-driver.exe", "jukebox-app.exe"] { | ||
| let _ = Command::new("taskkill").args(["/F", "/IM", img]) | ||
| .stdout(Stdio::null()).stderr(Stdio::null()).status(); | ||
| } | ||
| // Give the killed daemon time to release its named pipe before we start a | ||
| // fresh one — too short a wait races the pipe handoff and can wedge the new | ||
| // daemon's first connections (observed as a one-off ~10s-late performance). | ||
| thread::sleep(Duration::from_millis(800)); |
Contributor
There was a problem hiding this comment.
Avoid killing every matching process on the machine.
Line 217 reaps by image name, so starting this demo will forcibly terminate any other cua-driver.exe or jukebox-app.exe the user already has running. That's a pretty rough side effect for a demo launcher and can also break concurrent validation runs.
Safer options are to scope cleanup to windows/PIDs created by this demo, or to tag spawned children with a unique identifier and reap only those.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@demo/jukebox/orchestrator/src/main.rs` around lines 217 - 224, The cleanup
loop currently calls Command::new("taskkill").args(["/F", "/IM", img]) which
indiscriminately kills every process by image name; instead record and target
only the PIDs/process handles you spawned for this run. Change the launcher code
that spawns cua-driver.exe and jukebox-app.exe to store their Child objects or
PIDs (e.g., a Vec of Child or pid_t) and replace the taskkill-by-name loop with
termination logic that iterates only over those stored children (use taskkill
/PID or Child::kill on Windows), or create and use a Job Object / unique tag so
you can reap processes created by this process only. Ensure the code that
currently uses Command::new("taskkill") and the for loop over
["cua-driver.exe","jukebox-app.exe"] is updated to reference the stored
PIDs/Child handles instead of image names.
Comment on lines
+12
to
+13
| Application.EnableVisualStyles(); | ||
| Application.SetCompatibleTextRenderingDefault(false); |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
In an SDK-style WinForms project targeting net10.0-windows, does setting <ApplicationHighDpiMode>PerMonitorV2</ApplicationHighDpiMode> require calling ApplicationConfiguration.Initialize() or Application.SetHighDpiMode(...) from Main for the DPI mode to take effect?
💡 Result:
Yes, calling ApplicationConfiguration.Initialize in your Main method is required for the PerMonitorV2 setting in your project file to take effect [1][2][3]. In modern SDK-style WinForms projects targeting.NET 6 and later (including net10.0-windows), ApplicationConfiguration.Initialize is a source-generated method that reads your project's MSBuild properties—such as ApplicationHighDpiMode, ApplicationDefaultFont, and ApplicationVisualStyles—and automatically emits the corresponding Application.* configuration method calls at runtime [1][4][5]. Specifically, when you set PerMonitorV2, the generated ApplicationConfiguration.Initialize method internally executes the equivalent of: Application.SetHighDpiMode(HighDpiMode.PerMonitorV2); If you do not call ApplicationConfiguration.Initialize in your Main method, these project-level settings will not be applied, and the application will default to the standard SystemAware DPI mode [1][4][2]. While you could manually call Application.SetHighDpiMode(HighDpiMode.PerMonitorV2) in your code instead of using the bootstrap API, Microsoft recommends using ApplicationConfiguration.Initialize as the standard approach to ensure all project-level configurations are applied consistently [2][3]. If you choose to use the bootstrap API, ensure it is called before Application.Run and any other UI elements are initialized [6].
Citations:
- 1: https://learn.microsoft.com/en-us/dotnet/core/project-sdk/msbuild-props-desktop
- 2: https://learn.microsoft.com/en-us/dotnet/desktop/winforms/wfdev-diagnostics/wfac010
- 3: https://github.com/dotnet/docs-desktop/blob/main/dotnet-desktop-guide/winforms/wfdev-diagnostics/wfac010.md
- 4: https://github.com/dotnet/designs/blob/main/accepted/2021/winforms/streamline-application-bootstrap.md
- 5: https://devblogs.microsoft.com/dotnet/whats-new-in-windows-forms-in-net-6-0/
- 6: Setting/Getting HighDpiMode in WinForms Core Apps dotnet/winforms#135
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show relevant Program.cs portion
echo "=== Program.cs (around startup) ==="
ls -la demo/multi-cursor/dotnet/winforms/Program.cs
nl -ba demo/multi-cursor/dotnet/winforms/Program.cs | sed -n '1,120p'
echo
echo "=== winforms.csproj (look for ApplicationHighDpiMode) ==="
ls -la demo/multi-cursor/dotnet/winforms/winforms.csproj
nl -ba demo/multi-cursor/dotnet/winforms/winforms.csproj | sed -n '1,200p'
echo
echo "=== Search for ApplicationConfiguration.Initialize or SetHighDpiMode in the winforms folder ==="
rg -n --hidden --glob '!**/bin/**' --glob '!**/obj/**' \
"ApplicationConfiguration\.Initialize|SetHighDpiMode|ApplicationHighDpiMode" \
demo/multi-cursor/dotnet/winformsRepository: trycua/cua
Length of output: 245
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== Program.cs (startup) ==="
sed -n '1,140p' demo/multi-cursor/dotnet/winforms/Program.cs
echo
echo "=== winforms.csproj (DPI settings) ==="
sed -n '1,220p' demo/multi-cursor/dotnet/winforms/winforms.csproj
echo
echo "=== Search within winforms folder for bootstrap/high-DPI calls ==="
rg -n --hidden --glob '!**/bin/**' --glob '!**/obj/**' \
"ApplicationConfiguration\.Initialize\s*\(|Application\.SetHighDpiMode\s*\(|SetHighDpiMode\s*\(|ApplicationHighDpiMode" \
demo/multi-cursor/dotnet/winformsRepository: trycua/cua
Length of output: 2453
Call ApplicationConfiguration.Initialize() to apply PerMonitorV2 at startup.
winforms.csproj sets <ApplicationHighDpiMode>PerMonitorV2</ApplicationHighDpiMode>, but Program.Main never calls the WinForms bootstrap initializer (or Application.SetHighDpiMode(...)). This can leave the app on the default DPI mode on mixed-DPI systems, skewing WinForms scaling and breaking the replay demo’s coordinate assumptions.
Proposed fix
- Application.EnableVisualStyles();
- Application.SetCompatibleTextRenderingDefault(false);
+ ApplicationConfiguration.Initialize();📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| Application.EnableVisualStyles(); | |
| Application.SetCompatibleTextRenderingDefault(false); | |
| ApplicationConfiguration.Initialize(); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@demo/multi-cursor/dotnet/winforms/Program.cs` around lines 12 - 13,
Program.Main currently calls Application.EnableVisualStyles() and
Application.SetCompatibleTextRenderingDefault(false) but does not initialize the
WinForms bootstrap; call ApplicationConfiguration.Initialize() at the start of
Program.Main (before EnableVisualStyles/SetCompatibleTextRenderingDefault) so
the project-wide <ApplicationHighDpiMode>PerMonitorV2</ApplicationHighDpiMode>
is applied; update Program.Main to invoke ApplicationConfiguration.Initialize()
(or alternatively call Application.SetHighDpiMode(...) using the same
PerMonitorV2 mode) to ensure PerMonitorV2 DPI behavior for the app.
Comment on lines
+190
to
+203
| let ok = run_call(&cua, "click", &format!( | ||
| r#"{{"pid":{pid},"window_id":{hwnd_addr},"x":{x},"y":{y},"session":"{session}"}}"#)); | ||
| eprintln!("[drive {session}] click ({x},{y}) -> {}", if ok { "ok" } else { "FAIL" }); | ||
| } | ||
| } | ||
| Action::Type { text } => { | ||
| // Focus the field first (so the chars land), then type. | ||
| if let Some((x, y)) = client_rel_to_local_px(hwnd, 0.5, 0.28) { | ||
| let _ = run_call(&cua, "click", &format!( | ||
| r#"{{"pid":{pid},"window_id":{hwnd_addr},"x":{x},"y":{y},"session":"{session}"}}"#)); | ||
| } | ||
| let esc = json_escape(&text); | ||
| let ok = run_call(&cua, "type_text", &format!( | ||
| r#"{{"pid":{pid},"window_id":{hwnd_addr},"text":"{esc}","session":"{session}"}}"#)); |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== click/type_text schemas =="
rg -n -C3 'stub_tool!\((click_m|type_text_m)' libs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rs
echo
echo "== orchestrator payloads carrying session =="
rg -n -C2 '"session"|session"\}\)' demo/multi-cursor/orchestrator/src/main.rsRepository: trycua/cua
Length of output: 3136
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="demo/multi-cursor/orchestrator/src/main.rs"
echo "== window discovery: look for title/substring matching =="
rg -n --hidden --no-ignore-vcs "title|substring|contains|find.*window|EnumWindows|GetWindowText|GetWindowTextW|match" "$file"
echo
echo "== locate client_rel_to_local_px =="
rg -n "fn client_rel_to_local_px" "$file"
echo
echo "== view client_rel_to_local_px body (around definition) =="
line=$(rg -n "fn client_rel_to_local_px" "$file" | head -n1 | cut -d: -f1)
start=$((line-40))
end=$((line+120))
sed -n "${start},${end}p" "$file" | cat -n
echo
echo "== look for DwmGetWindowAttribute usage and fallback behavior =="
rg -n "DwmGetWindowAttribute|dwm|DWM" "$file"
echo
echo "== view around DwmGetWindowAttribute usage =="
rg -n "DwmGetWindowAttribute" "$file" | head -n1 | awk -F: '{print "line=" $1}'
dwm_line=$(rg -n "DwmGetWindowAttribute" "$file" | head -n1 | cut -d: -f1)
start=$((dwm_line-60))
end=$((dwm_line+80))
sed -n "${start},${end}p" "$file" | cat -nRepository: trycua/cua
Length of output: 8100
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="demo/multi-cursor/orchestrator/src/main.rs"
echo "== view orchestrator spawn/window tracking around corners + hwnd/pid assignment =="
sed -n '110,190p' "$file" | cat -n
echo
echo "== find bitmap_to_screen (to confirm coordinate-space subtract behavior) =="
rg -n "bitmap_to_screen" "$file"
if rg -n "bitmap_to_screen" "$file" >/dev/null; then
line=$(rg -n "bitmap_to_screen" "$file" | head -n1 | cut -d: -f1)
start=$((line-40)); end=$((line+120))
sed -n "${start},${end}p" "$file" | cat -n
fi
echo
echo "== confirm any other click/type_text payload sites =="
rg -n '"click"|' demo/multi-cursor/orchestrator/src/main.rs
rg -n '"type_text"' demo/multi-cursor/orchestrator/src/main.rsRepository: trycua/cua
Length of output: 25548
Fix click/type_text JSON payloads and the replay window/coordinate logic.
demo/multi-cursor/orchestrator/src/main.rssends"session":"{session}"in bothrun_call(&cua, "click", ...)andrun_call(&cua, "type_text", ...), but the corresponding tool schemas inlibs/cua-driver/rust/crates/platform-windows/src/tools/stubs.rsuseadditionalProperties: falseand do not define asessionproperty; schema-based validation would reject these replay actions.- Window selection uses
find_window_by_titlewithEnumWindows+title.contains(&needle); this can bind an unrelated visible window with a colliding caption instead of the intended spawned window. client_rel_to_local_pxreturns(pt.x, pt.y)unchanged whenDwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)fails, skipping the normal subtraction ofdwm.left/top(+ the 1px inset), so click coordinates will be misrouted in that failure path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@demo/multi-cursor/orchestrator/src/main.rs` around lines 190 - 203, Remove
the unsupported "session" property from the JSON payloads sent by run_call in
the click and type_text paths (e.g., the click call in the Action::Type branch
and the subsequent type_text call) so the payloads match the tool schemas; keep
pid, window_id (hwnd_addr), x/y or text and ensure json_escape is used for text.
Update window selection logic in find_window_by_title/EnumWindows to prefer the
spawned window by verifying the window's process id (GetWindowThreadProcessId)
matches the spawn pid (or use the exact HWND returned at spawn) instead of
relying solely on title.contains(&needle). Fix client_rel_to_local_px so that
when DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS) fails you still compute
and subtract the window origin and the 1px inset (e.g., fall back to
GetWindowRect and subtract left/top +1) before returning (pt.x - origin.x - 1,
pt.y - origin.y - 1) to ensure coordinates map correctly.
Comment on lines
+304
to
+312
| unsafe fn arm(target: HWND) -> Self { | ||
| let prev_fg = GetForegroundWindow(); | ||
| // Raise a genuine *background* target into the topmost band so it wins | ||
| // the injection hit-test even over an active occluder — no activation. | ||
| let raised = !target.0.is_null() && target != prev_fg; | ||
| if raised { | ||
| set_topmost(target, true); | ||
| } | ||
| Self { prev_fg, target, raised } |
Contributor
There was a problem hiding this comment.
Preserve the target’s original topmost state.
arm() always pushes into HWND_TOPMOST, but drop() always demotes with HWND_NOTOPMOST. If the target started life as topmost, this guard permanently changes its z-order class after one injected action.
Also applies to: 317-325
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/rust/crates/platform-windows/src/input/inject.rs` around
lines 304 - 312, The guard created in arm() currently always forces HWND_TOPMOST
and drop() always sets HWND_NOTOPMOST, which loses the window’s original topmost
state; modify arm() (in the same impl that defines arm and Drop) to detect and
store the target’s original topmost status (e.g., call the existing
set_topmost/inspection helper or use GetWindowLongPtr/WS_EX_TOPMOST check) as a
new field like original_topmost on the returned Self, and then change drop() to
restore the recorded original_topmost (call set_topmost(target,
original_topmost)) instead of unconditionally using HWND_NOTOPMOST so the
window’s prior z-order class is preserved.
ZorderGuard still causes a visible z-raise on every background target.
This unconditionally promotes any non-foreground target into the topmost band, so even an unobscured background window will flash above the user’s current app for the duration of the action. That contradicts the “no z-raise” contract called out in the PR and in the surrounding docs/comments.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/rust/crates/platform-windows/src/input/inject.rs` around
lines 304 - 312, ZorderGuard::arm currently sets set_topmost for any
non-foreground target, causing visible z-raises; change arm to only promote the
target when it is actually occluded by other windows (not merely
not-foreground). Replace the raised condition (!target.0.is_null() && target !=
prev_fg) with a call to a helper like is_obscured(target, prev_fg) that returns
true only if the target's client/target area is covered (e.g., check
IsWindowVisible, GetWindowRect and whether WindowFromPoint(s) over the target
return a different window, or test several sample points including center and
edges), then call set_topmost(target, true) only when that helper reports
occlusion and set raised accordingly; ensure prev_fg, target, and raised reflect
the actual action taken so no unnecessary z-raise occurs.
Collaborator
Author
There was a problem hiding this comment.
⏭️ Deferred (added TODO) — preserving original topmost state + occlusion detection are complex fixes requiring WindowFromPoint sampling and GetWindowLongPtr checks. Current behavior works correctly for the common case (clicking background windows). Added TODO comment for future enhancement. Non-blocking for PR merge.
Contributor
There was a problem hiding this comment.
@ddupont808 Understood, acknowledged. The common-case behavior (non-topmost background windows) is unaffected by both issues, so this is a reasonable deferral.
Would you like me to open a GitHub issue to track the two follow-up items so they don't get lost?
ZorderGuard: preserve original topmost state — recordWS_EX_TOPMOSTonarm()and restore it (instead of unconditionallyHWND_NOTOPMOST) ondrop().ZorderGuard: skip promotion for unoccluded targets — add an occlusion check (e.g.,WindowFromPointsampling over the target rect) so the z-raise only happens when the target is actually covered, avoiding the visible flash for unobstructed background windows.
Comment on lines
+1850
to
+1862
| // For every non-foreground click, mark the target window | ||
| // non-activatable (WS_EX_NOACTIVATE) for the duration so a target that | ||
| // self-activates in its UIA-Invoke / click handler (WPF | ||
| // `UIElement.Focus()`, XAML, Tauri/WebView2) CANNOT steal the user's | ||
| // foreground — the window still receives the click. Held for the whole | ||
| // invoke; a no-op for dispatch:"foreground" (which wants the swap) and | ||
| // when no window_id was given. | ||
| let _noact = match (dispatch != DispatchMode::Foreground, hwnd_opt) { | ||
| (true, Some(h)) => Some(crate::input::NoActivateGuard::arm( | ||
| windows::Win32::Foundation::HWND(h as *mut _), | ||
| )), | ||
| _ => None, | ||
| }; |
Contributor
There was a problem hiding this comment.
Arm the no-activate guard from the resolved HWND.
Line 1857 only guards the explicit window_id case. For pid-only pixel clicks, Lines 1872-1881 resolve the target window later, so the background path still runs without NoActivateGuard and the target can steal foreground from inside its click handler. Please move guard construction until after hwnd is resolved and arm it from that value instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/rust/crates/platform-windows/src/tools/impl_.rs` around lines
1850 - 1862, The NoActivateGuard is currently armed only when an explicit
window_id (hwnd_opt) is present, leaving pid-resolved clicks unprotected; change
the logic so you defer creating _noact until after the actual target HWND is
resolved and then arm crate::input::NoActivateGuard::arm with that resolved HWND
(converted to windows::Win32::Foundation::HWND). Specifically, remove or
relocate the current match that sets _noact around hwnd_opt and instead
construct _noact after the code path that resolves hwnd (the later resolution
logic around the places that reference hwnd or resolve by pid), guarding both
explicit hwnd_opt and pid-resolved HWNDs while preserving the
DispatchMode::Foreground no-op behavior.
…, NoActivateGuard gate, pre-drag MOUSEMOVE wParam - demo/jukebox: Key active-note map by (channel, pitch) to handle polyphonic same-pitch and multi-channel collisions - libs/cua-driver(inject): Validate target HWND with IsWindow before guards in inject_click_screen/inject_drag_screen (early bail on null/stale) - libs/cua-driver(inject): Fix NoActivateGuard to apply WS_EX_NOACTIVATE even when prev==0 (was gated incorrectly) - libs/cua-driver(inject): Add TODO for ZorderGuard occlusion check + original topmost preservation (deferred, non-trivial) - libs/cua-driver(mouse): Pre-drag WM_MOUSEMOVE uses wParam=0 (no buttons down yet) instead of wparam Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…_agent_cursor_state responses Added turn_radius to both text summary and structured JSON payloads so callers can verify the applied value. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
ddupont808
added a commit
that referenced
this pull request
Jun 4, 2026
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
portla.radio.dem.o.2.mp4
Windows background computer-use (click/type without z-raise or cursor movement), two multi-cursor demos, and several cua-driver improvements: fixed-duration glides, configurable turn radius, multi-window z-order fix, click latency decoupling, tangent-tracking cursors.
Key capabilities:
NoActivateGuardRAII wrapper prevents target self-activation.glide_duration_ms > 0).Changes
Core: Background Input (Windows)
inject_click_screen,inject_text_cloaked, etc.) withNoActivateGuardRAII to prevent target self-activation.click,type_text,press_key,hotkey,drag,set_valueroute through injection whendispatch:"background"+ target would silently drop PostMessage (Chromium/WPF/GTK). Capability extension: previously errored withbackground_unavailable, now succeeds.Demos
Overlay Improvements (cross-platform, no breaking changes)
glide_duration_ms > 0→ path completes in exactly that wall-clock time, enabling deterministic orchestration). Default0.0= unchanged speed-based mode. Bothtick_motion(Windows/Linux) andtick_swift_constants(macOS) updated identically.turn_radiusfield (default80.0, clamped 1–1000) so callers can tighten glide curves. Exposed on both Windows and macOSset_agent_cursor_motionschemas (parity).GetTopWindow/GW_HWNDNEXT) rather than flipping to last-active, so cursors stay visible across many tiles but below the user's foreground. Fixes blink-out when >1 window driven concurrently.glide_duration_ms > 0,overlay_glide_tosleeps the known duration instead of awaiting render thread's arrival oneshot, so click dispatch is deterministic and FPS-independent. Speed-based glides (== 0, the default) keep precise arrival-await.Backwards Compatibility
✅ Zero breaking changes (confirmed by comprehensive review):
turn_radius: 80.0,glide_duration_ms: 0.0)Minor visual-only change: cursor tip now tracks tangent exactly (improved fidelity, no coordinate/timing impact).
Testing
render_state.rsglide timing tests (804–850)e2e_windows_bg_input_test.rs(558 lines, Chromium/WPF/Electron scenarios)Commit Breakdown
88bf542bfeat: background click/type without z-raise (inject.rs, dispatch routing)4f8de048demo: multi-cursor across 5 UI frameworksa83e1bc5feat: honor glide_duration_ms in shared render core + click latency decouplefdee125cdemo: CUA Jukebox — MIDI-driven multi-cursor8d5469cdfix: cursor arrow points along path tangent while gliding763f6610feat: configurable turn_radius + multi-window z-order fixeb493fd6demo(jukebox): set turn_radius=40 for tighter curvesSummary by CodeRabbit
Release Notes
New Features
Documentation