Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 53 additions & 16 deletions libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,32 @@ best-effort: `SetForegroundWindow` from non-UIAccess processes is subject
to Windows' foreground-lock and may silently no-op — failures are logged
at `tracing::trace!` and not surfaced as errors.

**`hotkey` modifier dispatch on Chromium/Electron (Windows).** When
the target is detected as a Chromium-family window (`Chrome_WidgetWin_*`
class — covers every Chromium browser AND every Electron app: Chrome,
Edge, Brave, Arc, Vivaldi, Slack, VS Code, Discord, Teams, Notion),
`hotkey` with modifiers routes through `PostMessage(WM_KEYDOWN/UP)`
instead of `SendInput` — no foreground swap. Chromium's
`Browser::HandleKeyboardEvent` reads modifier state from the WM_KEYDOWN
LPARAM bits, NOT from `GetKeyState`, so PostMessage delivery works for
real accelerators (Ctrl+T, Ctrl+W, Ctrl+L, Ctrl+Shift+B, etc.). The
SendInput-swap path (`send_key_synthesized`) remains the dispatch for
**non-Chromium Win32 + modifiers** — classic apps (LibreOffice, FAR,
classic Notepad) use `TranslateAccelerator` which requires the system
modifier state updated, and PostMessage can't do that.

**Chromium pixel-click foreground polling restore.** `click({pid, x, y})`
on a Chromium target falls through to `send_click_synthesized` (SendInput
+ brief foreground swap) because Chromium's input thread filters by
queue-origin and PostMessage-delivered clicks don't fire DOM events. The
synchronous restore inside `send_click_synthesized` covers the
immediate swap; an additional polling guard (same shape as `launch_app`'s
`FocusRestoreGuard`) catches the **asynchronous** Chromium re-activation
that can happen as the renderer's input handler processes the click
(focus().activate() / WebContents::Activate() — 100-500 ms later). The
guard is gated on `GetWindowThreadProcessId(fg_now) == pid` so user
Alt-Tabs are respected.

## Defaults — always prefer cua-driver over shell shims

**Default transport is the `cua-driver` CLI** — `Bash` shelling out
Expand Down Expand Up @@ -654,28 +680,39 @@ browser instance identified by `(pid, window_id)`:
create it manually.
- **The Favorites bar must already be visible.** This is a
one-time setup: press `Ctrl+Shift+B` once inside the browser
and the setting persists across sessions. cua-driver does
**not** synthesize Ctrl+Shift+B for you any more, because the
synthesizer routed via `SetForegroundWindow(target) →
SendInput → SetForegroundWindow(prev_fg)` produces a visible
foreground flicker and the restore can fail silently under
UIPI / UIAccess constraints — both violations of the
no-foreground contract. If the bar is hidden when
`execute_javascript` is called, the bookmark path bails with a
clear error and falls through to the CDP path (if configured).
and the setting persists across sessions. **If the bar is
hidden, cua-driver now synthesizes `Ctrl+Shift+B` via
`PostMessage(WM_KEYDOWN/UP)` with no foreground swap** —
Chromium's `Browser::HandleKeyboardEvent` dispatches
accelerators from the WM_KEYDOWN LPARAM bits without
consulting `GetKeyState`, so PostMessage works without the
`SetForegroundWindow` dance that previously violated the
no-foreground contract. After a brief settle the path
re-checks for the bar; if it's still hidden (locked-down
browser policy / non-Chromium target) the call bails with a
clear error and falls through to the CDP path.
- The user's expression should be a single statement or block;
`return` inside the IIFE is honored. Bookmarks strip line
breaks, so multi-line expressions are joined with spaces.

**Known limitation — transient browser activation on Invoke.**
**Known limitation — Chromium's window activation on Invoke.**
When the bookmark is invoked via UIA `InvokePattern::Invoke`,
Chromium briefly raises the browser window because clicking a
Chromium activates the browser window because clicking a
bookmark is a user-initiated navigation in Chromium's input
model. This is a Chromium-side activation we cannot suppress
from the UIA caller. We document it rather than mask it; the
rest of the page-tool actions (`get_text`, `query_dom`) remain
no-foreground. If this matters for your flow, use the CDP path
instead — `Runtime.evaluate` does not trigger window activation.
model. The activation happens inside Chromium's window-aura
layer, not in our UIA call. **Mitigation in place**: a polling
foreground-restore guard runs immediately after the Invoke —
same pattern `launch_app` uses (PR #1668) — capturing the
user's foreground HWND beforehand and calling
`SetForegroundWindow(prev)` once Chromium grabs foreground.
The restore is gated on `GetWindowThreadProcessId(fg_now) ==
browser_pid` so we never yank focus from a window the user
legitimately Alt-Tabbed to. Without UIAccess (the daemon's
normal integrity) Windows' foreground lock may deny the
restore — in that case the browser dwell time is bounded to
the ~600 ms poll budget instead of "until next user action".
The `get_text` and `query_dom` actions don't share this issue
(no Invoke → no activation).

2. **CDP fallback** — `Runtime.evaluate` via raw WebSocket against
`--remote-debugging-port=N`. Requires the browser launched with
Expand Down
65 changes: 51 additions & 14 deletions libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1693,10 +1693,27 @@ impl Tool for ClickTool {
.await
.unwrap_or(false);
if chromium {
// Belt-and-suspenders foreground restore. `send_click_synthesized`
// does its own SetForegroundWindow(prev_fg) ~40ms after the
// click, BUT Chromium's reaction to the click can include an
// async re-activation (focus().activate() in the renderer-side
// event handler, sometimes 100-500 ms later). The launch_app
// FocusRestoreGuard (PR #1668) already proved the polling
// pattern works; reuse the same helper to catch the async case.
//
// Captured here (async-runtime side) rather than inside the
// blocking task so we have a known-stable "before" snapshot.
let prev_fg_addr = unsafe {
windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize
};
let send_result = tokio::task::spawn_blocking(move || {
crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, count, &btn)
})
.await;
// Kick off the polling restore regardless of click result
// — even a partially-inserted click might have started an
// async activation Chromium can't undo.
tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
Comment on lines +1713 to +1716

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Guard async restore by the activated target HWND, not PID-only.

This restore task can steal focus from a user who Alt-Tabs to another window in the same browser process, because PID match alone still passes. Gate on the clicked target HWND (or root owner) for the click path before restoring.

Proposed direction
- tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid));
+ let target_hwnd_addr = hwnd as usize;
+ tokio::spawn(restore_foreground_polling_after_click_best_effort(
+     prev_fg_addr,
+     pid,
+     target_hwnd_addr,
+ ));
// Click-specific guard idea:
// restore only when foreground is the same HWND we clicked (or same root owner),
// not just any window owned by `pid`.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs` around lines
1713 - 1716, The restore task currently spawned with
restore_foreground_polling_best_effort(prev_fg_addr, pid) uses only pid to
decide whether to restore focus; change it to guard by the clicked target HWND
(or its root owner) instead: capture the clicked HWND (or compute its root
owner) at click time, pass that HWND (e.g. clicked_hwnd or clicked_root_owner)
into the restore task instead of or in addition to pid, and in
restore_foreground_polling_best_effort compare the current foreground HWND/root
owner (via GetForegroundWindow/GetAncestor/RootOwner) with the captured HWND
before restoring; update the spawn call and the restore function signature (and
any callers) to accept this HWND and use it for the focus check rather than
PID-only logic.

match send_result {
Ok(Ok(())) => {
let click_word = match count {
Expand Down Expand Up @@ -2139,28 +2156,48 @@ impl Tool for HotkeyTool {
};
}

// Non-XAML Win32 path. Two routes available:
// 1. SendInput synthesized hotkey — pushes the events onto the
// *system input queue*. Updates GetKeyState's modifier state, so
// TranslateAccelerator-based apps (LibreOffice, FAR, classic
// Notepad, etc.) see Ctrl+S as a real accelerator. Trade-off:
// brief focus theft to ensure SendInput lands on the right HWND.
// 2. PostMessage WM_KEYDOWN/UP — no focus theft, but the
// synthesized Ctrl/Shift/Alt modifier never updates
// GetKeyState, so accelerators don't fire. Only useful for
// non-accelerator key sequences.
// We pick route 1 when modifiers are present (the accelerator case),
// route 2 otherwise (plain non-modifier keys still post fine).
// Non-XAML target. Three routing decisions:
// 1. PostMessage WM_KEYDOWN/UP — no foreground swap, no focus
// theft. Modifier state is encoded in the LPARAM bits, not
// in GetKeyState, so apps that read modifier state directly
// from the message (Chromium's `Browser::HandleKeyboardEvent`,
// Chromium renderer's input handler, every web app) see
// Ctrl+X correctly. Apps that consult `GetKeyState` via
// `TranslateAccelerator` (LibreOffice, FAR, classic Notepad)
// DON'T fire the accelerator — the modifier looks unset to
// them.
// 2. SendInput synthesized hotkey — pushes events onto the
// *system input queue*. Updates GetKeyState system-wide, so
// TranslateAccelerator-based apps see Ctrl+S as a real
// accelerator. Trade-off: brief foreground swap so SendInput
// lands on the right HWND. **This is the only path that
// violates the no-foreground contract** — we minimise its
// use to apps that genuinely require it.
//
// Routing:
// - No modifiers → PostMessage (no accelerator concern).
// - Modifiers + Chromium-family target → PostMessage. Chromium
// processes accelerators in `Browser::HandleKeyboardEvent`,
// which reads modifier state from the WM_KEYDOWN LPARAM
// directly (not via GetKeyState). Every Chromium browser
// (Chrome/Edge/Brave/Arc/Vivaldi) AND every Electron app
// (Slack/VS Code/Discord/Teams/Notion) detects as Chromium
// via `is_chromium_target_window` and gets the no-foreground
// path.
// - Modifiers + non-Chromium Win32 → SendInput (the
// TranslateAccelerator path). The foreground swap stays.
let has_modifiers = !mods.is_empty();
let is_chromium = crate::input::is_chromium_target_window(hwnd);
let use_send_input = has_modifiers && !is_chromium;
let result = tokio::task::spawn_blocking(move || {
let m: Vec<&str> = mods.iter().map(String::as_str).collect();
if has_modifiers {
if use_send_input {
crate::input::send_key_synthesized(hwnd, &key, &m)
} else {
crate::input::post_key(hwnd, &key, &m)
}
}).await;
let path = if has_modifiers { "SendInput" } else { "PostMessage" };
let path = if use_send_input { "SendInput" } else { "PostMessage" };
match result {
Ok(Ok(())) => ToolResult::text(format!(
"✅ Pressed {key_display} on pid {raw_pid} via {path} \
Expand Down
Loading
Loading