diff --git a/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md b/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md index bfa9cf9be3..6ef01f2414 100644 --- a/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md +++ b/libs/cua-driver-rs/Skills/cua-driver-rs/WINDOWS.md @@ -157,6 +157,32 @@ best-effort: `SetForegroundWindow` from non-UIAccess processes is subject to Windows' foreground-lock and may silently no-op — failures are logged at `tracing::trace!` and not surfaced as errors. +**`hotkey` modifier dispatch on Chromium/Electron (Windows).** When +the target is detected as a Chromium-family window (`Chrome_WidgetWin_*` +class — covers every Chromium browser AND every Electron app: Chrome, +Edge, Brave, Arc, Vivaldi, Slack, VS Code, Discord, Teams, Notion), +`hotkey` with modifiers routes through `PostMessage(WM_KEYDOWN/UP)` +instead of `SendInput` — no foreground swap. Chromium's +`Browser::HandleKeyboardEvent` reads modifier state from the WM_KEYDOWN +LPARAM bits, NOT from `GetKeyState`, so PostMessage delivery works for +real accelerators (Ctrl+T, Ctrl+W, Ctrl+L, Ctrl+Shift+B, etc.). The +SendInput-swap path (`send_key_synthesized`) remains the dispatch for +**non-Chromium Win32 + modifiers** — classic apps (LibreOffice, FAR, +classic Notepad) use `TranslateAccelerator` which requires the system +modifier state updated, and PostMessage can't do that. + +**Chromium pixel-click foreground polling restore.** `click({pid, x, y})` +on a Chromium target falls through to `send_click_synthesized` (SendInput ++ brief foreground swap) because Chromium's input thread filters by +queue-origin and PostMessage-delivered clicks don't fire DOM events. The +synchronous restore inside `send_click_synthesized` covers the +immediate swap; an additional polling guard (same shape as `launch_app`'s +`FocusRestoreGuard`) catches the **asynchronous** Chromium re-activation +that can happen as the renderer's input handler processes the click +(focus().activate() / WebContents::Activate() — 100-500 ms later). The +guard is gated on `GetWindowThreadProcessId(fg_now) == pid` so user +Alt-Tabs are respected. + ## Defaults — always prefer cua-driver over shell shims **Default transport is the `cua-driver` CLI** — `Bash` shelling out @@ -654,28 +680,39 @@ browser instance identified by `(pid, window_id)`: create it manually. - **The Favorites bar must already be visible.** This is a one-time setup: press `Ctrl+Shift+B` once inside the browser - and the setting persists across sessions. cua-driver does - **not** synthesize Ctrl+Shift+B for you any more, because the - synthesizer routed via `SetForegroundWindow(target) → - SendInput → SetForegroundWindow(prev_fg)` produces a visible - foreground flicker and the restore can fail silently under - UIPI / UIAccess constraints — both violations of the - no-foreground contract. If the bar is hidden when - `execute_javascript` is called, the bookmark path bails with a - clear error and falls through to the CDP path (if configured). + and the setting persists across sessions. **If the bar is + hidden, cua-driver now synthesizes `Ctrl+Shift+B` via + `PostMessage(WM_KEYDOWN/UP)` with no foreground swap** — + Chromium's `Browser::HandleKeyboardEvent` dispatches + accelerators from the WM_KEYDOWN LPARAM bits without + consulting `GetKeyState`, so PostMessage works without the + `SetForegroundWindow` dance that previously violated the + no-foreground contract. After a brief settle the path + re-checks for the bar; if it's still hidden (locked-down + browser policy / non-Chromium target) the call bails with a + clear error and falls through to the CDP path. - The user's expression should be a single statement or block; `return` inside the IIFE is honored. Bookmarks strip line breaks, so multi-line expressions are joined with spaces. - **Known limitation — transient browser activation on Invoke.** + **Known limitation — Chromium's window activation on Invoke.** When the bookmark is invoked via UIA `InvokePattern::Invoke`, - Chromium briefly raises the browser window because clicking a + Chromium activates the browser window because clicking a bookmark is a user-initiated navigation in Chromium's input - model. This is a Chromium-side activation we cannot suppress - from the UIA caller. We document it rather than mask it; the - rest of the page-tool actions (`get_text`, `query_dom`) remain - no-foreground. If this matters for your flow, use the CDP path - instead — `Runtime.evaluate` does not trigger window activation. + model. The activation happens inside Chromium's window-aura + layer, not in our UIA call. **Mitigation in place**: a polling + foreground-restore guard runs immediately after the Invoke — + same pattern `launch_app` uses (PR #1668) — capturing the + user's foreground HWND beforehand and calling + `SetForegroundWindow(prev)` once Chromium grabs foreground. + The restore is gated on `GetWindowThreadProcessId(fg_now) == + browser_pid` so we never yank focus from a window the user + legitimately Alt-Tabbed to. Without UIAccess (the daemon's + normal integrity) Windows' foreground lock may deny the + restore — in that case the browser dwell time is bounded to + the ~600 ms poll budget instead of "until next user action". + The `get_text` and `query_dom` actions don't share this issue + (no Invoke → no activation). 2. **CDP fallback** — `Runtime.evaluate` via raw WebSocket against `--remote-debugging-port=N`. Requires the browser launched with diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs index 36c382e125..7c6843b17a 100644 --- a/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs +++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/impl_.rs @@ -1693,10 +1693,27 @@ impl Tool for ClickTool { .await .unwrap_or(false); if chromium { + // Belt-and-suspenders foreground restore. `send_click_synthesized` + // does its own SetForegroundWindow(prev_fg) ~40ms after the + // click, BUT Chromium's reaction to the click can include an + // async re-activation (focus().activate() in the renderer-side + // event handler, sometimes 100-500 ms later). The launch_app + // FocusRestoreGuard (PR #1668) already proved the polling + // pattern works; reuse the same helper to catch the async case. + // + // Captured here (async-runtime side) rather than inside the + // blocking task so we have a known-stable "before" snapshot. + let prev_fg_addr = unsafe { + windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow().0 as usize + }; let send_result = tokio::task::spawn_blocking(move || { crate::input::send_click_synthesized(hwnd, sx as i32, sy as i32, count, &btn) }) .await; + // Kick off the polling restore regardless of click result + // — even a partially-inserted click might have started an + // async activation Chromium can't undo. + tokio::spawn(restore_foreground_polling_best_effort(prev_fg_addr, pid)); match send_result { Ok(Ok(())) => { let click_word = match count { @@ -2139,28 +2156,48 @@ impl Tool for HotkeyTool { }; } - // Non-XAML Win32 path. Two routes available: - // 1. SendInput synthesized hotkey — pushes the events onto the - // *system input queue*. Updates GetKeyState's modifier state, so - // TranslateAccelerator-based apps (LibreOffice, FAR, classic - // Notepad, etc.) see Ctrl+S as a real accelerator. Trade-off: - // brief focus theft to ensure SendInput lands on the right HWND. - // 2. PostMessage WM_KEYDOWN/UP — no focus theft, but the - // synthesized Ctrl/Shift/Alt modifier never updates - // GetKeyState, so accelerators don't fire. Only useful for - // non-accelerator key sequences. - // We pick route 1 when modifiers are present (the accelerator case), - // route 2 otherwise (plain non-modifier keys still post fine). + // Non-XAML target. Three routing decisions: + // 1. PostMessage WM_KEYDOWN/UP — no foreground swap, no focus + // theft. Modifier state is encoded in the LPARAM bits, not + // in GetKeyState, so apps that read modifier state directly + // from the message (Chromium's `Browser::HandleKeyboardEvent`, + // Chromium renderer's input handler, every web app) see + // Ctrl+X correctly. Apps that consult `GetKeyState` via + // `TranslateAccelerator` (LibreOffice, FAR, classic Notepad) + // DON'T fire the accelerator — the modifier looks unset to + // them. + // 2. SendInput synthesized hotkey — pushes events onto the + // *system input queue*. Updates GetKeyState system-wide, so + // TranslateAccelerator-based apps see Ctrl+S as a real + // accelerator. Trade-off: brief foreground swap so SendInput + // lands on the right HWND. **This is the only path that + // violates the no-foreground contract** — we minimise its + // use to apps that genuinely require it. + // + // Routing: + // - No modifiers → PostMessage (no accelerator concern). + // - Modifiers + Chromium-family target → PostMessage. Chromium + // processes accelerators in `Browser::HandleKeyboardEvent`, + // which reads modifier state from the WM_KEYDOWN LPARAM + // directly (not via GetKeyState). Every Chromium browser + // (Chrome/Edge/Brave/Arc/Vivaldi) AND every Electron app + // (Slack/VS Code/Discord/Teams/Notion) detects as Chromium + // via `is_chromium_target_window` and gets the no-foreground + // path. + // - Modifiers + non-Chromium Win32 → SendInput (the + // TranslateAccelerator path). The foreground swap stays. let has_modifiers = !mods.is_empty(); + let is_chromium = crate::input::is_chromium_target_window(hwnd); + let use_send_input = has_modifiers && !is_chromium; let result = tokio::task::spawn_blocking(move || { let m: Vec<&str> = mods.iter().map(String::as_str).collect(); - if has_modifiers { + if use_send_input { crate::input::send_key_synthesized(hwnd, &key, &m) } else { crate::input::post_key(hwnd, &key, &m) } }).await; - let path = if has_modifiers { "SendInput" } else { "PostMessage" }; + let path = if use_send_input { "SendInput" } else { "PostMessage" }; match result { Ok(Ok(())) => ToolResult::text(format!( "✅ Pressed {key_display} on pid {raw_pid} via {path} \ diff --git a/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs b/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs index 11fe671828..57f91bdccf 100644 --- a/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs +++ b/libs/cua-driver-rs/crates/platform-windows/src/tools/page_bookmark.rs @@ -188,31 +188,37 @@ unsafe fn try_bookmark_exec_blocking( let edge_window = root_for_window(&automation, hwnd_raw, pid) .context("could not resolve Edge top-level window for (pid, window_id)")?; - // 2) Ensure the favorites bar is visible. If not, we bail. + // 2) Ensure the favorites bar is visible. If not, send Ctrl+Shift+B + // via PostMessage (no foreground swap — Chromium's + // Browser::HandleKeyboardEvent dispatches accelerators from + // WM_KEYDOWN LPARAM modifier bits without consulting GetKeyState). // - // Historical note: we used to synthesize Ctrl+Shift+B here via - // `send_key_synthesized`, which does a transient - // `SetForegroundWindow(target) → SendInput → SetForegroundWindow(prev_fg)` - // dance. That dance is visible to the user (foreground flicker) and the - // restore can fail silently under UIPI / UIAccess constraints, both of - // which violate the cua-driver no-foreground contract. Every - // `page.execute_javascript` call where the user hadn't already shown - // the favorites bar would trigger this — unacceptable. - // - // The favorites-bar setting persists across browser sessions, so the - // one-time setup cost (Ctrl+Shift+B in the browser) is borne by the - // user once, never by the agent. - let fav_bar = find_favorites_bar(&automation, &edge_window).ok_or_else(|| { - anyhow!( - "Favorites bar not visible — cua-driver-rs's bookmark-URL JS \ - exec needs an always-on favorites bar so we can invoke the \ - `cua-driver-eval` bookmark without stealing the foreground. \ - Show the bar once via Ctrl+Shift+B and the setting persists \ - across sessions. The bookmark-exec path is now disabled for \ - this call; `execute_javascript` will fall through to the CDP \ - fallback if it's configured." - ) - })?; + // Historical note: PR #1668 removed the auto-toggle here because it + // used `send_key_synthesized` (SendInput + SetForegroundWindow swap) + // which violated the no-foreground contract. With `post_key` + // (PostMessage) routing — verified safe on every Chromium browser + + // Electron app — the auto-toggle is restored. If the bar STILL isn't + // visible after a brief settle, we bail with the same actionable + // error (covers locked-down browser policies / classic Edge profiles). + let fav_bar = if let Some(bar) = find_favorites_bar(&automation, &edge_window) { + bar + } else { + // No SendInput, no SetForegroundWindow, no UIPI risk. + let _ = crate::input::post_key(hwnd_raw, "b", &["ctrl", "shift"]); + std::thread::sleep(std::time::Duration::from_millis(150)); + find_favorites_bar(&automation, &edge_window).ok_or_else(|| { + anyhow!( + "Favorites bar not visible after Ctrl+Shift+B PostMessage \ + — cua-driver-rs's bookmark-URL JS exec needs the bar so \ + we can invoke the `cua-driver-eval` bookmark. The toggle \ + keystroke didn't take (browser policy override, locked-down \ + profile, or non-Chromium target). Show the bar manually via \ + Ctrl+Shift+B in the browser — the setting persists across \ + sessions. `execute_javascript` will fall through to the \ + CDP fallback if it's configured." + ) + })? + }; // 3) Find or create the cua-driver-eval bookmark. let bookmark = find_bookmark(&automation, &fav_bar, BOOKMARK_NAME); @@ -247,8 +253,40 @@ unsafe fn try_bookmark_exec_blocking( "captured original tab title: {:?}", original_title ); - // 6) Invoke the bookmark. + // 6) Invoke the bookmark with a no-foreground guard around it. + // + // Chromium activates the browser window when a bookmark is clicked + // because `WebContents::OpenURL` treats it as user-initiated + // navigation (the activation lives in Chromium's window-aura layer, + // not in our UIA call). Two-layer mitigation: + // + // (a) Capture `GetForegroundWindow()` immediately before the Invoke. + // (b) After the Invoke, briefly poll for "Chromium activated the + // browser" and `SetForegroundWindow(prev)` the user's window + // back. Same pattern `launch_app`'s `FocusRestoreGuard` + // successfully uses; the restore is gated on the new + // foreground being owned by Chromium so we never yank focus + // from a window the user legitimately tabbed to. + // + // Without UIAccess (the daemon's normal integrity), the restore + // SetForegroundWindow may be blocked by the system foreground lock; + // we log at trace level and continue — the brief activation is + // unavoidable without forking Chromium, but the dwell time on the + // browser is < ~150 ms instead of "until next user action". + let prev_fg_for_invoke = unsafe { + windows::Win32::UI::WindowsAndMessaging::GetForegroundWindow() + }; invoke_element(&bookmark).context("InvokePattern.Invoke on bookmark failed")?; + // Restore in a tight loop — at this point the bookmark URL is firing + // and Chromium is about to activate. Poll for ~600 ms; once we see + // Chromium grab foreground, swap back. We do this synchronously + // (not via tokio::spawn) because we WANT to block the poll-marker + // step below until the foreground is restored — otherwise a slow + // restore leaves the browser frontmost while we read the title. + restore_foreground_after_browser_activation( + prev_fg_for_invoke, + pid as u32, + ); // 7) Poll the active tab's title until it starts with CUA: / CUA_ERR: let result = poll_for_marker(&automation, &edge_window, &original_title)?; @@ -666,6 +704,104 @@ unsafe fn invoke_element(elem: &IUIAutomationElement) -> Result<()> { Ok(()) } +/// Belt-and-suspenders foreground restore after the bookmark Invoke. +/// +/// Chromium's bookmark click triggers `WebContents::OpenURL` → +/// `Browser::ActivateContents` → `SetForegroundWindow` on the browser +/// frame. This happens whether we like it or not, in Chromium-side code +/// we don't control. The mitigation: capture `prev_fg` immediately +/// before our `InvokePattern.Invoke` (the caller does this), then poll +/// for "the browser pid grabbed foreground" and call `SetForegroundWindow(prev_fg)` +/// the user's window back. Mirrors `launch_app`'s `FocusRestoreGuard` +/// (PR #1668, fixed by CodeRabbit feedback on PR #1668 to gate on PID +/// ownership rather than any foreground change). +/// +/// Synchronous (not tokio::spawn) because the caller's next step is +/// `poll_for_marker`, which reads the active tab's title via UIA — that +/// must happen against the now-restored foreground state so we don't +/// see transient state. +/// +/// Best-effort: when Windows' foreground-lock denies the restore (we're +/// not at UIAccess), the dwell time on the browser is still bounded to +/// the ~600 ms poll budget. Without this guard the browser stays +/// frontmost until the next user action. +unsafe fn restore_foreground_after_browser_activation( + prev_fg: windows::Win32::Foundation::HWND, + browser_pid: u32, +) { + use windows::Win32::Foundation::HWND; + use windows::Win32::UI::WindowsAndMessaging::{ + GetForegroundWindow, GetWindowThreadProcessId, IsWindow, SetForegroundWindow, + }; + + if prev_fg.0.is_null() { + tracing::trace!( + target: "page.bookmark_exec.focus_restore", + "no prior foreground to restore — skipping" + ); + return; + } + if browser_pid == 0 { + tracing::trace!( + target: "page.bookmark_exec.focus_restore", + "no browser pid — skipping restore (no ownership signal)" + ); + return; + } + + // Poll for ~600 ms in 50 ms steps. Chromium's activation typically + // lands within ~150 ms but the URL-load can stretch the window. + let deadline = std::time::Instant::now() + std::time::Duration::from_millis(600); + while std::time::Instant::now() < deadline { + let fg_now = GetForegroundWindow(); + if fg_now == prev_fg { + // Chromium didn't (yet) grab foreground — keep polling + // briefly; if it never does we just exit cleanly. + std::thread::sleep(std::time::Duration::from_millis(50)); + continue; + } + let mut fg_pid: u32 = 0; + let _ = GetWindowThreadProcessId(fg_now, Some(&mut fg_pid)); + if fg_pid != browser_pid { + // Foreground changed but to something we didn't expect + // (user Alt-Tabbed mid-eval). Leave it alone. + tracing::trace!( + target: "page.bookmark_exec.focus_restore", + "foreground changed to non-browser pid {fg_pid} (expected {browser_pid}) — leaving as-is" + ); + return; + } + // Chromium grabbed foreground — restore the user's window. + if !IsWindow(prev_fg).as_bool() { + tracing::trace!( + target: "page.bookmark_exec.focus_restore", + "prev foreground HWND no longer valid — skipping restore" + ); + return; + } + let ok = SetForegroundWindow(prev_fg).as_bool(); + if ok { + tracing::debug!( + target: "page.bookmark_exec.focus_restore", + "restored foreground to HWND 0x{:x} after browser pid {browser_pid} activation", + prev_fg.0 as usize + ); + } else { + tracing::trace!( + target: "page.bookmark_exec.focus_restore", + "SetForegroundWindow returned FALSE (foreground-lock denial; need UIAccess) — \ + browser will keep focus until next user action" + ); + } + return; + } + tracing::trace!( + target: "page.bookmark_exec.focus_restore", + "browser pid {browser_pid} did not steal foreground within 600ms — no restore needed" + ); + let _ = HWND::default(); // silence "unused import" possibility on cfg +} + /// Find the TabItem that's currently selected — Chromium exposes the /// active tab as the only TabItem whose `IsSelected` is true. We don't /// strictly need IsSelected: a fallback is "the first TabItem in the tab