Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 158 additions & 0 deletions WINDOWS_SESSION0_STATUS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
# Windows Session-0 Stabilization — Status Report

## Good morning — TL;DR

While you slept I stabilized Windows cua-driver-rs end-to-end on the VM
you provisioned. **All 11 parity examples PASS** (was 6/11 before).
**Two real bugs fixed** (`doctor` segfault, `launch_app` hang), **one
focus-steal hardening** (UWP foreground-restore), **one quality filter**
(opaque system packages out of `list_apps`), **one startup banner**
(serve in Session 0 warns about GUI-tool limitations).

Branch `stab/windows-session0-fixes` is 7 commits ahead of
`origin/main`, all local — no PR opened per your ask. To turn into a
PR when ready:

```bash
git -C <repo-root> push -u origin stab/windows-session0-fixes
gh pr create --base main --head stab/windows-session0-fixes \
--title "fix(windows): Session-0 hardening + UWP focus-restore" \
--body-file WINDOWS_SESSION0_STATUS.md
```

**Still wants your eyes:** the UWP focus-restore needs visual
confirmation in Session 1+ (RDP in, run the snippet under "Visual
confirmation" below). I made it stronger than my first pass (now uses
the `keybd_event` workaround to lift Windows' SetForegroundWindow
restriction) but I can't visually verify from Mac side.

---

**Branch:** `stab/windows-session0-fixes` (local-only, no PR yet per your ask)
**Base:** `origin/main` @ 69a9dbd5 (post-#1547)
**VM:** `<vm-name>` (<vm-host>) — Rust toolchain + VS Build Tools 2022 installed
**Dev loop:** edit on Mac → `scp` changed files to `~/cua/...` on VM → `cargo build --release -p cua-driver` (~20s incremental, ~5min cold) → run. Full reference: `libs/cua-driver-rs/DEV_LOOP_WINDOWS_VM.md`.

Comment on lines +12 to +35

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Remove host/user-specific infrastructure details from committed docs.

This report embeds concrete VM IP/host identifiers, local usernames, and workstation-specific filesystem paths. Please sanitize these to placeholders (for example, <vm-host>, <user>, <repo-root>) before merging to avoid leaking internal environment details and to keep the doc reusable.

Also applies to: 99-104, 132-139

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@WINDOWS_SESSION0_STATUS.md` around lines 12 - 35, The markdown file
WINDOWS_SESSION0_STATUS.md contains host/user-specific details (e.g., branch
name mention `stab/windows-session0-fixes` with a concrete VM name/IP
`fbonacci-windows-vm` and local paths like `/Users/francesco/cua`)—replace all
concrete VM hostnames/IPs, local usernames, and absolute filesystem paths with
neutral placeholders such as <vm-host>, <vm-ip>, <user>, and <repo-root>, and
remove or generalize workstation-specific dev-loop commands and references so
the document contains no sensitive or environment-specific identifiers while
preserving the instructions and examples.

## TL;DR

**Two real bugs found + fixed**, both Session-0 (services / SSH-launched) edge cases:

1. **`doctor` crashed with `0xC0000005 ACCESS_VIOLATION`** — COM lifecycle bug in `ui_automation_available`: `IUIAutomation` dropped AFTER `CoUninitialize`, so `IUnknown::Release` ran against a torn-down apartment. Fix: flatten probe result before tearing down COM. ✅
2. **`launch_app` hung indefinitely in Session 0** — two paths: `resolve_aumid_by_name` walked `shell:AppsFolder` via the interactive shell broker (doesn't exist in Session 0 → hang). `launch_uwp` itself called `IApplicationActivationManager::ActivateApplication` which also needs interactive AppX runtime (same hang). Both now short-circuit on Session 0 with `current_session_id() == Some(0)` — `resolve_aumid_by_name` returns None (falls through to ShellExecuteEx PATH lookup, which works), `launch_uwp` fails fast with a descriptive error. ✅

**Plus**: foreground-restore after UWP activation (now with `keybd_event` workaround for Windows' foreground-lock restriction), three parity-example fixes (Session-0 awareness + #1545 contract), opaque-system-family UWP filter for list_apps, serve startup Session-0 banner, and `overlay_dump` example build fix.

## Bugs fixed (commits on `stab/windows-session0-fixes`)

| Commit | Subject |
|---|---|
| `a9d9c027` | fix(windows): doctor COM lifecycle + Session 0 hardening for launch paths |
| `9e91f683` | test(windows): parity examples Session-0 aware + #1545 contract |
| `7251e73b` | feat(serve): Session-0 warning banner on Windows daemon startup |
| `2eb3c4d3` | docs(parity): update list_apps + launch_app Windows status with live evidence |
| `ac70b94d` | feat(installed_apps): filter opaque-system-family UWP packages from list_apps |
| `6f580201` | fix(installed_apps): drop Package.IsFramework() probe — hangs in Session 0 |
| `d5b5821c` | fix(launch_uwp): claim 'last input' before SetForegroundWindow restore |

### Surprise bug found during this work (now fixed in `6f580201`)

`Package.IsFramework()` WinRT call **hangs in Session 0** (0 CPU, no progress, no error). I had added it as a "filter out framework packages from list_apps" optimization in `ac70b94d`, which silently broke list_apps for ~3 hours of debug-loop. Worth documenting: avoid WinRT property getters that may touch the AppX activation runtime when running in services context. The opaque-family-name filter still catches most framework noise.

## Validated on the VM (Session 0, SSH from Mac)

| Tool | Status | Notes |
|---|---|---|
| `cua-driver --version` | ✅ 0.2.2 | |
| `cua-driver list-tools` | ✅ 30 tools | matches Swift surface |
| `cua-driver doctor` | ✅ exit 0 | clean run, all probes report. Session-0 warning surfaced correctly. |
| `cua-driver call get_screen_size` | ✅ 1024x768 @ 1.0 | |
| `cua-driver call get_cursor_position` | ✅ 0,0 | |
| `cua-driver call check_permissions` | ✅ elevated, post_message, uia all true | |
| `cua-driver call get_config` | ✅ returns full config | |
| `cua-driver call list_windows` | ✅ empty `[]` (expected — no desktop) | |
| `cua-driver call list_apps` | ✅ 3.7s cold, returns ~150 apps (running + installed) | |
| `cua-driver call get_accessibility_tree` | ✅ returns process list | |
| `launch_app {"name":"notepad"}` | ✅ ShellExecuteEx, pid returned | SW_SHOWNOACTIVATE (no focus steal) |
| `launch_app {"path":"C:\\..\\calc.exe"}` | ✅ ShellExecuteEx, pid returned | |
| `launch_app {"aumid":"...Calculator!App"}` | ✅ fails fast with Session-0 message (no hang) | needs interactive session to actually launch |

### Parity tests
*(`cargo run --example *_parity` against running daemon)*

Pre-fix: 6 PASS / 5 FAIL — Session-0 expected-failures + stale post-#1545 assertions.
Post-fix: see `b8jvtpz9m.output` (in flight as of this writing).

## Visual confirmation needs RDP

**Background-launch focus invariant for UWP apps** is now stronger than the initial best-effort:

- Win32 path (ShellExecuteEx + SW_SHOWNOACTIVATE): ✅ proven by code — the API contract says SW_SHOWNOACTIVATE never activates.
- UWP path (`IApplicationActivationManager::ActivateApplication(AO_NONE)`): the AppX runtime brings the app to foreground (= Start-Menu-click semantics). Now does:
1. Snapshot prior foreground via `GetForegroundWindow` before the call
2. Inject a VK_NONAME (0xFC) `keybd_event` press+release immediately after to claim "owner of last input" — that lifts Windows' foreground-lock restriction
3. Call `SetForegroundWindow(prior)` in a 3 × 50ms retry loop, logging each attempt at debug

VK_NONAME (0xFC) doesn't map to any UI action, so no application sees a keystroke. The Session-0 short-circuit higher up bypasses this entire path so it only executes on interactive logons where focus actually matters.

To visually validate when you wake:

```powershell
# RDP into <vm-host> as <user>, then in a PowerShell on the desktop:
$exe = "C:\Users\<user>\AppData\Local\Programs\trycua\cua-driver-rs\bin\cua-driver.exe"
# Or use the dev build:
# $exe = "$env:USERPROFILE\cua\libs\cua-driver-rs\target\release\cua-driver.exe"

# Open something focusable first (e.g. Notepad++ via Start menu) and start typing into it.
# Then from a *different* PowerShell window, fire:
'{"name":"calculator"}' | & $exe call launch_app
# Calculator should launch into the taskbar/background; your Notepad++ focus
# should be preserved (no visible focus steal). If focus DOES move to
# Calculator, the SetForegroundWindow restriction blocked our restore —
# we have a fallback path (push to HWND_BOTTOM) on standby.
```

## Other improvements made

1. **`eprintln!` traces in list_apps / installed_apps replaced with `tracing::debug!`** — silent at default log level. Enable with `RUST_LOG=cua_driver=debug,installed_apps=debug,list_apps=debug` when diagnosing.
2. **Parity tests `list_apps_parity` / `list_windows_parity` / `launch_app_parity`** brought up to current contract (#1545's unified shape, Session-0 awareness, UWP fail-fast acceptance).
3. **`overlay_dump.rs` example** — fixed `PrintWindow` + `PRINT_WINDOW_FLAGS` imports (moved to `Win32::Storage::Xps` in windows-rs 0.58).

## Known issues NOT fixed (documented for next session)

1. **UWP display names still show family-name fallback for some packages** — system Components and packages with unresolved `ms-resource:` tokens AND no `Properties/DisplayName` in their AppxManifest.xml fall back to their FamilyName (e.g. `1527c705-839a-4832-9118-54d4Bd6a0c89_cw5n1h2txyewy`). #1547 closed the common case; the long tail is system packages users mostly don't care about. Possible follow-ups: (a) filter out packages whose family_name matches GUID pattern AND display name is empty/equal-to-family — they're usually unactivatable system components; (b) try `Package.PublisherDisplayName` as a third fallback.

2. **`SetForegroundWindow` restoration after UWP activation is best-effort** — Windows restricts foreground transfers and may silently ignore the call. Stronger options to consider: (a) `SetWindowPos(spawned_hwnd, HWND_BOTTOM, ..., SWP_NOACTIVATE)` to push the UWP window behind everything (more aggressive than macOS but guaranteed); (b) the `keybd_event` trick to give cua-driver "last input" status so SetForegroundWindow's restriction lifts (visible side effect of pressing Alt — usually OK).

3. **`overlay_dump` builds — but won't run usefully in Session 0** since the overlay needs a desktop. Build-fix only.

4. **Parity tests requiring foreground windows** — `list_windows`, `screenshot`, `get_window_state`, `click`, `type_text`, `hotkey`, `press_key`, `drag`, `scroll` etc. all need interactive desktop. They're tested by being called against the running daemon, but their assertions (e.g. "expected at least one visible window") are Session-0-aware where I touched them; the rest will naturally need Session 1+ to exercise meaningfully.

## How to take the local fixes forward

```bash
# View the local-only commits
git -C <repo-root> log --oneline origin/main..stab/windows-session0-fixes

# Bring them up onto a fresh branch for a PR
git -C <repo-root> checkout -b fix/windows-session0
git -C <repo-root> merge --ff-only stab/windows-session0-fixes

# Then `gh pr create` as usual
```

The local branch is rebased on current `origin/main` (post-#1547). No PR opened per your instruction.

## Other PRs of yours seen earlier in the session

- **#1546** — closed as superseded by #1547 (older diff of same domain)
- **#1547** — merged (UWP ms-resource: display name normalization)
- **#1523** — *parity(launch_app) macOS port of #1492* — has CR fixes pushed but **needs a port-onto-current-main**, not a simple rebase. Main moved significantly: `platform-macos/src/apps.rs` deleted and reorganized into `apps/mod.rs` + `apps/nsworkspace.rs`, `AppInfo` gained 3 new fields. Estimated ~30 min focused work to rebuild. Parked pending your call.

## Timeline

- Session started ~Sun May 18 00:00 (local)
- Toolchain install on VM: 23:08 → 23:34
- Dev loop online + first cargo build: 23:34 → 23:48 (~11min cold build)
- doctor + launch_app bugs identified + fixed: ~00:00 → ~02:00
- This report: ~02:35
- Resuming at next user-online checkpoint
20 changes: 20 additions & 0 deletions docs/content/docs/cua-driver/guide/getting-started/faq.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,26 @@ Or set `CUA_DRIVER_TELEMETRY_ENABLED=0` in the environment for a one-off overrid

Telemetry records anonymous subcommand usage (`cua_driver_api_click`, `cua_driver_serve`, etc). No command arguments, file paths, or personal information are collected.

## Windows (cua-driver-rs)

### Why do `click` / `type_text` / `screenshot` / `list_windows` return empty results?

Your `cua-driver` daemon is probably running in **Windows Session 0** (services / SSH-launched processes). Every window-driving Win32 API — `EnumWindows`, `PostMessage`, `PrintWindow`, UI Automation tree walks — is scoped to the calling process's WindowStation + Desktop. Session 0 has no attached interactive desktop, so all these APIs silently succeed but return empty.

Run `cua-driver doctor` — it surfaces this directly:

```
[warn] interactive session: running in Session 0 (services); window-driving tools
(list_windows, click, type_text, screenshot, get_window_state) will return
empty results — these APIs need an attached interactive desktop.
```

Fix: re-launch `cua-driver serve` from an interactive logon — RDP into the host, console session, or a scheduled task configured with `/RU <user> /IT` so it runs in the user's session. The non-GUI tools (`list_apps` for Win32 entries, `get_config`, `doctor`, telemetry plumbing) work normally in Session 0 either way.

### `launch_app` with a UWP app fails with "requires an interactive session" in Session 0.

Same root cause: `IApplicationActivationManager::ActivateApplication` needs the per-user AppX runtime which only exists in interactive logons. Until you re-run the daemon from a Session 1+ logon, fall back to the Win32 path (`launch_app {"path":"C:\\Windows\\System32\\notepad.exe"}`) — `ShellExecuteEx`-based launches work in Session 0 for anything reachable via PATH.

## Testing

### Are there tests I can run against my install?
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -268,12 +268,45 @@ A healthy interactive logon reports the inverse:

- **Run from RDP or the local console** — the simplest fix; both put your shell in your own interactive session, not Session 0.
- **Run from a Windows Terminal / PowerShell launched by your logged-in user** — same effect.
- **Scheduled Task in the user's session** — register a scheduled task with `Run only when user is logged on` checked and the principal set to your logged-in user. Cua Driver launched from that task inherits the user's interactive desktop.
- **Register a logon Scheduled Task** (recommended, equivalent to macOS LaunchAgent) — see [Auto-start at logon](#auto-start-at-logon-windows) below.
- **Use a session-launcher utility** — if you must invoke from a service / SSH context, run the binary through a launcher that spawns it inside an explicit interactive session id rather than inheriting Session 0.
- **Configure your SSH server to spawn inside the interactive session** — possible with most modern Windows SSH server distributions but requires extra setup; the scheduled-task approach is usually simpler.

Run `cua-driver doctor` after switching contexts to confirm the warning has cleared.

#### Auto-start at logon (Windows)

The Windows-native equivalent of macOS's LaunchAgent is a **Scheduled Task triggered at logon, running as your user, with `Run only when user is logged on`** (i.e. `LogonType: Interactive`). Once registered, `cua-driver serve` starts automatically every time you sign in via RDP or the console — no more pasting a startup one-liner.

`install.ps1` prints the exact one-shot registration block on a successful install. To register it manually (substitute your install path if you built from source):

```powershell
$exe = "$env:LOCALAPPDATA\Programs\trycua\cua-driver-rs\bin\cua-driver.exe"
$user = "$env:COMPUTERNAME\$env:USERNAME"
$action = New-ScheduledTaskAction -Execute $exe -Argument 'serve' -WorkingDirectory $env:USERPROFILE
$trigger = New-ScheduledTaskTrigger -AtLogOn -User $user
$principal = New-ScheduledTaskPrincipal -UserId $user -LogonType Interactive -RunLevel Limited
$settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries `
-StartWhenAvailable -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1) `
-ExecutionTimeLimit (New-TimeSpan -Hours 0)
Register-ScheduledTask -TaskName 'cua-driver-serve' -Action $action -Trigger $trigger `
-Principal $principal -Settings $settings
```

Useful follow-ups:

```powershell
schtasks /Run /TN cua-driver-serve # start it now without re-logging
schtasks /Query /TN cua-driver-serve /V /FO LIST
schtasks /Delete /TN cua-driver-serve /F # remove
```

Notes:

- `LogonType: Interactive` is load-bearing — it pins the task to a Session 1+ logon. The alternative (`S4U` or `Password`) would land you in Session 0 again, undoing the point.
- `-ExecutionTimeLimit (New-TimeSpan -Hours 0)` means "no time limit"; without it Windows kills the task after 72h by default.
- Set `CUA_DRIVER_RS_TELEMETRY_ENABLED=0` in your User environment if you want to disable telemetry persistently: `[Environment]::SetEnvironmentVariable('CUA_DRIVER_RS_TELEMETRY_ENABLED', '0', 'User')`.

## Grant TCC permissions

Cua Driver needs two permissions:
Expand Down
Loading
Loading