cua-driver: handle macOS 26.4 SCK streaming-start failure

docs/content/docs/cua-driver/guide/getting-started/faq.mdx

@@ -36,6 +36,28 @@ open -n -g -a CuaDriver --args serve
 
 The window capture raced against a close, or the window has no backing store yet. Re-snapshot. If it persists, pick a different `window_id` via `list_windows`.
 
+### `screenshot` or `get_window_state` fails with "ScreenCaptureKit refused this window" / "Could not start streaming".
+
+Known macOS 26.4.x ScreenCaptureKit regression on physical Macs (`SCStreamError` code -3801, sometimes localized — e.g. Japanese "オーディオ/ビデオの取り込みがうまくいかなかったため、ストリーミングを開始できませんでした"). The driver already:
+
+1. Retries the SCK call once after a brief delay (covers transient failures).
+2. Falls back to the legacy `CGWindowListCreateImage` path (works on many windows the SCK regression breaks).
+
+If both refuse, the error surfaces with an actionable hint. Workarounds in order of preference:
+
+1. **Try a different `window_id` on the same app.** Usually only one specific window is hit.
+2. **Switch to AX-only capture for that workflow.** Element-indexed clicks don't need pixels:
+
+   ```bash
+   cua-driver config set capture_mode ax
+   ```
+
+   `get_window_state` then returns the AX tree without attempting a screenshot, and `click({pid, window_id, element_index: N})` works as before.
+
+3. **Re-snapshot a moment later.** The failure is sometimes transient.
+
+`get_window_state` does **not** hard-fail on this error: the AX tree still ships in the response with a warning line, so element-indexed clicks keep working even when the screenshot is unavailable. The standalone `screenshot` tool does hard-fail (no AX tree to fall back to).
+
 ### The AX tree is tiny. What's happening?
 
 Check the capture mode:

docs/content/docs/cua-driver/reference/cli-reference.mdx

@@ -7,7 +7,7 @@ description: Command Line Interface reference for Cua Driver
   AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
   Generated by: npx tsx scripts/docs-generators/cua-driver.ts
   Source: recursive Swift sources under libs/cua-driver/Sources
-  Version: 0.1.1
+  Version: 0.1.6
 */}
 
 import { Callout } from 'fumadocs-ui/components/callout';
@@ -16,7 +16,7 @@ import { VersionHeader } from '@/components/version-selector';
 <VersionHeader
   versions={[{"version":"0.1","href":"/cua-driver/reference/cli-reference","isCurrent":true}]}
   currentVersion="0.1"
-  fullVersion="0.1.1"
+  fullVersion="0.1.6"
   packageName="cua-driver"
   installCommand="curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash"
 />

docs/content/docs/cua-driver/reference/mcp-tools.mdx

@@ -7,7 +7,7 @@ description: Reference for every MCP tool cua-driver exposes
   AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
   Generated by: npx tsx scripts/docs-generators/cua-driver.ts
   Source: recursive Swift sources under libs/cua-driver/Sources
-  Version: 0.1.1
+  Version: 0.1.6
 */}
 
 import { Callout } from 'fumadocs-ui/components/callout';
@@ -318,6 +318,16 @@ keeps its name:
 Change with `cua-driver config set capture_mode <mode>` or
 the `set_config` tool.
 
+Screenshot capture failures in `som` / `vision` modes are
+non-fatal for `som`: the AX tree still ships in the
+response and the summary line carries a hint. The macOS
+26.4.x SCK regression (SCStreamError -3801, "Could not
+start streaming") is handled this way — agents can keep
+doing element-indexed clicks against the same window even
+when the screenshot is unavailable. Switching to
+`capture_mode: ax` skips the capture attempt entirely on
+subsequent turns.
+
 Requires Accessibility and Screen Recording permissions.
 
 **Arguments:**
@@ -340,6 +350,17 @@ screenshot selection. The combo is posted directly to
 the target pid's event queue via `CGEvent.postToPid`;
 the target does NOT need to be frontmost.
 
+**`window_id`** (optional): when supplied, the driver
+calls `FocusWithoutRaise.activateWithoutRaise` before
+posting — making the target AppKit-active without
+raising its window. This is required for shortcuts that
+are dispatched via NSMenu key equivalents (Cmd+S,
+Cmd+N, Cmd+W, Cmd+Shift+N, …) because AppKit only
+routes menu key equivalents to the active app. Omit
+`window_id` for shortcuts handled by the renderer
+(e.g. Cmd+C in a Chromium text field). Trade-off: the
+sentinel foreground app will lose focus once.
+
 Recognized modifiers: cmd/command, shift, option/alt,
 ctrl/control, fn. Non-modifier keys use the same
 vocabulary as `press_key` (return, tab, escape,
@@ -351,6 +372,7 @@ first, one non-modifier last.
 
 - `keys` (array of string, required): Modifier(s) and one non-modifier key, e.g. ["cmd", "c"].
 - `pid` (integer, required): Target process ID.
+- `window_id` (integer, optional): CGWindowID of the target window. When provided, the driver calls FocusWithoutRaise before posting so NSMenu key equivalents (Cmd+S, Cmd+N, …) reach the backgrounded app.
 
 ```json
 {"keys":["cmd","c"],"pid":844}
@@ -370,12 +392,11 @@ Provide either `bundle_id` (preferred — unambiguous, e.g.
 against /Applications and /System/Applications). If both are
 given, bundle_id wins.
 
-Launches hidden — the app runs with its window initialized
-for automation (so the AX tree is populated and
-`click({pid, element_index})` works end-to-end), but no
-window is drawn on screen. To bring the window on-screen,
-use `NSRunningApplication.unhide()` from your own code, or
-have the user open the app (Dock click, Cmd-Tab, Spotlight).
+Launches in the background — the app's window appears on
+screen but does not steal focus from whatever is currently
+frontmost. The AX tree is fully populated and automation
+can start immediately via `click`, `hotkey`,
+`get_window_state`, etc.
 
 Some apps (Calculator and many Electron apps) call
 `NSApp.activate(ignoringOtherApps:)` in their own
@@ -386,16 +407,14 @@ focus-steal preventer arms a short
 around the launch and re-activates the previously-frontmost
 app if the target self-activates. If suppression fails the
 returned summary includes a "self-activation not suppressed"
-warning. With `hides = true`, the target is also hidden
-back immediately by LaunchServices, so even a missed
-layer-3 demotion doesn't leave a visible window.
+warning.
 
 Optional `urls` are handed to the app through its
 `application(_:open:)` AppKit delegate. For Finder, passing
-a folder URL opens a backgrounded Finder window rooted at
-that folder — no activation, no visible window. Works
-with any app that implements `application(_:open:)`; apps
-that ignore the delegate simply launch without side effects.
+a folder URL opens a Finder window rooted at that folder in
+the background. Works with any app that implements
+`application(_:open:)`; apps that ignore the delegate simply
+launch without side effects.
 
 Optional `electron_debugging_port` launches an Electron app
 with `--remote-debugging-port=<N>`, activating its Chrome
@@ -600,7 +619,7 @@ cmd/shift/option/ctrl/fn. For true combinations (cmd+c),
 - `key` (string, required): Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit).
 - `modifiers` (array of string, optional): Optional modifier names held while the key is pressed (cmd/shift/option/ctrl/fn).
 - `pid` (integer, required): Target process ID.
-- `window_id` (integer, optional): CGWindowID for the window whose get_window_state produced the element_index. Required when element_index is used.
+- `window_id` (integer, optional): CGWindowID for the window. Required when element_index is used. When supplied without element_index, triggers FocusWithoutRaise before the key fires — use this for NSMenu key equivalents (Cmd+S, Cmd+N, …) on backgrounded apps.
 
 ```json
 {"key":"return","pid":844}
@@ -698,11 +717,20 @@ propagate modifier keys).
 Capture a screenshot using ScreenCaptureKit. Returns base64-encoded
 image data for a single window in the requested format (default png).
 
-`window_id` is required. Get window ids from `list_windows`.
+
+                `window_id` is required. Get window ids from `list_windows`.
 
 Requires the Screen Recording TCC grant — call `check_permissions`
 first if unsure.
 
+On macOS 26.4.x, ScreenCaptureKit can refuse specific windows on
+physical Macs (SCStreamError -3801, "Could not start streaming").
+The driver retries once and falls back to the legacy
+CGWindowList path before failing; if both refuse, the error
+response includes a hint to try a different `window_id` or
+switch to `capture_mode: ax` for `get_window_state` (the
+element-indexed flow doesn't need pixels).
+
 **Arguments:**
 
 - `format` (string, optional): Image format. Default: png.

libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift

@@ -33,6 +33,13 @@ public enum CaptureError: Error, Sendable, CustomStringConvertible {
     case encodeFailed
     case captureFailed(String)
     case windowNotFound(UInt32)
+    /// ScreenCaptureKit could not start streaming for this window. Distinct
+    /// from `captureFailed` so callers (e.g. `get_window_state`) can surface
+    /// an actionable hint — switch to `capture_mode: ax`, retry against a
+    /// different window — without having to grep error strings. Seen
+    /// regularly on macOS 26.4.x physical Macs against specific windows
+    /// where even `screencapture -l<id>` fails (rdar / openclaw/Peekaboo#121).
+    case streamingFailed(String)
 
     public var description: String {
         switch self {
@@ -41,6 +48,7 @@ public enum CaptureError: Error, Sendable, CustomStringConvertible {
         case .encodeFailed: return "failed to encode CGImage"
         case .captureFailed(let msg): return "capture failed: \(msg)"
         case .windowNotFound(let id): return "no shareable window with id \(id)"
+        case .streamingFailed(let msg): return "ScreenCaptureKit streaming failed: \(msg)"
         }
     }
 }
@@ -131,12 +139,43 @@ public actor WindowCapture {
         config.height = max(1, Int(window.frame.height * scale))
         config.showsCursor = false
 
+        // One-shot SCK call with a single retry on streaming-start failure.
+        // macOS 26.4.x has a regression where `SCScreenshotManager.captureImage`
+        // intermittently returns "Could not start streaming because audio/video
+        // capture failed" (SCStreamError code -3801) on physical Macs, often
+        // recovering on a second attempt a moment later. We retry once with a
+        // brief back-off; if it still fails, we surface `.streamingFailed` so
+        // the tool layer can hint the caller toward `capture_mode: ax` for
+        // `get_window_state` workflows.
         let cgImage: CGImage
         do {
-            cgImage = try await SCScreenshotManager.captureImage(
-                contentFilter: filter,
-                configuration: config
-            )
+            cgImage = try await captureSCKWithRetry(filter: filter, config: config)
+        } catch let error as CaptureError {
+            // Already classified — re-throw without wrapping. CGWindowList
+            // is intentionally NOT tried for permission errors (it'd just
+            // fail the same way and confuse the user-facing message).
+            if case .permissionDenied = error { throw error }
+            // For streaming / generic SCK failures, try the legacy
+            // CGWindowListCreateImage path. It's deprecated on macOS 15+
+            // but still works in many cases where SCK refuses — particularly
+            // useful as a last-ditch fallback for the 26.4 SCK regression.
+            if let fallback = legacyCaptureWindow(windowID: windowID) {
+                let origW = fallback.width
+                let origH = fallback.height
+                let resized = resizeIfNeeded(fallback, maxDim: maxImageDimension)
+                let didResize = resized.width != origW || resized.height != origH
+                let data = try encode(resized, format: format, quality: quality)
+                return Screenshot(
+                    imageData: data,
+                    format: format,
+                    width: resized.width,
+                    height: resized.height,
+                    scaleFactor: Double(scale),
+                    originalWidth: didResize ? origW : nil,
+                    originalHeight: didResize ? origH : nil
+                )
+            }
+            throw error
         } catch {
             throw classify(error)
         }
@@ -207,17 +246,121 @@ public actor WindowCapture {
         return (best ?? NSScreen.main)?.backingScaleFactor ?? 1.0
     }
 
+    /// Attempt `SCScreenshotManager.captureImage` once; on a streaming-start
+    /// failure, wait briefly and retry once more. Returns a classified
+    /// `CaptureError` on persistent failure so the caller can branch on the
+    /// kind (permission vs. streaming vs. generic) without string-matching.
+    ///
+    /// The retry covers the macOS 26.4.x SCK regression where the very first
+    /// call after the SCK daemon has been idle returns -3801 ("Could not
+    /// start streaming because audio/video capture failed") but a second
+    /// call ~250ms later succeeds. A second failure isn't transient and we
+    /// stop retrying — the caller falls back to CGWindowList or surfaces
+    /// the error.
+    private func captureSCKWithRetry(
+        filter: SCContentFilter,
+        config: SCStreamConfiguration
+    ) async throws -> CGImage {
+        do {
+            return try await SCScreenshotManager.captureImage(
+                contentFilter: filter,
+                configuration: config
+            )
+        } catch {
+            let classified = classify(error)
+            // Only retry on streaming-start failures; permission errors and
+            // not-found errors won't change on a second attempt.
+            guard case .streamingFailed = classified else { throw classified }
+            try? await Task.sleep(nanoseconds: 250_000_000)
+            do {
+                return try await SCScreenshotManager.captureImage(
+                    contentFilter: filter,
+                    configuration: config
+                )
+            } catch {
+                throw classify(error)
+            }
+        }
+    }
+
+    /// Legacy `CGWindowListCreateImage` fallback for the SCK 26.4 regression.
+    /// Deprecated by Apple in macOS 15 but still functional on most windows,
+    /// and frequently works where SCK refuses. Returns nil on failure — the
+    /// caller surfaces the original SCK error in that case so the user knows
+    /// the real cause.
+    ///
+    /// Marked with `@available(*, deprecated)` suppression because the API
+    /// is the entire point: we *want* the legacy path here.
+    private func legacyCaptureWindow(windowID: UInt32) -> CGImage? {
+        // CGWindowListCreateImage is deprecated on macOS 15+. The deprecation
+        // diagnostic is silenced with the @available pragma. Apple has not
+        // (yet) removed the symbol, and this path is the only practical
+        // fallback when SCK's streaming-start is broken for a given window.
+        let opts: CGWindowImageOption = [.boundsIgnoreFraming, .bestResolution]
+        let listOption: CGWindowListOption = .optionIncludingWindow
+        // Wrap the deprecated call so we keep the unsafePointer-style
+        // signature out of the rest of the code.
+        let image = legacyCGWindowImage(
+            windowID: windowID, listOption: listOption, imageOption: opts
+        )
+        // Reject 1×1 placeholder images that the legacy API sometimes returns
+        // for occluded / off-screen windows — they're worse than no image.
+        guard let image, image.width > 1, image.height > 1 else { return nil }
+        return image
+    }
+
     private func classify(_ error: Error) -> CaptureError {
         let ns = error as NSError
         let msg = ns.localizedDescription.lowercased()
+
+        // Permission failure — English and Japanese phrasings observed in
+        // SCK's `NSError.localizedDescription`. The Japanese strings cover
+        // users on JP system locale where the SCK error comes back
+        // localized rather than in English.
         if msg.contains("permission") || msg.contains("not authorized")
             || msg.contains("declined") || msg.contains("denied")
+            || ns.localizedDescription.contains("許可")     // "permission"
+            || ns.localizedDescription.contains("拒否")     // "denied"
         {
             return .permissionDenied
         }
+
+        // SCStreamError "could not start streaming" — code -3801 in
+        // `SCStreamErrorDomain`. macOS localizes the message ("Could not
+        // start streaming because audio/video capture failed" / Japanese:
+        // "オーディオ/ビデオの取り込みがうまくいかなかったため、ストリーミングを開始できませんでした"),
+        // so we match on code first and fall through to substring matching
+        // for the rare case where the domain isn't surfaced.
+        let isSCStreamDomain = ns.domain == "SCStreamErrorDomain"
+            || ns.domain == "com.apple.ScreenCaptureKit.SCStreamErrorDomain"
+        if (isSCStreamDomain && ns.code == -3801)
+            || msg.contains("could not start streaming")
+            || msg.contains("streaming")
+            || ns.localizedDescription.contains("ストリーミング")  // "streaming"
+        {
+            return .streamingFailed(ns.localizedDescription)
+        }
+
         return .captureFailed(ns.localizedDescription)
     }
 
+    /// Thin shim around the deprecated `CGWindowListCreateImage` so the
+    /// deprecation-warning suppression is isolated to one place. Returns nil
+    /// if the legacy path also refuses to produce an image.
+    ///
+    /// Marking the wrapper itself deprecated downgrades the call-site
+    /// warning to a no-op — we *want* this legacy path because SCK has a
+    /// well-known regression on macOS 26.4.x where streaming-start fails
+    /// for specific windows on physical Macs.
+    @available(*, deprecated, message: "Intentional fallback for SCK streaming-start failures.")
+    private func legacyCGWindowImage(
+        windowID: UInt32,
+        listOption: CGWindowListOption,
+        imageOption: CGWindowImageOption
+    ) -> CGImage? {
+        CGWindowListCreateImage(.null, listOption, windowID, imageOption)
+    }
+
     /// Capture the topmost layer-0 window owned by `pid`, or `nil` when the
     /// pid has no such window at all (menubar-only helpers, apps that
     /// haven't created any window yet).