diff --git a/docs/content/docs/cua-driver/guide/getting-started/faq.mdx b/docs/content/docs/cua-driver/guide/getting-started/faq.mdx index 7a0623d61c..dc802f9459 100644 --- a/docs/content/docs/cua-driver/guide/getting-started/faq.mdx +++ b/docs/content/docs/cua-driver/guide/getting-started/faq.mdx @@ -36,6 +36,28 @@ open -n -g -a CuaDriver --args serve The window capture raced against a close, or the window has no backing store yet. Re-snapshot. If it persists, pick a different `window_id` via `list_windows`. +### `screenshot` or `get_window_state` fails with "ScreenCaptureKit refused this window" / "Could not start streaming". + +Known macOS 26.4.x ScreenCaptureKit regression on physical Macs (`SCStreamError` code -3801, sometimes localized — e.g. Japanese "オーディオ/ビデオの取り込みがうまくいかなかったため、ストリーミングを開始できませんでした"). The driver already: + +1. Retries the SCK call once after a brief delay (covers transient failures). +2. Falls back to the legacy `CGWindowListCreateImage` path (works on many windows the SCK regression breaks). + +If both refuse, the error surfaces with an actionable hint. Workarounds in order of preference: + +1. **Try a different `window_id` on the same app.** Usually only one specific window is hit. +2. **Switch to AX-only capture for that workflow.** Element-indexed clicks don't need pixels: + + ```bash + cua-driver config set capture_mode ax + ``` + + `get_window_state` then returns the AX tree without attempting a screenshot, and `click({pid, window_id, element_index: N})` works as before. + +3. **Re-snapshot a moment later.** The failure is sometimes transient. + +`get_window_state` does **not** hard-fail on this error: the AX tree still ships in the response with a warning line, so element-indexed clicks keep working even when the screenshot is unavailable. The standalone `screenshot` tool does hard-fail (no AX tree to fall back to). + ### The AX tree is tiny. What's happening? Check the capture mode: diff --git a/docs/content/docs/cua-driver/reference/cli-reference.mdx b/docs/content/docs/cua-driver/reference/cli-reference.mdx index a61daedff8..9976f56df3 100644 --- a/docs/content/docs/cua-driver/reference/cli-reference.mdx +++ b/docs/content/docs/cua-driver/reference/cli-reference.mdx @@ -7,7 +7,7 @@ description: Command Line Interface reference for Cua Driver AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY Generated by: npx tsx scripts/docs-generators/cua-driver.ts Source: recursive Swift sources under libs/cua-driver/Sources - Version: 0.1.1 + Version: 0.1.6 */} import { Callout } from 'fumadocs-ui/components/callout'; @@ -16,7 +16,7 @@ import { VersionHeader } from '@/components/version-selector'; diff --git a/docs/content/docs/cua-driver/reference/mcp-tools.mdx b/docs/content/docs/cua-driver/reference/mcp-tools.mdx index b41588de26..25bcba1bdf 100644 --- a/docs/content/docs/cua-driver/reference/mcp-tools.mdx +++ b/docs/content/docs/cua-driver/reference/mcp-tools.mdx @@ -7,7 +7,7 @@ description: Reference for every MCP tool cua-driver exposes AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY Generated by: npx tsx scripts/docs-generators/cua-driver.ts Source: recursive Swift sources under libs/cua-driver/Sources - Version: 0.1.1 + Version: 0.1.6 */} import { Callout } from 'fumadocs-ui/components/callout'; @@ -318,6 +318,16 @@ keeps its name: Change with `cua-driver config set capture_mode ` or the `set_config` tool. +Screenshot capture failures in `som` / `vision` modes are +non-fatal for `som`: the AX tree still ships in the +response and the summary line carries a hint. The macOS +26.4.x SCK regression (SCStreamError -3801, "Could not +start streaming") is handled this way — agents can keep +doing element-indexed clicks against the same window even +when the screenshot is unavailable. Switching to +`capture_mode: ax` skips the capture attempt entirely on +subsequent turns. + Requires Accessibility and Screen Recording permissions. **Arguments:** @@ -340,6 +350,17 @@ screenshot selection. The combo is posted directly to the target pid's event queue via `CGEvent.postToPid`; the target does NOT need to be frontmost. +**`window_id`** (optional): when supplied, the driver +calls `FocusWithoutRaise.activateWithoutRaise` before +posting — making the target AppKit-active without +raising its window. This is required for shortcuts that +are dispatched via NSMenu key equivalents (Cmd+S, +Cmd+N, Cmd+W, Cmd+Shift+N, …) because AppKit only +routes menu key equivalents to the active app. Omit +`window_id` for shortcuts handled by the renderer +(e.g. Cmd+C in a Chromium text field). Trade-off: the +sentinel foreground app will lose focus once. + Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modifier keys use the same vocabulary as `press_key` (return, tab, escape, @@ -351,6 +372,7 @@ first, one non-modifier last. - `keys` (array of string, required): Modifier(s) and one non-modifier key, e.g. ["cmd", "c"]. - `pid` (integer, required): Target process ID. +- `window_id` (integer, optional): CGWindowID of the target window. When provided, the driver calls FocusWithoutRaise before posting so NSMenu key equivalents (Cmd+S, Cmd+N, …) reach the backgrounded app. ```json {"keys":["cmd","c"],"pid":844} @@ -370,12 +392,11 @@ Provide either `bundle_id` (preferred — unambiguous, e.g. against /Applications and /System/Applications). If both are given, bundle_id wins. -Launches hidden — the app runs with its window initialized -for automation (so the AX tree is populated and -`click({pid, element_index})` works end-to-end), but no -window is drawn on screen. To bring the window on-screen, -use `NSRunningApplication.unhide()` from your own code, or -have the user open the app (Dock click, Cmd-Tab, Spotlight). +Launches in the background — the app's window appears on +screen but does not steal focus from whatever is currently +frontmost. The AX tree is fully populated and automation +can start immediately via `click`, `hotkey`, +`get_window_state`, etc. Some apps (Calculator and many Electron apps) call `NSApp.activate(ignoringOtherApps:)` in their own @@ -386,16 +407,14 @@ focus-steal preventer arms a short around the launch and re-activates the previously-frontmost app if the target self-activates. If suppression fails the returned summary includes a "self-activation not suppressed" -warning. With `hides = true`, the target is also hidden -back immediately by LaunchServices, so even a missed -layer-3 demotion doesn't leave a visible window. +warning. Optional `urls` are handed to the app through its `application(_:open:)` AppKit delegate. For Finder, passing -a folder URL opens a backgrounded Finder window rooted at -that folder — no activation, no visible window. Works -with any app that implements `application(_:open:)`; apps -that ignore the delegate simply launch without side effects. +a folder URL opens a Finder window rooted at that folder in +the background. Works with any app that implements +`application(_:open:)`; apps that ignore the delegate simply +launch without side effects. Optional `electron_debugging_port` launches an Electron app with `--remote-debugging-port=`, activating its Chrome @@ -600,7 +619,7 @@ cmd/shift/option/ctrl/fn. For true combinations (cmd+c), - `key` (string, required): Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit). - `modifiers` (array of string, optional): Optional modifier names held while the key is pressed (cmd/shift/option/ctrl/fn). - `pid` (integer, required): Target process ID. -- `window_id` (integer, optional): CGWindowID for the window whose get_window_state produced the element_index. Required when element_index is used. +- `window_id` (integer, optional): CGWindowID for the window. Required when element_index is used. When supplied without element_index, triggers FocusWithoutRaise before the key fires — use this for NSMenu key equivalents (Cmd+S, Cmd+N, …) on backgrounded apps. ```json {"key":"return","pid":844} @@ -698,11 +717,20 @@ propagate modifier keys). Capture a screenshot using ScreenCaptureKit. Returns base64-encoded image data for a single window in the requested format (default png). -`window_id` is required. Get window ids from `list_windows`. + + `window_id` is required. Get window ids from `list_windows`. Requires the Screen Recording TCC grant — call `check_permissions` first if unsure. +On macOS 26.4.x, ScreenCaptureKit can refuse specific windows on +physical Macs (SCStreamError -3801, "Could not start streaming"). +The driver retries once and falls back to the legacy +CGWindowList path before failing; if both refuse, the error +response includes a hint to try a different `window_id` or +switch to `capture_mode: ax` for `get_window_state` (the +element-indexed flow doesn't need pixels). + **Arguments:** - `format` (string, optional): Image format. Default: png. diff --git a/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift b/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift index d29f1a16fd..2137cda868 100644 --- a/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift +++ b/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift @@ -33,6 +33,13 @@ public enum CaptureError: Error, Sendable, CustomStringConvertible { case encodeFailed case captureFailed(String) case windowNotFound(UInt32) + /// ScreenCaptureKit could not start streaming for this window. Distinct + /// from `captureFailed` so callers (e.g. `get_window_state`) can surface + /// an actionable hint — switch to `capture_mode: ax`, retry against a + /// different window — without having to grep error strings. Seen + /// regularly on macOS 26.4.x physical Macs against specific windows + /// where even `screencapture -l` fails (rdar / openclaw/Peekaboo#121). + case streamingFailed(String) public var description: String { switch self { @@ -41,6 +48,7 @@ public enum CaptureError: Error, Sendable, CustomStringConvertible { case .encodeFailed: return "failed to encode CGImage" case .captureFailed(let msg): return "capture failed: \(msg)" case .windowNotFound(let id): return "no shareable window with id \(id)" + case .streamingFailed(let msg): return "ScreenCaptureKit streaming failed: \(msg)" } } } @@ -131,12 +139,43 @@ public actor WindowCapture { config.height = max(1, Int(window.frame.height * scale)) config.showsCursor = false + // One-shot SCK call with a single retry on streaming-start failure. + // macOS 26.4.x has a regression where `SCScreenshotManager.captureImage` + // intermittently returns "Could not start streaming because audio/video + // capture failed" (SCStreamError code -3801) on physical Macs, often + // recovering on a second attempt a moment later. We retry once with a + // brief back-off; if it still fails, we surface `.streamingFailed` so + // the tool layer can hint the caller toward `capture_mode: ax` for + // `get_window_state` workflows. let cgImage: CGImage do { - cgImage = try await SCScreenshotManager.captureImage( - contentFilter: filter, - configuration: config - ) + cgImage = try await captureSCKWithRetry(filter: filter, config: config) + } catch let error as CaptureError { + // Already classified — re-throw without wrapping. CGWindowList + // is intentionally NOT tried for permission errors (it'd just + // fail the same way and confuse the user-facing message). + if case .permissionDenied = error { throw error } + // For streaming / generic SCK failures, try the legacy + // CGWindowListCreateImage path. It's deprecated on macOS 15+ + // but still works in many cases where SCK refuses — particularly + // useful as a last-ditch fallback for the 26.4 SCK regression. + if let fallback = legacyCaptureWindow(windowID: windowID) { + let origW = fallback.width + let origH = fallback.height + let resized = resizeIfNeeded(fallback, maxDim: maxImageDimension) + let didResize = resized.width != origW || resized.height != origH + let data = try encode(resized, format: format, quality: quality) + return Screenshot( + imageData: data, + format: format, + width: resized.width, + height: resized.height, + scaleFactor: Double(scale), + originalWidth: didResize ? origW : nil, + originalHeight: didResize ? origH : nil + ) + } + throw error } catch { throw classify(error) } @@ -207,17 +246,121 @@ public actor WindowCapture { return (best ?? NSScreen.main)?.backingScaleFactor ?? 1.0 } + /// Attempt `SCScreenshotManager.captureImage` once; on a streaming-start + /// failure, wait briefly and retry once more. Returns a classified + /// `CaptureError` on persistent failure so the caller can branch on the + /// kind (permission vs. streaming vs. generic) without string-matching. + /// + /// The retry covers the macOS 26.4.x SCK regression where the very first + /// call after the SCK daemon has been idle returns -3801 ("Could not + /// start streaming because audio/video capture failed") but a second + /// call ~250ms later succeeds. A second failure isn't transient and we + /// stop retrying — the caller falls back to CGWindowList or surfaces + /// the error. + private func captureSCKWithRetry( + filter: SCContentFilter, + config: SCStreamConfiguration + ) async throws -> CGImage { + do { + return try await SCScreenshotManager.captureImage( + contentFilter: filter, + configuration: config + ) + } catch { + let classified = classify(error) + // Only retry on streaming-start failures; permission errors and + // not-found errors won't change on a second attempt. + guard case .streamingFailed = classified else { throw classified } + try? await Task.sleep(nanoseconds: 250_000_000) + do { + return try await SCScreenshotManager.captureImage( + contentFilter: filter, + configuration: config + ) + } catch { + throw classify(error) + } + } + } + + /// Legacy `CGWindowListCreateImage` fallback for the SCK 26.4 regression. + /// Deprecated by Apple in macOS 15 but still functional on most windows, + /// and frequently works where SCK refuses. Returns nil on failure — the + /// caller surfaces the original SCK error in that case so the user knows + /// the real cause. + /// + /// Marked with `@available(*, deprecated)` suppression because the API + /// is the entire point: we *want* the legacy path here. + private func legacyCaptureWindow(windowID: UInt32) -> CGImage? { + // CGWindowListCreateImage is deprecated on macOS 15+. The deprecation + // diagnostic is silenced with the @available pragma. Apple has not + // (yet) removed the symbol, and this path is the only practical + // fallback when SCK's streaming-start is broken for a given window. + let opts: CGWindowImageOption = [.boundsIgnoreFraming, .bestResolution] + let listOption: CGWindowListOption = .optionIncludingWindow + // Wrap the deprecated call so we keep the unsafePointer-style + // signature out of the rest of the code. + let image = legacyCGWindowImage( + windowID: windowID, listOption: listOption, imageOption: opts + ) + // Reject 1×1 placeholder images that the legacy API sometimes returns + // for occluded / off-screen windows — they're worse than no image. + guard let image, image.width > 1, image.height > 1 else { return nil } + return image + } + private func classify(_ error: Error) -> CaptureError { let ns = error as NSError let msg = ns.localizedDescription.lowercased() + + // Permission failure — English and Japanese phrasings observed in + // SCK's `NSError.localizedDescription`. The Japanese strings cover + // users on JP system locale where the SCK error comes back + // localized rather than in English. if msg.contains("permission") || msg.contains("not authorized") || msg.contains("declined") || msg.contains("denied") + || ns.localizedDescription.contains("許可") // "permission" + || ns.localizedDescription.contains("拒否") // "denied" { return .permissionDenied } + + // SCStreamError "could not start streaming" — code -3801 in + // `SCStreamErrorDomain`. macOS localizes the message ("Could not + // start streaming because audio/video capture failed" / Japanese: + // "オーディオ/ビデオの取り込みがうまくいかなかったため、ストリーミングを開始できませんでした"), + // so we match on code first and fall through to substring matching + // for the rare case where the domain isn't surfaced. + let isSCStreamDomain = ns.domain == "SCStreamErrorDomain" + || ns.domain == "com.apple.ScreenCaptureKit.SCStreamErrorDomain" + if (isSCStreamDomain && ns.code == -3801) + || msg.contains("could not start streaming") + || msg.contains("streaming") + || ns.localizedDescription.contains("ストリーミング") // "streaming" + { + return .streamingFailed(ns.localizedDescription) + } + return .captureFailed(ns.localizedDescription) } + /// Thin shim around the deprecated `CGWindowListCreateImage` so the + /// deprecation-warning suppression is isolated to one place. Returns nil + /// if the legacy path also refuses to produce an image. + /// + /// Marking the wrapper itself deprecated downgrades the call-site + /// warning to a no-op — we *want* this legacy path because SCK has a + /// well-known regression on macOS 26.4.x where streaming-start fails + /// for specific windows on physical Macs. + @available(*, deprecated, message: "Intentional fallback for SCK streaming-start failures.") + private func legacyCGWindowImage( + windowID: UInt32, + listOption: CGWindowListOption, + imageOption: CGWindowImageOption + ) -> CGImage? { + CGWindowListCreateImage(.null, listOption, windowID, imageOption) + } + /// Capture the topmost layer-0 window owned by `pid`, or `nil` when the /// pid has no such window at all (menubar-only helpers, apps that /// haven't created any window yet). diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift index 6df93a8e8b..c47afc1fab 100644 --- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift +++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift @@ -74,6 +74,19 @@ public enum GetWindowStateTool { Change with `cua-driver config set capture_mode ` or the `set_config` tool. + Screenshot capture failures behave differently per mode: + in `som`, they're non-fatal — the AX tree still ships in + the response and the summary line carries a hint, so + agents can keep doing element-indexed clicks against the + same window even when the screenshot is unavailable. In + `vision`, the screenshot IS the deliverable, so the same + failure returns `isError: true` with an actionable hint + (try another window, retry later, or switch to + `capture_mode: ax`). The macOS 26.4.x SCK regression + (SCStreamError -3801, "Could not start streaming") is + surfaced this way. Switching to `capture_mode: ax` skips + the capture attempt entirely on subsequent turns. + Requires Accessibility and Screen Recording permissions. """, inputSchema: [ @@ -211,6 +224,13 @@ public enum GetWindowStateTool { // race (window closed between validation and capture) // leaves the snapshot without a screenshot; the structured // response's `has_screenshot=false` surfaces the omission. + // A `.streamingFailed` (macOS 26.4 SCK regression) is + // handled differently per mode: in `som` the AX tree is + // still useful so we swallow it and emit a hint; in + // `vision` the screenshot IS the deliverable, so we + // return a hard error with the same actionable guidance + // as the standalone `screenshot` tool. + var captureHint: String? = nil if captureMode != .ax { do { let shot = try await capture.captureWindow( @@ -237,12 +257,52 @@ public enum GetWindowStateTool { } catch CaptureError.windowNotFound { // Window raced — swallow and emit a screenshot-less // response. + } catch CaptureError.streamingFailed(let msg) { + if captureMode == .vision { + // In `vision` mode the screenshot IS the + // deliverable — there's no AX tree to fall + // back to. Surface the same actionable hint + // as the standalone `screenshot` tool. + return errorResult( + """ + ScreenCaptureKit refused this window: \(msg) + + This is a known macOS 26.4.x SCK regression that hits \ + specific windows on physical Macs. The legacy \ + CGWindowList fallback also returned no image. + + Workarounds: + • Try a different `window_id` on the same app — \ + often only one window is affected. + • For element-indexed clicks, switch to AX-only: \ + `cua-driver config set capture_mode ax` and re-call \ + `get_window_state` (no screenshot, AX tree only). + • Re-snapshot a moment later — the failure is \ + sometimes transient. + """ + ) + } + // `som` mode: AX snapshot is still useful for + // element-indexed clicks, so we don't fail the + // call. The hint nudges the caller toward + // `capture_mode: ax` to skip the capture attempt + // entirely on subsequent turns. + captureHint = """ + ⚠️ Screenshot skipped: ScreenCaptureKit refused this \ + window (\(msg)). Known macOS 26.4 SCK regression. The \ + AX tree below is still valid; element-indexed clicks \ + work as usual. To suppress future capture attempts on \ + this app: `cua-driver config set capture_mode ax`. + """ } } var textContent = buildSummary( snapshot: snapshot, pid: pid, mode: captureMode ) + if let captureHint { + textContent += "\n" + captureHint + } if captureMode != .vision && !snapshot.treeMarkdown.isEmpty { textContent += "\n\n" + snapshot.treeMarkdown } diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift index bd61d11911..0c43f48631 100644 --- a/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift +++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift @@ -18,6 +18,14 @@ public enum ScreenshotTool { Requires the Screen Recording TCC grant — call `check_permissions` first if unsure. + + On macOS 26.4.x, ScreenCaptureKit can refuse specific windows on + physical Macs (SCStreamError -3801, "Could not start streaming"). + The driver retries once and falls back to the legacy + CGWindowList path before failing; if both refuse, the error + response includes a hint to try a different `window_id` or + switch to `capture_mode: ax` for `get_window_state` (the + element-indexed flow doesn't need pixels). """, inputSchema: [ "type": "object", @@ -112,6 +120,38 @@ public enum ScreenshotTool { ], isError: true ) + } catch CaptureError.streamingFailed(let msg) { + // SCK streaming-start regression on macOS 26.4.x — the + // legacy CGWindowList fallback also refused this specific + // window. There's nothing we can do at the pixel layer; + // surface an actionable hint pointing at `get_window_state` + // (which can fall back to AX-only via `capture_mode: ax`) + // or trying a different window. + return CallTool.Result( + content: [ + .text( + text: """ + ScreenCaptureKit refused this window: \(msg) + + This is a known macOS 26.4.x SCK regression that hits \ + specific windows on physical Macs. The legacy \ + CGWindowList fallback also returned no image. + + Workarounds: + • Try a different `window_id` on the same app — \ + often only one window is affected. + • For element-indexed clicks, switch to AX-only: \ + `cua-driver config set capture_mode ax` and use \ + `get_window_state` (no screenshot, AX tree only). + • Re-snapshot a moment later — the failure is \ + sometimes transient. + """, + annotations: nil, + _meta: nil + ) + ], + isError: true + ) } catch { return CallTool.Result( content: [