cua-driver: handle macOS 26.4 SCK streaming-start failure - #1478
Conversation
…1467) ScreenCaptureKit on macOS 26.4.x regresses for specific windows on physical Macs — `SCScreenshotManager.captureImage` returns "Could not start streaming because audio/video capture failed" (SCStreamError -3801, sometimes localized in Japanese / other locales) and even `screencapture -l<id>` refuses the same window. Without a hint, `get_window_state` and `screenshot` callers had no actionable path forward. Changes: - `WindowCapture.captureWindow` now retries the SCK call once after a brief delay (covers transient -3801) and falls back to `CGWindowListCreateImage` (deprecated but still functional, works on many windows where SCK refuses). - New `CaptureError.streamingFailed` distinguishes the regression from generic capture failures so the tool layer can render an actionable hint instead of an opaque message. - `get_window_state` no longer hard-fails on streaming errors: AX tree still ships in the response with a warning line pointing at `capture_mode: ax`. Element-indexed clicks keep working. - `screenshot` returns a clear error response naming the regression and suggesting workarounds (different `window_id`, `capture_mode: ax`, retry). - Error classification now detects Japanese permission strings ("許可", "拒否") in addition to English. Docs: - FAQ entry covering the macOS 26.4 SCK regression and workarounds. - mcp-tools.mdx regenerated (tool descriptions updated). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR adds resilience for a macOS 26.4.x ScreenCaptureKit streaming regression where screenshot capture is refused for certain windows. It implements retry logic with legacy fallback in ChangesScreenCaptureKit Streaming Failure Resilience
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift`:
- Around line 224-227: The code currently swallows `.streamingFailed` into a
non-fatal path by setting `captureHint` to a hint string; change the error
handling in GetWindowStateTool (the branches around the `captureHint` usage) so
that when the error is `.streamingFailed` and the tool is running in `vision`
mode you do not treat it as non‑fatal—propagate or return the failure (i.e.,
produce an error result) instead of returning a "successful" result with an AX
hint; retain the non‑fatal behavior only for `som` mode. Apply the same change
to the other similar blocks mentioned (the handling around lines 254–268 and
275–277) so `.streamingFailed` is fatal for `vision` but preserved as hint-only
for `som`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: fc5a8253-61ed-4bc2-9a75-3926a5a06c20
📒 Files selected for processing (6)
docs/content/docs/cua-driver/guide/getting-started/faq.mdxdocs/content/docs/cua-driver/reference/cli-reference.mdxdocs/content/docs/cua-driver/reference/mcp-tools.mdxlibs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swiftlibs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swiftlibs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift
| // A `.streamingFailed` (macOS 26.4 SCK regression) is also | ||
| // swallowed: the AX tree is still useful, and the summary | ||
| // line gets an actionable hint instead of a hard error. | ||
| var captureHint: String? = nil |
There was a problem hiding this comment.
Don’t swallow .streamingFailed in vision mode.
This path currently treats streaming failure as non-fatal for both som and vision, but vision has no AX tree to preserve. It can return a “successful” result with no usable payload and a misleading AX hint.
💡 Suggested fix
- } catch CaptureError.streamingFailed(let msg) {
+ } catch CaptureError.streamingFailed(let msg) {
+ if captureMode == .vision {
+ throw CaptureError.streamingFailed(msg)
+ }
// Known macOS 26.4 SCK regression: streaming-start
// fails for specific windows on physical Macs and
// even CGWindowList can't get pixels back. We don't
// fail the call — the AX snapshot is still usefulAlso applies to: 254-268, 275-277
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift`
around lines 224 - 227, The code currently swallows `.streamingFailed` into a
non-fatal path by setting `captureHint` to a hint string; change the error
handling in GetWindowStateTool (the branches around the `captureHint` usage) so
that when the error is `.streamingFailed` and the tool is running in `vision`
mode you do not treat it as non‑fatal—propagate or return the failure (i.e.,
produce an error result) instead of returning a "successful" result with an AX
hint; retain the non‑fatal behavior only for `som` mode. Apply the same change
to the other similar blocks mentioned (the handling around lines 254–268 and
275–277) so `.streamingFailed` is fatal for `vision` but preserved as hint-only
for `som`.
…bbit) In `vision` capture mode the screenshot IS the deliverable, so swallowing `CaptureError.streamingFailed` and returning an AX-only response was misleading — the AX tree is also skipped in that mode, so the caller got neither a screenshot nor a tree. Surface the same actionable error the standalone `screenshot` tool already emits (try alternate window, switch to `capture_mode: ax`, or retry). `som` mode is unchanged: the AX tree is still useful for element-indexed clicks, so the streaming-start failure keeps emitting a non-fatal hint on the summary line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@coderabbitai Addressed in a8a92b5. In |
|
Tip For best results, initiate chat on the files or code changes.
If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! |
3 participants
Fixes #1467.
Problem
On macOS 26.4.1 (Build 25E253) on physical Apple Silicon Macs, ScreenCaptureKit intermittently refuses to capture specific windows:
SCScreenshotManager.captureImagereturnsSCStreamErrorcode -3801 ("Could not start streaming because audio/video capture failed", localized in Japanese as "オーディオ/ビデオの取り込みがうまくいかなかったため、ストリーミングを開始できませんでした"). Evenscreencapture -l<id>fails on the same window — so it's not just an SDK glitch; the SCK daemon itself is refusing that specific window.cua-driver screenshotandget_window_state(withcapture_mode: som/vision) surface this as an opaque error with no actionable next step, even thoughcapture_mode: axwould work fine for element-indexed workflows. Related: openclaw/Peekaboo#121.Summary
WindowCapture.captureWindowretries once after a 250 ms back-off when SCK reports a streaming-start failure (covers genuinely transient -3801s).CGWindowListCreateImagewhen SCK still refuses. The legacy API is deprecated on macOS 15+ but remains functional and frequently succeeds where the new SCK pipeline refuses on 26.4.x. The deprecation warning is silenced via an@available(*, deprecated)shim so the call-site stays clean.CaptureError.streamingFaileddistinguishes the SCK regression from generic capture failures, so the tool layer can render an actionable hint without string-matching.get_window_stateno longer hard-fails on streaming errors: it ships the AX tree with a warning line nudging the caller towardcapture_mode: ax. Element-indexed clicks keep working even when no screenshot is available.screenshotreturns a clear error response naming the regression and listing workarounds (differentwindow_id, switch tocapture_mode: ax, retry).mcp-tools.mdxregenerated from the updated tool descriptions.Files changed
libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift— retry, CGWindowList fallback, error classification.libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift— actionable error response.libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift— non-fatal streaming failure with hint.docs/content/docs/cua-driver/guide/getting-started/faq.mdx— new FAQ entry.docs/content/docs/cua-driver/reference/mcp-tools.mdx/cli-reference.mdx— regenerated.Test plan
swift buildclean (no new warnings)swift test— all 26 tests passbun run scripts/docs-generators/cua-driver.ts --check— docs in synccua-driver screenshot '{"window_id": <known-good-id>}'still works on healthy windows (i.e. the fallback path doesn't regress the happy case).cua-driver get_window_state '{"pid":<pid>,"window_id":<id>}'returns AX tree + hint when capture fails.Notes
CaptureError.streamingFaileddoes not propagate throughRecordingSession— recording still swallows any throw and emitsshot = nil, so the change is additive there..boundsIgnoreFraming+.bestResolutionand rejects 1×1 placeholder images (the legacy API returns those for occluded windows; they're worse than no image).🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
get_window_statenow gracefully handles screenshot failures, returning the AX tree with guidance.Documentation