diff --git a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx index 6b657edcde..daabda628b 100644 --- a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx +++ b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx @@ -90,7 +90,43 @@ Tools appear prefixed as `mcp_cua-driver_*`. cua-driver mcp-config --client opencode ``` -Paste the output into your `opencode.json`. +Paste the output into `~/.config/opencode/config.json` (global) or `opencode.json` at the project root. + + +**Always configure cua-driver as an MCP server — never rely on the CLI fallback.** If MCP is not wired up, OpenCode calls `cua-driver` as a shell subprocess. The `get_window_state` response no longer includes base64 by default, but the screenshot image block is silently dropped — the model receives only the AX tree with no visual context. Use `--screenshot-out-file` or the `screenshot_out_file` param to preserve the image when using the CLI path. + + +### Local vision models (Ollama) + +If you are using a vision-capable model via Ollama, you must also declare its input modalities in `config.json` — otherwise OpenCode strips images before they reach the model: + +```json +{ + "mcp": { + "cua-driver": { + "type": "local", + "command": ["/Users/you/.local/bin/cua-driver", "mcp"], + "enabled": true + } + }, + "provider": { + "ollama": { + "npm": "@ai-sdk/openai-compatible", + "options": { "baseURL": "http://localhost:11434/v1" }, + "models": { + "gemma4:26b": { + "modalities": { + "input": ["text", "image"], + "output": ["text"] + } + } + } + } + } +} +``` + +The `modalities` field is required because OpenCode's `@ai-sdk/openai-compatible` provider defaults to text-only when no capabilities are declared. Without it, screenshots are replaced with an error string and never reach the model. ## Hermes (NousResearch) diff --git a/docs/content/docs/cua-driver/reference/cli-reference.mdx b/docs/content/docs/cua-driver/reference/cli-reference.mdx index 16034b4b8a..c5af31770c 100644 --- a/docs/content/docs/cua-driver/reference/cli-reference.mdx +++ b/docs/content/docs/cua-driver/reference/cli-reference.mdx @@ -182,7 +182,7 @@ Pipe to `pbcopy` to put any output on the clipboard, or pass the printed `claude ### cua-driver recording start -Enable the trajectory recorder. Every subsequent action-tool call (`click`, `right_click`, `scroll`, `type_text`, `type_text_chars`, `press_key`, `hotkey`, `set_value`) writes a numbered turn folder under ``. +Enable the trajectory recorder. Every subsequent action-tool call (`click`, `right_click`, `scroll`, `type_text`, `press_key`, `hotkey`, `set_value`) writes a numbered turn folder under ``. ```bash cua-driver recording start ~/cua-trajectories/demo1 @@ -346,7 +346,6 @@ The following MCP tools are callable via `cua-driver `. For input schemas - `check_permissions` — Accessibility + Screen Recording TCC status. - `get_screen_size` — main display size in points + scale factor. - `get_cursor_position` — current mouse cursor position. -- `get_accessibility_tree` — lightweight desktop snapshot (apps + visible windows). **App lifecycle** @@ -368,8 +367,7 @@ The following MCP tools are callable via `cua-driver `. For input schemas **Keyboard** -- `type_text` — insert text via `AXSelectedText`. Pid-scoped. -- `type_text_chars` — character-by-character via `CGEvent.postToPid`. Reaches Chromium/Electron inputs. +- `type_text` — insert text via `AXSelectedText` with automatic CGEvent fallback for Chromium/Electron inputs. Pid-scoped. - `press_key` — single key press. Pid-scoped. - `hotkey` — modifier combo (e.g. `["cmd","c"]`). Pid-scoped. diff --git a/docs/content/docs/cua-driver/reference/mcp-tools.mdx b/docs/content/docs/cua-driver/reference/mcp-tools.mdx index 0b19c1f3b7..2b110dfceb 100644 --- a/docs/content/docs/cua-driver/reference/mcp-tools.mdx +++ b/docs/content/docs/cua-driver/reference/mcp-tools.mdx @@ -58,16 +58,6 @@ Return the current mouse cursor position in screen points (top-left origin). {} ``` -### get_accessibility_tree - -Lightweight desktop snapshot: running regular apps and on-screen visible windows with bounds, z-order, and owner pid. For a single window's internal UI, use `get_window_state`. - -**Arguments:** none. - -```json -{} -``` - ### screenshot Raw ScreenCaptureKit capture. Full main display, or a single window when `window_id` is set. Returns an image content block plus a text summary listing on-screen windows. @@ -271,7 +261,7 @@ Modifier combo as a single array, e.g. `["cmd", "c"]`. Requires at least two ent ### type_text -Insert text at the target's current cursor via `AXSelectedText`. Fast (single AX write) but skipped by apps with custom text layers; for Chromium / Electron inputs use `type_text_chars`. +Insert text at the target's current cursor. Attempts `AXSelectedText` write first (fast, single AX call); automatically falls back to character-by-character `CGEvent.postToPid` synthesis when the target input doesn't implement `AXSelectedText` — this makes it work for Chromium and Electron inputs without any manual switching. **Arguments:** @@ -279,22 +269,10 @@ Insert text at the target's current cursor via `AXSelectedText`. Fast (single AX - `text` (string, required): Text to insert at the target's cursor. - `element_index` (integer, optional): When present, the element is focused before the write. Requires `window_id`. - `window_id` (integer, optional): Required when `element_index` is used. +- `delay_ms` (integer, optional): Milliseconds between characters when the CGEvent fallback path is used, 0-200. Default 30. ```json {"pid": 844, "window_id": 10725, "element_index": 12, "text": "hello"} -``` - -### type_text_chars - -Character-by-character input via `CGEvent.postToPid`. Slower than `type_text` but reaches Chromium and Electron inputs that ignore AX writes. - -**Arguments:** - -- `pid` (integer, required): Target process ID. -- `text` (string, required): Text to type into the target's focused element. -- `delay_ms` (integer, optional): Milliseconds between characters, 0-200. Default 30. - -```json {"pid": 844, "text": "hello world", "delay_ms": 40} ``` @@ -450,7 +428,7 @@ Supported keys and ranges: see the [CLI reference](/cua-driver/reference/cli-ref ## Recording and replay -The trajectory recorder captures every action-tool call (`click`, `right_click`, `scroll`, `type_text`, `type_text_chars`, `press_key`, `hotkey`, `set_value`) into numbered turn folders. Recordings can be replayed turn-by-turn. +The trajectory recorder captures every action-tool call (`click`, `right_click`, `scroll`, `type_text`, `press_key`, `hotkey`, `set_value`) into numbered turn folders. Recordings can be replayed turn-by-turn. ### get_recording_state diff --git a/libs/cua-driver/Skills/cua-driver/RECORDING.md b/libs/cua-driver/Skills/cua-driver/RECORDING.md index 79fc526398..e60c3d2f8c 100644 --- a/libs/cua-driver/Skills/cua-driver/RECORDING.md +++ b/libs/cua-driver/Skills/cua-driver/RECORDING.md @@ -6,7 +6,7 @@ user explicitly asks to record — the skill does not auto-enable this. `set_recording` turns on a session-scoped trajectory recorder. While enabled, every action-tool call (`click`, `right_click`, `scroll`, -`type_text`, `type_text_chars`, `press_key`, `hotkey`, `set_value`) +`type_text`, `press_key`, `hotkey`, `set_value`) writes a numbered turn folder under a caller-chosen output directory. Read-only tools (`get_window_state`, `list_windows`, `screenshot`, `list_apps`, permission probes, agent-cursor getters / @@ -101,8 +101,7 @@ keyed on `(pid, window_id)`, so a recorded resolve today — the pid is usually different, the window_id always is. The call returns `Invalid element_index` or `No cached AX state`. Pixel clicks (`click({pid, x, y})`) and keyboard tools -(`press_key`, `type_text_chars`, `hotkey`, `type_text` without -element_index) replay cleanly; element-indexed actions require a +(`press_key`, `hotkey`, `type_text` without element_index) replay cleanly; element-indexed actions require a live snapshot that replay doesn't currently re-emit (read-only tools like `get_window_state` aren't recorded). For a reliable replay, either compose the trajectory from pixel + keyboard primitives, or capture diff --git a/libs/cua-driver/Skills/cua-driver/SKILL.md b/libs/cua-driver/Skills/cua-driver/SKILL.md index 3858f74922..e442825136 100644 --- a/libs/cua-driver/Skills/cua-driver/SKILL.md +++ b/libs/cua-driver/Skills/cua-driver/SKILL.md @@ -420,8 +420,7 @@ you're interacting with a long-lived process). In the default get_window_state → reason over PNG → pixel click`. When you need `element_index` dispatch (AX-addressable elements, backgrounded clicks), flip to `som` first: `cua-driver set_config '{"key": -"capture_mode", "value": "som"}'`, or call `get_accessibility_tree` -directly. The rest of this section walks through `som` mode, which +"capture_mode", "value": "som"}'`. The rest of this section walks through `som` mode, which is what you want once you've decided element-indexed addressing is required. @@ -432,33 +431,26 @@ In `som` mode the response carries: ~1600 elements, ~190 KB); when it exceeds token limits the MCP harness saves it to a file and returns the path. Use `Bash` + `jq -r '.tree_markdown'` + `grep` to pull the section you need. -- `screenshot_png_b64` + `screenshot_width` / `_height` / - `_scale_factor` — the window screenshot (actually JPEG-85 despite - the `_png_` field name, hard-coded in - `WindowCapture.captureFrontmostWindow`). Present in `som` mode - (spliced into the structured JSON alongside the tree). In `vision` - mode the image arrives as a native MCP image content block with no - structured wrapper. Omitted when the target has no on-screen - window. -- `has_screenshot: bool` — **gate on this before piping the PNG**. - Otherwise `jq -r '.screenshot_png_b64'` emits the literal - `"null"`, base64-decodes into 3 bytes of garbage, and downstream - vision APIs reject it with an opaque "Could not process image" - error. - -``` -# canonical, works in every capture mode — writes the image bytes -# wherever you point, stdout stays readable (tree in som, summary -# in vision). stderr warns (exit 0) if the response had no image. -cua-driver get_window_state '{"pid":N,"window_id":W}' --image-out /tmp/shot.png - -# som-only legacy path: pull the spliced base64 out of structuredContent. -# Prefer --image-out above — it's one flag vs a probe + pipe. -if [ "$(cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.has_screenshot')" = "true" ]; then - cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.screenshot_png_b64' | base64 -d > shot.png -fi +- `screenshot_file_path` — absolute path to the saved screenshot when + `screenshot_out_file` was passed. Absent otherwise. +- `screenshot_width` / `_height` / `_scale_factor` — dimensions of the + captured image. Present whenever a screenshot was taken. +**Getting the screenshot as a file (CLI and context-constrained agents):** + +```bash +# write to file — stdout stays readable (AX tree / summary only, no base64) +cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.jpg"}' + +# CLI --screenshot-out-file flag is equivalent and works for all capture modes +cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.jpg ``` +Pass `screenshot_out_file` when using `get_window_state` via CLI or from an +agent whose context window can't absorb ~31 KB of inline base64 (e.g. +OpenCode with a local Ollama model). The MCP image content block is omitted +from the response when this param is set — the model receives only the AX +tree and `screenshot_file_path`, then reads the image from disk. + **Reason over both the tree AND the screenshot — they're complementary, not redundant.** In `som` mode every turn's `get_window_state` gives you both halves and you should pull @@ -512,7 +504,7 @@ anchor the conversion against a specific window): | Focus + send key | `press_key({pid, key, window_id, element_index, modifiers})` | element_index sets AXFocused, then posts key | | Send key to pid | `press_key({pid, key, modifiers})` | no focus change; key goes to pid's current focus | | Modifier combo | `hotkey({pid, keys})` | e.g. `["cmd","c"]`; posted per-pid, not HID tap | -| Unicode keystrokes | `type_text_chars({pid, text, delay_ms})` | CGEvent-to-pid; reaches Chromium/Electron inputs | +| Unicode keystrokes | `type_text({pid, text, delay_ms})` | AX write with automatic CGEvent fallback; reaches Chromium/Electron inputs | **All keyboard/text primitives require `pid`.** There is no frontmost-routed variant — every key goes to the named target via @@ -556,7 +548,7 @@ below against the full-resolution file in that case. 1. `get_window_state({pid, window_id})` returns an image capped at 1568 long-side (default) plus its dimensions (`screenshot_width` / `screenshot_height`). Write the bytes to - disk with `--image-out ` in any capture mode — works + disk with `--screenshot-out-file ` in any capture mode — works identically in `vision` (where it's the only way) and `som` (where it sidesteps the jq + base64 dance on the spliced `screenshot_png_b64` field). diff --git a/libs/cua-driver/Skills/cua-driver/TESTS.md b/libs/cua-driver/Skills/cua-driver/TESTS.md index dde5fca226..fdc835b329 100644 --- a/libs/cua-driver/Skills/cua-driver/TESTS.md +++ b/libs/cua-driver/Skills/cua-driver/TESTS.md @@ -116,11 +116,11 @@ These are where CuaDriver's AX-activation trio matters: ### 8. Slack — sparse Electron, hotkey fallback **Prompt:** `In Slack, jump to the #pr-reviews channel using the quick switcher (⌘K).` -**Exercises:** "retry snapshot once, then `hotkey` + `type_text_chars`" fallback path. +**Exercises:** "retry snapshot once, then `hotkey` + `type_text`" fallback path. **Success:** - Re-snapshot shows Slack's channel title area (AX role `AXStaticText` or similar) contains `pr-reviews`. -- Claude uses `hotkey` and `type_text_chars` — NOT `simulate_click` / pixel coords. +- Claude uses `hotkey` and `type_text` — NOT `simulate_click` / pixel coords. **Fail signals:** Claude drops to `simulate_click` (guardrail violation — pixel fallback is not allowed on sparse AX trees), wrong channel joined, typed text echoes into the message composer instead of the switcher. @@ -168,7 +168,7 @@ These are where CuaDriver's AX-activation trio matters: ### 12. Chrome proper — omnibox **Prompt:** `In Google Chrome, open a new tab and navigate to https://trycua.com.` -**Exercises:** `hotkey(["cmd","t"])` + `type_text_chars` + Return. Chrome's omnibox typically isn't AX-exposed even after activation. +**Exercises:** `hotkey(["cmd","t"])` + `type_text` + Return. Chrome's omnibox typically isn't AX-exposed even after activation; `type_text` automatically falls back to CGEvent synthesis for Chromium/Electron inputs. **Success:** - A new Chrome tab whose AX title contains `Cua` or `trycua` is present. diff --git a/libs/cua-driver/Skills/cua-driver/WEB_APPS.md b/libs/cua-driver/Skills/cua-driver/WEB_APPS.md index 6fbaef9e21..312525d8af 100644 --- a/libs/cua-driver/Skills/cua-driver/WEB_APPS.md +++ b/libs/cua-driver/Skills/cua-driver/WEB_APPS.md @@ -46,10 +46,10 @@ pixels: `hotkey({pid, keys: ["cmd", "enter"]})`, `hotkey({pid, keys: ["cmd", "k"]})`, etc. Posted via `CGEvent.postToPid`, reaches the target regardless of AX state, no activation required. -3. For typing into web inputs where `type_text` silently drops - (input doesn't implement `AXSelectedText`), use `type_text_chars` - — pure CGEvent keystrokes reach any focused keyboard receiver, - including Unicode / emoji. +3. For typing into web inputs, use `type_text` — it automatically + falls back to CGEvent synthesis when the input doesn't implement + `AXSelectedText`, reaching any focused keyboard receiver including + Unicode / emoji. 4. If none of the above reaches the target, tell the user this interaction isn't reachable from the driver today and ask for guidance. @@ -86,7 +86,7 @@ documented only as historical context: ``` # DON'T DO THIS — ⌘L steals focus. Use launch_app above. hotkey({pid, keys: ["cmd", "l"]}) -type_text_chars({pid, text: "https://cua.ai", delay_ms: 30}) +type_text({pid, text: "https://cua.ai", delay_ms: 30}) get_window_state({pid, window_id}) click({pid, window_id, element_index: }) ``` @@ -157,7 +157,7 @@ When the target window is **minimized** (genie'd into the Dock): `AXFocused=true` on a minimized window's descendants doesn't propagate to real keyboard focus). Symptom: macOS system-alert beep, or silent no-op. Example: `hotkey cmd+L` + - `type_text_chars URL` + `press_key return` on minimized Chrome — + `type_text URL` + `press_key return` on minimized Chrome — the URL lands in the omnibox AX value but Return doesn't commit the navigation. - **Primary workaround — use `set_value` to commit directly**: For @@ -465,6 +465,7 @@ type_text({pid, window_id, element_index: , text: "…"}) ``` If it silently drops (some web inputs don't implement -`AXSelectedText`), click the field first, then use -`type_text_chars({pid, text})` — pure CGEvent keystrokes delivered -to the pid, reaching any focused keyboard receiver. +`AXSelectedText`), `type_text` automatically falls back to CGEvent +synthesis — pure CGEvent keystrokes delivered to the pid, reaching +any focused keyboard receiver. You can also click the field first +to ensure focus before typing. diff --git a/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift b/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift index 1f09722339..8320df85c3 100644 --- a/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift +++ b/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift @@ -83,13 +83,13 @@ struct CallCommand: AsyncParsableCommand { file path. `vision` and `screenshot` capture modes return a PNG as a native MCP image block with no accompanying structuredContent, so the CLI's text formatter would otherwise - drop the bytes silently. With `--image-out /tmp/shot.png` the - raw PNG lands on disk and downstream tooling (PIL, sips, + drop the bytes silently. With `--screenshot-out-file /tmp/shot.jpg` + the raw image lands on disk and downstream tooling (PIL, sips, ffprobe) can read it directly. Silently warns (no error exit) when the response carries no image. """ ) - var imageOut: String? + var screenshotOutFile: String? @OptionGroup var output: JSONOutputOptions @OptionGroup var daemon: DaemonForwardingOptions @@ -113,7 +113,7 @@ struct CallCommand: AsyncParsableCommand { arguments: arguments, socketPath: daemon.resolvedSocketPath, raw: raw, - imageOut: imageOut, + screenshotOutFile: screenshotOutFile, output: output ) return @@ -184,7 +184,7 @@ struct CallCommand: AsyncParsableCommand { throw ExitCode(Exit.software) } - if let path = imageOut { + if let path = screenshotOutFile { writeFirstImageContent(result.content, to: path) } @@ -251,10 +251,7 @@ struct CallCommand: AsyncParsableCommand { let encoder = JSONEncoder() encoder.outputFormatting = output.encoderOutputFormatting let encoded = try encoder.encode(structured) - let merged = mergeImageContentIntoJSON( - encoded, content: result.content, output: output - ) - printDataLine(merged) + printDataLine(encoded) return } @@ -352,7 +349,7 @@ func forwardCallToDaemon( arguments: [String: Value]?, socketPath: String, raw: Bool, - imageOut: String?, + screenshotOutFile: String?, output: JSONOutputOptions ) async throws { let request = DaemonRequest(method: "call", name: toolName, args: arguments) @@ -381,7 +378,7 @@ func forwardCallToDaemon( throw ExitCode(Exit.software) } - if let path = imageOut { + if let path = screenshotOutFile { writeFirstImageContent(result.content, to: path) } @@ -426,10 +423,7 @@ private func emitUnwrappedResultForDaemon( let encoder = JSONEncoder() encoder.outputFormatting = output.encoderOutputFormatting let encoded = try encoder.encode(structured) - let merged = mergeImageContentIntoJSON( - encoded, content: result.content, output: output - ) - FileHandle.standardOutput.write(merged) + FileHandle.standardOutput.write(encoded) FileHandle.standardOutput.write(Data("\n".utf8)) return } @@ -487,70 +481,23 @@ enum DaemonCLIError: Error { case protocolMismatch } -/// MCP delivers screenshot bytes as a native `.image()` content block -/// separate from `structuredContent`, so shell consumers of -/// `cua-driver ` only see the metadata half ("has_screenshot", -/// dimensions) and the base64 pixels silently vanish. This reunites them -/// at CLI emit time: if any image block is present in `content`, splice -/// its base64 into the outgoing JSON as `screenshot_png_b64` (plus -/// `screenshot_mime_type`) alongside the existing fields. The MCP wire -/// contract is unchanged — only the CLI's pretty-printed output gains -/// the pixels that downstream `jq -r '.screenshot_png_b64' | base64 -d` -/// pipelines expect. -/// -/// Returns the input `encoded` unchanged when there's no image block to -/// splice OR when the structured content didn't decode as a JSON object -/// (e.g. a bare array / scalar from some other tool). Never throws — -/// merge failures fall through silently so a malformed structured -/// response still emits its original bytes. -private func mergeImageContentIntoJSON( - _ encoded: Data, - content: [Tool.Content], - output: JSONOutputOptions -) -> Data { - var imageBase64: String? = nil - var imageMime: String? = nil - for item in content { - if case let .image(data, mime, _, _) = item { - imageBase64 = data - imageMime = mime - break - } - } - guard let imageBase64, let imageMime else { return encoded } - - guard - var object = (try? JSONSerialization.jsonObject(with: encoded)) - as? [String: Any] - else { return encoded } - object["screenshot_png_b64"] = imageBase64 - object["screenshot_mime_type"] = imageMime - - var writingOptions: JSONSerialization.WritingOptions = [ - .sortedKeys, .withoutEscapingSlashes, - ] - if !output.compact { writingOptions.insert(.prettyPrinted) } - return (try? JSONSerialization.data( - withJSONObject: object, options: writingOptions - )) ?? encoded -} /// Write the first `.image(...)` content block from a tool result to -/// `path`, decoded from base64. Used by the `--image-out` flag so +/// `path`, decoded from base64. Used by the `--screenshot-out-file` flag so /// vision-mode screenshots land on disk without callers needing to /// speak the raw daemon protocol or the MCP bridge. /// /// Never throws — an empty/malformed image, a write failure, or a /// response that carries no image at all each emit a stderr warning /// and return. The exit code is NOT affected: callers have already -/// gotten the textual result on stdout, and `--image-out` is +/// gotten the textual result on stdout, and `--screenshot-out-file` is /// advisory ("if the tool returned a PNG, put it here"), not a hard /// contract. func writeFirstImageContent(_ content: [Tool.Content], to path: String) { for item in content { if case let .image(base64, mime, _, _) = item { guard let bytes = Data(base64Encoded: base64) else { - printToStderr("--image-out: base64 decode failed for \(mime) block") + printToStderr("--screenshot-out-file: base64 decode failed for \(mime) block") return } let url = URL(fileURLWithPath: (path as NSString).expandingTildeInPath) @@ -558,7 +505,7 @@ func writeFirstImageContent(_ content: [Tool.Content], to path: String) { try bytes.write(to: url) } catch { printToStderr( - "--image-out: failed to write \(url.path): \(error.localizedDescription)" + "--screenshot-out-file: failed to write \(url.path): \(error.localizedDescription)" ) } return @@ -568,7 +515,7 @@ func writeFirstImageContent(_ content: [Tool.Content], to path: String) { // tools legitimately return nothing (hidden/minimized windows, // `ax` capture mode, list operations). Warn so the user notices // the file they expected didn't get written. - printToStderr("--image-out: no image content in tool response; file not written") + printToStderr("--screenshot-out-file: no image content in tool response; file not written") } private func printUnknownTool(_ name: String, registry: ToolRegistry) { diff --git a/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift b/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift index a463005c86..65b4cbfd71 100644 --- a/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift +++ b/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift @@ -54,6 +54,9 @@ public struct AppStateSnapshot: Sendable, Codable { public let screenshotOriginalWidth: Int? /// Original height before maxImageDimension resize. nil = no resize. public let screenshotOriginalHeight: Int? + /// File-system path to the saved screenshot (JPEG) when the caller + /// passed `screenshot_out_file`. Absent when no path was written. + public let screenshotFilePath: String? public init( pid: Int32, @@ -67,7 +70,8 @@ public struct AppStateSnapshot: Sendable, Codable { screenshotHeight: Int? = nil, screenshotScaleFactor: Double? = nil, screenshotOriginalWidth: Int? = nil, - screenshotOriginalHeight: Int? = nil + screenshotOriginalHeight: Int? = nil, + screenshotFilePath: String? = nil ) { self.pid = pid self.bundleId = bundleId @@ -81,6 +85,7 @@ public struct AppStateSnapshot: Sendable, Codable { self.screenshotScaleFactor = screenshotScaleFactor self.screenshotOriginalWidth = screenshotOriginalWidth self.screenshotOriginalHeight = screenshotOriginalHeight + self.screenshotFilePath = screenshotFilePath } private enum CodingKeys: String, CodingKey { @@ -95,6 +100,7 @@ public struct AppStateSnapshot: Sendable, Codable { case screenshotScaleFactor = "screenshot_scale_factor" case screenshotOriginalWidth = "screenshot_original_width" case screenshotOriginalHeight = "screenshot_original_height" + case screenshotFilePath = "screenshot_file_path" } /// Encodes the struct with ``screenshot_*`` keys omitted when nil. @@ -116,6 +122,7 @@ public struct AppStateSnapshot: Sendable, Codable { try container.encodeIfPresent(screenshotScaleFactor, forKey: .screenshotScaleFactor) try container.encodeIfPresent(screenshotOriginalWidth, forKey: .screenshotOriginalWidth) try container.encodeIfPresent(screenshotOriginalHeight, forKey: .screenshotOriginalHeight) + try container.encodeIfPresent(screenshotFilePath, forKey: .screenshotFilePath) } public init(from decoder: Decoder) throws { @@ -132,6 +139,7 @@ public struct AppStateSnapshot: Sendable, Codable { screenshotScaleFactor = try container.decodeIfPresent(Double.self, forKey: .screenshotScaleFactor) screenshotOriginalWidth = try container.decodeIfPresent(Int.self, forKey: .screenshotOriginalWidth) screenshotOriginalHeight = try container.decodeIfPresent(Int.self, forKey: .screenshotOriginalHeight) + screenshotFilePath = try container.decodeIfPresent(String.self, forKey: .screenshotFilePath) } } diff --git a/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift b/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift index 631f3b9ab7..d29f1a16fd 100644 --- a/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift +++ b/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift @@ -100,7 +100,7 @@ public actor WindowCapture { } /// Capture a single window by its CGWindowID / kCGWindowNumber. - /// Pairs with `get_accessibility_tree` which returns each window's `id`. + /// Get window ids from `list_windows`. public func captureWindow( windowID: UInt32, format: ImageFormat = .png, diff --git a/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift b/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift index c5ef3cd6ca..6a152bf3e1 100644 --- a/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift +++ b/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift @@ -37,7 +37,6 @@ public struct ToolRegistry: Sendable { "drag", "scroll", "type_text", - "type_text_chars", "press_key", "hotkey", "set_value", @@ -219,12 +218,11 @@ public struct ToolRegistry: Sendable { GetScreenSizeTool.handler, CheckPermissionsTool.handler, ScreenshotTool.handler, - GetAccessibilityTreeTool.handler, + GetCursorPositionTool.handler, MoveCursorTool.handler, ScrollTool.handler, TypeTextTool.handler, - TypeTextCharsTool.handler, PressKeyTool.handler, HotkeyTool.handler, GetWindowStateTool.handler, diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetAccessibilityTreeTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetAccessibilityTreeTool.swift deleted file mode 100644 index 6636acea24..0000000000 --- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetAccessibilityTreeTool.swift +++ /dev/null @@ -1,48 +0,0 @@ -import CuaDriverCore -import Foundation -import MCP - -public enum GetAccessibilityTreeTool { - public static let handler = ToolHandler( - tool: Tool( - name: "get_accessibility_tree", - description: """ - Return a lightweight snapshot of the desktop: running regular apps and - on-screen visible windows with their bounds, z-order, and owner pid. - - For the full AX subtree of a single window (with interactive element - indices you can click by), use `get_window_state` instead — that's the - heavy per-window tool. This one is a fast discovery read that needs no - TCC grants. - """, - inputSchema: [ - "type": "object", - "properties": [:], - "additionalProperties": false, - ], - annotations: .init( - readOnlyHint: true, - destructiveHint: false, - idempotentHint: true, - openWorldHint: false - ) - ), - invoke: { _ in - let apps = AppEnumerator.runningApps() - let windows = WindowEnumerator.visibleWindows() - var lines = ["✅ \(apps.count) running app(s), \(windows.count) visible window(s)"] - for app in apps { - lines.append("- \(app.name) (pid \(app.pid))") - } - let summary = lines.joined(separator: "\n") - return CallTool.Result( - content: [.text(text: summary, annotations: nil, _meta: nil)] - ) - } - ) - - struct Output: Codable, Sendable { - let applications: [AppInfo] - let windows: [WindowInfo] - } -} diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift index 18e0ee4d29..6df93a8e8b 100644 --- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift +++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift @@ -107,6 +107,19 @@ public enum GetWindowStateTool { For mutations or side effects use the `page` tool instead. """, ], + "screenshot_out_file": [ + "type": "string", + "description": """ + Optional absolute path to write the screenshot to (e.g. \ + "/tmp/shot.jpg"). When set, the screenshot bytes are written \ + to this file and the MCP image content block is omitted from \ + the response — `screenshot_file_path` is returned instead of \ + `screenshot_png_b64`. Useful for CLI callers and agents that \ + cannot consume inline base64 without saturating their context \ + window (e.g. OpenCode with a local Ollama model). The directory \ + must already exist; the file is created or overwritten. + """, + ], ], "additionalProperties": false, ], @@ -137,6 +150,7 @@ public enum GetWindowStateTool { } let query = arguments?["query"]?.stringValue let javascript = arguments?["javascript"]?.stringValue + let screenshotOutFile = arguments?["screenshot_out_file"]?.stringValue // Validate that the window belongs to this pid. The driver // never guesses which window to snapshot — the caller names @@ -254,16 +268,37 @@ public enum GetWindowStateTool { } } + // When the caller supplied screenshot_out_file, write the bytes + // to disk and omit the MCP image content block entirely — the + // path is surfaced via structuredContent.screenshot_file_path + // so the caller knows where to find the image without paying + // the base64-in-context token cost. + var resolvedScreenshotFilePath: String? = nil + if let outPath = screenshotOutFile, let b64 = snapshot.screenshotPngBase64 { + let expandedPath = (outPath as NSString).expandingTildeInPath + if let bytes = Data(base64Encoded: b64) { + let url = URL(fileURLWithPath: expandedPath) + do { + try bytes.write(to: url) + resolvedScreenshotFilePath = expandedPath + } catch { + // Write failed — fall through so the inline image + // content block is still emitted rather than silently + // dropping the screenshot entirely. + } + } + } + var content: [Tool.Content] = [] - if let b64 = snapshot.screenshotPngBase64 { + if resolvedScreenshotFilePath == nil, let b64 = snapshot.screenshotPngBase64 { content.append( .image(data: b64, mimeType: "image/jpeg", annotations: nil, _meta: nil) ) } content.append(.text(text: textContent, annotations: nil, _meta: nil)) // Strip the b64 bytes from the structured snapshot — the image - // is already the first content block and tests just need the - // metadata fields (screenshot_scale_factor, dimensions, etc.). + // is already the first content block (or on disk) and tests + // just need the metadata fields. let structuredSnapshot = AppStateSnapshot( pid: snapshot.pid, bundleId: snapshot.bundleId, @@ -276,7 +311,8 @@ public enum GetWindowStateTool { screenshotHeight: snapshot.screenshotHeight, screenshotScaleFactor: snapshot.screenshotScaleFactor, screenshotOriginalWidth: snapshot.screenshotOriginalWidth, - screenshotOriginalHeight: snapshot.screenshotOriginalHeight + screenshotOriginalHeight: snapshot.screenshotOriginalHeight, + screenshotFilePath: resolvedScreenshotFilePath ) if let result = try? CallTool.Result( content: content, diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift index c11e7e1c1e..5c87622094 100644 --- a/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift +++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift @@ -14,8 +14,7 @@ public enum ScreenshotTool { image data in the requested format (default png). Without `window_id`, captures the full main display. With `window_id`, - captures just that window (pair with `get_accessibility_tree` which - returns window ids). + captures just that window (get the id from `list_windows`). Requires the Screen Recording TCC grant — call `check_permissions` first if unsure. diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextCharsTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextCharsTool.swift deleted file mode 100644 index 7d1da21bb5..0000000000 --- a/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextCharsTool.swift +++ /dev/null @@ -1,111 +0,0 @@ -import CuaDriverCore -import Foundation -import MCP - -/// Character-by-character CGEvent typing, always targeting a specific pid. -/// Previously this posted to the system HID tap (frontmost-routed), which -/// was a footgun when a driver-backgrounded app typed characters into the -/// user's real foreground app. Making pid mandatory and routing via -/// `CGEvent.postToPid` removes the footgun. -public enum TypeTextCharsTool { - public static let handler = ToolHandler( - tool: Tool( - name: "type_text_chars", - description: """ - Type `text` one character at a time, delivered directly to - the target pid's event queue via `CGEvent.postToPid`. Each - character is posted as a synthesized key-down/key-up pair - whose Unicode payload is set via - `CGEventKeyboardSetUnicodeString`, bypassing virtual-key - mapping so accents, symbols, and emoji transmit verbatim. - - Use this when the AX-based `type_text` silently drops - characters — typical for Chromium / Electron text inputs - that don't expose `kAXSelectedText`. The target does NOT - need to be frontmost; keyboard focus within the target pid - determines where characters land, so focus the receiving - element first (e.g. via `click` on the input). - - `delay_ms` (0-200) spaces successive characters so - autocomplete and IME paths can keep up. Default 30. - """, - inputSchema: [ - "type": "object", - "required": ["pid", "text"], - "properties": [ - "pid": [ - "type": "integer", - "description": "Target process ID.", - ], - "text": [ - "type": "string", - "description": "Text to type into the target's focused element.", - ], - "delay_ms": [ - "type": "integer", - "minimum": 0, - "maximum": 200, - "description": - "Milliseconds between successive characters. Default 30.", - ], - ], - "additionalProperties": false, - ], - annotations: .init( - readOnlyHint: false, - destructiveHint: true, - idempotentHint: false, - openWorldHint: true - ) - ), - invoke: { arguments in - guard let rawPid = arguments?["pid"]?.intValue else { - return errorResult("Missing required integer field pid.") - } - guard let text = arguments?["text"]?.stringValue else { - return errorResult("Missing required string field text.") - } - let delayMs = arguments?["delay_ms"]?.intValue ?? 30 - guard let pid = Int32(exactly: rawPid) else { - return errorResult( - "pid \(rawPid) is outside the supported Int32 range.") - } - - do { - try KeyboardInput.typeCharacters( - text, - delayMilliseconds: delayMs, - toPid: pid - ) - let summary = - "✅ Typed \(text.count) character(s) on pid \(rawPid) with \(delayMs)ms delay." - return CallTool.Result( - content: [.text(text: summary, annotations: nil, _meta: nil)] - ) - } catch let error as KeyboardError { - return errorResult(error.description) - } catch { - return errorResult("Unexpected error: \(error)") - } - } - ) - - struct Result: Codable, Sendable { - let pid: Int - let characterCount: Int - let delayMilliseconds: Int - - private enum CodingKeys: String, CodingKey { - case pid - case characterCount = "character_count" - case delayMilliseconds = "delay_ms" - } - } - - private static func errorResult(_ message: String) -> CallTool.Result { - CallTool.Result( - content: [.text(text: message, annotations: nil, _meta: nil)], - isError: true - ) - } -} diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextTool.swift index 984474f842..16035c3599 100644 --- a/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextTool.swift +++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextTool.swift @@ -3,31 +3,28 @@ import Foundation import MCP /// Unified text-insertion primitive — always targets a specific pid. -/// Previously there were two tools (`type_text` which wrote to whatever -/// was system-focused, and `type_text_in` which targeted a pid + element). -/// The system-focused variant was a footgun because any driver-backgrounded -/// app that triggered it would write characters into the user's real -/// foreground app. Merging them under a mandatory-pid shape removes the -/// footgun; the old `type_text_in` is gone. /// -/// `element_index` is optional. When present, the element is looked up -/// from the last `get_window_state` snapshot and focused before the write — -/// the canonical path for "fill this specific text field." When absent, -/// the write targets whatever element currently has focus within the -/// target pid's AX tree (`AXUIElementCreateApplication(pid)` + -/// `AXFocusedUIElement`), which is the cheaper path when focus was -/// already established by a prior click. +/// Tries `AXSetAttribute(kAXSelectedText)` first (fast, bulk insert). +/// If the target element rejects the AX write (Chromium / Electron inputs +/// that don't expose `kAXSelectedText`), automatically falls back to +/// `KeyboardInput.typeCharacters` — character-by-character CGEvent synthesis +/// via `CGEvent.postToPid`. The fallback path is noted in the response summary +/// so callers can tell which path was taken. public enum TypeTextTool { public static let handler = ToolHandler( tool: Tool( name: "type_text", description: """ - Insert text into the target pid via - `AXSetAttribute(kAXSelectedText)`. Works for standard Cocoa - text fields and text views. No keystrokes are synthesized — - special keys (Return / Escape / arrows) go through - `press_key` / `hotkey`. For Chromium / Electron inputs that - don't implement `kAXSelectedText`, use `type_text_chars`. + Insert text into the target pid. Tries + `AXSetAttribute(kAXSelectedText)` first (fast bulk insert — + works for standard Cocoa text fields). If the target element + rejects the AX write, automatically falls back to + character-by-character `CGEvent.postToPid` synthesis — + reaches Chromium / Electron inputs and any surface that + doesn't implement `kAXSelectedText`. + + Special keys (Return, Escape, arrows, Tab) go through + `press_key` / `hotkey` — they are not text. Optional `element_index` + `window_id` (from the last `get_window_state` snapshot of that window) pre-focuses @@ -36,8 +33,11 @@ public enum TypeTextTool { targets the pid's currently-focused element — useful after a prior click already set focus. - Requires Accessibility. Returns isError=true when the - target element has no focus / rejects the attribute write. + `delay_ms` (0–200, default 30) spaces successive characters + in the CGEvent fallback path so autocomplete and IME can keep + up; ignored when the AX path succeeds. + + Requires Accessibility. """, inputSchema: [ "type": "object", @@ -61,6 +61,13 @@ public enum TypeTextTool { "description": "CGWindowID for the window whose get_window_state produced the element_index. Required when element_index is used.", ], + "delay_ms": [ + "type": "integer", + "minimum": 0, + "maximum": 200, + "description": + "Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds.", + ], ], "additionalProperties": false, ], @@ -80,6 +87,7 @@ public enum TypeTextTool { } let elementIndex = arguments?["element_index"]?.intValue let rawWindowId = arguments?["window_id"]?.intValue + let delayMs = arguments?["delay_ms"]?.intValue ?? 30 guard let pid = Int32(exactly: rawPid) else { return errorResult( "pid \(rawPid) is outside the supported Int32 range.") @@ -91,6 +99,9 @@ public enum TypeTextTool { + "the same window_id you used in `get_window_state`.") } + // Attempt AX bulk-insert. On any AXInputError fall back to + // CGEvent character synthesis, which reaches Chromium / Electron + // inputs that don't expose kAXSelectedText. do { if let index = elementIndex, let rawWindowId { guard let windowId = UInt32(exactly: rawWindowId) else { @@ -101,38 +112,58 @@ public enum TypeTextTool { pid: pid, windowId: windowId, elementIndex: index) - try await AppStateRegistry.focusGuard.withFocusSuppressed( - pid: pid, element: element - ) { + do { + try await AppStateRegistry.focusGuard.withFocusSuppressed( + pid: pid, element: element + ) { + try AXInput.setAttribute( + "AXSelectedText", + on: element, + value: text as CFTypeRef + ) + } + let target = AXInput.describe(element) + let summary = + "✅ Inserted \(text.count) char(s) into [\(index)] \(target.role ?? "?") \"\(target.title ?? "")\" on pid \(rawPid) via AX." + return CallTool.Result( + content: [.text(text: summary, annotations: nil, _meta: nil)] + ) + } catch is AXInputError { + // AX rejected — fall through to CGEvent synthesis. + } + } else { + do { + let element = try AXInput.focusedElement(pid: pid) try AXInput.setAttribute( "AXSelectedText", on: element, value: text as CFTypeRef ) + let target = AXInput.describe(element) + let summary = + "✅ Inserted \(text.count) char(s) into focused \(target.role ?? "?") \"\(target.title ?? "")\" on pid \(rawPid) via AX." + return CallTool.Result( + content: [.text(text: summary, annotations: nil, _meta: nil)] + ) + } catch is AXInputError { + // AX rejected — fall through to CGEvent synthesis. } - let target = AXInput.describe(element) - let summary = - "✅ Inserted \(text.count) char(s) into [\(index)] \(target.role ?? "?") \"\(target.title ?? "")\" on pid \(rawPid)." - return CallTool.Result( - content: [.text(text: summary, annotations: nil, _meta: nil)] - ) - } else { - let element = try AXInput.focusedElement(pid: pid) - try AXInput.setAttribute( - "AXSelectedText", - on: element, - value: text as CFTypeRef - ) - let target = AXInput.describe(element) - let summary = - "✅ Inserted \(text.count) char(s) into focused \(target.role ?? "?") \"\(target.title ?? "")\" on pid \(rawPid)." - return CallTool.Result( - content: [.text(text: summary, annotations: nil, _meta: nil)] - ) } + + // CGEvent fallback path. + try KeyboardInput.typeCharacters( + text, + delayMilliseconds: delayMs, + toPid: pid + ) + let summary = + "✅ Typed \(text.count) char(s) on pid \(rawPid) via CGEvent (AX fallback, \(delayMs)ms delay)." + return CallTool.Result( + content: [.text(text: summary, annotations: nil, _meta: nil)] + ) } catch let error as AppStateError { return errorResult(error.description) - } catch let error as AXInputError { + } catch let error as KeyboardError { return errorResult(error.description) } catch { return errorResult("Unexpected error: \(error)")