diff --git a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx
index 6b657edcde..daabda628b 100644
--- a/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx
+++ b/docs/content/docs/cua-driver/guide/getting-started/integrations.mdx
@@ -90,7 +90,43 @@ Tools appear prefixed as `mcp_cua-driver_*`.
cua-driver mcp-config --client opencode
```
-Paste the output into your `opencode.json`.
+Paste the output into `~/.config/opencode/config.json` (global) or `opencode.json` at the project root.
+
+
+**Always configure cua-driver as an MCP server — never rely on the CLI fallback.** If MCP is not wired up, OpenCode calls `cua-driver` as a shell subprocess. The `get_window_state` response no longer includes base64 by default, but the screenshot image block is silently dropped — the model receives only the AX tree with no visual context. Use `--screenshot-out-file` or the `screenshot_out_file` param to preserve the image when using the CLI path.
+
+
+### Local vision models (Ollama)
+
+If you are using a vision-capable model via Ollama, you must also declare its input modalities in `config.json` — otherwise OpenCode strips images before they reach the model:
+
+```json
+{
+ "mcp": {
+ "cua-driver": {
+ "type": "local",
+ "command": ["/Users/you/.local/bin/cua-driver", "mcp"],
+ "enabled": true
+ }
+ },
+ "provider": {
+ "ollama": {
+ "npm": "@ai-sdk/openai-compatible",
+ "options": { "baseURL": "http://localhost:11434/v1" },
+ "models": {
+ "gemma4:26b": {
+ "modalities": {
+ "input": ["text", "image"],
+ "output": ["text"]
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+The `modalities` field is required because OpenCode's `@ai-sdk/openai-compatible` provider defaults to text-only when no capabilities are declared. Without it, screenshots are replaced with an error string and never reach the model.
## Hermes (NousResearch)
diff --git a/docs/content/docs/cua-driver/reference/cli-reference.mdx b/docs/content/docs/cua-driver/reference/cli-reference.mdx
index 16034b4b8a..c5af31770c 100644
--- a/docs/content/docs/cua-driver/reference/cli-reference.mdx
+++ b/docs/content/docs/cua-driver/reference/cli-reference.mdx
@@ -182,7 +182,7 @@ Pipe to `pbcopy` to put any output on the clipboard, or pass the printed `claude
### cua-driver recording start
-Enable the trajectory recorder. Every subsequent action-tool call (`click`, `right_click`, `scroll`, `type_text`, `type_text_chars`, `press_key`, `hotkey`, `set_value`) writes a numbered turn folder under ``.
+Enable the trajectory recorder. Every subsequent action-tool call (`click`, `right_click`, `scroll`, `type_text`, `press_key`, `hotkey`, `set_value`) writes a numbered turn folder under ``.
```bash
cua-driver recording start ~/cua-trajectories/demo1
@@ -346,7 +346,6 @@ The following MCP tools are callable via `cua-driver `. For input schemas
- `check_permissions` — Accessibility + Screen Recording TCC status.
- `get_screen_size` — main display size in points + scale factor.
- `get_cursor_position` — current mouse cursor position.
-- `get_accessibility_tree` — lightweight desktop snapshot (apps + visible windows).
**App lifecycle**
@@ -368,8 +367,7 @@ The following MCP tools are callable via `cua-driver `. For input schemas
**Keyboard**
-- `type_text` — insert text via `AXSelectedText`. Pid-scoped.
-- `type_text_chars` — character-by-character via `CGEvent.postToPid`. Reaches Chromium/Electron inputs.
+- `type_text` — insert text via `AXSelectedText` with automatic CGEvent fallback for Chromium/Electron inputs. Pid-scoped.
- `press_key` — single key press. Pid-scoped.
- `hotkey` — modifier combo (e.g. `["cmd","c"]`). Pid-scoped.
diff --git a/docs/content/docs/cua-driver/reference/mcp-tools.mdx b/docs/content/docs/cua-driver/reference/mcp-tools.mdx
index 0b19c1f3b7..2b110dfceb 100644
--- a/docs/content/docs/cua-driver/reference/mcp-tools.mdx
+++ b/docs/content/docs/cua-driver/reference/mcp-tools.mdx
@@ -58,16 +58,6 @@ Return the current mouse cursor position in screen points (top-left origin).
{}
```
-### get_accessibility_tree
-
-Lightweight desktop snapshot: running regular apps and on-screen visible windows with bounds, z-order, and owner pid. For a single window's internal UI, use `get_window_state`.
-
-**Arguments:** none.
-
-```json
-{}
-```
-
### screenshot
Raw ScreenCaptureKit capture. Full main display, or a single window when `window_id` is set. Returns an image content block plus a text summary listing on-screen windows.
@@ -271,7 +261,7 @@ Modifier combo as a single array, e.g. `["cmd", "c"]`. Requires at least two ent
### type_text
-Insert text at the target's current cursor via `AXSelectedText`. Fast (single AX write) but skipped by apps with custom text layers; for Chromium / Electron inputs use `type_text_chars`.
+Insert text at the target's current cursor. Attempts `AXSelectedText` write first (fast, single AX call); automatically falls back to character-by-character `CGEvent.postToPid` synthesis when the target input doesn't implement `AXSelectedText` — this makes it work for Chromium and Electron inputs without any manual switching.
**Arguments:**
@@ -279,22 +269,10 @@ Insert text at the target's current cursor via `AXSelectedText`. Fast (single AX
- `text` (string, required): Text to insert at the target's cursor.
- `element_index` (integer, optional): When present, the element is focused before the write. Requires `window_id`.
- `window_id` (integer, optional): Required when `element_index` is used.
+- `delay_ms` (integer, optional): Milliseconds between characters when the CGEvent fallback path is used, 0-200. Default 30.
```json
{"pid": 844, "window_id": 10725, "element_index": 12, "text": "hello"}
-```
-
-### type_text_chars
-
-Character-by-character input via `CGEvent.postToPid`. Slower than `type_text` but reaches Chromium and Electron inputs that ignore AX writes.
-
-**Arguments:**
-
-- `pid` (integer, required): Target process ID.
-- `text` (string, required): Text to type into the target's focused element.
-- `delay_ms` (integer, optional): Milliseconds between characters, 0-200. Default 30.
-
-```json
{"pid": 844, "text": "hello world", "delay_ms": 40}
```
@@ -450,7 +428,7 @@ Supported keys and ranges: see the [CLI reference](/cua-driver/reference/cli-ref
## Recording and replay
-The trajectory recorder captures every action-tool call (`click`, `right_click`, `scroll`, `type_text`, `type_text_chars`, `press_key`, `hotkey`, `set_value`) into numbered turn folders. Recordings can be replayed turn-by-turn.
+The trajectory recorder captures every action-tool call (`click`, `right_click`, `scroll`, `type_text`, `press_key`, `hotkey`, `set_value`) into numbered turn folders. Recordings can be replayed turn-by-turn.
### get_recording_state
diff --git a/libs/cua-driver/Skills/cua-driver/RECORDING.md b/libs/cua-driver/Skills/cua-driver/RECORDING.md
index 79fc526398..e60c3d2f8c 100644
--- a/libs/cua-driver/Skills/cua-driver/RECORDING.md
+++ b/libs/cua-driver/Skills/cua-driver/RECORDING.md
@@ -6,7 +6,7 @@ user explicitly asks to record — the skill does not auto-enable this.
`set_recording` turns on a session-scoped trajectory recorder. While
enabled, every action-tool call (`click`, `right_click`, `scroll`,
-`type_text`, `type_text_chars`, `press_key`, `hotkey`, `set_value`)
+`type_text`, `press_key`, `hotkey`, `set_value`)
writes a numbered turn folder under a caller-chosen output
directory. Read-only tools (`get_window_state`, `list_windows`,
`screenshot`, `list_apps`, permission probes, agent-cursor getters /
@@ -101,8 +101,7 @@ keyed on `(pid, window_id)`, so a recorded
resolve today — the pid is usually different, the window_id always
is. The call returns `Invalid element_index` or `No cached AX
state`. Pixel clicks (`click({pid, x, y})`) and keyboard tools
-(`press_key`, `type_text_chars`, `hotkey`, `type_text` without
-element_index) replay cleanly; element-indexed actions require a
+(`press_key`, `hotkey`, `type_text` without element_index) replay cleanly; element-indexed actions require a
live snapshot that replay doesn't currently re-emit (read-only tools
like `get_window_state` aren't recorded). For a reliable replay, either
compose the trajectory from pixel + keyboard primitives, or capture
diff --git a/libs/cua-driver/Skills/cua-driver/SKILL.md b/libs/cua-driver/Skills/cua-driver/SKILL.md
index 3858f74922..e442825136 100644
--- a/libs/cua-driver/Skills/cua-driver/SKILL.md
+++ b/libs/cua-driver/Skills/cua-driver/SKILL.md
@@ -420,8 +420,7 @@ you're interacting with a long-lived process). In the default
get_window_state → reason over PNG → pixel click`. When you need
`element_index` dispatch (AX-addressable elements, backgrounded
clicks), flip to `som` first: `cua-driver set_config '{"key":
-"capture_mode", "value": "som"}'`, or call `get_accessibility_tree`
-directly. The rest of this section walks through `som` mode, which
+"capture_mode", "value": "som"}'`. The rest of this section walks through `som` mode, which
is what you want once you've decided element-indexed addressing is
required.
@@ -432,33 +431,26 @@ In `som` mode the response carries:
~1600 elements, ~190 KB); when it exceeds token limits the MCP
harness saves it to a file and returns the path. Use `Bash` +
`jq -r '.tree_markdown'` + `grep` to pull the section you need.
-- `screenshot_png_b64` + `screenshot_width` / `_height` /
- `_scale_factor` — the window screenshot (actually JPEG-85 despite
- the `_png_` field name, hard-coded in
- `WindowCapture.captureFrontmostWindow`). Present in `som` mode
- (spliced into the structured JSON alongside the tree). In `vision`
- mode the image arrives as a native MCP image content block with no
- structured wrapper. Omitted when the target has no on-screen
- window.
-- `has_screenshot: bool` — **gate on this before piping the PNG**.
- Otherwise `jq -r '.screenshot_png_b64'` emits the literal
- `"null"`, base64-decodes into 3 bytes of garbage, and downstream
- vision APIs reject it with an opaque "Could not process image"
- error.
-
-```
-# canonical, works in every capture mode — writes the image bytes
-# wherever you point, stdout stays readable (tree in som, summary
-# in vision). stderr warns (exit 0) if the response had no image.
-cua-driver get_window_state '{"pid":N,"window_id":W}' --image-out /tmp/shot.png
-
-# som-only legacy path: pull the spliced base64 out of structuredContent.
-# Prefer --image-out above — it's one flag vs a probe + pipe.
-if [ "$(cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.has_screenshot')" = "true" ]; then
- cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.screenshot_png_b64' | base64 -d > shot.png
-fi
+- `screenshot_file_path` — absolute path to the saved screenshot when
+ `screenshot_out_file` was passed. Absent otherwise.
+- `screenshot_width` / `_height` / `_scale_factor` — dimensions of the
+ captured image. Present whenever a screenshot was taken.
+**Getting the screenshot as a file (CLI and context-constrained agents):**
+
+```bash
+# write to file — stdout stays readable (AX tree / summary only, no base64)
+cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.jpg"}'
+
+# CLI --screenshot-out-file flag is equivalent and works for all capture modes
+cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.jpg
```
+Pass `screenshot_out_file` when using `get_window_state` via CLI or from an
+agent whose context window can't absorb ~31 KB of inline base64 (e.g.
+OpenCode with a local Ollama model). The MCP image content block is omitted
+from the response when this param is set — the model receives only the AX
+tree and `screenshot_file_path`, then reads the image from disk.
+
**Reason over both the tree AND the screenshot — they're
complementary, not redundant.** In `som` mode every
turn's `get_window_state` gives you both halves and you should pull
@@ -512,7 +504,7 @@ anchor the conversion against a specific window):
| Focus + send key | `press_key({pid, key, window_id, element_index, modifiers})` | element_index sets AXFocused, then posts key |
| Send key to pid | `press_key({pid, key, modifiers})` | no focus change; key goes to pid's current focus |
| Modifier combo | `hotkey({pid, keys})` | e.g. `["cmd","c"]`; posted per-pid, not HID tap |
-| Unicode keystrokes | `type_text_chars({pid, text, delay_ms})` | CGEvent-to-pid; reaches Chromium/Electron inputs |
+| Unicode keystrokes | `type_text({pid, text, delay_ms})` | AX write with automatic CGEvent fallback; reaches Chromium/Electron inputs |
**All keyboard/text primitives require `pid`.** There is no
frontmost-routed variant — every key goes to the named target via
@@ -556,7 +548,7 @@ below against the full-resolution file in that case.
1. `get_window_state({pid, window_id})` returns an image capped
at 1568 long-side (default) plus its dimensions
(`screenshot_width` / `screenshot_height`). Write the bytes to
- disk with `--image-out ` in any capture mode — works
+ disk with `--screenshot-out-file ` in any capture mode — works
identically in `vision` (where it's the only way) and `som`
(where it sidesteps the jq + base64 dance on the spliced
`screenshot_png_b64` field).
diff --git a/libs/cua-driver/Skills/cua-driver/TESTS.md b/libs/cua-driver/Skills/cua-driver/TESTS.md
index dde5fca226..fdc835b329 100644
--- a/libs/cua-driver/Skills/cua-driver/TESTS.md
+++ b/libs/cua-driver/Skills/cua-driver/TESTS.md
@@ -116,11 +116,11 @@ These are where CuaDriver's AX-activation trio matters:
### 8. Slack — sparse Electron, hotkey fallback
**Prompt:** `In Slack, jump to the #pr-reviews channel using the quick switcher (⌘K).`
-**Exercises:** "retry snapshot once, then `hotkey` + `type_text_chars`" fallback path.
+**Exercises:** "retry snapshot once, then `hotkey` + `type_text`" fallback path.
**Success:**
- Re-snapshot shows Slack's channel title area (AX role `AXStaticText` or similar) contains `pr-reviews`.
-- Claude uses `hotkey` and `type_text_chars` — NOT `simulate_click` / pixel coords.
+- Claude uses `hotkey` and `type_text` — NOT `simulate_click` / pixel coords.
**Fail signals:** Claude drops to `simulate_click` (guardrail violation — pixel fallback is not allowed on sparse AX trees), wrong channel joined, typed text echoes into the message composer instead of the switcher.
@@ -168,7 +168,7 @@ These are where CuaDriver's AX-activation trio matters:
### 12. Chrome proper — omnibox
**Prompt:** `In Google Chrome, open a new tab and navigate to https://trycua.com.`
-**Exercises:** `hotkey(["cmd","t"])` + `type_text_chars` + Return. Chrome's omnibox typically isn't AX-exposed even after activation.
+**Exercises:** `hotkey(["cmd","t"])` + `type_text` + Return. Chrome's omnibox typically isn't AX-exposed even after activation; `type_text` automatically falls back to CGEvent synthesis for Chromium/Electron inputs.
**Success:**
- A new Chrome tab whose AX title contains `Cua` or `trycua` is present.
diff --git a/libs/cua-driver/Skills/cua-driver/WEB_APPS.md b/libs/cua-driver/Skills/cua-driver/WEB_APPS.md
index 6fbaef9e21..312525d8af 100644
--- a/libs/cua-driver/Skills/cua-driver/WEB_APPS.md
+++ b/libs/cua-driver/Skills/cua-driver/WEB_APPS.md
@@ -46,10 +46,10 @@ pixels:
`hotkey({pid, keys: ["cmd", "enter"]})`, `hotkey({pid, keys:
["cmd", "k"]})`, etc. Posted via `CGEvent.postToPid`, reaches the
target regardless of AX state, no activation required.
-3. For typing into web inputs where `type_text` silently drops
- (input doesn't implement `AXSelectedText`), use `type_text_chars`
- — pure CGEvent keystrokes reach any focused keyboard receiver,
- including Unicode / emoji.
+3. For typing into web inputs, use `type_text` — it automatically
+ falls back to CGEvent synthesis when the input doesn't implement
+ `AXSelectedText`, reaching any focused keyboard receiver including
+ Unicode / emoji.
4. If none of the above reaches the target, tell the user this
interaction isn't reachable from the driver today and ask for
guidance.
@@ -86,7 +86,7 @@ documented only as historical context:
```
# DON'T DO THIS — ⌘L steals focus. Use launch_app above.
hotkey({pid, keys: ["cmd", "l"]})
-type_text_chars({pid, text: "https://cua.ai", delay_ms: 30})
+type_text({pid, text: "https://cua.ai", delay_ms: 30})
get_window_state({pid, window_id})
click({pid, window_id, element_index: })
```
@@ -157,7 +157,7 @@ When the target window is **minimized** (genie'd into the Dock):
`AXFocused=true` on a minimized window's descendants doesn't
propagate to real keyboard focus). Symptom: macOS system-alert
beep, or silent no-op. Example: `hotkey cmd+L` +
- `type_text_chars URL` + `press_key return` on minimized Chrome —
+ `type_text URL` + `press_key return` on minimized Chrome —
the URL lands in the omnibox AX value but Return doesn't commit
the navigation.
- **Primary workaround — use `set_value` to commit directly**: For
@@ -465,6 +465,7 @@ type_text({pid, window_id, element_index: , text: "…"})
```
If it silently drops (some web inputs don't implement
-`AXSelectedText`), click the field first, then use
-`type_text_chars({pid, text})` — pure CGEvent keystrokes delivered
-to the pid, reaching any focused keyboard receiver.
+`AXSelectedText`), `type_text` automatically falls back to CGEvent
+synthesis — pure CGEvent keystrokes delivered to the pid, reaching
+any focused keyboard receiver. You can also click the field first
+to ensure focus before typing.
diff --git a/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift b/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift
index 1f09722339..8320df85c3 100644
--- a/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift
+++ b/libs/cua-driver/Sources/CuaDriverCLI/CallCommand.swift
@@ -83,13 +83,13 @@ struct CallCommand: AsyncParsableCommand {
file path. `vision` and `screenshot` capture modes return a
PNG as a native MCP image block with no accompanying
structuredContent, so the CLI's text formatter would otherwise
- drop the bytes silently. With `--image-out /tmp/shot.png` the
- raw PNG lands on disk and downstream tooling (PIL, sips,
+ drop the bytes silently. With `--screenshot-out-file /tmp/shot.jpg`
+ the raw image lands on disk and downstream tooling (PIL, sips,
ffprobe) can read it directly. Silently warns (no error exit)
when the response carries no image.
"""
)
- var imageOut: String?
+ var screenshotOutFile: String?
@OptionGroup var output: JSONOutputOptions
@OptionGroup var daemon: DaemonForwardingOptions
@@ -113,7 +113,7 @@ struct CallCommand: AsyncParsableCommand {
arguments: arguments,
socketPath: daemon.resolvedSocketPath,
raw: raw,
- imageOut: imageOut,
+ screenshotOutFile: screenshotOutFile,
output: output
)
return
@@ -184,7 +184,7 @@ struct CallCommand: AsyncParsableCommand {
throw ExitCode(Exit.software)
}
- if let path = imageOut {
+ if let path = screenshotOutFile {
writeFirstImageContent(result.content, to: path)
}
@@ -251,10 +251,7 @@ struct CallCommand: AsyncParsableCommand {
let encoder = JSONEncoder()
encoder.outputFormatting = output.encoderOutputFormatting
let encoded = try encoder.encode(structured)
- let merged = mergeImageContentIntoJSON(
- encoded, content: result.content, output: output
- )
- printDataLine(merged)
+ printDataLine(encoded)
return
}
@@ -352,7 +349,7 @@ func forwardCallToDaemon(
arguments: [String: Value]?,
socketPath: String,
raw: Bool,
- imageOut: String?,
+ screenshotOutFile: String?,
output: JSONOutputOptions
) async throws {
let request = DaemonRequest(method: "call", name: toolName, args: arguments)
@@ -381,7 +378,7 @@ func forwardCallToDaemon(
throw ExitCode(Exit.software)
}
- if let path = imageOut {
+ if let path = screenshotOutFile {
writeFirstImageContent(result.content, to: path)
}
@@ -426,10 +423,7 @@ private func emitUnwrappedResultForDaemon(
let encoder = JSONEncoder()
encoder.outputFormatting = output.encoderOutputFormatting
let encoded = try encoder.encode(structured)
- let merged = mergeImageContentIntoJSON(
- encoded, content: result.content, output: output
- )
- FileHandle.standardOutput.write(merged)
+ FileHandle.standardOutput.write(encoded)
FileHandle.standardOutput.write(Data("\n".utf8))
return
}
@@ -487,70 +481,23 @@ enum DaemonCLIError: Error {
case protocolMismatch
}
-/// MCP delivers screenshot bytes as a native `.image()` content block
-/// separate from `structuredContent`, so shell consumers of
-/// `cua-driver ` only see the metadata half ("has_screenshot",
-/// dimensions) and the base64 pixels silently vanish. This reunites them
-/// at CLI emit time: if any image block is present in `content`, splice
-/// its base64 into the outgoing JSON as `screenshot_png_b64` (plus
-/// `screenshot_mime_type`) alongside the existing fields. The MCP wire
-/// contract is unchanged — only the CLI's pretty-printed output gains
-/// the pixels that downstream `jq -r '.screenshot_png_b64' | base64 -d`
-/// pipelines expect.
-///
-/// Returns the input `encoded` unchanged when there's no image block to
-/// splice OR when the structured content didn't decode as a JSON object
-/// (e.g. a bare array / scalar from some other tool). Never throws —
-/// merge failures fall through silently so a malformed structured
-/// response still emits its original bytes.
-private func mergeImageContentIntoJSON(
- _ encoded: Data,
- content: [Tool.Content],
- output: JSONOutputOptions
-) -> Data {
- var imageBase64: String? = nil
- var imageMime: String? = nil
- for item in content {
- if case let .image(data, mime, _, _) = item {
- imageBase64 = data
- imageMime = mime
- break
- }
- }
- guard let imageBase64, let imageMime else { return encoded }
-
- guard
- var object = (try? JSONSerialization.jsonObject(with: encoded))
- as? [String: Any]
- else { return encoded }
- object["screenshot_png_b64"] = imageBase64
- object["screenshot_mime_type"] = imageMime
-
- var writingOptions: JSONSerialization.WritingOptions = [
- .sortedKeys, .withoutEscapingSlashes,
- ]
- if !output.compact { writingOptions.insert(.prettyPrinted) }
- return (try? JSONSerialization.data(
- withJSONObject: object, options: writingOptions
- )) ?? encoded
-}
/// Write the first `.image(...)` content block from a tool result to
-/// `path`, decoded from base64. Used by the `--image-out` flag so
+/// `path`, decoded from base64. Used by the `--screenshot-out-file` flag so
/// vision-mode screenshots land on disk without callers needing to
/// speak the raw daemon protocol or the MCP bridge.
///
/// Never throws — an empty/malformed image, a write failure, or a
/// response that carries no image at all each emit a stderr warning
/// and return. The exit code is NOT affected: callers have already
-/// gotten the textual result on stdout, and `--image-out` is
+/// gotten the textual result on stdout, and `--screenshot-out-file` is
/// advisory ("if the tool returned a PNG, put it here"), not a hard
/// contract.
func writeFirstImageContent(_ content: [Tool.Content], to path: String) {
for item in content {
if case let .image(base64, mime, _, _) = item {
guard let bytes = Data(base64Encoded: base64) else {
- printToStderr("--image-out: base64 decode failed for \(mime) block")
+ printToStderr("--screenshot-out-file: base64 decode failed for \(mime) block")
return
}
let url = URL(fileURLWithPath: (path as NSString).expandingTildeInPath)
@@ -558,7 +505,7 @@ func writeFirstImageContent(_ content: [Tool.Content], to path: String) {
try bytes.write(to: url)
} catch {
printToStderr(
- "--image-out: failed to write \(url.path): \(error.localizedDescription)"
+ "--screenshot-out-file: failed to write \(url.path): \(error.localizedDescription)"
)
}
return
@@ -568,7 +515,7 @@ func writeFirstImageContent(_ content: [Tool.Content], to path: String) {
// tools legitimately return nothing (hidden/minimized windows,
// `ax` capture mode, list operations). Warn so the user notices
// the file they expected didn't get written.
- printToStderr("--image-out: no image content in tool response; file not written")
+ printToStderr("--screenshot-out-file: no image content in tool response; file not written")
}
private func printUnknownTool(_ name: String, registry: ToolRegistry) {
diff --git a/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift b/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift
index a463005c86..65b4cbfd71 100644
--- a/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift
+++ b/libs/cua-driver/Sources/CuaDriverCore/AppState/AppState.swift
@@ -54,6 +54,9 @@ public struct AppStateSnapshot: Sendable, Codable {
public let screenshotOriginalWidth: Int?
/// Original height before maxImageDimension resize. nil = no resize.
public let screenshotOriginalHeight: Int?
+ /// File-system path to the saved screenshot (JPEG) when the caller
+ /// passed `screenshot_out_file`. Absent when no path was written.
+ public let screenshotFilePath: String?
public init(
pid: Int32,
@@ -67,7 +70,8 @@ public struct AppStateSnapshot: Sendable, Codable {
screenshotHeight: Int? = nil,
screenshotScaleFactor: Double? = nil,
screenshotOriginalWidth: Int? = nil,
- screenshotOriginalHeight: Int? = nil
+ screenshotOriginalHeight: Int? = nil,
+ screenshotFilePath: String? = nil
) {
self.pid = pid
self.bundleId = bundleId
@@ -81,6 +85,7 @@ public struct AppStateSnapshot: Sendable, Codable {
self.screenshotScaleFactor = screenshotScaleFactor
self.screenshotOriginalWidth = screenshotOriginalWidth
self.screenshotOriginalHeight = screenshotOriginalHeight
+ self.screenshotFilePath = screenshotFilePath
}
private enum CodingKeys: String, CodingKey {
@@ -95,6 +100,7 @@ public struct AppStateSnapshot: Sendable, Codable {
case screenshotScaleFactor = "screenshot_scale_factor"
case screenshotOriginalWidth = "screenshot_original_width"
case screenshotOriginalHeight = "screenshot_original_height"
+ case screenshotFilePath = "screenshot_file_path"
}
/// Encodes the struct with ``screenshot_*`` keys omitted when nil.
@@ -116,6 +122,7 @@ public struct AppStateSnapshot: Sendable, Codable {
try container.encodeIfPresent(screenshotScaleFactor, forKey: .screenshotScaleFactor)
try container.encodeIfPresent(screenshotOriginalWidth, forKey: .screenshotOriginalWidth)
try container.encodeIfPresent(screenshotOriginalHeight, forKey: .screenshotOriginalHeight)
+ try container.encodeIfPresent(screenshotFilePath, forKey: .screenshotFilePath)
}
public init(from decoder: Decoder) throws {
@@ -132,6 +139,7 @@ public struct AppStateSnapshot: Sendable, Codable {
screenshotScaleFactor = try container.decodeIfPresent(Double.self, forKey: .screenshotScaleFactor)
screenshotOriginalWidth = try container.decodeIfPresent(Int.self, forKey: .screenshotOriginalWidth)
screenshotOriginalHeight = try container.decodeIfPresent(Int.self, forKey: .screenshotOriginalHeight)
+ screenshotFilePath = try container.decodeIfPresent(String.self, forKey: .screenshotFilePath)
}
}
diff --git a/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift b/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift
index 631f3b9ab7..d29f1a16fd 100644
--- a/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift
+++ b/libs/cua-driver/Sources/CuaDriverCore/Capture/WindowCapture.swift
@@ -100,7 +100,7 @@ public actor WindowCapture {
}
/// Capture a single window by its CGWindowID / kCGWindowNumber.
- /// Pairs with `get_accessibility_tree` which returns each window's `id`.
+ /// Get window ids from `list_windows`.
public func captureWindow(
windowID: UInt32,
format: ImageFormat = .png,
diff --git a/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift b/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift
index c5ef3cd6ca..6a152bf3e1 100644
--- a/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift
+++ b/libs/cua-driver/Sources/CuaDriverServer/ToolRegistry.swift
@@ -37,7 +37,6 @@ public struct ToolRegistry: Sendable {
"drag",
"scroll",
"type_text",
- "type_text_chars",
"press_key",
"hotkey",
"set_value",
@@ -219,12 +218,11 @@ public struct ToolRegistry: Sendable {
GetScreenSizeTool.handler,
CheckPermissionsTool.handler,
ScreenshotTool.handler,
- GetAccessibilityTreeTool.handler,
+
GetCursorPositionTool.handler,
MoveCursorTool.handler,
ScrollTool.handler,
TypeTextTool.handler,
- TypeTextCharsTool.handler,
PressKeyTool.handler,
HotkeyTool.handler,
GetWindowStateTool.handler,
diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetAccessibilityTreeTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetAccessibilityTreeTool.swift
deleted file mode 100644
index 6636acea24..0000000000
--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetAccessibilityTreeTool.swift
+++ /dev/null
@@ -1,48 +0,0 @@
-import CuaDriverCore
-import Foundation
-import MCP
-
-public enum GetAccessibilityTreeTool {
- public static let handler = ToolHandler(
- tool: Tool(
- name: "get_accessibility_tree",
- description: """
- Return a lightweight snapshot of the desktop: running regular apps and
- on-screen visible windows with their bounds, z-order, and owner pid.
-
- For the full AX subtree of a single window (with interactive element
- indices you can click by), use `get_window_state` instead — that's the
- heavy per-window tool. This one is a fast discovery read that needs no
- TCC grants.
- """,
- inputSchema: [
- "type": "object",
- "properties": [:],
- "additionalProperties": false,
- ],
- annotations: .init(
- readOnlyHint: true,
- destructiveHint: false,
- idempotentHint: true,
- openWorldHint: false
- )
- ),
- invoke: { _ in
- let apps = AppEnumerator.runningApps()
- let windows = WindowEnumerator.visibleWindows()
- var lines = ["✅ \(apps.count) running app(s), \(windows.count) visible window(s)"]
- for app in apps {
- lines.append("- \(app.name) (pid \(app.pid))")
- }
- let summary = lines.joined(separator: "\n")
- return CallTool.Result(
- content: [.text(text: summary, annotations: nil, _meta: nil)]
- )
- }
- )
-
- struct Output: Codable, Sendable {
- let applications: [AppInfo]
- let windows: [WindowInfo]
- }
-}
diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift
index 18e0ee4d29..6df93a8e8b 100644
--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift
+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/GetWindowStateTool.swift
@@ -107,6 +107,19 @@ public enum GetWindowStateTool {
For mutations or side effects use the `page` tool instead.
""",
],
+ "screenshot_out_file": [
+ "type": "string",
+ "description": """
+ Optional absolute path to write the screenshot to (e.g. \
+ "/tmp/shot.jpg"). When set, the screenshot bytes are written \
+ to this file and the MCP image content block is omitted from \
+ the response — `screenshot_file_path` is returned instead of \
+ `screenshot_png_b64`. Useful for CLI callers and agents that \
+ cannot consume inline base64 without saturating their context \
+ window (e.g. OpenCode with a local Ollama model). The directory \
+ must already exist; the file is created or overwritten.
+ """,
+ ],
],
"additionalProperties": false,
],
@@ -137,6 +150,7 @@ public enum GetWindowStateTool {
}
let query = arguments?["query"]?.stringValue
let javascript = arguments?["javascript"]?.stringValue
+ let screenshotOutFile = arguments?["screenshot_out_file"]?.stringValue
// Validate that the window belongs to this pid. The driver
// never guesses which window to snapshot — the caller names
@@ -254,16 +268,37 @@ public enum GetWindowStateTool {
}
}
+ // When the caller supplied screenshot_out_file, write the bytes
+ // to disk and omit the MCP image content block entirely — the
+ // path is surfaced via structuredContent.screenshot_file_path
+ // so the caller knows where to find the image without paying
+ // the base64-in-context token cost.
+ var resolvedScreenshotFilePath: String? = nil
+ if let outPath = screenshotOutFile, let b64 = snapshot.screenshotPngBase64 {
+ let expandedPath = (outPath as NSString).expandingTildeInPath
+ if let bytes = Data(base64Encoded: b64) {
+ let url = URL(fileURLWithPath: expandedPath)
+ do {
+ try bytes.write(to: url)
+ resolvedScreenshotFilePath = expandedPath
+ } catch {
+ // Write failed — fall through so the inline image
+ // content block is still emitted rather than silently
+ // dropping the screenshot entirely.
+ }
+ }
+ }
+
var content: [Tool.Content] = []
- if let b64 = snapshot.screenshotPngBase64 {
+ if resolvedScreenshotFilePath == nil, let b64 = snapshot.screenshotPngBase64 {
content.append(
.image(data: b64, mimeType: "image/jpeg", annotations: nil, _meta: nil)
)
}
content.append(.text(text: textContent, annotations: nil, _meta: nil))
// Strip the b64 bytes from the structured snapshot — the image
- // is already the first content block and tests just need the
- // metadata fields (screenshot_scale_factor, dimensions, etc.).
+ // is already the first content block (or on disk) and tests
+ // just need the metadata fields.
let structuredSnapshot = AppStateSnapshot(
pid: snapshot.pid,
bundleId: snapshot.bundleId,
@@ -276,7 +311,8 @@ public enum GetWindowStateTool {
screenshotHeight: snapshot.screenshotHeight,
screenshotScaleFactor: snapshot.screenshotScaleFactor,
screenshotOriginalWidth: snapshot.screenshotOriginalWidth,
- screenshotOriginalHeight: snapshot.screenshotOriginalHeight
+ screenshotOriginalHeight: snapshot.screenshotOriginalHeight,
+ screenshotFilePath: resolvedScreenshotFilePath
)
if let result = try? CallTool.Result(
content: content,
diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift
index c11e7e1c1e..5c87622094 100644
--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift
+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/ScreenshotTool.swift
@@ -14,8 +14,7 @@ public enum ScreenshotTool {
image data in the requested format (default png).
Without `window_id`, captures the full main display. With `window_id`,
- captures just that window (pair with `get_accessibility_tree` which
- returns window ids).
+ captures just that window (get the id from `list_windows`).
Requires the Screen Recording TCC grant — call `check_permissions`
first if unsure.
diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextCharsTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextCharsTool.swift
deleted file mode 100644
index 7d1da21bb5..0000000000
--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextCharsTool.swift
+++ /dev/null
@@ -1,111 +0,0 @@
-import CuaDriverCore
-import Foundation
-import MCP
-
-/// Character-by-character CGEvent typing, always targeting a specific pid.
-/// Previously this posted to the system HID tap (frontmost-routed), which
-/// was a footgun when a driver-backgrounded app typed characters into the
-/// user's real foreground app. Making pid mandatory and routing via
-/// `CGEvent.postToPid` removes the footgun.
-public enum TypeTextCharsTool {
- public static let handler = ToolHandler(
- tool: Tool(
- name: "type_text_chars",
- description: """
- Type `text` one character at a time, delivered directly to
- the target pid's event queue via `CGEvent.postToPid`. Each
- character is posted as a synthesized key-down/key-up pair
- whose Unicode payload is set via
- `CGEventKeyboardSetUnicodeString`, bypassing virtual-key
- mapping so accents, symbols, and emoji transmit verbatim.
-
- Use this when the AX-based `type_text` silently drops
- characters — typical for Chromium / Electron text inputs
- that don't expose `kAXSelectedText`. The target does NOT
- need to be frontmost; keyboard focus within the target pid
- determines where characters land, so focus the receiving
- element first (e.g. via `click` on the input).
-
- `delay_ms` (0-200) spaces successive characters so
- autocomplete and IME paths can keep up. Default 30.
- """,
- inputSchema: [
- "type": "object",
- "required": ["pid", "text"],
- "properties": [
- "pid": [
- "type": "integer",
- "description": "Target process ID.",
- ],
- "text": [
- "type": "string",
- "description": "Text to type into the target's focused element.",
- ],
- "delay_ms": [
- "type": "integer",
- "minimum": 0,
- "maximum": 200,
- "description":
- "Milliseconds between successive characters. Default 30.",
- ],
- ],
- "additionalProperties": false,
- ],
- annotations: .init(
- readOnlyHint: false,
- destructiveHint: true,
- idempotentHint: false,
- openWorldHint: true
- )
- ),
- invoke: { arguments in
- guard let rawPid = arguments?["pid"]?.intValue else {
- return errorResult("Missing required integer field pid.")
- }
- guard let text = arguments?["text"]?.stringValue else {
- return errorResult("Missing required string field text.")
- }
- let delayMs = arguments?["delay_ms"]?.intValue ?? 30
- guard let pid = Int32(exactly: rawPid) else {
- return errorResult(
- "pid \(rawPid) is outside the supported Int32 range.")
- }
-
- do {
- try KeyboardInput.typeCharacters(
- text,
- delayMilliseconds: delayMs,
- toPid: pid
- )
- let summary =
- "✅ Typed \(text.count) character(s) on pid \(rawPid) with \(delayMs)ms delay."
- return CallTool.Result(
- content: [.text(text: summary, annotations: nil, _meta: nil)]
- )
- } catch let error as KeyboardError {
- return errorResult(error.description)
- } catch {
- return errorResult("Unexpected error: \(error)")
- }
- }
- )
-
- struct Result: Codable, Sendable {
- let pid: Int
- let characterCount: Int
- let delayMilliseconds: Int
-
- private enum CodingKeys: String, CodingKey {
- case pid
- case characterCount = "character_count"
- case delayMilliseconds = "delay_ms"
- }
- }
-
- private static func errorResult(_ message: String) -> CallTool.Result {
- CallTool.Result(
- content: [.text(text: message, annotations: nil, _meta: nil)],
- isError: true
- )
- }
-}
diff --git a/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextTool.swift b/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextTool.swift
index 984474f842..16035c3599 100644
--- a/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextTool.swift
+++ b/libs/cua-driver/Sources/CuaDriverServer/Tools/TypeTextTool.swift
@@ -3,31 +3,28 @@ import Foundation
import MCP
/// Unified text-insertion primitive — always targets a specific pid.
-/// Previously there were two tools (`type_text` which wrote to whatever
-/// was system-focused, and `type_text_in` which targeted a pid + element).
-/// The system-focused variant was a footgun because any driver-backgrounded
-/// app that triggered it would write characters into the user's real
-/// foreground app. Merging them under a mandatory-pid shape removes the
-/// footgun; the old `type_text_in` is gone.
///
-/// `element_index` is optional. When present, the element is looked up
-/// from the last `get_window_state` snapshot and focused before the write —
-/// the canonical path for "fill this specific text field." When absent,
-/// the write targets whatever element currently has focus within the
-/// target pid's AX tree (`AXUIElementCreateApplication(pid)` +
-/// `AXFocusedUIElement`), which is the cheaper path when focus was
-/// already established by a prior click.
+/// Tries `AXSetAttribute(kAXSelectedText)` first (fast, bulk insert).
+/// If the target element rejects the AX write (Chromium / Electron inputs
+/// that don't expose `kAXSelectedText`), automatically falls back to
+/// `KeyboardInput.typeCharacters` — character-by-character CGEvent synthesis
+/// via `CGEvent.postToPid`. The fallback path is noted in the response summary
+/// so callers can tell which path was taken.
public enum TypeTextTool {
public static let handler = ToolHandler(
tool: Tool(
name: "type_text",
description: """
- Insert text into the target pid via
- `AXSetAttribute(kAXSelectedText)`. Works for standard Cocoa
- text fields and text views. No keystrokes are synthesized —
- special keys (Return / Escape / arrows) go through
- `press_key` / `hotkey`. For Chromium / Electron inputs that
- don't implement `kAXSelectedText`, use `type_text_chars`.
+ Insert text into the target pid. Tries
+ `AXSetAttribute(kAXSelectedText)` first (fast bulk insert —
+ works for standard Cocoa text fields). If the target element
+ rejects the AX write, automatically falls back to
+ character-by-character `CGEvent.postToPid` synthesis —
+ reaches Chromium / Electron inputs and any surface that
+ doesn't implement `kAXSelectedText`.
+
+ Special keys (Return, Escape, arrows, Tab) go through
+ `press_key` / `hotkey` — they are not text.
Optional `element_index` + `window_id` (from the last
`get_window_state` snapshot of that window) pre-focuses
@@ -36,8 +33,11 @@ public enum TypeTextTool {
targets the pid's currently-focused element — useful
after a prior click already set focus.
- Requires Accessibility. Returns isError=true when the
- target element has no focus / rejects the attribute write.
+ `delay_ms` (0–200, default 30) spaces successive characters
+ in the CGEvent fallback path so autocomplete and IME can keep
+ up; ignored when the AX path succeeds.
+
+ Requires Accessibility.
""",
inputSchema: [
"type": "object",
@@ -61,6 +61,13 @@ public enum TypeTextTool {
"description":
"CGWindowID for the window whose get_window_state produced the element_index. Required when element_index is used.",
],
+ "delay_ms": [
+ "type": "integer",
+ "minimum": 0,
+ "maximum": 200,
+ "description":
+ "Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds.",
+ ],
],
"additionalProperties": false,
],
@@ -80,6 +87,7 @@ public enum TypeTextTool {
}
let elementIndex = arguments?["element_index"]?.intValue
let rawWindowId = arguments?["window_id"]?.intValue
+ let delayMs = arguments?["delay_ms"]?.intValue ?? 30
guard let pid = Int32(exactly: rawPid) else {
return errorResult(
"pid \(rawPid) is outside the supported Int32 range.")
@@ -91,6 +99,9 @@ public enum TypeTextTool {
+ "the same window_id you used in `get_window_state`.")
}
+ // Attempt AX bulk-insert. On any AXInputError fall back to
+ // CGEvent character synthesis, which reaches Chromium / Electron
+ // inputs that don't expose kAXSelectedText.
do {
if let index = elementIndex, let rawWindowId {
guard let windowId = UInt32(exactly: rawWindowId) else {
@@ -101,38 +112,58 @@ public enum TypeTextTool {
pid: pid,
windowId: windowId,
elementIndex: index)
- try await AppStateRegistry.focusGuard.withFocusSuppressed(
- pid: pid, element: element
- ) {
+ do {
+ try await AppStateRegistry.focusGuard.withFocusSuppressed(
+ pid: pid, element: element
+ ) {
+ try AXInput.setAttribute(
+ "AXSelectedText",
+ on: element,
+ value: text as CFTypeRef
+ )
+ }
+ let target = AXInput.describe(element)
+ let summary =
+ "✅ Inserted \(text.count) char(s) into [\(index)] \(target.role ?? "?") \"\(target.title ?? "")\" on pid \(rawPid) via AX."
+ return CallTool.Result(
+ content: [.text(text: summary, annotations: nil, _meta: nil)]
+ )
+ } catch is AXInputError {
+ // AX rejected — fall through to CGEvent synthesis.
+ }
+ } else {
+ do {
+ let element = try AXInput.focusedElement(pid: pid)
try AXInput.setAttribute(
"AXSelectedText",
on: element,
value: text as CFTypeRef
)
+ let target = AXInput.describe(element)
+ let summary =
+ "✅ Inserted \(text.count) char(s) into focused \(target.role ?? "?") \"\(target.title ?? "")\" on pid \(rawPid) via AX."
+ return CallTool.Result(
+ content: [.text(text: summary, annotations: nil, _meta: nil)]
+ )
+ } catch is AXInputError {
+ // AX rejected — fall through to CGEvent synthesis.
}
- let target = AXInput.describe(element)
- let summary =
- "✅ Inserted \(text.count) char(s) into [\(index)] \(target.role ?? "?") \"\(target.title ?? "")\" on pid \(rawPid)."
- return CallTool.Result(
- content: [.text(text: summary, annotations: nil, _meta: nil)]
- )
- } else {
- let element = try AXInput.focusedElement(pid: pid)
- try AXInput.setAttribute(
- "AXSelectedText",
- on: element,
- value: text as CFTypeRef
- )
- let target = AXInput.describe(element)
- let summary =
- "✅ Inserted \(text.count) char(s) into focused \(target.role ?? "?") \"\(target.title ?? "")\" on pid \(rawPid)."
- return CallTool.Result(
- content: [.text(text: summary, annotations: nil, _meta: nil)]
- )
}
+
+ // CGEvent fallback path.
+ try KeyboardInput.typeCharacters(
+ text,
+ delayMilliseconds: delayMs,
+ toPid: pid
+ )
+ let summary =
+ "✅ Typed \(text.count) char(s) on pid \(rawPid) via CGEvent (AX fallback, \(delayMs)ms delay)."
+ return CallTool.Result(
+ content: [.text(text: summary, annotations: nil, _meta: nil)]
+ )
} catch let error as AppStateError {
return errorResult(error.description)
- } catch let error as AXInputError {
+ } catch let error as KeyboardError {
return errorResult(error.description)
} catch {
return errorResult("Unexpected error: \(error)")