Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@

- transport 是 stdio 上的 line-framed JSON-RPC 2.0:一行一个 JSON value,UTF-8,`\n` 结尾,没有 `Content-Length`。stdout 只承载 JSON-RPC,诊断一律走 stderr。
- 两层错误刻意分开:JSON-RPC `error` 只表示请求本身不可用(`-32000` 版本不匹配、`-32001` 未握手、`-32002` 未知 session、`-32003` 正在退出);世界的状态一律是 `result` 里的 `{ ok: false, error: { code, message, detail } }`,`code` 是闭集,`message` 由 `code` 决定,`detail` 只有枚举和数字。
- 当前 method:`host.hello`、`session.begin` / `session.end`、`observe`、`window.list`、`apps.list`、`permissions.check`、`apps.launch`、`dispatch.element` / `dispatch.point` / `dispatch.key`、`screen.capture`,以及 `$/cancel` notification。`capture.start` / `capture.next` / `capture.stop` 已预留,v1 一律返回 domain 结果 `not_implemented`,而不是 `-32601`——这样 feature detection 是一次稳定的字段读取,方法名也不会被别的东西占掉。
- 当前 method:`host.hello`、`session.begin` / `session.end`、`observe`、`window.list`、`apps.list`、`permissions.check`、`apps.launch`、`dispatch.element` / `dispatch.key`、`screen.capture`,以及 `$/cancel` notification。`dispatch.point` 仅作为旧 host 的兼容拒绝端点保留,固定返回 `unsupported_action`,不会解析 snapshot、读取目标或发送输入。`capture.start` / `capture.next` / `capture.stop` 已预留,v1 一律返回 domain 结果 `not_implemented`,而不是 `-32601`——这样 feature detection 是一次稳定的字段读取,方法名也不会被别的东西占掉。
- 握手带回 executor 版本、capabilities 和全部 limits。host 不允许硬编码 executor 负责执行的边界,因为过去两侧各写一份、谁都发现不了漂移。
- 并发按 lane 组织:`control`、`target:<pid>:<windowId>`、`misc`、`capture:<streamId>`。同 lane 严格 FIFO,跨 lane 并发;所以同一窗口的 dispatch 永远不会超过产出它 snapshot 的 observe,而未来的 capture 长轮询也不会挡住 dispatch。
- 每个 snapshot id 都带一个 128-bit 的 per-process nonce。executor 重启后,上一代的 id 一定 `snapshot_unknown`,不会静默命中新状态。
Expand All @@ -56,7 +56,7 @@

### 3. Service 层

- host protocol 自己实现 observe 与 dispatch:`observe` 产出结构化的 AX 树 + element token,`dispatch.*` 只对 token 指名的那个元素执行指名的那个动作。它不复用 `ComputerUseService` 的 index 定址入口——把协议接到 index 上就等于把协议存在的理由接回来了。
- host protocol 自己实现 observe 与 dispatch:`observe` 产出结构化的 AX 树 + element token,可执行 mutation 通过 `dispatch.element` 或绑定已验证焦点的 `dispatch.key` 完成。生产动作面不接受坐标输入,也不把 semantic action 降级成 point event。它不复用 `ComputerUseService` 的 index 定址入口——把协议接到 index 上就等于把协议存在的理由接回来了。
- mutation 消费 snapshot 的 dispatch authority,但 spent snapshot 的 revision 仍可作为下一次 post-action observation 的差分基线;expired/evicted snapshot 不参与。这样动作后的模型输入可以只写有效变化,又不放宽 single-use snapshot 合同。
- `{kind: "app"}` 按 window inventory 的前到后顺序选择当前 sheet/窗口;`{kind: "window"}` 严格按 PID + window ID。AppKit sheet 在 CGWindow 侧是独立窗口、在 AX 侧是主窗口的 `AXSheet` / `AXDrawer` child,匹配顺序固定为 direct AXWindow 后 child sheet。
- CGWindow 已出现但对应 AXWindow/AXSheet 尚未发布时,会对同一 PID/window/frame 最多重读 250ms;window inventory 本身不存在目标时仍立即 `window_gone`,不会回退到别的窗口。
Expand Down Expand Up @@ -139,6 +139,7 @@
- 当前权限引导已经具备可运行 app、深链、拖拽辅助,以及一版更接近官方的 accessory panel 入场动画和返回 affordance;点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑,并且在 overlay 可见期间会持续重申“排在目标 window 之上”,避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住;但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。
- host protocol 的截图一律以文件路径返回,写在握手声明的 `imageDir` 里,生命周期与 snapshot 绑定;line-framed 通道上内联 base64 是 4/3 膨胀,而且一条 8 MB 的行会把其它待回的响应全部堵住。调试命令仍走 `ScreenCaptureKit` 捕获目标窗口,不再把普通 app 截图落盘到仓库或临时目录;编码前会按最大尺寸和目标字节数自适应缩小,避免复杂页面的大 PNG 触发 host 侧 MCP result 降级,同时 coordinate tools 继续按实际返回的 screenshot pixel 尺寸映射坐标;单次 ScreenCaptureKit capture 会设置超时,超时后省略 image block 而不是卡住整个 `get_app_state`。
- host protocol 的会话状态是进程内内存态:每个 session 持有自己的 snapshot 集合、element token 字典和保留的 `AXUIElement` 引用;`session.end` 会一次性释放 snapshot、删除本会话写出的图片,并清掉 executor 画的 cursor,同时把释放计数报回去,好让这类回归有断言可写。
- 本仓库旧 MCP/CLI 产品面仍有历史坐标 API;它和 Maka 的 `maka.cu/2` host protocol 是不同边界。Windows、macOS 和后续平台接入 Maka 时必须共享 semantic-only host contract,不能从旧 MCP/CLI schema 派生第二套 model action space 或 fallback ladder。

## 主要验证路径

Expand Down
128 changes: 48 additions & 80 deletions docs/HOST_PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,8 +197,7 @@ executor is already correct; the host is the side that changes.
"capabilities": {
"captureStream": false,
"elementActions": ["click", "set_value", "select_text", "secondary_action", "scroll"],
"pointActions": ["move", "left_click", "right_click", "middle_click", "double_click",
"triple_click", "mouse_down", "mouse_up", "drag", "scroll"],
"pointActions": [],
"keyActions": ["type", "key"],
"imageFormats": ["png", "jpeg"]
},
Expand Down Expand Up @@ -1044,11 +1043,9 @@ carries a `title`. A consumer that reads `label` alone gets an anonymous tree.
why it was already missing.

**`elements` and `menu.elements` are separate, and `windowDigest` is over
`elements` only.** The digest anchors `dispatch.point` and is recomputed on
every settle sample — one Accessibility round trip per recorded element, per
look — and a menu folded into it would make the same window digest differently
depending on whether menus had been asked for, while charging every settle for
elements that cannot change when the window does.
`elements` only.** A menu folded into it would make the same window digest
differently depending on whether menus had been asked for, while charging every
settle for elements that cannot change when the window does.

#### Scope, because a menu bar is not one size

Expand Down Expand Up @@ -1264,8 +1261,9 @@ command is not there, and stops looking.

#### `observeAfter.menu`

`dispatch.element`, `dispatch.point` and `dispatch.key` take `menu` in
`observeAfter` on the same terms, absent meaning `false`. It exists because a
`dispatch.element` and `dispatch.key` take `menu` in `observeAfter` on the same
terms, absent meaning `false`. The legacy `dispatch.point` shape still parses
the field but never performs a post-action observation. The field exists because a
menu press changes what the rest of the menu will do: `文件 > 打开…` brings a
document up, and `存储`, `导出为PDF…` and `关闭` all move from disabled to
enabled with it. Without it the host would have to spend a second `observe` to
Expand Down Expand Up @@ -1698,49 +1696,41 @@ became expressible when refusals started carrying `path` (§1.1).
} }
```

- `space` is `"image_px"`. It is required and single-valued: a required field
with one legal value is how a second space gets added later without either side
guessing which one it was handed.
- `expectWindowDigest` is required. A point has no element to anchor to, so the
whole window is the anchor — which is what the host already does for coordinate
actions (`cua-driver-target-resolution.ts:336-365`). The executor MUST
**recompute** that digest against the live window before dispatching, and MUST
refuse `window_changed` when it differs; comparing the echo against its own
record and stopping there checks the host against itself, and inside the TTL
the click goes to whatever the window has become — a resize rescales the point
silently, because the screen point is derived from the *current* bounds.
The recompute is over the elements the snapshot recorded, read the one way
§4.3 requires. This is the whole of point dispatch's binding, so an executor
whose two ends disagree by one field on one element does not lose an edge case:
it refuses every point dispatch ever made against it, and answers
`window_changed` for a window sitting still. See §12 vector 52.
- `occlusionPolicy` defaults to `"any"` here, not `"same_app"`. A pixel is a
pixel: anything on top of it owns it.
- `startPoint` is present only for `drag`.

**Path selection is declared, not discovered.** `path` in the response is one of:
This method is retained only as a compatibility endpoint for a host that still
sends the old request shape. Point mutation is not an executor capability:
`host.hello.capabilities.pointActions` is always empty.

After the normal handshake and session checks, every structurally valid request
returns:

```json
{ "ok": false, "toolCallId": "call_2",
"outcome": "refused", "tier": "coordinate-background", "path": "none",
"effect": "unverifiable",
"verification": { "method": "none", "observedChange": false },
"error": { "code": "unsupported_action",
"message": "The requested action is not supported by this executor." } }
```

The executor MUST NOT resolve or consume `snapshotId`, compare
`expectWindowDigest`, inspect window inventory, choose a dispatch path, post an
input event, or perform `observeAfter`. A stale, missing, or mismatched target
therefore does not replace `unsupported_action` with a target error. This keeps
old hosts on a typed domain refusal while making coordinate execution
unavailable on every platform.

The paths used by executable dispatch methods are:

| `path` | mechanism | permitted when |
| --- | --- | --- |
| `ax_action` | `AXUIElementPerformAction` on the element under the point | always |
| `ax_action` | `AXUIElementPerformAction` on the bound element | always |
| `ax_attribute` | `AXUIElementSetAttributeValue` | always |
| `ax_select` | set `AXSelectedChildren` on the containing list | always |
| `cg_event_pid` | `CGEventPostToPid` — target-bound, no cursor warp | always |
| `skylight_pid` | `SLEventPostToPid` — background window path, including WebContent-aware host-window routing | always |
| `cg_event_global` | `CGEventPost` — **moves the system cursor** | only when `allowGlobalPointer: true` |
| `cg_event_pid` | PID-bound key or element-scroll fallback | `dispatch.key` or bound `dispatch.element` only |
| `skylight_pid` | bound WebContent element activation | `dispatch.element` only |
| `cg_event_global` | global pointer dispatch path | never permitted by Maka |
| `none` | nothing was dispatched | refusals |

When `allowGlobalPointer` is `false` and no permitted path can reach the target,
the executor MUST return `dispatch_refused` with
`detail: { "wouldRequirePath": "cg_event_global" }`. It MUST NOT fall back. This
is the invariant Maka refuses to trade: no cursor warp, no z-order change. The
current backend enforces it by refusing when no app window owns the click point
(`cua-driver-backend.ts:1954-1963`) — a check that only works because the host
knows which driver path a pid-bound click takes. Under this protocol the executor
states the path and the host verifies it: a response whose `path` was not
permitted is a protocol violation, and the host MUST treat the session as
compromised rather than accept the result.

`tier` and `path` are both declared, and their pairing is fixed:

| `tier` | permitted `path` |
Expand Down Expand Up @@ -2165,12 +2155,10 @@ Which method belongs to which action, in full:
| | `dispatch.key` | `kind: "type"` |
| `selection_readback` | `dispatch.element` | `select_text` |
| `tree_delta` | `dispatch.element` | `click`, `scroll` — with `settle: "quiesce"` |
| | `dispatch.point` | every action — with `settle: "quiesce"` |
| | `dispatch.key` | `kind: "key"` — with `settle: "quiesce"` |
| `action_result` | `dispatch.element` | `secondary_action`, and `click`/`scroll` without a settle |
| | `dispatch.point` | every action without a settle |
| `none` | `dispatch.key` | `kind: "key"` without a settle |
| | any | every refusal (§6.5), and a readback with nothing on either side to read |
| | any | every refusal (§6.5), including every `dispatch.point`, and a readback with nothing on either side to read |

`none` is where a key without a settle lands rather than `action_result` because
there is no result to report: the events were written to the target pid and
Expand Down Expand Up @@ -2258,10 +2246,6 @@ empty image at half the resolution its `scale` claims, and a host that mirrors
the frame renders the empty part as a black margin — which is how this was
found, two layers away from the cause.

Every pixel statement in the protocol rests on this. `image_px` in §6.3 is read
from the image's origin, so a frame whose content is drawn at a different scale
than it declares puts every point dispatch off by the ratio between the two.

The executor therefore sizes the output buffer from the same source it renders
from, so the two cannot disagree, and measures `scale` from the bitmap it got
back (§5.3). Neither number is chosen twice.
Expand Down Expand Up @@ -2312,8 +2296,7 @@ gets the same answer — a new member.
What the host does with it:

- Surfaces it to the model as an outcome, with the executor's `detail` (enums and
numbers only, §1.2) as evidence — including `wouldRequirePath` when nothing was
attempted (§6.3).
numbers only, §1.2) as evidence.
- Does **not** re-observe automatically. A refused dispatch leaves the frame live
(§4.1), so the model may retry against the same frame with different arguments.
- Does not treat it as a permission problem. `permission_missing` is a separate
Expand Down Expand Up @@ -2598,9 +2581,9 @@ Declared schema:
`suspected_noop`, not `ok`.
13. `click` with `settle: "none"` never reports `effect: "confirmed"` via
`tree_delta`.
14. With `allowGlobalPointer: false`, a target reachable only by
`cg_event_global` yields `dispatch_refused` with
`wouldRequirePath: "cg_event_global"` — and the system cursor does not move.
14. `dispatch.point` returns `unsupported_action`, `path: "none"` and
`verification.method: "none"` without resolving or consuming the snapshot,
inspecting a window, or posting an event.

Versioning:

Expand Down Expand Up @@ -2788,29 +2771,14 @@ Observing a window that is expensive to read (§5.2):
is per-attempt rather than per-observation passes the unit half and fails
this one, because §7.5 walks the tree up to four times.

Dispatching a point at the frame just observed (§4.3, §6.3):

52. `observe` a window, then immediately `dispatch.point` against the snapshot it
returned: the answer is not `window_changed`. Nothing moved between the two
calls, so the anchor the observation recorded must still recompute to the
same bytes.

The vector that fails against an executor whose two ends of §4.3 read one
field of one element differently. Measured: the walk recorded the snapshot
root's ancestor chain live as `["AXApplication"]` while the binding probe
answered `[]` for the root, so 1 element of 65 differed, the window digest
differed, and `dispatch.point` refused `window_changed` on every call against
every application on both displays. No element dispatch could see it —
`strictness: "element"` checks the element it targets — so an executor can
hold this defect with a full element matrix passing.

Its unit half puts a tree in front of the walk whose nodes report a live
parent chain, and asserts the root records neither an ancestor nor a sibling
index; an executor that answers the binding probe from its own record rather
than by recomputing passes every other point vector and cannot fail this one,
which is why the vector also has a live half. The live half needs a window
that does not change on its own — against a window with a clock in it,
`window_changed` is the correct answer and the vector proves nothing.
Refusing point mutation (§6.3):

52. Begin a valid session, install a live snapshot, then send `dispatch.point`
with a deliberately wrong `expectWindowDigest`, an empty live window
inventory, and an event sink configured to fail if called. The answer is the
stable `unsupported_action` refusal, no event is posted, and the snapshot
remains live. This proves the compatibility endpoint does not inspect or
mutate the target before refusing.

Judging a key by something that bears on it (§6.5):

Expand Down
37 changes: 37 additions & 0 deletions docs/histories/2026-09/20260902-1200-reject-point-dispatch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
## [2026-09-02 12:00] | Task: Reject host point dispatch

### Execution Context

- Agent: Codex
- Runtime: Codex desktop

### User Query

> Make the Maka Computer Use execution boundary semantic-only while retaining a typed compatibility response for older hosts.

### Changes

- `host.hello` now advertises an empty `pointActions` capability.
- `dispatch.point` remains parseable but always returns `unsupported_action`.
- The refusal does not resolve or consume a snapshot, inspect a window, select a path, post an event, or run post-action observation.
- Removed the unused host-protocol point path selector and obsolete point-success tests.
- Updated the protocol and architecture documents to distinguish the semantic Maka host boundary from the repository's legacy MCP/CLI coordinate APIs.

### Design Intent

Maka exposes one model action space across platforms. Native executors implement revision-bound AX/UIA semantic actions and verified keyboard targeting; they do not add platform-specific coordinate actions or silently fall back from semantic intent to pixels.

### Verification

- `swift test --filter HostDispatchTests`
- `swift test`

### Files Modified

- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift`
- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift`
- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostDispatchPolicy.swift`
- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift`
- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift`
- `docs/HOST_PROTOCOL.md`
- `docs/ARCHITECTURE.md`
Loading