diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 7e3567a..807c91d 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -47,7 +47,7 @@ - transport 是 stdio 上的 line-framed JSON-RPC 2.0：一行一个 JSON value，UTF-8，`\n` 结尾，没有 `Content-Length`。stdout 只承载 JSON-RPC，诊断一律走 stderr。 - 两层错误刻意分开：JSON-RPC `error` 只表示请求本身不可用（`-32000` 版本不匹配、`-32001` 未握手、`-32002` 未知 session、`-32003` 正在退出）；世界的状态一律是 `result` 里的 `{ ok: false, error: { code, message, detail } }`，`code` 是闭集，`message` 由 `code` 决定，`detail` 只有枚举和数字。 -- 当前 method：`host.hello`、`session.begin` / `session.end`、`observe`、`window.list`、`apps.list`、`permissions.check`、`apps.launch`、`dispatch.element` / `dispatch.point` / `dispatch.key`、`screen.capture`，以及 `$/cancel` notification。`capture.start` / `capture.next` / `capture.stop` 已预留，v1 一律返回 domain 结果 `not_implemented`，而不是 `-32601`——这样 feature detection 是一次稳定的字段读取，方法名也不会被别的东西占掉。 +- 当前 method：`host.hello`、`session.begin` / `session.end`、`observe`、`window.list`、`apps.list`、`permissions.check`、`apps.launch`、`dispatch.element` / `dispatch.key`、`screen.capture`，以及 `$/cancel` notification。`dispatch.point` 仅作为旧 host 的兼容拒绝端点保留，固定返回 `unsupported_action`，不会解析 snapshot、读取目标或发送输入。`capture.start` / `capture.next` / `capture.stop` 已预留，v1 一律返回 domain 结果 `not_implemented`，而不是 `-32601`——这样 feature detection 是一次稳定的字段读取，方法名也不会被别的东西占掉。 - 握手带回 executor 版本、capabilities 和全部 limits。host 不允许硬编码 executor 负责执行的边界，因为过去两侧各写一份、谁都发现不了漂移。 - 并发按 lane 组织：`control`、`target::`、`misc`、`capture:`。同 lane 严格 FIFO，跨 lane 并发；所以同一窗口的 dispatch 永远不会超过产出它 snapshot 的 observe，而未来的 capture 长轮询也不会挡住 dispatch。 - 每个 snapshot id 都带一个 128-bit 的 per-process nonce。executor 重启后，上一代的 id 一定 `snapshot_unknown`，不会静默命中新状态。 @@ -56,7 +56,7 @@ ### 3. Service 层 -- host protocol 自己实现 observe 与 dispatch：`observe` 产出结构化的 AX 树 + element token，`dispatch.*` 只对 token 指名的那个元素执行指名的那个动作。它不复用 `ComputerUseService` 的 index 定址入口——把协议接到 index 上就等于把协议存在的理由接回来了。 +- host protocol 自己实现 observe 与 dispatch：`observe` 产出结构化的 AX 树 + element token，可执行 mutation 通过 `dispatch.element` 或绑定已验证焦点的 `dispatch.key` 完成。生产动作面不接受坐标输入，也不把 semantic action 降级成 point event。它不复用 `ComputerUseService` 的 index 定址入口——把协议接到 index 上就等于把协议存在的理由接回来了。 - mutation 消费 snapshot 的 dispatch authority，但 spent snapshot 的 revision 仍可作为下一次 post-action observation 的差分基线；expired/evicted snapshot 不参与。这样动作后的模型输入可以只写有效变化，又不放宽 single-use snapshot 合同。 - `{kind: "app"}` 按 window inventory 的前到后顺序选择当前 sheet/窗口；`{kind: "window"}` 严格按 PID + window ID。AppKit sheet 在 CGWindow 侧是独立窗口、在 AX 侧是主窗口的 `AXSheet` / `AXDrawer` child，匹配顺序固定为 direct AXWindow 后 child sheet。 - CGWindow 已出现但对应 AXWindow/AXSheet 尚未发布时，会对同一 PID/window/frame 最多重读 250ms；window inventory 本身不存在目标时仍立即 `window_gone`，不会回退到别的窗口。 @@ -139,6 +139,7 @@ - 当前权限引导已经具备可运行 app、深链、拖拽辅助，以及一版更接近官方的 accessory panel 入场动画和返回 affordance；点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑，并且在 overlay 可见期间会持续重申“排在目标 window 之上”，避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住；但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。 - host protocol 的截图一律以文件路径返回，写在握手声明的 `imageDir` 里，生命周期与 snapshot 绑定；line-framed 通道上内联 base64 是 4/3 膨胀，而且一条 8 MB 的行会把其它待回的响应全部堵住。调试命令仍走 `ScreenCaptureKit` 捕获目标窗口，不再把普通 app 截图落盘到仓库或临时目录；编码前会按最大尺寸和目标字节数自适应缩小，避免复杂页面的大 PNG 触发 host 侧 MCP result 降级，同时 coordinate tools 继续按实际返回的 screenshot pixel 尺寸映射坐标；单次 ScreenCaptureKit capture 会设置超时，超时后省略 image block 而不是卡住整个 `get_app_state`。 - host protocol 的会话状态是进程内内存态：每个 session 持有自己的 snapshot 集合、element token 字典和保留的 `AXUIElement` 引用；`session.end` 会一次性释放 snapshot、删除本会话写出的图片，并清掉 executor 画的 cursor，同时把释放计数报回去，好让这类回归有断言可写。 +- 本仓库旧 MCP/CLI 产品面仍有历史坐标 API；它和 Maka 的 `maka.cu/2` host protocol 是不同边界。Windows、macOS 和后续平台接入 Maka 时必须共享 semantic-only host contract，不能从旧 MCP/CLI schema 派生第二套 model action space 或 fallback ladder。 ## 主要验证路径 diff --git a/docs/HOST_PROTOCOL.md b/docs/HOST_PROTOCOL.md index 8f70c06..c81c60f 100644 --- a/docs/HOST_PROTOCOL.md +++ b/docs/HOST_PROTOCOL.md @@ -197,8 +197,7 @@ executor is already correct; the host is the side that changes. "capabilities": { "captureStream": false, "elementActions": ["click", "set_value", "select_text", "secondary_action", "scroll"], - "pointActions": ["move", "left_click", "right_click", "middle_click", "double_click", - "triple_click", "mouse_down", "mouse_up", "drag", "scroll"], + "pointActions": [], "keyActions": ["type", "key"], "imageFormats": ["png", "jpeg"] }, @@ -1044,11 +1043,9 @@ carries a `title`. A consumer that reads `label` alone gets an anonymous tree. why it was already missing. **`elements` and `menu.elements` are separate, and `windowDigest` is over -`elements` only.** The digest anchors `dispatch.point` and is recomputed on -every settle sample — one Accessibility round trip per recorded element, per -look — and a menu folded into it would make the same window digest differently -depending on whether menus had been asked for, while charging every settle for -elements that cannot change when the window does. +`elements` only.** A menu folded into it would make the same window digest +differently depending on whether menus had been asked for, while charging every +settle for elements that cannot change when the window does. #### Scope, because a menu bar is not one size @@ -1264,8 +1261,9 @@ command is not there, and stops looking. #### `observeAfter.menu` -`dispatch.element`, `dispatch.point` and `dispatch.key` take `menu` in -`observeAfter` on the same terms, absent meaning `false`. It exists because a +`dispatch.element` and `dispatch.key` take `menu` in `observeAfter` on the same +terms, absent meaning `false`. The legacy `dispatch.point` shape still parses +the field but never performs a post-action observation. The field exists because a menu press changes what the rest of the menu will do: `文件 > 打开…` brings a document up, and `存储`, `导出为PDF…` and `关闭` all move from disabled to enabled with it. Without it the host would have to spend a second `observe` to @@ -1698,49 +1696,41 @@ became expressible when refusals started carrying `path` (§1.1). } } ``` -- `space` is `"image_px"`. It is required and single-valued: a required field - with one legal value is how a second space gets added later without either side - guessing which one it was handed. -- `expectWindowDigest` is required. A point has no element to anchor to, so the - whole window is the anchor — which is what the host already does for coordinate - actions (`cua-driver-target-resolution.ts:336-365`). The executor MUST - **recompute** that digest against the live window before dispatching, and MUST - refuse `window_changed` when it differs; comparing the echo against its own - record and stopping there checks the host against itself, and inside the TTL - the click goes to whatever the window has become — a resize rescales the point - silently, because the screen point is derived from the *current* bounds. - The recompute is over the elements the snapshot recorded, read the one way - §4.3 requires. This is the whole of point dispatch's binding, so an executor - whose two ends disagree by one field on one element does not lose an edge case: - it refuses every point dispatch ever made against it, and answers - `window_changed` for a window sitting still. See §12 vector 52. -- `occlusionPolicy` defaults to `"any"` here, not `"same_app"`. A pixel is a - pixel: anything on top of it owns it. -- `startPoint` is present only for `drag`. - -**Path selection is declared, not discovered.** `path` in the response is one of: +This method is retained only as a compatibility endpoint for a host that still +sends the old request shape. Point mutation is not an executor capability: +`host.hello.capabilities.pointActions` is always empty. + +After the normal handshake and session checks, every structurally valid request +returns: + +```json +{ "ok": false, "toolCallId": "call_2", + "outcome": "refused", "tier": "coordinate-background", "path": "none", + "effect": "unverifiable", + "verification": { "method": "none", "observedChange": false }, + "error": { "code": "unsupported_action", + "message": "The requested action is not supported by this executor." } } +``` + +The executor MUST NOT resolve or consume `snapshotId`, compare +`expectWindowDigest`, inspect window inventory, choose a dispatch path, post an +input event, or perform `observeAfter`. A stale, missing, or mismatched target +therefore does not replace `unsupported_action` with a target error. This keeps +old hosts on a typed domain refusal while making coordinate execution +unavailable on every platform. + +The paths used by executable dispatch methods are: | `path` | mechanism | permitted when | | --- | --- | --- | -| `ax_action` | `AXUIElementPerformAction` on the element under the point | always | +| `ax_action` | `AXUIElementPerformAction` on the bound element | always | | `ax_attribute` | `AXUIElementSetAttributeValue` | always | | `ax_select` | set `AXSelectedChildren` on the containing list | always | -| `cg_event_pid` | `CGEventPostToPid` — target-bound, no cursor warp | always | -| `skylight_pid` | `SLEventPostToPid` — background window path, including WebContent-aware host-window routing | always | -| `cg_event_global` | `CGEventPost` — **moves the system cursor** | only when `allowGlobalPointer: true` | +| `cg_event_pid` | PID-bound key or element-scroll fallback | `dispatch.key` or bound `dispatch.element` only | +| `skylight_pid` | bound WebContent element activation | `dispatch.element` only | +| `cg_event_global` | global pointer dispatch path | never permitted by Maka | | `none` | nothing was dispatched | refusals | -When `allowGlobalPointer` is `false` and no permitted path can reach the target, -the executor MUST return `dispatch_refused` with -`detail: { "wouldRequirePath": "cg_event_global" }`. It MUST NOT fall back. This -is the invariant Maka refuses to trade: no cursor warp, no z-order change. The -current backend enforces it by refusing when no app window owns the click point -(`cua-driver-backend.ts:1954-1963`) — a check that only works because the host -knows which driver path a pid-bound click takes. Under this protocol the executor -states the path and the host verifies it: a response whose `path` was not -permitted is a protocol violation, and the host MUST treat the session as -compromised rather than accept the result. - `tier` and `path` are both declared, and their pairing is fixed: | `tier` | permitted `path` | @@ -2165,12 +2155,10 @@ Which method belongs to which action, in full: | | `dispatch.key` | `kind: "type"` | | `selection_readback` | `dispatch.element` | `select_text` | | `tree_delta` | `dispatch.element` | `click`, `scroll` — with `settle: "quiesce"` | -| | `dispatch.point` | every action — with `settle: "quiesce"` | | | `dispatch.key` | `kind: "key"` — with `settle: "quiesce"` | | `action_result` | `dispatch.element` | `secondary_action`, and `click`/`scroll` without a settle | -| | `dispatch.point` | every action without a settle | | `none` | `dispatch.key` | `kind: "key"` without a settle | -| | any | every refusal (§6.5), and a readback with nothing on either side to read | +| | any | every refusal (§6.5), including every `dispatch.point`, and a readback with nothing on either side to read | `none` is where a key without a settle lands rather than `action_result` because there is no result to report: the events were written to the target pid and @@ -2258,10 +2246,6 @@ empty image at half the resolution its `scale` claims, and a host that mirrors the frame renders the empty part as a black margin — which is how this was found, two layers away from the cause. -Every pixel statement in the protocol rests on this. `image_px` in §6.3 is read -from the image's origin, so a frame whose content is drawn at a different scale -than it declares puts every point dispatch off by the ratio between the two. - The executor therefore sizes the output buffer from the same source it renders from, so the two cannot disagree, and measures `scale` from the bitmap it got back (§5.3). Neither number is chosen twice. @@ -2312,8 +2296,7 @@ gets the same answer — a new member. What the host does with it: - Surfaces it to the model as an outcome, with the executor's `detail` (enums and - numbers only, §1.2) as evidence — including `wouldRequirePath` when nothing was - attempted (§6.3). + numbers only, §1.2) as evidence. - Does **not** re-observe automatically. A refused dispatch leaves the frame live (§4.1), so the model may retry against the same frame with different arguments. - Does not treat it as a permission problem. `permission_missing` is a separate @@ -2598,9 +2581,9 @@ Declared schema: `suspected_noop`, not `ok`. 13. `click` with `settle: "none"` never reports `effect: "confirmed"` via `tree_delta`. -14. With `allowGlobalPointer: false`, a target reachable only by - `cg_event_global` yields `dispatch_refused` with - `wouldRequirePath: "cg_event_global"` — and the system cursor does not move. +14. `dispatch.point` returns `unsupported_action`, `path: "none"` and + `verification.method: "none"` without resolving or consuming the snapshot, + inspecting a window, or posting an event. Versioning: @@ -2788,29 +2771,14 @@ Observing a window that is expensive to read (§5.2): is per-attempt rather than per-observation passes the unit half and fails this one, because §7.5 walks the tree up to four times. -Dispatching a point at the frame just observed (§4.3, §6.3): - -52. `observe` a window, then immediately `dispatch.point` against the snapshot it - returned: the answer is not `window_changed`. Nothing moved between the two - calls, so the anchor the observation recorded must still recompute to the - same bytes. - - The vector that fails against an executor whose two ends of §4.3 read one - field of one element differently. Measured: the walk recorded the snapshot - root's ancestor chain live as `["AXApplication"]` while the binding probe - answered `[]` for the root, so 1 element of 65 differed, the window digest - differed, and `dispatch.point` refused `window_changed` on every call against - every application on both displays. No element dispatch could see it — - `strictness: "element"` checks the element it targets — so an executor can - hold this defect with a full element matrix passing. - - Its unit half puts a tree in front of the walk whose nodes report a live - parent chain, and asserts the root records neither an ancestor nor a sibling - index; an executor that answers the binding probe from its own record rather - than by recomputing passes every other point vector and cannot fail this one, - which is why the vector also has a live half. The live half needs a window - that does not change on its own — against a window with a clock in it, - `window_changed` is the correct answer and the vector proves nothing. +Refusing point mutation (§6.3): + +52. Begin a valid session, install a live snapshot, then send `dispatch.point` + with a deliberately wrong `expectWindowDigest`, an empty live window + inventory, and an event sink configured to fail if called. The answer is the + stable `unsupported_action` refusal, no event is posted, and the snapshot + remains live. This proves the compatibility endpoint does not inspect or + mutate the target before refusing. Judging a key by something that bears on it (§6.5): diff --git a/docs/histories/2026-09/20260902-1200-reject-point-dispatch.md b/docs/histories/2026-09/20260902-1200-reject-point-dispatch.md new file mode 100644 index 0000000..ee2b514 --- /dev/null +++ b/docs/histories/2026-09/20260902-1200-reject-point-dispatch.md @@ -0,0 +1,37 @@ +## [2026-09-02 12:00] | Task: Reject host point dispatch + +### Execution Context + +- Agent: Codex +- Runtime: Codex desktop + +### User Query + +> Make the Maka Computer Use execution boundary semantic-only while retaining a typed compatibility response for older hosts. + +### Changes + +- `host.hello` now advertises an empty `pointActions` capability. +- `dispatch.point` remains parseable but always returns `unsupported_action`. +- The refusal does not resolve or consume a snapshot, inspect a window, select a path, post an event, or run post-action observation. +- Removed the unused host-protocol point path selector and obsolete point-success tests. +- Updated the protocol and architecture documents to distinguish the semantic Maka host boundary from the repository's legacy MCP/CLI coordinate APIs. + +### Design Intent + +Maka exposes one model action space across platforms. Native executors implement revision-bound AX/UIA semantic actions and verified keyboard targeting; they do not add platform-specific coordinate actions or silently fall back from semantic intent to pixels. + +### Verification + +- `swift test --filter HostDispatchTests` +- `swift test` + +### Files Modified + +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostDispatchPolicy.swift` +- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift` +- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift` +- `docs/HOST_PROTOCOL.md` +- `docs/ARCHITECTURE.md` diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostDispatchPolicy.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostDispatchPolicy.swift index 9be6976..59d18ef 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostDispatchPolicy.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostDispatchPolicy.swift @@ -112,52 +112,11 @@ public enum HostKeyAction: Equatable, Sendable { case key(name: String, modifiers: [HostKeyModifier]) } -// MARK: - Path selection - -/// §6.3 — path selection is declared, not discovered. When `allowGlobalPointer` -/// is false and no permitted path can reach the target, the executor refuses; it -/// must never fall back. This is the invariant Maka refuses to trade: no cursor -/// warp, no z-order change. -public func hostPointDispatchPath( - action: HostPointAction, - point: CGPoint, - startPoint: CGPoint?, - windowBounds: CGRect, - allowGlobalPointer: Bool -) -> Result { - // A pointer *move* has no pid-bound form: the only way to make the cursor - // appear somewhere is to warp the system cursor, which is the one thing this - // executor will not do on Maka's behalf. - if action == .move { - return allowGlobalPointer - ? .success(.cgEventGlobal) - : .failure(HostDomainError(.dispatchRefused, detail: .wouldRequirePath(.cgEventGlobal))) - } - - guard windowBounds.contains(point) else { - return .failure(HostDomainError(.invalidPoint)) - } - - if action.needsStartPoint { - guard let startPoint else { - return .failure(HostDomainError(.invalidPoint)) - } - - // A drag that begins outside the target window crosses a window boundary, - // and only a global-tap drag is delivered to whichever window happens to - // be under each intermediate point. - guard windowBounds.contains(startPoint) else { - return allowGlobalPointer - ? .success(.cgEventGlobal) - : .failure(HostDomainError(.dispatchRefused, detail: .wouldRequirePath(.cgEventGlobal))) - } - } - - return .success(.cgEventPid) -} +// MARK: - Path validation -/// §6.3 — the host MUST reject an inconsistent `tier`/`path` pair as a protocol -/// violation rather than prefer one. The executor never emits one. +/// §6.5 — the host MUST reject an inconsistent `tier`/`path` pair as a protocol +/// violation rather than prefer one. `dispatch.point` never selects a path; its +/// compatibility refusal always reports `path: none`. public func hostTierIsConsistent(tier: HostDispatchTier, path: HostDispatchPath) -> Bool { guard let expected = path.tier else { // `path: none` only ever accompanies a refusal, which reports the tier it diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostElementBinding.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostElementBinding.swift index a18800d..fc319c3 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostElementBinding.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostElementBinding.swift @@ -72,10 +72,7 @@ public func hostNormalizedActions(_ rawAXActions: [String]) -> [HostElementActio /// `AXParent` on the other, which disagree on every Chromium tree. /// - `ancestorRoles` for the **root**: the walker read the live chain for every /// node, so a window recorded `["AXApplication"]`, while the probe answered -/// `[]`. One element out of sixty-five, and no element dispatch could see it — -/// it checks the element it targets. `dispatch.point` has no element to target, -/// anchors on the whole window, and so refused `window_changed` on every call -/// against every application. +/// `[]`. /// /// `depth == 0` is the snapshot's root, and inside the frame the root has no /// ancestors and no siblings: the walk is rooted at the window and never sees the @@ -197,8 +194,7 @@ public func hostElementDigest(_ input: HostElementDigestInput) -> String { return HostDigest.sha256(HostDigest.canonicalArray(parts)) } -/// §4.3 — the window digest is the anchor for point dispatch, which has no -/// element to bind to. +/// §4.3 — the canonical digest for the observed window. public func hostWindowDigest( elementDigests: [String], bounds: CGRect?, diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostImageStore.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostImageStore.swift index 08b3e8a..956c6c0 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostImageStore.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostImageStore.swift @@ -278,13 +278,9 @@ public enum HostCapture { return CGImage?.none } filter = SCContentFilter(display: display, excludingWindows: []) - // §5.3 / §6.3 — `image.scale` is measured against the target - // window and `dispatch.point` reads `image_px` from the - // window's origin, so a desktop-scope image has to be the - // window's rectangle *as composited* — everything stacked on - // top included. Handing back a display-origin crop the size of - // the window kept both fields but moved the pixels, and every - // point dispatch under this scope landed somewhere else. + // §5.3 — `image.scale` is measured against the target window, + // so a desktop-scope image has to be the window's rectangle + // as composited, including everything stacked on top. let sourceRect = hostDesktopSourceRect( windowFrame: window.frame, displayFrame: display.frame diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift index 7164f4b..54719ae 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift @@ -1546,146 +1546,14 @@ extension HostProtocolServer { return } - // §6.5 — a point dispatch would have used a coordinate path, so that is - // the tier a refusal reports even though nothing was dispatched. - func refuse(_ error: HostDomainError) { - emit(id: id, toolCallId: params.toolCallId, dispatchFailure: error, tier: .coordinateBackground) - } - - let registry = currentRegistry() - let snapshot: HostSnapshot - switch registry.resolve(session: params.session, snapshotId: params.snapshotId, now: hostNowMs()) { - case .success(let resolved): - snapshot = resolved - case .failure(let error): - refuse(error) - return - } - - // The echo is host bookkeeping, not evidence about the world: a digest - // that is not the one this snapshot recorded means the host paired a - // window digest with the wrong `snapshotId`. Reporting `window_changed` - // for it sent the host round the re-observe loop with the same wrong - // pairing, which is the collapse §6.2 separates one level down. - guard params.expectWindowDigest == snapshot.windowDigest else { - refuse(HostDomainError(.elementDigestMismatch)) - return - } - - if cancellations.isCancelledBeforeDispatch(id: id) { - refuse(HostDomainError(.aborted)) - return - } - - let windows = environment.onScreenWindows() - guard let window = windows.first(where: { $0.pid == snapshot.pid && $0.windowId == snapshot.windowId }) else { - refuse(HostDomainError(.windowGone)) - return - } - - // §6.3 — a point has no element to anchor to, so the whole window is the - // anchor, and an anchor is only an anchor if it is recomputed. Comparing - // the echo against the recorded digest and stopping there checked the - // host against itself: inside the TTL the click went to whatever the - // window had become, and because the screen point is derived from the - // *current* bounds a resize rescaled it silently. - let probe = environment.bindingProbe(windowBounds: window.bounds) - guard hostRecomputeWindowDigest(snapshot: snapshot, window: window, probe: probe) == snapshot.windowDigest else { - refuse(HostDomainError(.windowChanged)) - return - } - - // `image_px` is only meaningful against the image the quoted snapshot - // carried, and its measured scale is the only conversion we trust. - guard let scale = snapshot.payload.image?.scale, scale > 0 else { - refuse(HostDomainError(.invalidPoint)) - return - } - - let screenPoint = CGPoint( - x: window.bounds.minX + params.point.x / scale, - y: window.bounds.minY + params.point.y / scale - ) - let screenStart = params.startPoint.map { - CGPoint( - x: window.bounds.minX + $0.x / scale, - y: window.bounds.minY + $0.y / scale - ) - } - - let path: HostDispatchPath - switch hostPointDispatchPath( - action: params.action, - point: screenPoint, - startPoint: screenStart, - windowBounds: window.bounds, - allowGlobalPointer: globalPointerAllowed() - ) { - case .success(let selected): - path = selected - case .failure(let error): - refuse(error) - return - } - - let obscuring = windows - .filter { $0.layer == 0 && $0.zIndex > window.zIndex && $0.windowId != window.windowId } - .filter { !HostWindowInventory.isFullScreenDockSurface($0) } - .map { (pid: $0.pid, rect: $0.bounds) } - - // §6.3 — the default is `any` here, not `same_app`. A pixel is a pixel: - // anything on top of it owns it. - if hostIsOccluded( - policy: params.occlusionPolicy ?? .any, - targetPid: snapshot.pid, - targetPoint: screenPoint, - obscuringWindows: obscuring - ) { - refuse(HostDomainError(.windowOccluded)) - return - } - - let settleMode = params.observeAfter?.settle ?? HostSettleMode.none - cancellations.markDispatched(id: id) - - do { - try environment.postPointEvent( - params.action, - at: screenPoint, - from: screenStart, - pid: snapshot.pid, - path: path - ) - } catch let error as HostDomainError where error.code != .dispatchRefused { - // The event was never built, so nothing was attempted. - refuse(error) - return - } catch { - // §6.5 — the path was taken and the OS rejected it: `failed`, naming - // the path attempted, not `refused` with `path: none`. - emit( - id: id, - toolCallId: params.toolCallId, - dispatchFailure: HostDomainError(.dispatchRefused), - outcome: .failed, - tier: path.tier ?? .coordinateBackground, - path: path, - verdict: hostEffectFromActionResult() - ) - return - } - - finishDispatch( + // Keep the method as a stable refusal endpoint for older hosts. Point + // mutation is not an executor capability: do not resolve or consume the + // snapshot, inspect the target, select a path, or post an event. + emit( id: id, toolCallId: params.toolCallId, - snapshot: snapshot, - outcome: .ok, - path: path, - verificationIsTreeDelta: true, - fallbackVerdict: hostEffectFromActionResult(), - settleMode: settleMode, - observeAfter: params.observeAfter, - window: window + dispatchFailure: HostDomainError(.unsupportedAction), + tier: .coordinateBackground ) } diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift index f6444b2..ea5e8d9 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift @@ -585,10 +585,7 @@ public struct HostCapabilities: Codable, Equatable, Sendable { "click", "set_value", "select_text", "secondary_action", "scroll", "move_window", "resize_window", "minimize_window", ] - public var pointActions: [String] = [ - "move", "left_click", "right_click", "middle_click", "double_click", - "triple_click", "drag", "scroll", - ] + public var pointActions: [String] = [] public var keyActions: [String] = ["type", "key"] public var imageFormats: [HostImageFormat] = [.png] diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSystemEnvironment.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSystemEnvironment.swift index ffa2d26..aaab474 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSystemEnvironment.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSystemEnvironment.swift @@ -83,8 +83,8 @@ public protocol HostSystemEnvironment { /// caller re-reads `focusedElement(pid:)` before posting anything. func setFocusedElement(_ element: AXUIElement, pid: pid_t) -> Bool func bindingProbe(windowBounds: CGRect) -> HostElementBindingProbe - /// §6.3 — the path has already been selected and permitted by - /// `hostPointDispatchPath`; this only posts it. + /// Posts an executor-derived, PID-bound event for an already bound semantic + /// element action. Model-provided point dispatch never reaches this seam. func postPointEvent( _ action: HostPointAction, at point: CGPoint, @@ -225,8 +225,8 @@ public struct HostLiveEnvironment: HostSystemEnvironment { pid: pid_t, path: HostDispatchPath ) throws { - // Only the pid-bound paths are reachable when `allowGlobalPointer` is - // false, and `hostPointDispatchPath` has already refused anything else. + // Host element dispatch passes only PID-bound paths here. The retained + // `dispatch.point` compatibility endpoint never calls this method. switch action { case .move: try InputSimulation.moveGlobally(to: point) diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift index a0ae3a4..717fe73 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift @@ -676,14 +676,12 @@ final class HostDispatchTests: XCTestCase { XCTAssertEqual(try errorCode(harness.awaitResult()), "element_disabled") } - // MARK: - Point dispatch (§6.3) + // MARK: - Point dispatch compatibility refusal (§6.3) - func testPointDispatchRecomputesTheWindowAnchorRatherThanTrustingTheEcho() throws { - // The echo proves only that the host remembered its own digest. Within - // the TTL the window can be resized, and because the screen point is - // derived from the *current* bounds that silently rescaled the click. + func testPointDispatchIsAStableRefusalAndDoesNotTouchTheTarget() throws { var environment = FakeEnvironment() - environment.windows = [hostTestWindow(bounds: CGRect(x: 0, y: 0, width: 200, height: 200))] + environment.windows = [] + environment.pointEvents.failure = HostDomainError(.dispatchRefused) let harness = ServerHarness(environment: environment) try harness.begin() @@ -694,128 +692,31 @@ final class HostDispatchTests: XCTestCase { ) harness.install(snapshot) - harness.send(dispatchPoint(snapshot: snapshot, expectDigest: snapshot.windowDigest)) - XCTAssertEqual(try errorCode(harness.awaitResult()), "window_changed") - XCTAssertTrue(harness.environment.pointEvents.posted.isEmpty, "a refused point dispatch posts nothing") - } - - /// §12 vector 52 — a window that did not change is dispatchable at a point. - /// - /// Every other point vector installs a snapshot whose digest the fixture - /// computed, and verifies it against a probe that answers from the record. So - /// the recompute always agreed with itself, and the executor could ship with - /// the two ends of §4.3 reading the same unchanged element differently: - /// against every real application, `dispatch.point` refused `window_changed` - /// on the frame it had just been handed. - /// - /// Here the snapshot comes from the real tree walk and the probe recomputes - /// from the nodes, so the two ends are both present and neither is the other. - func testAPointDispatchAgainstAWindowThatDidNotChangeIsNotRefused() throws { - // A window, one group, one button. Every node reports the live parent - // chain the probe will read — including the root, whose chain runs up - // into the application element the walk never sees. - let button = FakeNode( - role: "AXButton", - label: "Send", - liveAncestorRoles: ["AXGroup", "AXWindow"] - ) - let group = FakeNode(role: "AXGroup", liveAncestorRoles: ["AXWindow"], children: [button]) - let root = FakeNode(role: "AXWindow", liveAncestorRoles: ["AXApplication"], children: [group]) - - var environment = FakeEnvironment() - environment.windows = [hostTestWindow()] - let probe = FakeRecomputingProbe() - environment.probe = probe - - let harness = ServerHarness(environment: environment) - try harness.begin() - let (snapshot, walk) = hostTestWalkedSnapshot( - registry: harness.server.currentRegistry(), - session: "s1", - root: root - ) - - probe.nodes = [ - walk.elements[0].token: root, - walk.elements[1].token: group, - walk.elements[2].token: button, - ] - probe.siblingIndexes = [ - // The root's live index is not its traversal index: this window is - // third in its application's `AXWindows`. §4.3's root rule is what - // keeps a *different* window coming forward out of this digest. - walk.elements[0].token: 3, - walk.elements[1].token: 0, - walk.elements[2].token: 0, - ] - harness.install(snapshot) - - harness.send(dispatchPoint(snapshot: snapshot, expectDigest: snapshot.windowDigest, x: 100, y: 100)) - let result = try harness.awaitResult() - - XCTAssertNil( - (result["error"] as? [String: Any])?["code"] as? String, - "nothing in the window moved, so the anchor must still hold" - ) - XCTAssertEqual(result["ok"] as? Bool, true) - XCTAssertEqual(result["path"] as? String, "cg_event_pid") - XCTAssertFalse(harness.environment.pointEvents.posted.isEmpty) - } - - func testPointDispatchTellsAHostEchoMistakeApartFromAChangedWindow() throws { - var environment = FakeEnvironment() - environment.windows = [hostTestWindow()] - - let harness = ServerHarness(environment: environment) - try harness.begin() - let snapshot = hostTestSnapshot( - registry: harness.server.currentRegistry(), - session: "s1", - image: hostTestImage() + harness.send( + dispatchPoint( + snapshot: snapshot, + expectDigest: "sha256:" + String(repeating: "c", count: 64), + x: 100, + y: 100, + observeAfter: #"{"includeImage":false,"settle":"quiesce"}"# + ) ) - harness.install(snapshot) - - harness.send(dispatchPoint(snapshot: snapshot, expectDigest: "sha256:" + String(repeating: "c", count: 64))) let result = try harness.awaitResult() - // Reporting `window_changed` for a host bookkeeping fault sent the host - // round the re-observe loop with the same wrong pairing. - XCTAssertEqual(try errorCode(result), "element_digest_mismatch") + XCTAssertEqual(result["ok"] as? Bool, false) XCTAssertEqual(result["outcome"] as? String, "refused") XCTAssertEqual(result["tier"] as? String, "coordinate-background") + XCTAssertEqual(result["path"] as? String, "none") + XCTAssertEqual(result["effect"] as? String, "unverifiable") + let verification = try XCTUnwrap(result["verification"] as? [String: Any]) + XCTAssertEqual(verification["method"] as? String, "none") + XCTAssertEqual(verification["observedChange"] as? Bool, false) + XCTAssertEqual(try errorCode(result), "unsupported_action") XCTAssertTrue(harness.environment.pointEvents.posted.isEmpty) - } - - func testAPointDispatchThatLandsSpendsTheFrameAndDeclaresItsPath() throws { - var environment = FakeEnvironment() - environment.windows = [hostTestWindow()] - - let harness = ServerHarness(environment: environment) - try harness.begin() - let snapshot = hostTestSnapshot( - registry: harness.server.currentRegistry(), - session: "s1", - image: hostTestImage() - ) - harness.install(snapshot) - - harness.send(dispatchPoint(snapshot: snapshot, expectDigest: snapshot.windowDigest, x: 100, y: 100)) - let result = try harness.awaitResult() - - XCTAssertEqual(result["ok"] as? Bool, true) - XCTAssertEqual(result["outcome"] as? String, "ok") - XCTAssertEqual(result["path"] as? String, "cg_event_pid") - XCTAssertEqual(result["tier"] as? String, "coordinate-background") - - // `image_px` is divided by the image's measured scale, never by a - // backing scale factor read off the screen. - let posted = try XCTUnwrap(harness.environment.pointEvents.posted.first) - XCTAssertEqual(posted.point, CGPoint(x: 50, y: 50)) - - // §4.1 — a mutating dispatch spends the frame it quoted. XCTAssertEqual( harness.server.currentRegistry().snapshotState(session: "s1", snapshotId: snapshot.id), - .spent + .live, + "a compatibility refusal must not consume snapshot authority" ) } @@ -851,99 +752,6 @@ final class HostDispatchTests: XCTestCase { XCTAssertEqual(posted.action, .scroll(direction: .down, pages: 0.5)) } - func testAnAttemptedPointDispatchTheOSRejectedIsFailedNotRefused() throws { - var environment = FakeEnvironment() - environment.windows = [hostTestWindow()] - environment.pointEvents.failure = HostDomainError(.dispatchRefused) - - let harness = ServerHarness(environment: environment) - try harness.begin() - let snapshot = hostTestSnapshot( - registry: harness.server.currentRegistry(), - session: "s1", - image: hostTestImage() - ) - harness.install(snapshot) - - harness.send(dispatchPoint(snapshot: snapshot, expectDigest: snapshot.windowDigest, x: 100, y: 100)) - let result = try harness.awaitResult() - - // §6.5 — `failed` names the path attempted. Reporting `refused` with - // `path: none` here erases the difference between "we never tried" and - // "we tried and it said no". - XCTAssertEqual(result["outcome"] as? String, "failed") - XCTAssertEqual(result["path"] as? String, "cg_event_pid") - XCTAssertEqual(try errorCode(result), "dispatch_refused") - XCTAssertEqual(result["effect"] as? String, "unverifiable") - } - - func testATargetReachableOnlyByTheGlobalPathIsRefusedOverTheWireToo() throws { - // Vector 14, through the handler rather than the policy function: the - // refusal carries `wouldRequirePath`, and the pointer never moves. - var environment = FakeEnvironment() - environment.windows = [hostTestWindow()] - - let harness = ServerHarness(environment: environment) - try harness.begin() - let snapshot = hostTestSnapshot( - registry: harness.server.currentRegistry(), - session: "s1", - image: hostTestImage() - ) - harness.install(snapshot) - - harness.send( - dispatchPoint( - snapshot: snapshot, - expectDigest: snapshot.windowDigest, - action: #"{"kind":"move"}"# - ) - ) - let result = try harness.awaitResult() - - XCTAssertEqual(try errorCode(result), "dispatch_refused") - XCTAssertEqual(result["outcome"] as? String, "refused") - XCTAssertEqual(result["path"] as? String, "none") - let detail = try XCTUnwrap((result["error"] as? [String: Any])?["detail"] as? [String: Any]) - XCTAssertEqual(detail["wouldRequirePath"] as? String, "cg_event_global") - XCTAssertTrue(harness.environment.pointEvents.posted.isEmpty, "the system cursor must not move") - } - - func testAFailedPostObservationIsReportedAsAnErrorObjectBesideTheOutcome() throws { - // §6.1 — the action happened and must be reported even though the frame - // after it could not be. The field is an error object like every other - // error on this wire; a bare code string made it the one failure the host - // had to parse differently. - var environment = FakeEnvironment() - environment.windows = [hostTestWindow()] - - let harness = ServerHarness(environment: environment) - try harness.begin() - let snapshot = hostTestSnapshot( - registry: harness.server.currentRegistry(), - session: "s1", - image: hostTestImage() - ) - harness.install(snapshot) - - harness.send( - dispatchPoint( - snapshot: snapshot, - expectDigest: snapshot.windowDigest, - x: 100, - y: 100, - observeAfter: #"{"includeImage":false,"settle":"none"}"# - ) - ) - let result = try harness.awaitResult() - - XCTAssertEqual(result["ok"] as? Bool, true) - XCTAssertNil(result["snapshot"] as? [String: Any]) - let postError = try XCTUnwrap(result["postObservationError"] as? [String: Any]) - XCTAssertEqual(postError["code"] as? String, "window_gone") - XCTAssertEqual(postError["message"] as? String, HostDomainErrorCode.windowGone.message) - } - // MARK: - Key dispatch (§6.4) func testKeyDispatchVerifiesTheBindingAndTheFocusBeforePostingAnything() throws { diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObservationTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObservationTests.swift index d405b69..5c50ee5 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObservationTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObservationTests.swift @@ -182,13 +182,11 @@ final class HostMenuObservationTests: XCTestCase { // MARK: - Vector 58: the menu has its own budget, and stays out of the window's func testTheMenuDoesNotChangeTheWindowDigest() throws { - // §5.8 — the reason the menu is a second array and not a second root. - // `windowDigest` anchors `dispatch.point` and is recomputed on every - // settle sample, one Accessibility round trip per recorded element. A - // menu folded into `elements` would have joined that hash, so the same + // §5.8 — the reason the menu is a second array and not a second root. A + // menu folded into `elements` would join the window hash, so the same // window would digest differently depending on whether the host had - // asked for menus — and every settle would have paid for menu elements - // that cannot change while the window does. + // asked for menus, and every settle would pay for menu elements that + // cannot change while the window does. let harness = try menuHarness() harness.send(observe(id: 3, menu: nil)) diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveContractTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveContractTests.swift index 0a81917..de98dcf 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveContractTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveContractTests.swift @@ -312,7 +312,7 @@ final class HostObserveContractTests: XCTestCase { func testDesktopScopeCapturesTheWindowRectangleAndNotADisplayOriginCrop() { // A whole-display filter sized to the window produced a display-origin // crop while `image.scale` and `image_px` both anchor at the window's - // origin, so every point dispatch under this scope landed elsewhere. + // origin, so the returned pixels did not describe the declared window. let rect = hostDesktopSourceRect( windowFrame: CGRect(x: 1512, y: 200, width: 800, height: 600), displayFrame: CGRect(x: 1512, y: 0, width: 1512, height: 982) diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostPointDispatchLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostPointDispatchLiveTests.swift deleted file mode 100644 index ba12f01..0000000 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostPointDispatchLiveTests.swift +++ /dev/null @@ -1,228 +0,0 @@ -import AppKit -import ApplicationServices -import CoreGraphics -import Foundation -import XCTest -@testable import OpenComputerUseKit - -/// §12 vector 52 — the half of `dispatch.point`'s anchor that no unit test can -/// reach: whether the executor's two ends of §4.3 read the same live window the -/// same way. -/// -/// OPEN_COMPUTER_USE_RUN_POINT_LIVE_TEST=1 \ -/// swift test --filter HostPointDispatchLiveTests -/// -/// Every unit vector for point dispatch installs a snapshot whose digest the -/// fixture computed, and verifies it against a probe that answers from the -/// record. The recompute therefore always agreed with itself, and the executor -/// shipped with the walk and the probe disagreeing about the snapshot root: -/// observe, then dispatch a point at the frame that observation just handed you, -/// and the answer was `window_changed` — on every application, on both displays, -/// with nothing on screen having moved. Point dispatch was unreachable in full, -/// and the whole suite was green. -/// -/// So this test does the one thing that catches it: it observes a real window and -/// immediately dispatches a point against it. Nothing changed in between, so -/// `window_changed` is a wrong answer by construction. -/// -/// Calculator is the target because it has no clock, no cursor and no animation -/// in it — a window that changes on its own would make `window_changed` a correct -/// answer and the test meaningless. It is launched in the background and left -/// running: this test does not own it. -/// -/// Every wait here is a semaphore. `wait(for:)` and `RunLoop.run` both spin the -/// main run loop, which thaws exactly the timings a live test exists to catch — -/// the executor's main thread sits in `readLine` and spins nothing. -final class HostPointDispatchLiveTests: XCTestCase { - func testAPointDispatchAgainstTheFrameJustObservedIsNotWindowChanged() throws { - guard ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_RUN_POINT_LIVE_TEST"] == "1" else { - throw XCTSkip("Set OPEN_COMPUTER_USE_RUN_POINT_LIVE_TEST=1 to run the live point dispatch test") - } - - // `AXIsProcessTrusted` asks; the prompting variant would block the run on - // a dialog nobody is there to answer. - guard AXIsProcessTrusted() else { - throw XCTSkip("Accessibility is not granted to the process running these tests") - } - guard PermissionDiagnostics.current().screenCaptureGranted else { - throw XCTSkip("Screen Recording is not granted, and `image_px` needs a captured image") - } - guard !hostScreenIsLocked() else { - throw XCTSkip("The screen is locked, so the tree is the menu bar and nothing else") - } - - let app = try backgroundLaunchedCalculator() - let window = try XCTUnwrap(try windowOf(pid: app.pid), "Calculator came up with no window to observe") - let frontBefore = NSWorkspace.shared.frontmostApplication?.processIdentifier - - let imageDirectory = try makeImageDirectory() - defer { try? FileManager.default.removeItem(at: imageDirectory) } - - let inbox = ResponseInbox() - let server = HostProtocolServer( - output: HostOutputWriter { inbox.append($0) }, - environment: HostLiveEnvironment() - ) - - // `allowGlobalPointer` is false, as it is in production: no cursor warp. - server.handle(line: #""" - {"jsonrpc":"2.0","id":1,"method":"host.hello","params":{"protocol":"\#(makaCuProtocolVersion)","hostPid":\#(ProcessInfo.processInfo.processIdentifier),"imageDir":"\#(imageDirectory.path)","allowGlobalPointer":false}} - """#) - _ = try inbox.next() - server.handle(line: #"{"jsonrpc":"2.0","id":2,"method":"session.begin","params":{"session":"s1","captureScope":"window"}}"#) - _ = try inbox.next() - - // An image, because `image_px` is only meaningful against the image the - // quoted snapshot carried. - server.handle(line: #""" - {"jsonrpc":"2.0","id":3,"method":"observe","params":{"session":"s1","target":{"kind":"window","pid":\#(window.pid),"windowId":\#(window.windowId)},"includeImage":true}} - """#) - let observed = try inbox.next(timeout: 30) - let observation = try XCTUnwrap(observed["result"] as? [String: Any]) - guard observation["ok"] as? Bool == true else { - return XCTFail("observe: \((observation["error"] as? [String: Any])?["code"] ?? "?")") - } - - let snapshot = try XCTUnwrap(observation["snapshot"] as? [String: Any]) - let snapshotId = try XCTUnwrap(snapshot["snapshotId"] as? String) - let windowDigest = try XCTUnwrap(snapshot["windowDigest"] as? String) - let image = try XCTUnwrap(snapshot["image"] as? [String: Any]) - let widthPx = try XCTUnwrap(image["widthPx"] as? Int) - let heightPx = try XCTUnwrap(image["heightPx"] as? Int) - let centre = (x: Double(widthPx) / 2, y: Double(heightPx) / 2) - - // 1. The reproduction, exactly: a move against the frame just handed - // over. It is refused, because a pointer move has no target-bound form - // and `allowGlobalPointer` is false — but it is refused for *that*, - // after the window anchor was recomputed and held. `window_changed` - // here is the executor disagreeing with its own observation. - server.handle(line: #""" - {"jsonrpc":"2.0","id":4,"method":"dispatch.point","params":{"session":"s1","snapshotId":"\#(snapshotId)","toolCallId":"call_move","expectWindowDigest":"\#(windowDigest)","point":{"x":\#(centre.x),"y":\#(centre.y)},"space":"image_px","action":{"kind":"move"}}} - """#) - let moved = try XCTUnwrap(try inbox.next(timeout: 30)["result"] as? [String: Any]) - let moveCode = (moved["error"] as? [String: Any])?["code"] as? String - - XCTAssertNotEqual( - moveCode, - "window_changed", - "nothing moved between the observation and the dispatch, so the anchor it recorded must still hold" - ) - XCTAssertEqual(moveCode, "dispatch_refused", "a move without a global pointer is refused for the pointer") - XCTAssertEqual( - ((moved["error"] as? [String: Any])?["detail"] as? [String: Any])?["wouldRequirePath"] as? String, - "cg_event_global" - ) - - // 2. §4.1 — that refusal did not spend the frame, so the same snapshot - // can carry a dispatch that actually posts. A scroll, because - // Calculator has nothing to scroll: the event is delivered to the pid - // and the window is left exactly as the user had it. - // - // `occlusionPolicy: "none"` because the anchor is the subject here and - // the desktop's stacking is not. Point dispatch defaults to `"any"` — - // correct, a pixel belongs to whatever is on top of it — but a - // background-launched window starts at the bottom of the z-order, so - // the default would make this assertion a report on what the user - // happened to have open rather than on the executor. - server.handle(line: #""" - {"jsonrpc":"2.0","id":5,"method":"dispatch.point","params":{"session":"s1","snapshotId":"\#(snapshotId)","toolCallId":"call_scroll","expectWindowDigest":"\#(windowDigest)","point":{"x":\#(centre.x),"y":\#(centre.y)},"space":"image_px","occlusionPolicy":"none","action":{"kind":"scroll","direction":"down","pages":1}}} - """#) - let scrolled = try XCTUnwrap(try inbox.next(timeout: 30)["result"] as? [String: Any]) - - XCTAssertNotEqual( - (scrolled["error"] as? [String: Any])?["code"] as? String, - "window_changed", - "the anchor has to hold all the way to a dispatch, not merely to the path table" - ) - XCTAssertEqual(scrolled["ok"] as? Bool, true, "\(scrolled)") - XCTAssertEqual(scrolled["outcome"] as? String, "ok") - XCTAssertEqual(scrolled["path"] as? String, "cg_event_pid", "target-bound, and the cursor never moved") - XCTAssertEqual(scrolled["tier"] as? String, "coordinate-background") - - // Nothing in either dispatch may take the foreground. - XCTAssertEqual( - NSWorkspace.shared.frontmostApplication?.processIdentifier, - frontBefore, - "a background dispatch does not change who is in front" - ) - - server.handle(line: #"{"jsonrpc":"2.0","id":99,"method":"session.end","params":{"session":"s1"}}"#) - _ = try inbox.next() - } - - // MARK: - Helpers - - /// Calculator, running, without having taken the foreground to get there. It - /// is left running: quitting an application the user may have opened is a - /// side effect a test has no business having. - private func backgroundLaunchedCalculator() throws -> RunningAppDescriptor { - guard FileManager.default.fileExists(atPath: "/System/Applications/Calculator.app") else { - throw XCTSkip("Calculator is not installed") - } - - let resolved = Outcome() - let finished = DispatchSemaphore(value: 0) - Thread.detachNewThread { - resolved.result = Result { try AppDiscovery.resolve("Calculator", waitFor: 20) } - finished.signal() - } - - guard finished.wait(timeout: .now() + 40) == .success else { - throw ComputerUseError.message("Calculator never resolved") - } - return try XCTUnwrap(resolved.result).get() - } - - /// Its on-screen ordinary window, waited for: a launch returns before the - /// window is mapped, measured at 1.3–3.2 s against 2.3–4.5 s (§5.7). - private func windowOf(pid: pid_t) throws -> HostWindowInfo? { - let deadline = Date().addingTimeInterval(15) - while Date() < deadline { - let windows = HostWindowInventory.onScreenWindows() - .filter { $0.pid == pid && $0.layer == 0 && $0.bounds.width > 64 && $0.bounds.height > 64 } - if let first = windows.first { - return first - } - Thread.sleep(forTimeInterval: 0.2) - } - - return nil - } - - private func makeImageDirectory() throws -> URL { - let directory = FileManager.default.temporaryDirectory - .appendingPathComponent("maka-cu-point-live-\(UUID().uuidString)", isDirectory: true) - try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true) - return directory - } - - private final class Outcome: @unchecked Sendable { - var result: Result? - } - - /// Line-delimited responses collected off whichever lane produced them, and - /// handed to the main thread through a semaphore rather than a run loop. - private final class ResponseInbox: @unchecked Sendable { - private let lock = NSLock() - private var lines: [Data] = [] - private let arrived = DispatchSemaphore(value: 0) - - func append(_ data: Data) { - lock.lock() - lines.append(data) - lock.unlock() - arrived.signal() - } - - func next(timeout: TimeInterval = 5) throws -> [String: Any] { - guard arrived.wait(timeout: .now() + timeout) == .success else { - throw ComputerUseError.message("no response within \(timeout)s") - } - - lock.lock() - let data = lines.removeFirst() - lock.unlock() - return try XCTUnwrap(try JSONSerialization.jsonObject(with: data) as? [String: Any]) - } - } -} diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTestSupport.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTestSupport.swift index d024cc1..c220832 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTestSupport.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTestSupport.swift @@ -638,9 +638,7 @@ func hostTestBinding( ) } -/// A snapshot of `window`, carrying one element and a window digest computed the -/// way `dispatch.point`'s live check recomputes it, so an unchanged window -/// matches and a changed one does not. +/// A snapshot of `window`, carrying one element and a canonical window digest. func hostTestSnapshot( registry: HostSnapshotRegistry, session: String, diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift index 97e89fb..50a5172 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift @@ -252,9 +252,8 @@ final class HostProtocolTests: XCTestCase { func testTheTwoEndsOfTheBindingCheckAgreeOnAnElementThatDidNotMove() throws { // The walk records the inputs and the probe recomputes them, and §4.3 // holds only if the two produce the same bytes from the same unchanged - // element. Asserted at the root, where they disagreed: no element - // dispatch could see it — it checks the element it targets — and - // `dispatch.point`, which anchors on the whole window, saw it every time. + // element. Asserted at the root, where they once disagreed despite the + // element itself remaining unchanged. let root = FakeNode(role: "AXWindow", title: "Untitled", liveAncestorRoles: ["AXApplication"]) let walk = hostWalkTree( @@ -608,73 +607,6 @@ final class HostProtocolTests: XCTestCase { XCTAssertNotEqual(neverChecked.method, inconclusive.method) } - // MARK: - Path selection (§6.3) - - func testTargetReachableOnlyByTheGlobalPathIsRefusedRatherThanWarped() { - let bounds = CGRect(x: 0, y: 0, width: 100, height: 100) - - let refusal = hostPointDispatchPath( - action: .move, - point: CGPoint(x: 10, y: 10), - startPoint: nil, - windowBounds: bounds, - allowGlobalPointer: false - ) - - XCTAssertEqual(failureCode(refusal), .dispatchRefused) - if case .failure(let error) = refusal { - XCTAssertEqual(error.detail, .wouldRequirePath(.cgEventGlobal)) - } else { - XCTFail("a pointer move has no target-bound form and must be refused") - } - - // The same request is allowed only when the host said so in the handshake. - XCTAssertEqual( - try? hostPointDispatchPath( - action: .move, - point: CGPoint(x: 10, y: 10), - startPoint: nil, - windowBounds: bounds, - allowGlobalPointer: true - ).get(), - .cgEventGlobal - ) - } - - func testDragLeavingTheTargetWindowIsRefusedWithoutGlobalPointer() { - let bounds = CGRect(x: 0, y: 0, width: 100, height: 100) - - let refusal = hostPointDispatchPath( - action: .drag, - point: CGPoint(x: 50, y: 50), - startPoint: CGPoint(x: 400, y: 400), - windowBounds: bounds, - allowGlobalPointer: false - ) - XCTAssertEqual(failureCode(refusal), .dispatchRefused) - - let inside = hostPointDispatchPath( - action: .drag, - point: CGPoint(x: 50, y: 50), - startPoint: CGPoint(x: 10, y: 10), - windowBounds: bounds, - allowGlobalPointer: false - ) - XCTAssertEqual(try? inside.get(), .cgEventPid) - } - - func testPointOutsideTheTargetWindowIsAnInvalidPointNotARefusal() { - let result = hostPointDispatchPath( - action: .leftClick(count: 1), - point: CGPoint(x: 900, y: 900), - startPoint: nil, - windowBounds: CGRect(x: 0, y: 0, width: 100, height: 100), - allowGlobalPointer: false - ) - - XCTAssertEqual(failureCode(result), .invalidPoint) - } - // MARK: - Occlusion (§6.2) func testSameAppOcclusionIgnoresForeignWindowsStackedAbove() {