Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
711 changes: 675 additions & 36 deletions libs/cua-driver/contract/manifest.json

Large diffs are not rendered by default.

25 changes: 19 additions & 6 deletions libs/cua-driver/docs/action-result-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,10 @@ Every successful action returns a closed `structuredContent` object:
| `effect` | `confirmed`, `partial`, `unverifiable`, `suspected_noop`, `refused` |
| `route` | `accessibility`, `synthetic_events`, `global_input`, `dom`, `trusted_input` |
| `delivery.mode` | `background`, `foreground`, `not_applicable`, `unknown` |
| `evidence[].kind` | `value_readback`, `window_change` |
| `escalation.target` | `pixel`, `foreground`, `page`, `session` |
| `escalation.reason` | `route_unavailable`, `delivery_failed`, `effect_unconfirmed`, `suspected_noop`, `permission_required` |
| `evidence[].kind` | `value_readback` |
| `window_change.new_windows[]` | target-scoped, owner-verified `pid`, `window_id`, application, and title |
| `escalation.target` | `pixel`, `foreground`, `page`, `session`, `rebind` |
| `escalation.reason` | `route_unavailable`, `delivery_failed`, `effect_unconfirmed`, `suspected_noop`, `permission_required`, `surface_changed` |

The action-result tools are:

Expand All @@ -51,7 +52,7 @@ scope, targets, platform transport names, diagnostic pointers, or the old

The invariants are:

- `confirmed` has publishable readback or window-change evidence;
- `confirmed` has publishable value readback; topology alone cannot confirm an action;
- `partial` has `delivery.delivered_count`;
- `refused` has neither delivery nor evidence.

Expand All @@ -72,8 +73,19 @@ resolution semantics; the guard does not replace or reinterpret them.

An action that reached an actuator but lacks a trusted readback is
`unverifiable`, not `confirmed`. Screenshot change, native API acceptance,
event receipt, and operator observation may remain useful internal diagnostics,
but they do not independently justify `confirmed`.
event receipt, window topology, and operator observation may remain useful
facts, but they do not independently justify `confirmed`.

On macOS, one action decorator snapshots accessibility roots for the target PID
before and after dispatch. It reports new windows, dialogs, sheets, and popovers
without activating them. An exact `rebind` target is present when exactly one
new owner-verified root appeared; ambiguous changes require `list_windows`.
Owner verification preserves the exact WindowServer identity
reported by the appeared AX root. AppKit can expose an addressable same-process
proxy beside an AX-empty XPC duplicate, so a foreign PID is not required; the
returned target is the identity callers can resnapshot. Windows and Linux
currently emit no topology record rather than substituting a global desktop
heuristic.

## Verification remains separate

Expand Down Expand Up @@ -123,6 +135,7 @@ An optional escalation is advice, not an automatic retry:
| `foreground` | explicitly select foreground delivery when session policy permits |
| `page` | bind the native window to a supported browser page route |
| `session` | prepare or explicitly widen the session only when policy permits |
| `rebind` | refresh the validated window target without activation; use `list_windows` when no exact target is present |

SDK integrators, OpenClaw, Hermes, and other agent hosts can implement different
policies above this same narrow fact contract without duplicating platform
Expand Down
4 changes: 4 additions & 0 deletions libs/cua-driver/python/src/cua_driver/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@
ActionEvidenceKind,
ActionResult,
ActionRoute,
ActionWindowChange,
ActionWindowTarget,
BoundsExpectation,
CaptureScope,
ClickButton,
Expand Down Expand Up @@ -180,6 +182,8 @@ def _create_private_worker_python_sdk(cls, options):
"ActionEvidenceKind",
"ActionResult",
"ActionRoute",
"ActionWindowChange",
"ActionWindowTarget",
"BoundsExpectation",
"CaptureScope",
"ClickButton",
Expand Down
Loading
Loading