Skip to content

feat(cu): add desktop presentation lifecycle - #5

Closed
hqhq1025 wants to merge 1 commit into
codex/cu-desktop-wiringfrom
codex/cu-presentation-lifecycle
Closed

hqhq1025 wants to merge 1 commit into
codex/cu-desktop-wiringfrom
codex/cu-presentation-lifecycle

Conversation

@hqhq1025

Copy link
Copy Markdown
Owner

Goal

Add the Desktop Computer Use presentation lifecycle as a reviewable layer above verified targeting and dispatch.

What this PR does

  • adds a focusless, click-through, transparent agent-cursor BrowserWindow
  • builds the overlay renderer and receive-mostly preload in normal build/dev flows
  • maps only Runtime-bound observation coordinates into screen presentation coordinates
  • waits up to 1 second for readyForInteraction, then fails open into native dispatch
  • never waits for visual finished before returning the tool result
  • commits visual completion only after Runtime accepts the post-dispatch lease and fresh observation
  • cancels presentation for failed, stale, intervened, aborted, or otherwise discarded results
  • validates mouse_move against the bound window and returns an executor-resolved completion point without injecting native pointer input
  • serializes operations through the single visual cursor while keeping no-overlay session execution independent
  • immediately releases active and queued presentation waits on session clear
  • binds renderer acknowledgements to session, action, BrowserWindow generation, and current window identity
  • tears down on load failure, renderer loss, display-layout change, session terminal boundaries, main-window close, and app shutdown
  • supports the union bounds of multiple displays, including negative origins

Reverse-engineering constraints applied

The implementation follows the local Codex Computer Use lab findings:

  • presentation identity is late-bound and is not execution evidence
  • the visible completion coordinate comes only from executor resolvedScreenPoint
  • presentation readiness is bounded and cannot redefine the execution target
  • intervention/stale/stop states discard the result and must not show a success pulse
  • stale renderer acknowledgements cannot settle a new operation
  • the overlay cannot take focus or receive mouse input

Non-goals

  • provider-specific Computer Use adapters or model loops
  • physical-input, lock/unlock, blocked-URL, or service-loss event producers
  • deterministic cumulative Electron E2E
  • release packaging/signing/notarization
  • Headless presentation support

Verification

  • npm --workspace @maka/runtime test — full suite passed
  • Runtime focused Computer Use tests — 47 passed
  • npm --workspace @maka/computer-use test — 103 passed
  • npm --workspace @maka/desktop run typecheck
  • npm --workspace @maka/desktop test — 2419 passed
  • npm --workspace @maka/desktop run build
  • overlay controller/preload focused tests — 16 passed
  • git diff --check

Real Electron smoke boundary

An earlier protocol revision produced a visible nonblank cursor screenshot. After adding session+generation acknowledgements, this desktop environment could not reliably launch a second isolated Electron app while Codex/ChatGPT Electron was active, so the current protocol's real-window smoke is intentionally deferred to the cumulative PR G harness rather than claimed as complete here.

Stacked on #4.

@hqhq1025
hqhq1025 force-pushed the codex/cu-desktop-wiring branch from 6d8cf35 to 4b88d54 Compare July 13, 2026 10:50
@hqhq1025
hqhq1025 force-pushed the codex/cu-presentation-lifecycle branch from ae59405 to 323d903 Compare July 13, 2026 10:51
@hqhq1025
hqhq1025 force-pushed the codex/cu-desktop-wiring branch from 4b88d54 to 359ded7 Compare July 13, 2026 11:06
@hqhq1025
hqhq1025 force-pushed the codex/cu-presentation-lifecycle branch from 323d903 to 7e605b5 Compare July 13, 2026 11:07
@hqhq1025
hqhq1025 force-pushed the codex/cu-desktop-wiring branch from 359ded7 to 80a87eb Compare July 13, 2026 11:20
@hqhq1025
hqhq1025 force-pushed the codex/cu-presentation-lifecycle branch from 7e605b5 to 701b05c Compare July 13, 2026 11:21
@hqhq1025
hqhq1025 force-pushed the codex/cu-desktop-wiring branch from 80a87eb to 0f43f33 Compare July 13, 2026 11:23
@hqhq1025
hqhq1025 force-pushed the codex/cu-presentation-lifecycle branch from 701b05c to 23381b4 Compare July 13, 2026 11:23
@hqhq1025
hqhq1025 force-pushed the codex/cu-desktop-wiring branch from 0f43f33 to 5ca7da5 Compare July 13, 2026 15:50
@hqhq1025

Copy link
Copy Markdown
Owner Author

The exact net diff represented here is now published upstream as apache#896. Closing this fork-local review PR to keep the active review queue in the upstream repository.

@hqhq1025 hqhq1025 closed this Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant