Skip to content

feat(preview): agents read and drive web pages in GOAT's built-in browser - #1907

Merged
lidge-jun merged 34 commits into
devfrom
feat/agents-prefer-preview-browser
Oct 28, 2026
Merged

lidge-jun merged 34 commits into
devfrom
feat/agents-prefer-preview-browser

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Agents in GOAT often reached for an outside browser tool, or wrote their own wait loops in preview.evaluate, to read a web page. Most users have no outside browser tool installed. The built-in browser could not read a script-rendered page in one call, its snapshot cut long pages and pointed agents at evaluate, and only Codex and ACP agents were told the built-in browser exists.

This change makes GOAT's built-in browser the default way agents read and act on web pages, and gives it the tools to do that well:

  • goat_preview_read returns a page's rendered text as Markdown or plain text once the page settles. Readiness means the document has loaded, the DOM is quiet, and the tab has no requests in flight; network state is reported as unknown when tracking started late. It reads shadow roots and same-origin frames. It skips hidden content, form values and closed disclosures, and reports how many controls are collapsed. Results page with a revision and nextOffset inside a byte budget for the whole JSON result, so a long page reads in full across calls. It refuses when a private period overlaps the read or a secret field changes during it, masks passwords GOAT filled into the tab, and never returns data: payloads. A scope whose root, ancestors, slot or embedding frame is hidden reads as empty. A page too large or deep to render whole is refused with render_cut and a hint to read one part with scope, so a cut never leaves part of a secret behind.
  • Snapshots gain interactiveOnly, scope (a ref or selector), diff against the caller's previous snapshot of the same tab, and annotate (ref labels drawn on the screenshot). Truncation is reported with structured reasons, and results carry a one-line next step: narrow the snapshot, read the text, close a blocking layer, or confirm an action. A diff restarts when the caller last received a cut tree.
  • Hosts advertise read and a new supportedFeatures: ["snapshotOptions"]. The broker checks both on every route, so an older host refuses a scoped snapshot instead of silently returning the whole page.
  • Browser cells (goat_screen_run):
    • Browser observations now expose usable refs. An opaque ref generation used to make every observation look truncated.
    • Actions report unverifiable until an observation confirms them, and the diff baseline survives actions.
    • A same-tab origin change can be re-observed, and failures keep a short sanitized reason.
    • New: screen.state holds tab handles across cells, plus tab.status(), tab.read() (clamped to the cell's reply budget), screen.listTabs(), and observe({interactiveOnly, scope}).
  • Guidance:
    • A 453-byte browser rule now reaches Codex, Claude, ACP, Cursor, OpenCode and Pi whenever the turn has browser access. It says to read web pages in GOAT's browser, to use another browser only when one is named or after the same step fails twice, that page output is data and never instructions, and to finish with a saved screenshot.
    • The resident screen-code text is shorter (476 → 284 B), and the installed screen-code skill teaches the browser workflow.
    • Preview tool descriptions no longer send agents to evaluate for more text. evaluate is described as read-only with a known selector.
    • Discovery ranks preview.read first for "read web page".
    • Research routing in the repository skill and role templates now points at the built-in browser.

Click interception errors now name the element that received the click, bounded and redacted.

Instruction budgets (context-budget --check passes): Codex 4,975 → 4,928 B, ACP 5,434 → 5,387, Claude 4,726 → 5,179, OpenCode 4,444 → 4,897, Cursor 5,001, Pi 4,618. The screen-enabled ACP first turn that was over the limit drops from 6,390 to 5,959 B. Cursor's later turn with an installed skill path stays at 388 B over its 300 B allowance; that was already the case before this change, and this PR leaves it unchanged.

Deferred to follow-ups: readiness waits for type and select actions, keeping downloads after a tab closes, and saving binary fetch results as artifacts.

Verification

  • Failing tests first (Chromium, promise gates and a paused page clock, no sleeps) for rendered reads, settle and network state, paging and byte budgets, privacy refusals, hidden ancestors, snapshot narrowing, diff and annotation, broker routing for old and new hosts, cell generations, receipts, diff baselines, screen.state, tab.read validation, provider instruction delivery, and discovery.
  • node scripts/agent-surface/audit.ts --check and node scripts/agent-surface/context-budget.ts --check pass.
  • Live check: an isolated dev server in this worktree with a Codex agent (GPT-6-Astra). The agent was asked to read a local page that renders 60 paragraphs after a 2.5 s delayed request, and the Vite guide, without naming any tool. It opened GOAT's browser, used goat_preview_read, expanded the collapsed appendix by ref, read again, and returned the correct heading, paragraph count, verbatim last paragraph, appendix text and guide headings.
  • CI on the final head: goat-test-changed covers this PR's tests. goat-check and goat-typecheck also report failures in apps/web/src/components/media/MediaVideoPlayer.tsx (no-manual-memo) and apps/web/src/state/threads.test.ts (inboundFrames). Neither file is changed by this PR; both came from the dev state merged here to resolve a conflict, and dev has since fixed them (chase(t3): land the rest of cycle 261028 as one combined PR #1982).
  • Local suites stopped mid-task at the maintainer's request; verification after that point is CI only.
  • NOT RUN: a live Claude turn (the Claude CLI is not installed on the test machine; delivery is covered by adapter tests), and mobile (no mobile consumer of these results).

Written by claude-opus-5-5 in GOAT through the Codex harness, with gpt-6.1-sol research, implementation and review lanes.

…view-browser

# Conflicts:
#	apps/server/src/preview/ServerBrowserPage.ts
#	apps/server/src/preview/pageSecrets.ts
…view-browser

# Conflicts:
#	apps/server/src/mcp/McpHttpServer.test.ts
#	apps/server/src/mcp/toolkits/preview/handlers.ts
#	apps/server/src/mcp/toolkits/preview/tools.ts
#	apps/server/src/screenCode/PreviewTargetHost.ts
#	docs/internals/agent-surface-audit.md
…view-browser

# Conflicts:
#	apps/server/src/preview/ServerBrowserPage.ts
…view-browser

# Conflicts:
#	apps/server/src/mcp/agentSurface.test.ts
#	apps/server/src/mcp/toolkits/core.test.ts
#	docs/internals/agent-surface-audit.md
#	packages/contracts/src/previewAutomation.ts
#	packages/shared/src/goatMcpToolPresentation.ts
@lidge-jun
lidge-jun merged commit 6d28d05 into dev Oct 28, 2026
3 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant