Repository navigation
feat(preview): agents read and drive web pages in GOAT's built-in browser - #1907
Merged
Merged
Conversation
…refer-preview-browser
…efer-preview-browser
…first in discovery
…ad, and diffs key by the resolved tab
…view-browser # Conflicts: # apps/server/src/preview/ServerBrowserPage.ts # apps/server/src/preview/pageSecrets.ts
…view-browser # Conflicts: # apps/server/src/mcp/McpHttpServer.test.ts # apps/server/src/mcp/toolkits/preview/handlers.ts # apps/server/src/mcp/toolkits/preview/tools.ts # apps/server/src/screenCode/PreviewTargetHost.ts # docs/internals/agent-surface-audit.md
… a new cell after a failure
…view-browser # Conflicts: # apps/server/src/preview/ServerBrowserPage.ts
…view-browser # Conflicts: # apps/server/src/mcp/agentSurface.test.ts # apps/server/src/mcp/toolkits/core.test.ts # docs/internals/agent-surface-audit.md # packages/contracts/src/previewAutomation.ts # packages/shared/src/goatMcpToolPresentation.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agents in GOAT often reached for an outside browser tool, or wrote their own wait loops in
preview.evaluate, to read a web page. Most users have no outside browser tool installed. The built-in browser could not read a script-rendered page in one call, its snapshot cut long pages and pointed agents at evaluate, and only Codex and ACP agents were told the built-in browser exists.This change makes GOAT's built-in browser the default way agents read and act on web pages, and gives it the tools to do that well:
goat_preview_readreturns a page's rendered text as Markdown or plain text once the page settles. Readiness means the document has loaded, the DOM is quiet, and the tab has no requests in flight; network state is reported asunknownwhen tracking started late. It reads shadow roots and same-origin frames. It skips hidden content, form values and closed disclosures, and reports how many controls are collapsed. Results page with a revision andnextOffsetinside a byte budget for the whole JSON result, so a long page reads in full across calls. It refuses when a private period overlaps the read or a secret field changes during it, masks passwords GOAT filled into the tab, and never returnsdata:payloads. A scope whose root, ancestors, slot or embedding frame is hidden reads as empty. A page too large or deep to render whole is refused withrender_cutand a hint to read one part withscope, so a cut never leaves part of a secret behind.interactiveOnly,scope(a ref or selector),diffagainst the caller's previous snapshot of the same tab, andannotate(ref labels drawn on the screenshot). Truncation is reported with structured reasons, and results carry a one-line next step: narrow the snapshot, read the text, close a blocking layer, or confirm an action. A diff restarts when the caller last received a cut tree.readand a newsupportedFeatures: ["snapshotOptions"]. The broker checks both on every route, so an older host refuses a scoped snapshot instead of silently returning the whole page.goat_screen_run):unverifiableuntil an observation confirms them, and the diff baseline survives actions.screen.stateholds tab handles across cells, plustab.status(),tab.read()(clamped to the cell's reply budget),screen.listTabs(), andobserve({interactiveOnly, scope}).evaluatefor more text.evaluateis described as read-only with a known selector.preview.readfirst for "read web page".Click interception errors now name the element that received the click, bounded and redacted.
Instruction budgets (
context-budget --checkpasses): Codex 4,975 → 4,928 B, ACP 5,434 → 5,387, Claude 4,726 → 5,179, OpenCode 4,444 → 4,897, Cursor 5,001, Pi 4,618. The screen-enabled ACP first turn that was over the limit drops from 6,390 to 5,959 B. Cursor's later turn with an installed skill path stays at 388 B over its 300 B allowance; that was already the case before this change, and this PR leaves it unchanged.Deferred to follow-ups: readiness waits for type and select actions, keeping downloads after a tab closes, and saving binary fetch results as artifacts.
Verification
screen.state,tab.readvalidation, provider instruction delivery, and discovery.node scripts/agent-surface/audit.ts --checkandnode scripts/agent-surface/context-budget.ts --checkpass.goat_preview_read, expanded the collapsed appendix by ref, read again, and returned the correct heading, paragraph count, verbatim last paragraph, appendix text and guide headings.goat-test-changedcovers this PR's tests.goat-checkandgoat-typecheckalso report failures inapps/web/src/components/media/MediaVideoPlayer.tsx(no-manual-memo) andapps/web/src/state/threads.test.ts(inboundFrames). Neither file is changed by this PR; both came from thedevstate merged here to resolve a conflict, anddevhas since fixed them (chase(t3): land the rest of cycle 261028 as one combined PR #1982).Written by claude-opus-5-5 in GOAT through the Codex harness, with gpt-6.1-sol research, implementation and review lanes.