Skip to content

feat(chat): agents can render interactive views inline in the chat - #139

Merged
tusharbhardwaj-bk merged 10 commits into
expbkmainfrom
t3code/mechoui
Aug 28, 2026
Merged

feat(chat): agents can render interactive views inline in the chat#139
tusharbhardwaj-bk merged 10 commits into
expbkmainfrom
t3code/mechoui

Conversation

@tusharbhardwaj-bk

@tusharbhardwaj-bk tusharbhardwaj-bk commented Aug 28, 2026

Copy link
Copy Markdown

The problem

Agents can describe a chart, a diagram or a table, but they cannot show one.
Everything they produce arrives as text or as a collapsed tool row, so anything
visual has to be rebuilt in the reader's head — or opened somewhere outside the
conversation, which loses the place in the work it belonged to.

Hosts like Claude Desktop solved this with MCP Apps (SEP-1865, the first
official MCP extension, finalized 2026-01-26): a tool result points at a ui://
resource and the host renders it in a sandboxed iframe inline in the chat. This
brings the same capability to T3.

What this adds

A new t3_show_ui tool on T3's own MCP server. An agent calls it with a
self-contained HTML document (or an https URL) and the chat mounts it as a
sandboxed box in the transcript, right where the tool call happened.

The iframe has an opaque origin: its styles and scripts run, so charts and
interactions work, but it cannot reach T3 Code, its cookies, or the network as
the signed-in user.

Renders are immutable. Calling the tool again shows a new box rather than
mutating an old one, so scrolling back still shows what the agent actually
produced at that point in the thread.

Why it is built this way

The document never travels on an activity payload. The tool stores it and
returns only a short render handle; ActivityPayloadProjection carves that
handle out before it summarizes the result away, and the chat fetches the body
on demand through a new thread-scoped fork RPC.

That matters for two reasons. It keeps oversized documents off the websocket and
clear of the 32 KiB activity string cap. And it works on every provider
including Claude Code, whose CLI flattens MCP tool results to text and would
otherwise drop an embedded ui:// resource entirely. Codex passes MCP content
blocks through verbatim; Claude does not. Routing through our own tool sidesteps
the difference.

The handle extractor reads three result shapes (text content array,
structuredContent, bare result object) because providers disagree about where
a result lands and a miss here fails silently.

Fork surface

Deliberately small, per AGENTS.md. New fork-owned modules carry the feature:

  • apps/server/src/agentui/, persistence/AgentUiRenders.ts, migration 1022
  • packages/contracts/src/agentUi.ts, packages/client-runtime/src/state/agentUi.ts
  • apps/web/src/fork/agentUiSurface.tsx

Upstream files take one marked seam each. The timeline seam is one import plus
an early return that wraps the unchanged upstream row — composition, not
modification — so turning the setting off leaves exactly the original row
behind.

Gated by Settings → Experiments → Agent views in chat, on by default.

Verification

  • AgentUiService.test.ts — 7 tests: storage round-trip, thread scoping (a
    render does not leak across threads), height clamping, https-only URL scheme
    (rejects http, javascript:, file:, data:), html/url exclusivity, size
    cap, missing render.
  • ActivityPayloadProjection.agentUi.test.ts — 7 tests pinning handle survival
    through the projection in both provider nestings and all three result shapes,
    plus that ordinary MCP results are untouched and the body is still summarized.
  • agentUiSurface.test.ts — 5 tests on the pure resolver, which runs on every
    work-log row and must stay total.
  • Typecheck clean: contracts, server, web. Fork marker check passes. Lint clean.

Screenshots to follow from the expbkt3 deploy.

Written by Claude Opus 5 in T3 Code.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…ive-bkt3-useremail-system-val-38e14c02

fix(server): use T3 sender for Claude userEmail (TEC-1031)
The appended T3 identity block named the right sender, but Claude Code's
own `# userEmail` context section still arrived per-turn with the shared,
rotating subscription account's address and won — two contradictory
identity claims coexisted and the model followed the later one, so a live
session answered "who am I" with the rotated account holder.

The CLI has no switch to suppress that section, so both branches of the
appended block now name it explicitly and tell the model to ignore it for
user attribution. No adapter wiring changed.

TEC-1048

Model: claude-opus-5. Harness: T3 Code (Claude Code).
fix(server): countermand Claude's native userEmail section (bkmain)
The TEC-1048 system-prompt countermand holds on Opus 5 but loses to
Claude Code's native `# userEmail` section on Sonnet 5 at medium effort
(3/3 failures against the deployed build, countermand quoted back
verbatim). Position beats wording: the native section is user context and
outranks anything appended to the system prompt.

Register an SDK `UserPromptSubmit` hook that returns the resolved
`BK_MESSAGE_SENDER_EMAIL` as `additionalContext` on every turn, which
lands at user-message position and does win (Sonnet 5 medium, 2/2,
including adversarial wording). The system-prompt append stays as the
belt; both are gated on `BK_IDENTITY_RUNTIME`, so upstream Claude
sessions are unchanged. The helper merges with any hooks already
registered rather than replacing them.

TEC-1051
fix(server): inject T3 sender identity on every Claude turn (bkmain)
Agents could describe a chart, a diagram or a table, but never show one.
Everything they produced arrived as text or as a collapsed tool row, so
anything visual had to be rebuilt in the user's head — or opened somewhere
outside the conversation.

This adds the piece MCP Apps hosts like Claude Desktop have: a sandboxed box
rendered inline in the transcript, where the tool call happened. An agent calls
the new `t3_show_ui` MCP tool with a self-contained HTML document (or an https
URL) and the chat mounts it in an iframe with an opaque origin — its scripts run,
so charts and interactions work, but it cannot reach T3 Code, its cookies, or the
network as the signed-in user.

The document never travels on an activity payload. The tool stores it and
returns a short render handle; the projection layer carves that handle out
before it summarizes the result away, and the chat fetches the body on demand
through a new thread-scoped fork RPC. That keeps oversized documents off the
websocket and clear of the 32 KiB activity string cap, and it works on every
provider — including Claude Code, whose CLI flattens MCP results to text.

Renders are immutable: calling the tool again shows a new box rather than
mutating an old one, so a row keeps showing what the agent actually produced at
that point in the thread.

Fork surface is deliberately small: the feature lives in new fork-owned modules
(`apps/server/src/agentui/`, `apps/web/src/fork/agentUiSurface.tsx`), and the
timeline seam is one import plus an early return that wraps the unchanged
upstream row. Gated by an Experiments setting, on by default.

Written by Claude Opus 5 in T3 Code.
…vider sends

Providers disagree about where an MCP tool result lands: the text content array,
`structuredContent`, or the bare result object. Claude Code has shipped the
second in place of the first, and a miss here fails silently — the view just
turns back into an ordinary tool row — so read all three.

Written by Claude Opus 5 in T3 Code.
Adds the user-facing page for the feature and registers it in the docs index
and the expbkt3 customization boundary registry.

Written by Claude Opus 5 in T3 Code.
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XL labels Aug 28, 2026
Written by Claude Opus 5 in T3 Code.
@tusharbhardwaj-bk
tusharbhardwaj-bk merged commit 071d2ed into expbkmain Aug 28, 2026
18 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant