📊 feat: Inline Visuals in Assistant Replies

[usnavy13]

Pull Request

Summary

The major assistants now answer with visuals, not just text. OpenAI launched Intelligent UI with GPT-6 on October 7: ChatGPT replies can now include charts, graphics, buttons, forms and interactive elements that appear as the model writes (OpenAI's announcement post). Anthropic shipped custom visuals in chat in March, where Claude writes HTML charts and diagrams inline and decides when to use them. Google has generative UI in the Gemini app, where Gemini 3 outputs HTML, CSS and JS for a custom interactive response. All three ship only in their own consumer apps, with no API for them, so LibreChat users have had no way to get this.

LibreChat today can draw Mermaid diagrams inline and can show HTML or React in the Artifacts side panel, but a model has no way to put a chart, a calculator or a small interactive explanation directly in its answer. Users have asked for this for a while: inline Plotly/Recharts charts (#5871), graphs without the paid Code Interpreter (#6576), an agent-level chart toggle (#8214), Vega-Lite charts in replies (#12993), and Plotly output inline (#4366). The move of Mermaid from the artifact panel into the conversation (#7848) showed the same preference for visuals in place.

This PR adds inline visuals. When the setting is on, the model is told it may write a self-contained HTML page inside a :::visual{title="…"} container around an html fence. The page renders inline in the reply, in a sandboxed frame themed with the app's colors and sized to its content. That covers charts, diagrams, timelines, comparisons, calculators, quizzes and small games, with or without web search. It uses the approach Claude and Gemini use, model-written HTML, rather than OpenAI's fixed component library, so a model can draw anything a page can and nothing ties it to one provider. The frame bootstrap is adapted from T3 Code's HTML renders (pingdotgg/t3code#15968, MIT), and the frame speaks the MCP Apps message names.

Admins control it with interface.visuals (default on, overridable per role like other interface settings). Each user controls it under Settings > Chat > Inline visuals (default on, saved in the browser like Maximize chat space); it applies to every chat, including saved agents. When the setting is off the model is not asked for visuals; when the admin turns visuals off, existing visuals show a short notice instead of rendering. Artifacts are unchanged and can be used at the same time; the instructions tell the model to use a visual for a figure inside an answer and an artifact for something the user will keep or edit. Any visual can open in the artifacts panel with its source.

Related to #5871, #6576, #8214, #12993, #4366. Partly addresses #13373 (interactive outputs, without the model callback). Adjacent to #10641 (MCP Apps), whose sandbox model this shares.

How it works

The visual travels as text in the reply, so it needs no new content part, tool call or storage. The client renders it in a frame served by the app with its own CSP, because a srcdoc frame would inherit the app's nonce CSP and block the page's scripts.

sequenceDiagram
    participant Settings as Settings (user)
    participant Client
    participant API as packages/api
    participant Model
    participant Frame as /api/visuals/frame
    Settings->>Client: Inline visuals on
    Client->>API: POST agents chat, visuals: true
    API->>Model: system prompt + visuals instructions (if interface.visuals allows)
    Model->>Client: streams text with :::visual{title} + html fence
    Client->>Client: "Building visual…" until the container closes
    Client->>Frame: load shell (sandbox, own CSP)
    Frame->>Client: ui/notifications/sandbox-proxy-ready
    Client->>Frame: sandbox-resource-ready { html + theme bootstrap }
    Frame->>Client: ui/notifications/size-changed { height }

Loading

packages/api/src/visuals/
├── prompt.ts     # the instructions, and the gate: user setting + interface.visuals
└── frame.ts      # frame shell and CSP: sandbox allow-scripts, scripts inline or from visuals.sources, no other network
client/src/components/Visuals/
├── plugin.ts     # :::visual container with an html fence -> <visual>; text around the fence stays in the reply
├── Visual.tsx    # title, actions (panel, download, copy), placeholder while streaming, 640px cap with Show more
├── Frame.tsx     # sandboxed iframe, handshake, auto height, live theme updates, links on user gesture
├── bootstrap.ts  # theme variables + element defaults injected into the page, size and link reporting
└── theme.ts      # reads the app theme into hex CSS variables (--background, --chart-1 … --chart-8)

Security: the frame has an opaque origin (sandbox allow-scripts, no allow-same-origin), so a page cannot read the app's cookies, storage or DOM. Its CSP allows scripts only inline and from visuals.sources (jsDelivr, cdnjs, unpkg and esm.sh by default), blocks every other request including the app's own API, blocks form posts, and uses the app's CSP_FRAME_ANCESTORS. Links open in a new tab only on a live user gesture.

Type of change

• Feature

Testing

1. Start LibreChat with an agents-capable endpoint. Leave interface.visuals unset (on).
2. Ask "Explain how compound interest works and let me play with the numbers." The reply contains a working calculator inline; moving a slider recalculates it.
3. Ask "Explain the Monty Hall problem, and let me play it myself to see why switching wins." The reply contains a playable game; picking a door makes the host open a goat door and offer Stay or Switch.
4. Turn on Web Search and ask "How has the price of gold moved over the past year? Chart it for me." The reply contains a sourced chart, followed by cited prose.
5. Turn off Settings > Chat > Inline visuals and repeat; the model answers without a visual, and the request carries visuals: false.
6. Set interface.visuals: false; the setting disappears and existing visuals show "Visuals are turned off for this app."

Tested environments/configuration:

• Chromium (Playwright) at 1920×1080, 1440×900 and 390×844, light, dark, high contrast and the ClickHouse theme
• Claude Sonnet 5.5, GPT-5.6 Terra, Gemini 3.8 Flash and Kimi K3 through OpenRouter, with and without LibreChat web search (Serper, Firecrawl, Cohere)

Automated tests:

• packages/api: src/visuals/frame.spec.ts, src/visuals/prompt.spec.ts, visuals cases in src/agents/__tests__/initialize.test.ts, src/app/interface.spec.ts
• packages/data-provider: specs/visuals.spec.ts; packages/data-schemas: loadVisualsConfig in src/app/service.spec.ts
• client: src/components/Visuals/__tests__/ (plugin, bootstrap, theme, frame handshake and sizing, Visual states), the settings registry, rich-text copy and the block splitter
• e2e/specs/mock/visuals.spec.ts: renders a visual in the real frame (script runs, parent and storage isolated, theme applied, fetch blocked, frame fits the page), opens it in the panel, and checks the request flag with the setting on and off
• npx tsc --noEmit in data-provider, data-schemas, api and client; npm run static-checks -- --against origin/dev

Screenshots / recordings

A new surface, so there is no "before". All shots use the ClickHouse theme in dark mode.

Settings > Chat > Messages:

Claude, Monty Hall you can play (after picking a door), 1920×1080:

Claude, interactive compound interest calculator, 1440×900:

Risk / compatibility

• On by default. On upgrade, every agents request from an updated client carries visuals: true, which adds about 750 tokens of instructions and lets models answer with visuals. Admins who want the old behavior set interface.visuals: false; users can turn it off in Settings.
• New configuration. interface.visuals (boolean) and a visuals.sources section (https origins, librechat.yaml only, like mcpAppSandbox). An invalid source list allows no origins rather than the defaults.
• New route. GET /api/visuals/frame is unauthenticated because it serves an empty shell; it carries no user data.
• Request field. Agents requests gain an optional visuals boolean. Older clients do not send it, so they get no visuals instructions. Assistants endpoints are not supported.
• Agents that get the instructions. The answering agent and graph agents (handoffs and subagents); background agents such as the memory agent do not.
• Residual risks. A page can navigate its own frame, and after a user click it can ask the host to open a link, so data the model wrote into a page could leave that way; CDN script loads are also outbound requests. This matches T3 Code's renderer. Setting visuals.sources: [] removes the CDN path.
• Known limits. Visual state (checked items, scores) resets on reload, a visual cannot send a message back to the conversation, and text inside visuals uses the system font because the sandboxed frame cannot load the app's Inter files.