Skip to content

feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown - #52935

Merged
OutThisLife merged 5 commits into
mainfrom
bb/desktop-inline-rendering
Jun 26, 2026
Merged

feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown#52935
OutThisLife merged 5 commits into
mainfrom
bb/desktop-inline-rendering

Conversation

@OutThisLife

@OutThisLife OutThisLife commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Inline rich rendering for the desktop chat markdown surface. A bare autolinked
URL unfurls to a provider embed; ```mermaid / ```svg fences render as
diagrams; > [!NOTE]-style blockquotes become GFM alert callouts. Diagrams open
into a reusable pan/zoom viewer. Embeds are privacy-gated — nothing reaches a
third party until the user consents. Detection is pure and testable; each
renderer is a lazy split chunk pulled only when first used.

Screenshot 2026-06-26 at 02 10 33 Screenshot 2026-06-26 at 02 10 40 Screenshot 2026-06-26 at 02 10 55 Screenshot 2026-06-26 at 02 11 39 Screenshot 2026-06-26 at 02 12 55

Renderers

  • YouTube / Vimeo — plain iframe, 16:9, capped at 33dvh.
  • Google Maps / OpenStreetMap — plain iframe; a ScrollGate blocks wheel until ⌘ is held (zoom), so maps never hijack transcript scroll.
  • X / Instagram — the provider's official blockquote + embed script rendered in-document so it self-sizes (no measuring); X is theme-aware + transparent.
  • TikTok — the official dark video player (player/v1), 9:16.
  • Pinterestassets.pinterest.com/ext/embed.html iframe at the small pin size.
  • Spotifytheme=0 dark player.
  • Mermaid — lazy mermaid, theme-aware; click to open the diagram in the pan/zoom viewer (copy-as-PNG). Falls back to source while streaming/on error.
  • Inline SVG — lazy, DOMPurify-sanitised, left-aligned and size-capped.
  • GFM alerts — note / tip / important / warning / caution.

Privacy consent (per @ethernet8023's review)

  • Embeds don't fetch until the user consents. Default ask: a placeholder sized to the embed shows a commit-style split button — Load (this embed) with a caret for Always allow (persisted per service).
  • Appearance → Inline Embeds global control: Ask / Always / Off (Off keeps plain links), with a one-click reset for the allowed-services list.
  • Purely client-side (persistentAtom: $embedMode + per-service allowlist) — the renderer makes the request, so it never touches the gateway/config.yaml.
  • Local renderers (Mermaid, SVG, alerts) make no outbound request and are never gated.

Zoom/pan primitive

  • components/ui/use-zoom-pan.ts — headless transform hook: wheel zooms toward the cursor, drag pans, buttons zoom/reset (clamped 0.25×–8×).
  • components/ui/zoomable.tsx<Zoomable>: click any content to open a full overlay with the pan/zoom stage, a toolbar, and an optional copy/export action. Content-agnostic.
  • lib/svg-image.tscopySvgAsPng() rasterises an SVG to PNG for the clipboard (text fallback).

Sizing & theming (pure CSS, no measuring)

  • Ratio embeds cap their WIDTH off the ratio (min(maxW, 100%, calc(33dvh * ratio))) so height tops out at 33dvh while scaling.
  • Plain iframes let the wheel chain to the transcript; only maps gate scroll.
  • Embeds are color-scheme-matched to their content to avoid the browser's opaque white Canvas backdrop (the white-corner artifact).

Structure

  • embeds/providers/* — pure URL → descriptor matchers (+ detect.test.ts).
  • embeds/registry.tsx — fenced-language → lazy renderer table.
  • embeds/url-embed.tsx — consent gate + inline card dispatch + content-visibility virtualization.
  • embeds/embed-consent.tsx + store/embed-consent.ts — the placeholder + consent state.
  • embeds/<provider>-embed.tsx — one lazy renderer per kind.
  • embeds/{rich-boundary,fail,scroll-gate,escape-html,use-is-dark,embed-size} — shared primitives.
  • electron/embed-referer.cjs — YouTube Referer stamp on the embed webview partition.

Test plan

  • npm run typecheck (apps/desktop)
  • eslint clean
  • vitest run src/components/assistant-ui/embeds (27 tests)
  • Paste bare YouTube / Vimeo / maps / X / Instagram / Pinterest / TikTok / Spotify URLs → placeholder, then Load / Always allow
  • ```mermaid + ```svg fences and > [!NOTE] blockquotes render
  • Click a mermaid diagram → pan/zoom viewer, copy-as-PNG
  • Appearance → Inline Embeds: Ask / Always / Off behave; Always-allow persists per service; reset clears it

@OutThisLife
OutThisLife requested a review from a team June 26, 2026 07:07
@OutThisLife OutThisLife changed the title feat(desktop): inline rich embeds in assistant markdown feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown Jun 26, 2026
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/desktop Electron desktop app (apps/desktop/*) labels Jun 26, 2026
Pure, synchronous URL→descriptor matchers for YouTube, Vimeo, Instagram,
Pinterest, TikTok, X, Spotify, Google Maps and OpenStreetMap, plus the shared
embed primitives (error boundary, fail card, escape-html, dark-mode hook,
sizing token). Declares the mermaid + dompurify deps used by the fenced
renderers.
Per-kind renderers, each a lazy split chunk: plain-iframe video/maps (wheel
chains to the transcript; maps gate scroll behind ⌘), the in-document
blockquote-script path for X/Instagram, the dark Spotify player, and the
YouTube iframe. Adds Mermaid and DOMPurify-sanitised SVG fences and GFM alert
callouts, all sized to 33dvh and theme-matched to avoid white color-scheme
artifacts. Main-process stamps a Referer on YouTube embed requests.
Wire the embeds module into the markdown surface: bare provider autolinks unfurl
to inline embeds, ```mermaid/```svg fences route to the rich renderers, and
`> [!NOTE]`-style blockquotes become alert callouts. Labeled links stay plain.
@OutThisLife
OutThisLife force-pushed the bb/desktop-inline-rendering branch 3 times, most recently from 2692e88 to c9b8c9e Compare June 26, 2026 08:38
Add a content-agnostic Zoomable primitive (useZoomPan hook + overlay viewer):
click to open full-screen, wheel-zoom toward the cursor, drag to pan, toolbar
zoom/reset, and an optional copy action. Wire Mermaid diagrams into it with
copy-as-PNG; reusable for other inline content later.
@OutThisLife
OutThisLife force-pushed the bb/desktop-inline-rendering branch from c9b8c9e to da0ed97 Compare June 26, 2026 08:40
@OutThisLife
OutThisLife enabled auto-merge June 26, 2026 10:29

@ethernet8023 ethernet8023 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i know a lot of people would be upset to see hermes agent sending traffic to youtube/etc without asking first. wdyt about an "allow embeds" setting, and a button that says "load youtube embed" that's like the size of the player, until it's turned on? maybe with an "always allow embeds" setting..?

other than that, this looks fine, i'm just hesitant about reaching out to third party websites (and thus giving em info they can use to track me as a user) without explicitly enabling it

@OutThisLife

Copy link
Copy Markdown
Collaborator Author

Great call — fully agree, shipped a consent gate in 6cd84f9.

Modeled on the tool-approval UX so it feels native:

  • Default is ask: each embed renders a placeholder sized exactly like the player (no layout shift) with Load (this embed only) and a dropdown with Always allow (persisted), Open in browser, and Never show embeds.
  • Per-service, not just global: "Always allow YouTube" won't allow X, etc.
  • Appearance → Inline Embeds setting for the global default: Ask / Always / Off (Off keeps plain links).
  • Purely client-side (desktop-local persistentAtom: $embedMode + per-service allowlist) — the renderer is what makes the request, so it never touches the gateway/config.yaml and gates with zero round-trip.
  • Local renderers (Mermaid, inline SVG, GFM alerts) are never gated — they make no outbound request.

So nothing reaches a third party until the user opts in, per-embed or per-service. Thanks for the nudge.

@OutThisLife
OutThisLife force-pushed the bb/desktop-inline-rendering branch from 6cd84f9 to 292716f Compare June 26, 2026 18:27
@OutThisLife

Copy link
Copy Markdown
Collaborator Author

Polished the consent UI a bit since the first pass: the placeholder now uses the same SplitButton primitive as the commit button — primary Load , caret for Always allow — sized to the embed's footprint, with the global Ask/Always/Off in Appearance settings. Behavior is unchanged from my earlier comment; just wanted it to match the rest of the app rather than a one-off control.

Embeds reach out to third parties on render, so default to a placeholder that
mirrors the tool-approval UX: "Load <service>" (this embed) or "Always allow
<service>" (persisted). A desktop-local store ($embedMode ask|always|off +
per-service allowlist) gates the fetch with zero gateway round-trip; an
Appearance setting controls the global default. Local renderers (mermaid, svg,
alerts) are never gated. Addresses review feedback on outbound third-party
requests.
@OutThisLife
OutThisLife force-pushed the bb/desktop-inline-rendering branch from 292716f to db6ced4 Compare June 26, 2026 18:35
@OutThisLife
OutThisLife disabled auto-merge June 26, 2026 18:35
@OutThisLife
OutThisLife enabled auto-merge June 26, 2026 18:35
@OutThisLife
OutThisLife merged commit ed96210 into main Jun 26, 2026
21 checks passed
@OutThisLife
OutThisLife deleted the bb/desktop-inline-rendering branch June 26, 2026 18:36
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…line-rendering

feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
teknium1 added a commit that referenced this pull request Jul 7, 2026
…mbeds registry

Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path,
theme frozen at first load) and wire preview-file.tsx's MarkdownCode through
the existing RichCodeBlock registry from #52935 instead. One mermaid init
path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error
fallback — and the preview pane gets svg fences for free. Shiki block stays
as the fallback for all other languages.
teknium1 added a commit that referenced this pull request Jul 7, 2026
…mbeds registry

Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path,
theme frozen at first load) and wire preview-file.tsx's MarkdownCode through
the existing RichCodeBlock registry from #52935 instead. One mermaid init
path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error
fallback — and the preview pane gets svg fences for free. Shiki block stays
as the fallback for all other languages.
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
…line-rendering

feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…line-rendering

feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…mbeds registry

Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path,
theme frozen at first load) and wire preview-file.tsx's MarkdownCode through
the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init
path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error
fallback — and the preview pane gets svg fences for free. Shiki block stays
as the fallback for all other languages.
justemu pushed a commit to justemu/hermes-agent that referenced this pull request Jul 18, 2026
…mbeds registry

Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path,
theme frozen at first load) and wire preview-file.tsx's MarkdownCode through
the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init
path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error
fallback — and the preview pane gets svg fences for free. Shiki block stays
as the fallback for all other languages.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…line-rendering

feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…mbeds registry

Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path,
theme frozen at first load) and wire preview-file.tsx's MarkdownCode through
the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init
path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error
fallback — and the preview pane gets svg fences for free. Shiki block stays
as the fallback for all other languages.
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…line-rendering

feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…mbeds registry

Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path,
theme frozen at first load) and wire preview-file.tsx's MarkdownCode through
the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init
path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error
fallback — and the preview pane gets svg fences for free. Shiki block stays
as the fallback for all other languages.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…mbeds registry

Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path,
theme frozen at first load) and wire preview-file.tsx's MarkdownCode through
the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init
path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error
fallback — and the preview pane gets svg fences for free. Shiki block stays
as the fallback for all other languages.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/desktop Electron desktop app (apps/desktop/*) P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants