feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown - #52935
Conversation
Pure, synchronous URL→descriptor matchers for YouTube, Vimeo, Instagram, Pinterest, TikTok, X, Spotify, Google Maps and OpenStreetMap, plus the shared embed primitives (error boundary, fail card, escape-html, dark-mode hook, sizing token). Declares the mermaid + dompurify deps used by the fenced renderers.
Per-kind renderers, each a lazy split chunk: plain-iframe video/maps (wheel chains to the transcript; maps gate scroll behind ⌘), the in-document blockquote-script path for X/Instagram, the dark Spotify player, and the YouTube iframe. Adds Mermaid and DOMPurify-sanitised SVG fences and GFM alert callouts, all sized to 33dvh and theme-matched to avoid white color-scheme artifacts. Main-process stamps a Referer on YouTube embed requests.
Wire the embeds module into the markdown surface: bare provider autolinks unfurl to inline embeds, ```mermaid/```svg fences route to the rich renderers, and `> [!NOTE]`-style blockquotes become alert callouts. Labeled links stay plain.
2692e88 to
c9b8c9e
Compare
Add a content-agnostic Zoomable primitive (useZoomPan hook + overlay viewer): click to open full-screen, wheel-zoom toward the cursor, drag to pan, toolbar zoom/reset, and an optional copy action. Wire Mermaid diagrams into it with copy-as-PNG; reusable for other inline content later.
c9b8c9e to
da0ed97
Compare
ethernet8023
left a comment
There was a problem hiding this comment.
i know a lot of people would be upset to see hermes agent sending traffic to youtube/etc without asking first. wdyt about an "allow embeds" setting, and a button that says "load youtube embed" that's like the size of the player, until it's turned on? maybe with an "always allow embeds" setting..?
other than that, this looks fine, i'm just hesitant about reaching out to third party websites (and thus giving em info they can use to track me as a user) without explicitly enabling it
|
Great call — fully agree, shipped a consent gate in 6cd84f9. Modeled on the tool-approval UX so it feels native:
So nothing reaches a third party until the user opts in, per-embed or per-service. Thanks for the nudge. |
6cd84f9 to
292716f
Compare
|
Polished the consent UI a bit since the first pass: the placeholder now uses the same |
Embeds reach out to third parties on render, so default to a placeholder that mirrors the tool-approval UX: "Load <service>" (this embed) or "Always allow <service>" (persisted). A desktop-local store ($embedMode ask|always|off + per-service allowlist) gates the fetch with zero gateway round-trip; an Appearance setting controls the global default. Local renderers (mermaid, svg, alerts) are never gated. Addresses review feedback on outbound third-party requests.
292716f to
db6ced4
Compare
…line-rendering feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
…mbeds registry Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path, theme frozen at first load) and wire preview-file.tsx's MarkdownCode through the existing RichCodeBlock registry from #52935 instead. One mermaid init path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error fallback — and the preview pane gets svg fences for free. Shiki block stays as the fallback for all other languages.
…mbeds registry Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path, theme frozen at first load) and wire preview-file.tsx's MarkdownCode through the existing RichCodeBlock registry from #52935 instead. One mermaid init path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error fallback — and the preview pane gets svg fences for free. Shiki block stays as the fallback for all other languages.
…line-rendering feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
…line-rendering feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
…mbeds registry Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path, theme frozen at first load) and wire preview-file.tsx's MarkdownCode through the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error fallback — and the preview pane gets svg fences for free. Shiki block stays as the fallback for all other languages.
…mbeds registry Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path, theme frozen at first load) and wire preview-file.tsx's MarkdownCode through the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error fallback — and the preview pane gets svg fences for free. Shiki block stays as the fallback for all other languages.
…line-rendering feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
…mbeds registry Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path, theme frozen at first load) and wire preview-file.tsx's MarkdownCode through the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error fallback — and the preview pane gets svg fences for free. Shiki block stays as the fallback for all other languages.
…line-rendering feat(desktop): inline rich embeds, diagrams & alerts in assistant markdown
…mbeds registry Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path, theme frozen at first load) and wire preview-file.tsx's MarkdownCode through the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error fallback — and the preview pane gets svg fences for free. Shiki block stays as the fallback for all other languages.
…mbeds registry Drop the duplicate mermaid-block.tsx (own mermaid.initialize + render path, theme frozen at first load) and wire preview-file.tsx's MarkdownCode through the existing RichCodeBlock registry from NousResearch#52935 instead. One mermaid init path, theme-flip re-init, Zoomable + copy-as-PNG, RichBoundary error fallback — and the preview pane gets svg fences for free. Shiki block stays as the fallback for all other languages.
Summary
Inline rich rendering for the desktop chat markdown surface. A bare autolinked
URL unfurls to a provider embed;
```mermaid/```svgfences render asdiagrams;
> [!NOTE]-style blockquotes become GFM alert callouts. Diagrams openinto a reusable pan/zoom viewer. Embeds are privacy-gated — nothing reaches a
third party until the user consents. Detection is pure and testable; each
renderer is a lazy split chunk pulled only when first used.
Renderers
player/v1), 9:16.assets.pinterest.com/ext/embed.htmliframe at the small pin size.theme=0dark player.mermaid, theme-aware; click to open the diagram in the pan/zoom viewer (copy-as-PNG). Falls back to source while streaming/on error.Privacy consent (per @ethernet8023's review)
ask: a placeholder sized to the embed shows a commit-style split button — Load (this embed) with a caret for Always allow (persisted per service).Ask/Always/Off(Offkeeps plain links), with a one-click reset for the allowed-services list.persistentAtom:$embedMode+ per-service allowlist) — the renderer makes the request, so it never touches the gateway/config.yaml.Zoom/pan primitive
components/ui/use-zoom-pan.ts— headless transform hook: wheel zooms toward the cursor, drag pans, buttons zoom/reset (clamped 0.25×–8×).components/ui/zoomable.tsx—<Zoomable>: click any content to open a full overlay with the pan/zoom stage, a toolbar, and an optional copy/export action. Content-agnostic.lib/svg-image.ts—copySvgAsPng()rasterises an SVG to PNG for the clipboard (text fallback).Sizing & theming (pure CSS, no measuring)
min(maxW, 100%, calc(33dvh * ratio))) so height tops out at 33dvh while scaling.color-scheme-matched to their content to avoid the browser's opaque white Canvas backdrop (the white-corner artifact).Structure
embeds/providers/*— pure URL → descriptor matchers (+detect.test.ts).embeds/registry.tsx— fenced-language → lazy renderer table.embeds/url-embed.tsx— consent gate + inline card dispatch + content-visibility virtualization.embeds/embed-consent.tsx+store/embed-consent.ts— the placeholder + consent state.embeds/<provider>-embed.tsx— one lazy renderer per kind.embeds/{rich-boundary,fail,scroll-gate,escape-html,use-is-dark,embed-size}— shared primitives.electron/embed-referer.cjs— YouTube Referer stamp on the embed webview partition.Test plan
npm run typecheck(apps/desktop)eslintcleanvitest run src/components/assistant-ui/embeds(27 tests)```mermaid+```svgfences and> [!NOTE]blockquotes render