feat: autovisualiser of structured data with mcp-ui #4153

michaelneale · 2025-08-18T07:59:40Z

This proposal uses mcp-ui with an (opional) fast, lightweight and 0 dependency built in MCP server called "Auto visualizer" which uses libraries like d3 (which are bundled into inline HTML so no external access required) to make charts and graphs and all sorts of visualisation.

Benefits: you don't need to know or be experts in presentation, you can just ask for goose to make an appropriate visualisation of anything you want and it can find and use the appropriate diagram for you to interact with inline, to complement the textual output.

and interactive things, all as tool calls.

Currently

Todo:

need to make sure that the blobs are not too large if part of session, d3 type resources should be JIT served not inlined to html

michaelneale · 2025-08-18T08:02:05Z

crates/goose-cli/src/commands/configure.rs

    match extension_id {
        "developer" => "Developer Tools".to_string(),
        "computercontroller" => "Computer Controller".to_string(),
+        "autovisualiser" => "Auto Visualiser".to_string(),


this really only applies to gui (MCP-ui) but not sure of protocol here - if we don't have it in CLI then can't turn it on there (but maybe that is ok?)

index.html

aharvard · 2025-08-18T21:01:24Z

crates/goose-mcp/src/autovisualiser/chart_template.html

+    <title>Interactive Chart</title>
+
+    <script>
+        {{CHART_MIN}}


Over in https://github.com/block/goose/pull/4164/files#diff-10caa690bbc229198a1a7c3c862db533dcdf27944a4b92931d5c47aabe7fa500R63 I have upgraded MCP-UI and started to experiment with the iframe sandbox value by adding allow-forms ...

I mention this because I think we might be able to use a <script src="<d3-script-here.js"> if we adjust the sandbox value to include allow-same-origin, and if we don't want to inline it like this.

IDK how insecure the allow-same-origin value is. If we needed, we could only set that value when it comes from one of our native tools like this one (and turn it off when we load an MCP UI from an 3rd party extension)

How do you feel about storing the minified JS in Goose as opposed to getting it from a CDN via <script src="" />

I think for these cases, can go either way, but for now having it served up from goose is very fast and zero risk which I think feels right (for now) but anything more complicated, yeah.

for anything recommended/out of the box, would ideally not pull in external deps at display time I think - so as this works is ideal for this case (for an arbitrary MCP that people may or may not use - I think can be more open).

Agreed, inlining the deps feels right

* main: (67 commits) blog: Transforming AI Assistance with Goose Mentor Mode (#4151) upgraded all npm packages and fixed related issues (#4072) Docs: @-mentions in goosehints (#4171) fix: consistent font sizing in ToolCallWithResponse (#4167) Temporarily disable TODO Tool (#4158) docs: add integrated MCP server config to jetbrains tutorial (#4120) docs: remove figma MCP from suggested servers (#4123) Blog: The AI Skeptic’s Guide to Context Windows (#4152) Docs: Auto-compact context (#4116) chore(deps): bump brace-expansion from 1.1.11 to 1.1.12 in /documentation (#4149) Recipe config to limit tool availability (#4020) docs: fix warning message (#4148) feat: adds cursor-agent as a cli provider (#4101) chore: remove vector search tool selection strategy (#3933) docs: add streamable_http install links (#4130) feat: iterating on summarize oneshot prompt (#4113) feat(mcp): Persist OAuth credentials to keyring (#4007) Sanitize Tags Unicode Block at prompt level (#4047) Fixing typos (#4114) chore(release): release version 1.4.0 (#4069) ...

michaelneale · 2025-08-19T00:42:28Z

hrm current problem:

A React Element from an older version of React was rendered. This is not supported. It can happen if:
- Multiple copies of the "react" package is used.
- A library pre-bundled an old copy of "react" or "react/jsx-runtime".
- A compiler tries to "inline" JSX instead of using the runtime.
- ```

aharvard · 2025-08-20T01:00:17Z

FYI - a fix is in the works MCP-UI-Org/mcp-ui#91

michaelneale · 2025-08-22T07:57:48Z

fyi @aharvard think this is ready to have a good look. Seems to work now (improved the data validation). there isn't any smarts in the templates for responding to dark mode yet, or posting tings back, but this is a start and still pretty simple and fast.

aharvard

This is excellent!

Works well with other tools to acquire proper data.

I did get stuck in an agent loop once; it kept trying different visualizations, and I don't know why. There might be room to add some lightweight data validation against the D3 rendering requirements. Then, we could return a text statement that clearly indicates whether the visualization was successful or failed for some reason. Result text like that might properly exit the loop. I'd love to learn what you think about this choreography!

Here's a recording of my experience:

goose-dataviz-small.mov

alexhancock · 2025-08-22T18:09:01Z

ui/desktop/src/components/ui/Pill.tsx

 }: PillProps) {
-  const baseStyles = 'inline-flex items-center justify-center rounded-full transition-all duration-300 ease-out font-medium';
-
+  const baseStyles =


Are these just formatting changes? Looks like it?

oh good catch - yeah not intended. I am using opus + a fast apply model and sometimes it gets carried away

yeah - looks like the prettier/checker requires it? odd...

michaelneale · 2025-08-23T00:15:11Z

"Then, we could return a text statement that clearly indicates whether the visualization was successful or failed for some reason" - yeah I like that - might be a way to validate the js/total thing in rust side and feed it back (it does push back if the json is invalid, so that can cause it to iterate a bit)

* main: docs: Plan tutorial (#4309) Extensions Modal Improvements (#4293) docs: fixed cicd tutorial pipeline in docs (#4223) Read oltp config from config and env (#4292) release/1.6.0 (#4280) docs: fix broken links in Docker tutorial (#4285) Remove half-second wait, rework auto submit (#4282) Block send until extensions are ready (#4271) fix: improve OpenAI-compatible error handling and add test coverage (#4175) Move To-Do Tool to Session Scope from Agent Scope (#4157) fix: recipe params not being replaced all the time (#4207) chore: removing little-used session sharing feature (#4249) Stop auto scrolling when agent responds and let scroll area handle scrolling to bottom (#4257) restore cli projects from accidental removal during cleanup (#4266) Fix: deep link extension installation to show dialog for headers configuration (#4150)

* main: (24 commits) feat: autovisualiser of structured data with mcp-ui (#4153) docs: Plan tutorial (#4309) Extensions Modal Improvements (#4293) docs: fixed cicd tutorial pipeline in docs (#4223) Read oltp config from config and env (#4292) release/1.6.0 (#4280) docs: fix broken links in Docker tutorial (#4285) Remove half-second wait, rework auto submit (#4282) Block send until extensions are ready (#4271) fix: improve OpenAI-compatible error handling and add test coverage (#4175) Move To-Do Tool to Session Scope from Agent Scope (#4157) fix: recipe params not being replaced all the time (#4207) chore: removing little-used session sharing feature (#4249) Stop auto scrolling when agent responds and let scroll area handle scrolling to bottom (#4257) restore cli projects from accidental removal during cleanup (#4266) Fix: deep link extension installation to show dialog for headers configuration (#4150) feat: Add message queue system with interruption handling (#4179) Start extensions concurrently (#4234) Add X-Title and referer headers on exchange to tetrate (#4250) docs: update View/Edit Recipe menu item name (#4267) ...

* 'main' of github.com:block/goose: fix slow typing with long sessions (#4291) docs: improvements to release docs (#4317) blog: mcp-ui recap blog (#4146) added more logging for recipe creation failures (#4299) Remove a bit from pre-commit that husky says we should (#4286) feat: autovisualiser of structured data with mcp-ui (#4153) docs: Plan tutorial (#4309) Extensions Modal Improvements (#4293) docs: fixed cicd tutorial pipeline in docs (#4223) Read oltp config from config and env (#4292) # Conflicts: # ui/desktop/src/App.tsx

Signed-off-by: Alex Rosenzweig <arosenzweig@squareup.com>

Signed-off-by: Dorien Koelemeijer <dkoelemeijer@squareup.com>

michaelneale added 16 commits August 12, 2025 18:29

checkpoint

3da3bea

flow

6fd0acd

treemap

f9fbc09

just moving around

d523b7e

placeholder for mcp ui server example - to remove

9fb55af

correct resource usage

81d080c

moving to its own mcp server

8053108

sankey with tmp debug

fb5260d

trying base64

6d4dcc5

inline d3

01f43be

embed d3 via template when needed

183de97

now working

137d21c

tidy up

788f757

stripping down

79d9978

adding radar chart

c318bb3

now has radar

ec0496b

michaelneale self-assigned this Aug 18, 2025

michaelneale requested review from aharvard and zanesq August 18, 2025 07:59

michaelneale commented Aug 18, 2025

View reviewed changes

michaelneale added 2 commits August 18, 2025 18:24

adding pie and donut

71e3e72

line and scatter and bar charts

e112829

aharvard reviewed Aug 18, 2025

View reviewed changes

index.html Outdated Show resolved Hide resolved

aharvard reviewed Aug 18, 2025

View reviewed changes

michaelneale added 4 commits August 19, 2025 09:55

more cleaning up

901106a

cleaning up cc

f02c7a8

migrate to new errors

7854062

need to use goose react version for mcp-ui

2e2369f

michaelneale added 6 commits August 19, 2025 13:31

maps

6217d4b

clarify

046c3df

chord diagram

869a2ff

final treemap

fba77b0

Merge branch 'main' into micn/hi-viz-autovisualiser

8c33d60

fmt for new router for autoviz

3328bde

switch to appropriate audience

11a1433

michaelneale marked this pull request as ready for review August 20, 2025 10:46

michaelneale added 5 commits August 21, 2025 10:35

Merge branch 'main' into micn/hi-viz-autovisualiser

4a60548

updating

3049e45

tightening up

277a81a

test coverage and tightening up the prompts

b245e44

cleaning up

2720fc5

fmt

90f7f69

aharvard approved these changes Aug 22, 2025

View reviewed changes

alexhancock reviewed Aug 22, 2025

View reviewed changes

alexhancock approved these changes Aug 22, 2025

View reviewed changes

michaelneale added 2 commits August 25, 2025 08:59

wording and small prompt guide

fc515b4

michaelneale merged commit 3e1557b into main Aug 25, 2025
10 checks passed

michaelneale deleted the micn/hi-viz-autovisualiser branch August 25, 2025 00:42

shellz-n-stuff pushed a commit to shellz-n-stuff/goose that referenced this pull request Aug 27, 2025

feat: autovisualiser of structured data with mcp-ui (block#4153)

11cde98

Signed-off-by: Alex Rosenzweig <arosenzweig@squareup.com>

This was referenced Aug 27, 2025

chore(release): release version 1.7.0 #4372

Closed

chore(release): release version 1.7.0 #4391

Merged

dorien-koelemeijer pushed a commit to dorien-koelemeijer/goose that referenced this pull request Sep 2, 2025

feat: autovisualiser of structured data with mcp-ui (block#4153)

c78f628

Signed-off-by: Dorien Koelemeijer <dkoelemeijer@squareup.com>

feat: autovisualiser of structured data with mcp-ui #4153

feat: autovisualiser of structured data with mcp-ui #4153

Uh oh!

Conversation

michaelneale commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelneale commented Aug 19, 2025

Uh oh!

aharvard commented Aug 20, 2025

Uh oh!

michaelneale commented Aug 22, 2025

Uh oh!

aharvard left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelneale commented Aug 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

michaelneale commented Aug 18, 2025 •

edited

Loading