🎨 feat: MCP UI basic integration by samuelpath · Pull Request #9299 · danny-avila/LibreChat

samuelpath · 2025-08-27T10:24:15Z

Summary

Here's a video describing my intention.

There's a project called MCP UI. Its goal is to allow MCP server tool calls to send back not just text content to the client, but also UI elements in various forms (it could be HTML or URLs to embed as iFrames). That library also defines a way for the client and the iFrames elements to communicate through message passing.

Some popular clients like Postman (see LinkedIn announcement) and Goose by Block (see blog post announcement) have already built support.

A few weeks ago, Shopify announced agent-kit (see tweet by Tobi the CEO), which contains MCP servers serving UI resources so that any client can integrate Shopify commerce components into chat AI experiences. All the work on Shopify side to test these components was done on a branch on our local fork of LibreChat.

Now we want to make such integrations available to a wider public, so that any LibreChat user can start tinkering with MCP Servers sending back UI resources (coming from Shopify or elsewhere).

Please note that I'm first waiting to have the general idea validated before writing detailed tests.

Example of a MCP Server tool serving UI Resources as URL

If you use Anthropic's MCP inspector, you can use the following public MCP Server:

Transport Type: Streamable HTTP
URL: https://mcpstorefront.com/?storedomain=allbirds.com

Connect to it, then go to the the tools, and make a query to search_shop_catalog and enter the following value in both the query and context parameters: "Looking for men's sneakers in red color under $200":

You'll get this as full result.

We see that the content array contains one object of type text and then multiple objects of type resource (see the MCP docs on Resources). Here's a sample one:

{
  "type": "resource",
  "resource": {
    "uri": "ui://product/gid://shopify/Product/7009816019024",
    "mimeType": "text/uri-list",
    "text": "https://mcpstorefront.com/img/storefront/product.component.html?store_domain=allbirds.com&product_handle=womens-tree-toppers-natural-black-blizzard&product_id=gid://shopify/Product/7009816019024&mode=tool"
  }
}

The way these UI Resources are identified is by their uri starting with the ui:// prefix.

Example of a MCP Server tool serving UI Resources as HTML

Now in Anthropic's MCP Inspector, use the following public MCP Server:

Transport Type: Streamable HTTP
URL: https://mcp-aharvard.netlify.app/mcp

Make a query to the get-weather tool:

You'll get this as full result.

The UI Resource looks like this:

{
  "type": "resource",
  "resource": {
    "uri": "ui://mcp-aharvard/weather-card",
    "mimeType": "text/html",
    "text": "<HTML CODE>"
  },
  // ...
}

So we see that we can have either mimeType as text/uri-list with the text property being a URL to embed, or mimeType as text/html having the text property as HTML code to render by the client.

The current behaviour

So what happens before our changes if we connect LibreChat to that MCP server and make a request to that tool?

Let's add 2 MCP servers supporting UI Resources to the Librechat.yaml configuration file:

mcpServers:
  mcp-ui-allbirds:
    type: 'streamable-http'
    url: https://mcpstorefront.com/?storedomain=allbirds.com

  mcp-aharvard:
      type: 'streamable-http'
      url: https://mcp-aharvard.netlify.app/mcp

This is what we get in the section where we see the tool call details (see this gist for the exact value):

If we replace the \n by actual line breaks, this is what we get:

https://mcpstorefront.com/img/storefront/product.component.html?store_domain=allbirds.com&product_handle=mens-tree-runner-go-blizzard-bold-red&product_id=gid://shopify/Product/7091625328720&mode=tool
Resource URI: ui://product/gid://shopify/Product/7091625328720
Type: text/uri-list

https://mcpstorefront.com/img/storefront/product.component.html?store_domain=allbirds.com&product_handle=mens-tree-runners-blizzard-bold-red&product_id=gid://shopify/Product/7091621527632&mode=tool
Resource URI: ui://product/gid://shopify/Product/7091621527632
Type: text/uri-list

# and so on…

This is coming from that following code in packages/api/src/mcp/parsers.ts:

LibreChat/packages/api/src/mcp/parsers.ts

Lines 143 to 161 in 3ab1bd6

    
           resource: (item) => { 
        
             const resourceText = []; 
        
             if (item.resource.text != null && item.resource.text) { 
        
               resourceText.push(item.resource.text); 
        
             } 
        
             if (item.resource.uri.length) { 
        
               resourceText.push(`Resource URI: ${item.resource.uri}`); 
        
             } 
        
             if (item.resource.name) { 
        
               resourceText.push(`Resource: ${item.resource.name}`); 
        
             } 
        
             if (item.resource.description) { 
        
               resourceText.push(`Description: ${item.resource.description}`); 
        
             } 
        
             if (item.resource.mimeType != null && item.resource.mimeType) { 
        
               resourceText.push(`Type: ${item.resource.mimeType}`); 
        
             } 
        
             currentTextBlock += (currentTextBlock ? '\n\n' : '') + resourceText.join('\n'); 
        
           },

What the parser is currently doing is converting the UI resource coming from the MCP tool response into text content to be sent to the LLM, which simply doesn't know what to do about it currently.

What we are changing

Now, when the backend receives such a response containing UI resources, we don't parse them as text elements with simple line breaks (which don't get rendered properly in the front-end anyway).

Instead, we accumulate them in an array, and then encode its value as b64 as an additional object in the formattedContent array with metadata: 'ui_resources' in order to identify it on the frontend.

This way, once the front-end receives that output, it can parse it back into UI resources more easily.

Another option we explored was to not use the text content at all but rather use the artifact variable, which Llangchain provides as a way to pass data down through the pipeline without sending it to the LLM and also without having to serialize and deserialize it. The logic is handled below:

LibreChat/packages/api/src/mcp/parsers.ts

Lines 178 to 183 in 3ab1bd6

    
           const artifacts = imageUrls.length ? { content: imageUrls } : undefined; 
        
           if (CONTENT_ARRAY_PROVIDERS.has(provider)) { 
        
             return [formattedContent, artifacts]; 
        
           } 
        
           return [currentTextBlock, artifacts];

However, we realized that unlike what was described in the Llangchain docs, the LibreChat agents library was parsing the artifacts as text to add them to the text content which is sent to the LLM. I proposed an update for that which was rejected since that approach was used as a workaround for other purposes. Also, getting the artifacts in the frontend where we also get the content wasn't straightforward, so it seemed easier for me here to use the existing data structure rather than sending the artifacts as such to the frontend rendering code.

Once the front-end receives the text output, it extracts then parses the encoded resources and removes them from the text output:

LibreChat/client/src/components/Chat/Messages/Content/ToolCallInfo.tsx

Lines 57 to 70 in 87b95d8

    
           // Extract ui_resources from the output to display them in the UI 
        
           let uiResources: UIResource[] = []; 
        
           if (output?.includes('ui_resources')) { 
        
             const parsedOutput = JSON.parse(output); 
        
             const uiResourcesItem = parsedOutput.find( 
        
               (contentItem) => contentItem.metadata === 'ui_resources', 
        
             ); 
        
             if (uiResourcesItem?.text) { 
        
               uiResources = JSON.parse(atob(uiResourcesItem.text)) as UIResource[]; 
        
             } 
        
             output = JSON.stringify( 
        
               parsedOutput.filter((contentItem) => contentItem.metadata !== 'ui_resources'), 
        
             ); 
        
           }

It then displays the UI elements in MCP UI's UIResourceRenderer component, which abstracts away the the iFrame, if there is one UI resource, or in a custom UIResourceGrid component if there are multiple elements, which allows to display UI resources as a grid:

LibreChat/client/src/components/Chat/Messages/Content/ToolCallInfo.tsx

Lines 89 to 106 in 9cb70b7

    
           <div className="my-2 text-sm font-medium text-text-primary"> 
        
             {localize('com_ui_ui_resources')} 
        
           </div> 
        
           <div> 
        
             {uiResources.length > 1 && <UIResourceGrid uiResources={uiResources} />} 
        
             {uiResources.length === 1 && ( 
        
               <UIResourceRenderer 
        
                 resource={uiResources[0]} 
        
                 onUIAction={async (result) => { 
        
                   console.log('Action:', result); 
        
                 }} 
        
                 htmlProps={{ 
        
                   autoResizeIframe: { width: true, height: true }, 
        
                 }} 
        
               /> 
        
             )} 
        
           </div>

We get something like this for multiple UI resources shared as embeddable URLs:

In this current iteration, when the user clicks on a UI element that should trigger an action, we simply log the intent in the console:

And we get something like this for a single UI Resource embedded as HTML rendered by the client:

What we will do next

In a follow-up PR, we would like to:

Add two new types of artifacts, one for a single UI element and one for multiple UI elements, using a syntax like the one below

:::artifact{identifier="p1" type="mcp-ui-single-element" title="Men's Court Graffik Shoes - Black/Red"}
https://cdn.shopify.com/storefront/product-catalog-details.component?store_domain=www.dcshoes.com&product_handle=dc-shoes-mens-court-graffik-shoes-blackred-blr
:::

Update the prompt sent to the LLM when UI resources are present so that it knows it can use MCP UI artifacts in the response interspersed with text.
Have the client handle the intent payload sent by the UI elements so that the LLM can trigger further actions, like other tool calls (for instance if the user clicks on Add to cart,

Change Type

New feature (non-breaking change which adds functionality)
This change requires a documentation update (potentially, it could be nice to explicit this support)

Testing

Please describe your test process and include instructions so that we can reproduce your test. If there are any important variables for your testing configuration, list them here.

Test Configuration:

Checklist

Please delete any irrelevant options.

My code adheres to this project's style guidelines
I have performed a self-review of my own code
I have commented in any complex areas of my code
I have made pertinent documentation changes (I will do this once this PR is accepted, so as not to waste effort)
My changes do not introduce new warnings
I have written tests demonstrating that my changes are effective or that my feature works (I am working on it, but I'm already looking for feedback so as not to write tests for nothing)
Local unit tests pass with my changes
Any changes dependent on mine have been merged and published in downstream modules.
A pull request for updating the documentation has been submitted (I will do this once this PR is accepted, so as not to waste effort)

packages/api/src/mcp/parsers.ts

mawburn

Looks good for the most part! A few comments:

This could be a security issue by allowing untrusted sources run iframes, without something like this:

 <iframe
    sandbox="allow-scripts allow-same-origin allow-forms"
    src={resource.text}
  />

I don't see the iframe specific code in this, but it looks like you can inject this without directly modifying those pieces.

Should we have loading states for the iframes or am I missing that?

We're missing a lot of tests, especially for the parser logic and component rendering

mawburn · 2025-08-27T17:38:39Z

client/src/components/Chat/Messages/Content/ToolCallInfo.tsx

+    } catch (error) {
+      console.error('Error parsing ui_resources:', error);
+    }


Should we have something to fallback to? Will this break the UI?

I'm going to test by sending badly formatted JSON, thanks!

@mawburn actually this is what we get with a parsing error. I'm going to also wrap the UI Resources title conditionally upon parsing success, good catch man!

mawburn · 2025-08-27T17:40:41Z

client/src/components/Chat/Messages/Content/ToolCallInfo.tsx

+                  onUIAction={async (result) => {
+                    console.log('Action:', result);
+                  }}


Should this do something?

This is what I say in the description, it's a basic version for now:

liady · 2025-08-27T18:21:56Z

Hi @mawburn, thanks for the comments! I'm from the mcp-ui team, and also worked on integrating it into Shopify.
As for security - <UIResourceRenderer> already renders the iframes securely as sandboxed, using only allow-scripts allow-same-origin. Communication in and out of the frames happens only using postMessages.

Check it out here: https://mcpui.dev/guide/client/resource-renderer#security-considerations

mawburn · 2025-08-27T18:27:36Z

@liady

I'm from the mcp-ui team

I know, Sam is on my team. 😄 Thanks for your correction! I am also the one who deployed LibreChat for us originally. Blog Post

samuelpath · 2025-08-27T18:39:42Z

@mawburn Regarding the tests, this is what I said in the description. I first want to validate the idea by @danny-avila before taking the time to write extensive tests, so as not to waste efforts.

sbruel · 2025-08-28T13:32:03Z

packages/api/src/mcp/parsers.ts

  'bedrock',
 ]);
-const CONTENT_ARRAY_PROVIDERS = new Set(['google', 'anthropic', 'azureopenai', 'openai']);
+const CONTENT_ARRAY_PROVIDERS = new Set(['google', 'anthropic', 'openai']);


is removing azureopenai on purpose?

sbruel · 2025-08-28T13:51:05Z

packages/api/src/mcp/parsers.ts

  }

+  if (uiResources.length) {
+    currentTextBlock += `<ui_resources>${JSON.stringify(uiResources)}</ui_resources>`;


formattedContent is already an array of objects that have a type, couldn't we do something like

formattedContent.push({ type: 'text', text: currentTextBlock }); formattedContent.push({ type: 'ui_resources', data: uiResources });

so we don't have to parse the text with regexp later?

Thanks @sbruel, this was actually my initial approach. The issue here is that this formattedContent is sent to the LLM, and so we can only use types that the main LLMs recognize.

Here's the error message from OpenAI when using that approach:

An error occurred while processing the request: 400 Invalid value: 'resource'. Supported values are: 'text', 'image_url', 'input_audio', 'refusal', 'audio', and 'file'.

…l info

sbruel · 2025-08-29T17:07:15Z

client/src/components/Chat/Messages/Content/ToolCallInfo.tsx


+  // Extract ui_resources from the output to display them in the UI
+  let uiResources: UIResource[] = [];
+  if (output?.includes('ui_resources')) {


we probably need more solid error handling if the output includes ui_resources as text, for some reason, without being a proper ui_resource

samuelpath force-pushed the feat/mcp-ui-basic-integration branch from b7fcede to 9cb70b7 Compare August 27, 2025 10:32

samuelpath marked this pull request as ready for review August 27, 2025 15:55

samuelpath changed the title ~~Feat/mcp UI basic integration~~ Feat: MCP UI basic integration Aug 27, 2025

liady reviewed Aug 27, 2025

View reviewed changes

packages/api/src/mcp/parsers.ts Show resolved Hide resolved

mawburn reviewed Aug 27, 2025

View reviewed changes

samuelpath force-pushed the feat/mcp-ui-basic-integration branch 2 times, most recently from a20a338 to 7352f36 Compare August 27, 2025 19:07

danny-avila changed the base branch from main to dev August 27, 2025 19:22

sbruel reviewed Aug 28, 2025

View reviewed changes

samuelpath force-pushed the feat/mcp-ui-basic-integration branch 2 times, most recently from 4b0adcd to b649e3c Compare August 28, 2025 14:50

🚦 feat: add basic support for UI resources, displayed in the tool cal…

87b95d8

…l info

samuelpath force-pushed the feat/mcp-ui-basic-integration branch from b649e3c to 87b95d8 Compare August 29, 2025 16:48

danny-avila changed the title ~~Feat: MCP UI basic integration~~ 🎨 feat: MCP UI basic integration Aug 29, 2025

sbruel reviewed Aug 29, 2025

View reviewed changes

danny-avila merged commit d16f93b into danny-avila:dev Aug 29, 2025
7 checks passed

pgangwani mentioned this pull request Aug 30, 2025

[Feature]: Support for MCP-UI in xmcp basementstudio/xmcp#127

Closed

samuelpath deleted the feat/mcp-ui-basic-integration branch September 2, 2025 07:33

This was referenced Sep 2, 2025

🚦 feat: Simplify MCP UI integration and add unit tests #9418

Merged

⚙️ fix: filter out content items with metadata property danny-avila/agents#19

Closed

lihe8811 pushed a commit to lihe8811/LibreChat that referenced this pull request Sep 4, 2025

🎨 feat: MCP UI basic integration (danny-avila#9299)

4fbd0d7

pedrojreis pushed a commit to nosportugal/LibreChat that referenced this pull request Sep 4, 2025

🎨 feat: MCP UI basic integration (danny-avila#9299)

076199e

samuelpath mentioned this pull request Sep 8, 2025

✂️ refactor: use artifacts and callbacks to pass UI resources #9472

Closed

8 tasks

danny-avila mentioned this pull request Sep 11, 2025

✂️ refactor: Artifacts and Tool Callbacks to Pass UI Resources #9581

Merged

8 tasks

samuelpath mentioned this pull request Sep 17, 2025

💻 feat: Deeper MCP UI integration in the Chat UI #9669

Merged

8 tasks

arbreton pushed a commit to arbreton/LibreChat that referenced this pull request Oct 9, 2025

🎨 feat: MCP UI basic integration (danny-avila#9299)

69313e3

Guiraud pushed a commit to Guiraud/LibreChat that referenced this pull request Nov 21, 2025

🎨 feat: MCP UI basic integration (danny-avila#9299)

fa2ee33

dustinhealy mentioned this pull request Dec 10, 2025

💻 feat: deeper MCP UI integration in the chat UI #10898

Closed

8 tasks

patricksn3ll pushed a commit to patricksn3ll/LibreChat that referenced this pull request Dec 11, 2025

🎨 feat: MCP UI basic integration (danny-avila#9299)

49a4db4

	resource: (item) => {
	const resourceText = [];
	if (item.resource.text != null && item.resource.text) {
	resourceText.push(item.resource.text);
	}
	if (item.resource.uri.length) {
	resourceText.push(`Resource URI: ${item.resource.uri}`);
	}
	if (item.resource.name) {
	resourceText.push(`Resource: ${item.resource.name}`);
	}
	if (item.resource.description) {
	resourceText.push(`Description: ${item.resource.description}`);
	}
	if (item.resource.mimeType != null && item.resource.mimeType) {
	resourceText.push(`Type: ${item.resource.mimeType}`);
	}
	currentTextBlock += (currentTextBlock ? '\n\n' : '') + resourceText.join('\n');
	},

	const artifacts = imageUrls.length ? { content: imageUrls } : undefined;
	if (CONTENT_ARRAY_PROVIDERS.has(provider)) {
	return [formattedContent, artifacts];
	}

	return [currentTextBlock, artifacts];

	// Extract ui_resources from the output to display them in the UI
	let uiResources: UIResource[] = [];
	if (output?.includes('ui_resources')) {
	const parsedOutput = JSON.parse(output);
	const uiResourcesItem = parsedOutput.find(
	(contentItem) => contentItem.metadata === 'ui_resources',
	);
	if (uiResourcesItem?.text) {
	uiResources = JSON.parse(atob(uiResourcesItem.text)) as UIResource[];
	}
	output = JSON.stringify(
	parsedOutput.filter((contentItem) => contentItem.metadata !== 'ui_resources'),
	);
	}

	<div className="my-2 text-sm font-medium text-text-primary">
	{localize('com_ui_ui_resources')}
	</div>
	<div>
	{uiResources.length > 1 && <UIResourceGrid uiResources={uiResources} />}

	{uiResources.length === 1 && (
	<UIResourceRenderer
	resource={uiResources[0]}
	onUIAction={async (result) => {
	console.log('Action:', result);
	}}
	htmlProps={{
	autoResizeIframe: { width: true, height: true },
	}}
	/>
	)}
	</div>

Uh oh!

Conversation

samuelpath commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Example of a MCP Server tool serving UI Resources as URL

Example of a MCP Server tool serving UI Resources as HTML

The current behaviour

What we are changing

What we will do next

Change Type

Testing

Test Configuration:

Checklist

Uh oh!

Uh oh!

mawburn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liady commented Aug 27, 2025

Uh oh!

mawburn commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samuelpath commented Aug 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelpath Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

samuelpath commented Aug 27, 2025 •

edited

Loading

mawburn commented Aug 27, 2025 •

edited

Loading

samuelpath Aug 28, 2025 •

edited

Loading