fix(tools): forward ImageContent blocks from MCP tool results - #20701
Closed
jackControls wants to merge 1 commit into
Closed
fix(tools): forward ImageContent blocks from MCP tool results#20701jackControls wants to merge 1 commit into
jackControls wants to merge 1 commit into
Conversation
Collaborator
The MCP spec allows tools to return ImageContent blocks alongside
TextContent (data = base64 bytes, mimeType = e.g. image/png). The
result extractor in tools/mcp_tool.py only walks block.text:
parts: List[str] = []
for block in (result.content or []):
if hasattr(block, "text"):
parts.append(block.text)
ImageContent has no .text attribute, so any image returned by an MCP
tool is silently dropped. Tools that produce diagrams, screenshots,
OCR previews, charts, or rendered pages are unusable: the agent has
no way to inspect their visual output.
Cache ImageContent blocks via cache_image_from_bytes (same image
cache the Telegram photo path uses) and surface the resulting paths
in the tool's text result so the agent can vision_analyze them on a
follow-up turn. This stays compatible with the OpenAI role: "tool"
chat-completions contract (which requires string content) and reuses
the existing image cache infrastructure.
No new dependencies (base64 + mimetypes from stdlib,
cache_image_from_bytes from gateway.platforms.base).
jackControls
force-pushed
the
fix/mcp-image-content-forwarding
branch
from
May 6, 2026 12:15
87ee384 to
2b8515e
Compare
Author
19 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & Why
The MCP spec allows tools to return
ImageContentblocks alongsideTextContent(data= base64 bytes,mimeType= e.g.image/png). The result extractor intools/mcp_tool.pyonly walksblock.text:ImageContenthas no.textattribute, so any image returned by an MCP tool is silently dropped. Tools that produce diagrams, screenshots, OCR previews, charts, or rendered pages are unusable — the agent has no way to inspect their visual output.A workaround pattern in the wild is for an MCP tool to do its own LLM call inside the tool body and return text. That works but defeats the point of MCP — the agent should choose how to interpret the image.
Fix: cache
ImageContentblocks viacache_image_from_bytes(same image cache the Telegram photo path uses) and surface the resulting local paths in the text result so the agent canvision_analyzethem on a follow-up turn. Stays compatible with the OpenAIrole: "tool"chat-completions contract (which requires string content) and reuses the existing image-cache infrastructure.For a richer future where tool results carry image parts on a synthetic user turn, this provides the same caching primitive the
_pending_native_image_pathsplumbing inrun_agent.pycould later consume.Net change: ~30 lines in the
_call()helper, no API changes, no new dependencies (base64+mimetypesfrom stdlib,cache_image_from_bytesfromgateway.platforms.base).How to test
ImageContentblock — image cached, path surfaced in tool result,vision_analyzedescribes itTested on
mcp.types.ImageContentblocks via the Python MCP SDK, vLLM serving a Qwen-VL multimodal main model withauxiliary.vision.provider: "main"sovision_analyzelands on the same model.