Skip to content

fix(tools): forward ImageContent blocks from MCP tool results - #20701

Closed
jackControls wants to merge 1 commit into
NousResearch:mainfrom
jackControls:fix/mcp-image-content-forwarding
Closed

fix(tools): forward ImageContent blocks from MCP tool results#20701
jackControls wants to merge 1 commit into
NousResearch:mainfrom
jackControls:fix/mcp-image-content-forwarding

Conversation

@jackControls

@jackControls jackControls commented May 6, 2026

Copy link
Copy Markdown

What & Why

The MCP spec allows tools to return ImageContent blocks alongside TextContent (data = base64 bytes, mimeType = e.g. image/png). The result extractor in tools/mcp_tool.py only walks block.text:

parts: List[str] = []
for block in (result.content or []):
    if hasattr(block, "text"):
        parts.append(block.text)
text_result = "\n".join(parts) if parts else ""

ImageContent has no .text attribute, so any image returned by an MCP tool is silently dropped. Tools that produce diagrams, screenshots, OCR previews, charts, or rendered pages are unusable — the agent has no way to inspect their visual output.

A workaround pattern in the wild is for an MCP tool to do its own LLM call inside the tool body and return text. That works but defeats the point of MCP — the agent should choose how to interpret the image.

Fix: cache ImageContent blocks via cache_image_from_bytes (same image cache the Telegram photo path uses) and surface the resulting local paths in the text result so the agent can vision_analyze them on a follow-up turn. Stays compatible with the OpenAI role: "tool" chat-completions contract (which requires string content) and reuses the existing image-cache infrastructure.

For a richer future where tool results carry image parts on a synthetic user turn, this provides the same caching primitive the _pending_native_image_paths plumbing in run_agent.py could later consume.

Net change: ~30 lines in the _call() helper, no API changes, no new dependencies (base64 + mimetypes from stdlib, cache_image_from_bytes from gateway.platforms.base).

How to test

  • MCP tool that returns text only — unchanged behaviour
  • MCP tool that returns one ImageContent block — image cached, path surfaced in tool result, vision_analyze describes it
  • MCP tool that returns mixed text + image — text preserved, image path appended in a clearly-marked footer
  • Image cache eviction still works (new entries land in the same per-day directory the Telegram path uses)

Tested on

  • End-to-end (production): Linux x86_64 (Debian 13 / "trixie") in a Docker container, custom MCP server returning mcp.types.ImageContent blocks via the Python MCP SDK, vLLM serving a Qwen-VL multimodal main model with auxiliary.vision.provider: "main" so vision_analyze lands on the same model.
  • Unit tests (locally on macOS 15, Python 3.11, against this branch):
    pytest tests/tools/test_mcp_tool.py \
           tests/tools/test_mcp_tool_issue_948.py \
           tests/tools/test_mcp_tool_401_handling.py \
           tests/tools/test_mcp_tool_session_expired.py -v
    ============================= 210 passed in 9.60s ==============================
    

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists tool/mcp MCP client and OAuth labels May 6, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #10848 (and related: #9570, #17915) — all fix MCP ImageContent blocks being silently dropped in tools/mcp_tool.py. See also the original issue #10759.

The MCP spec allows tools to return ImageContent blocks alongside
TextContent (data = base64 bytes, mimeType = e.g. image/png). The
result extractor in tools/mcp_tool.py only walks block.text:

    parts: List[str] = []
    for block in (result.content or []):
        if hasattr(block, "text"):
            parts.append(block.text)

ImageContent has no .text attribute, so any image returned by an MCP
tool is silently dropped. Tools that produce diagrams, screenshots,
OCR previews, charts, or rendered pages are unusable: the agent has
no way to inspect their visual output.

Cache ImageContent blocks via cache_image_from_bytes (same image
cache the Telegram photo path uses) and surface the resulting paths
in the tool's text result so the agent can vision_analyze them on a
follow-up turn. This stays compatible with the OpenAI role: "tool"
chat-completions contract (which requires string content) and reuses
the existing image cache infrastructure.

No new dependencies (base64 + mimetypes from stdlib,
cache_image_from_bytes from gateway.platforms.base).
@jackControls
jackControls force-pushed the fix/mcp-image-content-forwarding branch from 87ee384 to 2b8515e Compare May 6, 2026 12:15
@jackControls jackControls changed the title tools/mcp_tool: forward ImageContent blocks from MCP tool results fix(tools): forward ImageContent blocks from MCP tool results May 6, 2026
@jackControls

Copy link
Copy Markdown
Author

Thanks — closing as a duplicate of #10848 / #9570 / #17915 (per maintainer feedback). Will track those for the merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Medium — degraded but workaround exists tool/mcp MCP client and OAuth type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants