Skip to content

fix(agent): guard multimodal tool content behind provider profile flag - #27597

Closed
zccyman wants to merge 2 commits into
NousResearch:mainfrom
atyou2happy:fix/multimodal-tool-content-guard
Closed

fix(agent): guard multimodal tool content behind provider profile flag#27597
zccyman wants to merge 2 commits into
NousResearch:mainfrom
atyou2happy:fix/multimodal-tool-content-guard

Conversation

@zccyman

@zccyman zccyman commented May 17, 2026

Copy link
Copy Markdown
Contributor

Summary

When computer_use captures a screenshot and returns _multimodal content (list of text + image_url parts), _tool_result_content_for_active_model currently only checks _model_supports_vision() before passing through the multimodal content list.

However, many providers (e.g. Xiaomi MiMo) support images in user messages but require content to be a string in role: "tool" messages. Sending a list causes a 400 error.

Changes

Approach

Option 2 from the issue — conservative default. Only providers explicitly marked with supports_multimodal_tool_content=True receive multimodal tool content. All others (including new/unrecognized providers) get the text summary fallback.

This is safe because:

  • Default False means no existing provider behavior changes unless explicitly enabled
  • Providers that don't support it already fail at the API layer — this just prevents the failure
  • The fallback path (text summary) already exists and is well-tested

Fixes #27344

When computer_use returns image content, _tool_result_content_for_active_model
currently checks only _model_supports_vision() before passing through the
multimodal content list.  However, many providers (e.g. Xiaomi MiMo) support
images in user messages but require string-only content in role: "tool"
messages, causing a 400 error.

Add supports_multimodal_tool_content to ProviderProfile (default False) and
check it in addition to vision support.  Only providers known to accept
multimodal tool content (Anthropic, OpenAI Codex, OpenRouter, xAI) have
the flag set to True.

Fixes NousResearch#27344
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/plugins Plugin system and bundled plugins tool/vision Vision analysis and image generation P2 Medium — degraded but workaround exists labels May 17, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #27351 — both PRs fix #27344 with the same approach (provider profile flag for multimodal tool content support).

@BoardJames-Bot

Copy link
Copy Markdown

CI triage note from rock-turning: this PR is already called out above as a duplicate of #27351. Its current blocking check is not a code failure; the test job was cancelled in run 26002320915 while e2e and the rest of the checks had passed. I attempted gh run rerun 26002320915 --failed, but rerun is blocked for this account by repo admin permission (Must have admin rights to Repository). Next owner: repo maintainer should either close this as duplicate of #27351 or rerun the cancelled test job if keeping this PR open.

@zccyman

zccyman commented May 18, 2026

Copy link
Copy Markdown
Contributor Author

Closing as duplicate of #27351 (confirmed by alt-glitch). Both PRs implement the same provider profile flag approach for multimodal tool content support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/plugins Plugin system and bundled plugins P2 Medium — degraded but workaround exists tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] computer_use multimodal tool message causes 400 error on providers that don't support multimodal tool content (e.g. Xiaomi MiMo)

3 participants