Repository navigation
fix(vision): honor supports_vision_tool_messages=False in tool-result media gates - #89987
liuhao1024 wants to merge 1 commit into
Conversation
… media gates A provider profile declaring supports_vision=True together with supports_vision_tool_messages=False (xiaomi/MiMo — user-message images work, list-type tool-result content answers 400 "text is not set") still passed _supports_media_in_tool_results: the profile check only read supports_vision. The native fast path then wrapped every vision result in the multimodal tool-result envelope, which such a provider rejects outright — the image never entered context and the turn errored (NousResearch#47742, re-reported as NousResearch#89981). Two gates now treat an explicit supports_vision_tool_messages=False as a hard veto: - _supports_media_in_tool_results returns False before the supports_vision check (providers that do not declare the flag keep the existing behavior). - _should_use_native_vision_fast_path applies the same veto ahead of its capability-lookup escape hatch, so a models.dev / custom_providers vision-capable marking cannot re-open the envelope route. With the veto, these providers fall back to the legacy aux-LLM vision path — the designed degradation for tool-result-media-incapable providers.
|
Same pre-existing main regression as reported on #89968: slice 12 fails in |
|
Merged via #101947 (commit d6bb94a) — your commit was cherry-picked onto current main with authorship preserved, trimmed to the two gate checks + 2 invariant tests. We also declared |
What does this PR do?
Fixes the native vision fast path sending image media in tool-result messages to providers that only accept images in user messages (#47742, re-reported as #89981 — same root cause, fixing the canonical issue).
A provider profile can declare
supports_vision=True(user-message images work) together withsupports_vision_tool_messages=False(list-type tool-result content is rejected). xiaomi/MiMo does exactly this and answers such content with400 "text is not set". But_supports_media_in_tool_results()only readsupports_visionfrom the profile, so the gate flipped to True, the fast path wrapped every vision result in the multimodal tool-result envelope, and the provider rejected it outright — the image never entered context and the turn errored.Two gates now treat an explicit
supports_vision_tool_messages=Falseas a hard veto:_supports_media_in_tool_resultsreturns False before thesupports_visioncheck. Providers that do not declare the flag at all default to True and keep the existing behavior._should_use_native_vision_fast_pathapplies the same veto ahead of its capability-lookup escape hatch, so a models.dev / custom_providers vision-capable marking cannot re-open the envelope route for a provider that rejects the wire shape.With the veto, these providers take the legacy aux-LLM vision path — the designed degradation for tool-result-media-incapable providers (images are analyzed by the aux vision model and return as text). The report's suggested deeper enhancement (re-injecting the image as a synthetic user message in
tool_executor) composes on top of this and is intentionally left out of scope.Related Issue
Fixes #47742
(also resolves the re-report #89981, marked duplicate of it)
Type of Change
Changes Made
tools/vision_tools.py:_supports_media_in_tool_results— the profile branch checkssupports_vision_tool_messagesfirst and returns False on an explicit False;_should_use_native_vision_fast_path— the same veto is applied before the capability-lookup disjunction.How to Test
pytest tests/tools/test_vision_native_fast_path.py -q— 14 passed (2 new tests).xiaomi/mimoresolve to True; the fast path fires even with the capability lookup patched to True).tests/tools/test_vision_tools.py34 passed,tests/agent/test_image_routing.py38 passed._supports_media_in_tool_results("xiaomi", "mimo-v2.5")is now False, so the fast path falls through to the aux-LLM path instead of emitting the multimodal envelope that the provider 400s.Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/A