Repository navigation
fix(vision): providers that reject images in tool results no longer get the multimodal envelope — meta-ai/Muse Spark 400 (#101668, salvages #89987) - #101947
Merged
Conversation
… media gates A ProviderProfile that declares supports_vision_tool_messages=False accepts images in user messages but rejects list-type tool-result content with 400 (xiaomi/MiMo "text is not set"). supports_vision=True alone used to flip _supports_media_in_tool_results to True, and a vision-capable capability lookup could re-open _should_use_native_vision_fast_path — so the native multimodal envelope landed in a role:tool message and 400'd every turn. Both gates now go through one _profile_rejects_tool_media() veto. Refs #89981 (cherry picked from commit daed88f, trimmed)
…k 400s on image tool results Muse Spark accepts images on user turns but returns HTTP 400 invalid_request_error 'messages[N].content did not match any supported type' when the vision_analyze multimodal envelope lands in a role:tool message. With the profile veto now honored by the vision fast-path gates, declaring the limitation routes tool-result images through the aux-LLM text path while user-message vision stays enabled. Fixes #101668 Refs #47742
૮ >ﻌ< ა ci reviewran on bdcc96a — fix(meta-ai): declare supports_vision_tool_messages=False —
|
alt-glitch
added
type/bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Providers whose profile declares
supports_vision_tool_messages=Falseno longer receive thevision_analyzemultimodal envelope inside arole:toolmessage — and the bundled meta-ai profile now declares it, fixing the Muse Spark HTTP 400messages[N].content did not match any supported type(#101668).Changes
tools/vision_tools.py— new_profile_rejects_tool_media(provider)predicate (profile exists andsupports_vision_tool_messages is False). Both fast-path gates route through it:_supports_media_in_tool_results()returns False up front (sosupports_vision=Truecan no longer override the veto), and_should_use_native_vision_fast_path()checks it ahead of the_lookup_supports_vision(...)arm (so a models.dev /custom_providerscapability hit can't re-open the path either). Cherry-picked from fix(vision): honor supports_vision_tool_messages=False in tool-result media gates #89987 by @liuhao1024; the two inline try/except blocks in the original were collapsed into the single shared predicate.plugins/model-providers/meta-ai/__init__.py—supports_vision_tool_messages=Falsewith a one-line comment citing the 400.supports_vision=Trueis unchanged, so user-turn image input stays native; only the tool-result envelope falls back to the aux-LLM text path (same mechanism xiaomi already uses).supports_vision=True; veto still holds when the capability lookup says vision-capable. One assertion added to the existingtest_profile_registeredintests/providers/test_meta_ai_profile.py.On current
mainnothing intools/readsupports_vision_tool_messages— onlyrun_agent.py's post-hoc summary path did, and the vision fast path built the envelope before that ever ran.Validation
pytest tests/tools/test_vision_native_fast_path.py tests/providers/test_meta_ai_profile.pytests/run_agent/test_vision_tool_messages.py+tests/hermes_cli/test_cli_custom_provider_vision.pytools/vision_tools.pyreverted tomainHERMES_HOME:_supports_media_in_tool_results("meta-ai","muse-spark-1.2")("xiaomi","mimo-v2.5")("anthropic","claude-opus-4-6")(control)ruff checkon touched filesscripts/audit_pr_attribution.py --fixLive repro: the Meta wire-level 400 was not reproduced live (no
META_API_KEYavailable here); the reporter's trace in #101668 is relied on for the API behavior. The gate flip above was verified with real imports against the registered provider profiles.Fixes #101668
Refs #47742 #89981
Credits: @liuhao1024 — cherry-picked from #89987 with authorship preserved.
Infographic