fix(vision): honor model.supports_vision in vision_analyze + browser_vision - #34562
Merged
Conversation
…elper The fast-path decision (native routing + provider allowlist OR supports_vision override) lived inline in vision_analyze and was copied into browser_vision. Extract it to _should_use_native_vision_fast_path() so both tools share one source of truth. - vision_tools: gate logic now one helper; vision_analyze calls it in 3 lines - browser_tool: thin envelope decoration over the shared helper, not a copy - browser_vision typed Union[str, Dict] to match its real return shape - tests slimmed to target the override path + text-mode-wins invariant
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
invalid-argument-type |
2 |
First entries
tests/tools/test_browser_lightpanda.py:481: [invalid-argument-type] invalid-argument-type: Argument to function `loads` is incorrect: Expected `str | bytes | bytearray`, found `str | dict[str, Any]`
tests/tools/test_browser_console.py:322: [invalid-argument-type] invalid-argument-type: Argument to function `loads` is incorrect: Expected `str | bytes | bytearray`, found `str | dict[str, Any]`
✅ Fixed issues: none
Unchanged: 4899 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
alt-glitch
added
type/bug
Collaborator
|
Salvage of #29987 by @tillfalko onto current main. Supersedes #29987. Adds browser_vision native fast path not present in the original. |
17 tasks
This was referenced Aug 3, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Vision tools now honor the user's
model.supports_visionoverride, so a vision-capable model on acustom/local provider sees images directly instead of being forced through the auxiliary vision LLM.browser_visiongains a native fast path it never had — previously it always called an aux model regardless of provider.Salvage of #29987 by @tillfalko, cherry-picked onto current
mainwith authorship preserved, plus a consolidation refactor on top.Root cause
The fast-path gate keyed only on
_supports_media_in_tool_results(), a static provider allowlist. Custom/unknown providers returnFalsethere, so even when a user explicitly declared their model vision-capable,vision_analyzefell through to the aux LLM.browser_visionhad no fast path at all.Changes
tools/vision_tools.py: new_should_use_native_vision_fast_path()— single source of truth for the gate (native routing AND (provider allowlist ORsupports_visionoverride)).vision_analyzecalls it in 3 lines.tools/browser_tool.py:browser_visionuses the shared helper; thin multimodal-envelope decoration instead of a duplicated gate block. Return type corrected toUnion[str, Dict[str, Any]]; tool description updated for native-vision routing.tests/: focused coverage for the override path and the text-mode-wins invariant.scripts/release.py: AUTHOR_MAP entry for the contributor.Text-mode routing still wins over the override (explicit
image_input_mode: textblocks the fast path).Validation
supports_vision: trueimage_input_mode: textTargeted suite: 49/49 pass. E2E verified the real resolution chain (no helper mocking) for all four scenarios above.
Closes #29987.
Infographic