fix(vision): apply proactive embed cap to browser_vision screenshots - #47468
Open
Stromweld wants to merge 2 commits into
Open
fix(vision): apply proactive embed cap to browser_vision screenshots#47468Stromweld wants to merge 2 commits into
Stromweld wants to merge 2 commits into
Conversation
browser_vision takes full-page screenshots (--full) that can easily exceed provider dimension limits (8000px per side) on long web pages. Unlike vision_analyze which applies the proactive embed cap (_EMBED_TARGET_BYTES / _EMBED_MAX_DIMENSION) before embedding images into conversation history, browser_vision was passing raw full-resolution base64 directly to _build_native_vision_tool_result. Once an oversized image is baked into immutable conversation history, it triggers non-retryable 400 errors from the provider on every subsequent turn, permanently bricking the session (retries cannot clear bytes/pixels that are already in the request history). This patch imports and applies the same proactive resize checks that vision_tools._vision_analyze_native uses: - Check if base64 > _EMBED_TARGET_BYTES (4 MB) - Check if longest dimension > _EMBED_MAX_DIMENSION (7900px) - Auto-resize with Pillow before embedding if either limit is exceeded The resize applies to both the native fast path and the auxiliary vision LLM path since they share the data_url variable. Reproducer: use browser_vision on any page taller than ~8000px (most documentation pages, GitHub PRs, Confluence pages). The resulting full-page PNG will be 10,000-15,000+ pixels tall, pass the byte check but fail the dimension check, and brick the session on the next turn. Signed-off-by: Corey Hemminger <hemminger@hotmail.com>
alt-glitch
added
type/bug
Collaborator
|
Duplicate of #39226 — same fix: apply the proactive embed cap (_EMBED_MAX_DIMENSION / _image_exceeds_dimension) to browser_vision screenshots before embedding into conversation history. #39226 (open, 2026-06-04) is the canonical fix-PR; #44922 is also a duplicate of it. Related: dimension-clamp PR #25838 and the originating issue #47467. The proactive cap is still absent on main (only the reactive resize-on-400 path exists in tools/browser_tool.py), so the underlying bug remains live. |
teknium1
reviewed
Jul 14, 2026
teknium1
left a comment
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for isolating the native browser_vision gap. Current main still sends the raw full-page screenshot into the native tool-result envelope (tools/browser_tool.py:4021,4060-4080), while vision_analyze applies the shared cap (tools/vision_tools.py:1018-1028).
Problems
- The new resize call at PR
tools/browser_tool.py:3259is not a guaranteed cap:_resize_image_for_visionreturns its last candidate after five unsuccessful attempts (tools/vision_tools.py:737-742), and the PR immediately attaches that result at lines 3269-3274. A candidate still exceeding the byte or pixel cap can therefore enter immutable history. - No regression test is included. Existing native browser coverage only verifies that an image part is returned (
tests/tools/test_browser_console.py:403-440).
Suggested changes
- Validate both post-resize byte size and decoded dimensions before native attachment; return a tool error if either constraint remains unmet.
- Add native-path regressions for tall low-byte and byte-heavy screenshots that inspect the emitted data URL.
Automated hermes-sweeper review.
| "(bytes_over=%s, dims_over=%s), auto-resizing...", | ||
| _over_bytes, _over_dims, | ||
| ) | ||
| data_url = _resize_image_for_vision( |
Contributor
There was a problem hiding this comment.
_resize_image_for_vision returns its final candidate after five attempts even if it still fails the requested byte or pixel constraints (tools/vision_tools.py:737-742). Validate the returned data URL's byte size and decoded dimensions before the native envelope at line 3270, and return a tool error if it is still over either cap.
Author
There was a problem hiding this comment.
new commit should have suggested changes
teknium1
added
the
sweeper:blast-contained
…achment Per review #4696844968: _resize_image_for_vision returns its best attempt after 5 rounds even when constraints aren't met. Validate byte size and decoded pixel dimensions before passing data_url to the native envelope. Return tool_error if either cap is still exceeded to prevent baking an oversized embed into immutable history. Also add 3 regression tests for the native path: - tall low-byte screenshot → resize triggers, native path succeeds - byte-heavy screenshot → resize triggers, native path succeeds - resize failure (5-attempt exhaustion) → tool_error returned Note: Pillow imports inside browser_vision are lazy so patches target tools.vision_tools.* not tools.browser_tool.* AI assistance: Hermes Agent v0.16.0, claude-sonnet-4.6
This was referenced Aug 3, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #47467 —
browser_visionnow applies the same proactive embed cap thatvision_analyzeuses before embedding screenshots into conversation history.Problem
browser_visiontakes full-page screenshots (--full) that routinely exceed 7900px on long web pages (docs, PRs, Confluence). Without the proactive resize check, these oversized images get baked into immutable conversation history and trigger non-retryable 400 errors from providers (Anthropic 8000px limit, Copilot similar), permanently bricking the session.The
vision_analyzetool already has this protection (lines 758-766 ofvision_tools.py), butbrowser_visioninbrowser_tool.pywas bypassing it entirely.Changes
_resize_image_for_vision,_image_exceeds_dimension,_EMBED_TARGET_BYTES, and_EMBED_MAX_DIMENSIONfromvision_tools_build_native_vision_tool_resultdata_urlis shared between the native fast path AND the auxiliary vision LLM path, so both are protectedTesting
Verified against the actual problematic screenshot (1280×14567 px, 2.12 MB):
_image_exceeds_dimension()correctly detects it as over the 7900px cap_resize_image_for_vision()successfully downsizes itNote
This PR was created with AI assistance (Hermes Agent v0.16.0, Claude Opus 4.6).