fix(browser): apply proactive embed cap to browser_vision native fast path - #44922
fix(browser): apply proactive embed cap to browser_vision native fast path#44922reghar-bot wants to merge 1 commit into
Conversation
… path browser_vision's native image routing attached screenshots at full resolution with no size or dimension check, while vision_analyze's native path resizes to the embed target (4 MB / 7900px) and refuses payloads over the hard ceiling. Oversized screenshots (especially tall full-page captures) hit non-retryable provider rejections that retries re-send verbatim, killing the turn. Mirror _vision_analyze_native's proactive cap before the native attach; if the payload still exceeds the hard ceiling after resizing, fall back to the aux vision LLM path instead of erroring. Field incident: 2026-06-11 22:40, a 2.36 MB base64 screenshot was rejected 3x by openai-codex, aborting the turn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Duplicate of #39226 — same fix: |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for targeting the native browser-vision path; the premise is still live on current main (tools/browser_tool.py:4060-4080), while the sibling native vision path has the proactive cap (tools/vision_tools.py:1018-1047).
Problems
tools/browser_tool.py:3255only rejects a resized result above the 20 MB hard limit._resize_image_for_visiondeliberately returns its best candidate after five unsuccessful attempts (tools/vision_tools.py:701-742), so an under-20-MB result can still exceed the requested 4 MB or 7900px embed cap and be attached natively.- The PR has no regression tests for this new browser-specific cap path; the existing browser native-path test is only an ordinary envelope-shape test (
tests/tools/test_browser_console.py:403-440).
Suggested changes
- Validate that the resized result satisfies both embed constraints before native attachment; otherwise take the existing auxiliary fallback.
- Add byte-cap, dimension-cap, in-bounds, and failed-resize-fallback tests.
Automated hermes-sweeper review.
| max_base64_bytes=_EMBED_TARGET_BYTES, | ||
| max_dimension=_EMBED_MAX_DIMENSION, | ||
| ) | ||
| if len(data_url) > _MAX_BASE64_BYTES: |
There was a problem hiding this comment.
This fallback only detects the 20 MB hard limit. _resize_image_for_vision() returns its last candidate after five failed attempts (tools/vision_tools.py:701-742), so an under-20-MB result can still violate the requested 4 MB or 7900px cap and be embedded. Validate both requested constraints here, or take the auxiliary fallback.
Problem
browser_vision's native image routing fast path attaches screenshots to the main model at full resolution with no size or dimension check:https://github.com/NousResearch/hermes-agent/blob/main/tools/browser_tool.py#L3214-L3217
Meanwhile
vision_analyze's native path (_vision_analyze_nativeintools/vision_tools.py) proactively resizes to the embed target (_EMBED_TARGET_BYTES4 MB /_EMBED_MAX_DIMENSION7900px) and refuses payloads over_MAX_BASE64_BYTES, precisely because an oversized embed triggers a non-retryable provider rejection that retries re-send verbatim.Tall full-page screenshots routinely exceed the per-side pixel ceiling even when their byte size looks harmless.
Field incident
2026-06-11, macOS gateway, openai-codex / gpt-5.5 with native image routing:
A 2.36 MB base64 full-page screenshot was attached natively, the provider rejected the request three times (same payload each retry), and the turn aborted — the user got an error instead of an answer. (Related problem class: #513, #23767.)
Fix
Mirror
_vision_analyze_native's proactive embed cap inbrowser_vision's fast path before the native attach:_resize_image_for_visiondown to the embed targetNo behavior change for screenshots already under the caps.
Testing
py_compileclean; all imported helpers exist intools/vision_tools.py🤖 Generated with Claude Code