Skip to content

fix(browser): apply proactive embed cap to browser_vision native fast path - #44922

Open
reghar-bot wants to merge 1 commit into
NousResearch:mainfrom
reghar-bot:pr/fix-browser-vision-embed-cap
Open

fix(browser): apply proactive embed cap to browser_vision native fast path#44922
reghar-bot wants to merge 1 commit into
NousResearch:mainfrom
reghar-bot:pr/fix-browser-vision-embed-cap

Conversation

@reghar-bot

Copy link
Copy Markdown

Problem

browser_vision's native image routing fast path attaches screenshots to the main model at full resolution with no size or dimension check:

https://github.com/NousResearch/hermes-agent/blob/main/tools/browser_tool.py#L3214-L3217

Meanwhile vision_analyze's native path (_vision_analyze_native in tools/vision_tools.py) proactively resizes to the embed target (_EMBED_TARGET_BYTES 4 MB / _EMBED_MAX_DIMENSION 7900px) and refuses payloads over _MAX_BASE64_BYTES, precisely because an oversized embed triggers a non-retryable provider rejection that retries re-send verbatim.

Tall full-page screenshots routinely exceed the per-side pixel ceiling even when their byte size looks harmless.

Field incident

2026-06-11, macOS gateway, openai-codex / gpt-5.5 with native image routing:

22:40:10 tool browser_vision completed (0.73s, 2359836 chars)
22:40:10 WARNING Request context is very large (~679,562 tokens, model=gpt-5.5)...
22:40:14 WARNING API call failed (attempt 1/3) error_type=APIError provider=openai-codex
22:40:20 WARNING API call failed (attempt 2/3) ...
22:40:34 ERROR   API call failed after 3 retries.

A 2.36 MB base64 full-page screenshot was attached natively, the provider rejected the request three times (same payload each retry), and the turn aborted — the user got an error instead of an answer. (Related problem class: #513, #23767.)

Fix

Mirror _vision_analyze_native's proactive embed cap in browser_vision's fast path before the native attach:

  • over byte target or dimension ceiling → _resize_image_for_vision down to the embed target
  • still over the hard ceiling after resizing → fall back to the existing aux-LLM analysis path instead of erroring (a text description beats a guaranteed provider rejection)

No behavior change for screenshots already under the caps.

Testing

  • py_compile clean; all imported helpers exist in tools/vision_tools.py
  • Running live on the gateway that hit the incident

🤖 Generated with Claude Code

… path

browser_vision's native image routing attached screenshots at full
resolution with no size or dimension check, while vision_analyze's
native path resizes to the embed target (4 MB / 7900px) and refuses
payloads over the hard ceiling. Oversized screenshots (especially tall
full-page captures) hit non-retryable provider rejections that retries
re-send verbatim, killing the turn.

Mirror _vision_analyze_native's proactive cap before the native attach;
if the payload still exceeds the hard ceiling after resizing, fall back
to the aux vision LLM path instead of erroring.

Field incident: 2026-06-11 22:40, a 2.36 MB base64 screenshot was
rejected 3x by openai-codex, aborting the turn.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/tools Tool registry, model_tools, toolsets tool/browser Browser automation (CDP, Playwright) tool/vision Vision analysis and image generation duplicate This issue or pull request already exists labels Jun 12, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #39226 — same fix: browser_vision's native fast path attaches the full-resolution screenshot without the embed-time pixel/byte cap that _vision_analyze_native applies, so tall full-page screenshots trip the provider's per-side pixel ceiling and get rejected non-retryably. #39226 mirrors the proactive resize + aux-LLM fallback.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for targeting the native browser-vision path; the premise is still live on current main (tools/browser_tool.py:4060-4080), while the sibling native vision path has the proactive cap (tools/vision_tools.py:1018-1047).

Problems

  • tools/browser_tool.py:3255 only rejects a resized result above the 20 MB hard limit. _resize_image_for_vision deliberately returns its best candidate after five unsuccessful attempts (tools/vision_tools.py:701-742), so an under-20-MB result can still exceed the requested 4 MB or 7900px embed cap and be attached natively.
  • The PR has no regression tests for this new browser-specific cap path; the existing browser native-path test is only an ordinary envelope-shape test (tests/tools/test_browser_console.py:403-440).

Suggested changes

  • Validate that the resized result satisfies both embed constraints before native attachment; otherwise take the existing auxiliary fallback.
  • Add byte-cap, dimension-cap, in-bounds, and failed-resize-fallback tests.

Automated hermes-sweeper review.

Comment thread tools/browser_tool.py
max_base64_bytes=_EMBED_TARGET_BYTES,
max_dimension=_EMBED_MAX_DIMENSION,
)
if len(data_url) > _MAX_BASE64_BYTES:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fallback only detects the 20 MB hard limit. _resize_image_for_vision() returns its last candidate after five failed attempts (tools/vision_tools.py:701-742), so an under-20-MB result can still violate the requested 4 MB or 7900px cap and be embedded. Validate both requested constraints here, or take the auxiliary fallback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/browser Browser automation (CDP, Playwright) tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants