fix(vision): cap browser_vision native embeds (full-page screenshots OOM local models, wedge Anthropic sessions)

[bnikanjam]

What does this PR do?

Applies the proactive embed caps (4 MB / 7900 px) to browser_vision's native vision fast path. When the main model supports vision, browser_vision embeds its full-page screenshot into conversation history at full resolution — despite the code comment claiming "Consistent with vision_analyze", which has capped its embeds since the wedged-session incident (tools/vision_tools.py:548-566 explains why: history-embedded images are re-sent every turn, so an oversized embed permanently wedges Anthropic sessions with non-retryable 400s). A full-page capture is arbitrarily tall, making this the worst-case producer of oversized embeds.

Real-world evidence: on a 36 GB Apple Silicon machine running an MLX VLM (qwen3.6-27b via LM Studio), accumulated browser_vision screenshots at ~50k-token context crashed the model four consecutive times as retries re-sent the same context:

[METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
Fatal Python error: Aborted

Related Issue

Related: #63849 (the remaining accumulation half of the problem — capped screenshots still pile up on the OpenAI-compatible path; filed as a design question because eviction interacts with the prompt-caching invariant).

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• tools/vision_tools.py — extract the cap logic into a shared _apply_embed_caps(image_path, image_data_url, mime_type) helper (identity passthrough under both caps; resize via the existing _resize_image_for_vision; returns an error string instead of embedding when even resizing can't get under the 20 MB hard ceiling). _vision_analyze_native rewires to the helper — semantics-preserving, net-negative diff, guarded by the existing test_oversized_image_resized_under_embed_cap regression test.
• tools/browser_tool.py — the native fast path now caps the screenshot before embedding, scoped strictly inside the _should_use_native_vision_fast_path() branch. The auxiliary path is byte-identical: it sends the full-resolution image once, outside history, and keeps its existing reactive resize-and-retry.
• tests/tools/test_vision_native_fast_path.py — new TestApplyEmbedCaps: oversized-by-bytes → resized under 4 MB; oversized-by-dimension (200×9000 tall page, tiny bytes) → longest side ≤ 7900 px; under-cap image → identity passthrough (no recompression).
• tests/tools/test_browser_vision_embed_caps.py (new) — integration through the real browser_vision with only the browser daemon faked: oversized screenshot embeds under the cap; tiny screenshot embeds byte-for-byte at full resolution (guards over-eager capping).

How to Test

1. On a vision-capable main model, call browser_vision on a long page (full-page screenshot > 4 MB or > 7900 px tall).
2. Before: the multimodal tool result carries the full-resolution PNG (arbitrarily large). After: the embedded data URL is ≤ 4 MB and ≤ 7900 px per side; small screenshots are untouched byte-for-byte.
3. scripts/run_tests.sh tests/tools/test_vision_native_fast_path.py tests/tools/test_browser_vision_embed_caps.py tests/tools/test_vision_tools.py → 115 passed.
4. Full browser suite: scripts/run_tests.sh tests/tools/test_browser* → 488 passed.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(vision): …)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run the tests and all pass (via scripts/run_tests.sh, the CI-parity wrapper)
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 26 / Darwin 25.4 (Apple Silicon)

Documentation & Housekeeping

• I've updated relevant documentation — N/A (internal behavior; no doc describes the uncapped embed)
• I've updated cli-config.yaml.example if I added/changed config keys — N/A (no config keys changed)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — pure image-processing logic; Pillow-absent behavior unchanged (resize is a best-effort no-op, same as vision_analyze today)
• I've updated tool descriptions/schemas if I changed tool behavior — N/A (tool schema unchanged; only payload size)

Screenshots / Logs

scripts/run_tests.sh tests/tools/test_vision_native_fast_path.py tests/tools/test_browser_vision_embed_caps.py tests/tools/test_vision_tools.py
=== Summary: 3 files, 115 tests passed, 0 failed (100% complete) in 3.4s (20 workers) ===

scripts/run_tests.sh tests/tools/test_browser*  (35 files)
=== Summary: 35 files, 488 tests passed, 0 failed (100% complete) in 35.8s (20 workers) ===

🤖 Generated with Claude Code

[tonydwb]

Code Review Summary

Verdict: LGTM

Good fix. Vision caps for native embeds prevent oversized full-page screenshots from silently blowing context budget. Test coverage at both the browser-tool and native-fast-path levels confirms the cap is enforced at every entry point that contributes to the embed.

Strengths

• Cap is symmetric across browser_vision and the native fast path — single boundary, no double-bypass.
• Tests assert pixel/dimension limits, not arbitrary counts, so the invariant survives future intent changes.

Light follow-up (non-blocking)

• Consider documenting the cap value in a config constant so it's discoverable, not just enforced.

---

Reviewed by Hermes Agent in batch mode

[bnikanjam]

Thanks for the review! On the non-blocking follow-up: the cap values are module constants with full rationale docs — _EMBED_TARGET_BYTES (4 MB) and _EMBED_MAX_DIMENSION (7900 px) in tools/vision_tools.py (~lines 548-566), including why they sit under Anthropic's 5 MB / 8000 px ceilings. I deliberately didn't promote them to a user-facing config key in this PR to keep it surgical (a config key would also need cli-config.yaml.example + DEFAULT_CONFIG wiring per the contributing guide) — happy to do that as a follow-up if maintainers want the caps tunable. Related: #63849 tracks the remaining accumulation half (images never evicted on the OpenAI-compatible path), with an implementation in progress.

[teknium1]

Thanks for the focused fix. Current main still takes browser_vision screenshots with --full (tools/browser_tool.py:4017-4022), base64-encodes them at full resolution (tools/browser_tool.py:4060-4063), and directly embeds that URL on the native path (tools/browser_tool.py:4074-4080).

vision_analyze already applies both byte and dimension checks before its native embed (tools/vision_tools.py:1018-1041), so the proposed shared helper closes a verified sibling-path gap without adding tool or configuration surface. The PR's integration test covers the browser path, while the shared-helper tests cover byte, dimension, and identity behavior.

Automated hermes-sweeper review.

[GottZ]

This was generated by AI during triage.

Summary

Seventeen PRs touch this browser-vision complex: #11752 addresses remote screenshot persistence; #19647, #19765, #20118, and #21833 make full-page capture optional; #39226, #44922, #47468, #47551, and #63850 add native-embed caps; the others concern media handling, vision routing, native fast paths, or numeric configuration rather than the reported capture or embed causes.

Related pull requests

• fix(browser): persist remote browser_vision screenshots locally #11752 best fix — (+125/-7) — n/a: Avoids passing Hermes-local screenshot paths to Browserbase and copies the returned temporary file into cache, directly addressing Bug: browser_vision screenshot fails on WSL2 with Browserbase remote backend #11729, but the keep_open review requires direct-CDP coverage and adaptation to the current Chrome pre-route.
• fix(feishu): suppress group progress noise and skip placeholder media #13128 related — (+181/-20) — n/a: Changes Feishu progress behavior, placeholder media extraction, and the browser_vision failure note; its keep_open verdict preserves the failure-note change but blocks the Feishu portion pending maintainer policy direction.
• fix(browser): add full_page parameter to browser_vision for viewport-only screenshots #19647 partial — (+110/-10) — n/a: Adds an agent-facing full_page option for browser_vision unconditionally passes --full, producing unreadable screenshots on long pages #19620/browser_vision always captures full-page screenshots; consider viewport-default for context cost #54371, but its keep_open review requires dropping unrelated WhatsApp and invalid file-schema changes and porting the browser work with real normal, Lightpanda, and Camofox tests.
• Fix: browser_vision unconditionally passes --full, producing unreadable screenshots on long pages (#19620) #19765 partial — (+4/-2) — n/a: Makes --full conditional for browser_vision unconditionally passes --full, producing unreadable screenshots on long pages #19620, but the option is not exposed through the schema or registry and lacks current-path tests; despite its keep_open review, its diff is a narrower duplicate of fix(browser): add full_page parameter to browser_vision for viewport-only screenshots #19647/fix(tools): make browser_vision full-page capture opt-in via flag #20118.
• fix(tools): make browser_vision full-page capture opt-in via flag #20118 [closed] best fix — (+94/-9) — n/a: Closed duplicate reference for browser_vision unconditionally passes --full, producing unreadable screenshots on long pages #19620 that exposes and forwards full_page, conditions --full, documents Camofox behavior, and adds focused tests; it remains relevant as the recorded best existing implementation without implying reopening.
• feat(browser): add full parameter to browser_vision tool and default to false #21833 partial — (+12/-3) — n/a: Makes viewport capture the default and conditions normal and fallback screenshot arguments, but its keep_open review requires a current-layout port, Camofox behavior, and regression coverage.
• fix: read auxiliary.vision.model from config.yaml in vision/browser tools #24872 [closed] best fix — (+48/-2) — n/a: Closed configuration-routing fix for Bug: vision_analyze and browser_vision hardcoded to Gemma, ignore auxiliary.vision config #24842, still relevant as a superseded implementation because the automated close verdict records the behavior on main in commit 1496414 and merged fix(vision): forward custom-endpoint credentials + read aux model from config.yaml #57651.
• fix(vision): respect auxiliary vision routing #24875 partial — (+43/-37) — n/a: Routes browser vision through task-based auxiliary resolution for Bug: vision_analyze and browser_vision hardcoded to Gemma, ignore auxiliary.vision config #24842, but its keep_open review requires retaining the documented environment-only model fallback while making config authoritative.
• fix(vision): make vision tools honor model.supports_vision #29987 [closed] related — (+271/-39) — n/a: Closed precursor that introduced the browser native-vision fast path and was superseded by merged fix(vision): honor model.supports_vision in vision_analyze + browser_vision #34562, which rebased its four commits with authorship preserved.
• fix(vision): honor model.supports_vision in vision_analyze + browser_vision #34562 [merged] related — (+213/-33) — n/a: Merged reference implementation for the native browser-vision path, relevant because it embeds the screenshot directly and therefore defines the path affected by the missing proactive cap.
• fix(browser): apply vision embed-time pixel/byte caps in browser_vision #39226 duplicate — (+154/-0) — n/a: Adds byte- and dimension-triggered resizing plus useful cap tests, but its high-salvage keep_open review requires rejecting resized output that still exceeds the 7900px dimension cap.
• fix(browser): apply proactive embed cap to browser_vision native fast path #44922 partial — (+39/-1) — n/a: Duplicates fix(browser): apply vision embed-time pixel/byte caps in browser_vision #39226's proactive native-embed resize, but its high-salvage keep_open review requires post-resize validation of both targets and browser-specific regression tests.
• fix(vision): apply proactive embed cap to browser_vision screenshots #47468 best fix — (+230/-6) — n/a: The recorded best fix for browser_vision skips proactive embed cap — oversized full-page screenshots brick sessions #47467 checks byte and dimension triggers and adds native-path tests, but its keep_open review blocks attachment until both post-resize byte size and decoded dimensions are validated.
• fix(browser): proactively resize oversized screenshots before vision API (#47467) #47551 [closed] partial — (+20/-0) — n/a: Closed byte-only duplicate of fix(vision): apply proactive embed cap to browser_vision screenshots #47468 that remains relevant as an earlier proactive-resize attempt but misses the low-byte, over-dimension failure central to browser_vision skips proactive embed cap — oversized full-page screenshots brick sessions #47467.
• fix(browser): default non-finite Camofox vision config #51978 related — (+47/-2) — n/a: Validates finite Camofox vision timeout and temperature values; its keep_open review requires salvaging this as one shared helper across Camofox, regular browser, image, and video vision paths.
• fix(browser): default non-finite vision config #51981 related — (+46/-2) — n/a: Validates finite timeout and temperature values on the regular browser path; its keep_open review requires extending the same validation to image and video vision.
• fix(vision): cap browser_vision native embeds (full-page screenshots OOM local models, wedge Anthropic sessions) #63850 partial — (+244/-33) — n/a: Extracts a shared embed-cap helper and supplies strong byte, dimension, identity, and browser integration tests, but despite its high-salvage keep_open verdict the diff only rejects post-resize output above 20 MB and does not recheck the requested 4 MB and decoded 7900px limits.

Duplicates

Viewport-control family: #19647, #19765, #20118, and #21833, with closed #20118 the most complete reference and #21833 differing mainly on default policy. Embed-cap family: #39226, #44922, #47468, #47551, and #63850; #44922 and #47468 are documented duplicates of #39226, while #47551 is a narrower byte-only duplicate. Routing lineage: #29987 was superseded by merged #34562; #24872 and #24875 compete on #24842; #51978 and #51981 are backend-specific halves of the same finite-config bug class.

Suggested consolidation

Keep #47468 open with a salvage path: rebase it onto current main, preserve its byte-and-dimension trigger, incorporate #63850's shared helper and real-image tests, and require verified post-resize compliance with both _EMBED_TARGET_BYTES and _EMBED_MAX_DIMENSION before native attachment. After that validation lands in #47468, #39226, #44922, #47551, and #63850 can be closed as duplicates; this explicitly departs from the keep_open reviews on #39226, #44922, and #63850 only because their visible diffs implement the same cap path while none closes the post-resize validation gap, so their unique tests and helper work must first be preserved in #47468.

Complex graph

flowchart TD
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I19620(["issue #19620 (open)"])
    I47467(["issue #47467 (open)"])
    I54371(["issue #54371 (open)"])
    subgraph Dup39226 ["PRs duplicating each other"]
        P39226["PR #39226 (open)"]
        P44922["PR #44922 (open)"]
        P47468["PR #47468 (open)"]
        P47551["PR #47551 (closed)"]
        P63850["PR #63850 (open)"]
    end
    P63850 -.->|partial| I19620
    P63850 -.->|partial| I47467
    P63850 -.->|partial| I54371
    class I19620 open
    class I47467 open
    class I54371 open
    class P39226 open
    class P44922 open
    class P47468 open
    class P47551 closed
    class P63850 open
    class P47468 best
    class P63850 target
    click I19620 "https://github.com/NousResearch/hermes-agent/issues/19620"
    click I47467 "https://github.com/NousResearch/hermes-agent/issues/47467"
    click I54371 "https://github.com/NousResearch/hermes-agent/issues/54371"
    click P39226 "https://github.com/NousResearch/hermes-agent/pull/39226"
    click P44922 "https://github.com/NousResearch/hermes-agent/pull/44922"
    click P47468 "https://github.com/NousResearch/hermes-agent/pull/47468"
    click P47551 "https://github.com/NousResearch/hermes-agent/pull/47551"
    click P63850 "https://github.com/NousResearch/hermes-agent/pull/63850"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 17 pull requests and 5 issues in this complex. Each diff was read against this issue; Assessment working set: 150 kB of PR diffs, 43 kB of issue/PR text, 17 kB of discussion (32 comments), 38 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[bnikanjam]

Heads up for a maintainer: this PR's CI run has been stuck in action_required since it was opened (2026-07-13) — GitHub queued the workflow run but it's awaiting manual approval for a fork PR and was never approved (gh api repos/NousResearch/hermes-agent/actions/runs?head_sha=98d3939d0e02bd43cb5462ddab382f03cd5539d9 shows conclusion: action_required, zero check-runs ever executed). It's already been reviewed LGTM by @tonydwb, so it's just waiting on the CI approval click to move forward. Could someone with write access approve the workflow run? Rebasing won't help — it would just queue a new run behind the same gate.