Skip to content

fix(compressor): strip _multimodal tool-result images after compress() - #47412

Closed
srojk34 wants to merge 1 commit into
NousResearch:mainfrom
srojk34:fix/strip-multimodal-screenshots-after-compress
Closed

fix(compressor): strip _multimodal tool-result images after compress()#47412
srojk34 wants to merge 1 commit into
NousResearch:mainfrom
srojk34:fix/strip-multimodal-screenshots-after-compress

Conversation

@srojk34

@srojk34 srojk34 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Problem

browser_vision and computer_use return screenshots as a _multimodal envelope:

{"_multimodal": True, "content": [...base64...], "text_summary": "..."}

Each screenshot can be several megabytes. _strip_historical_media only stripped user-attached images, so _multimodal tool results accumulated across turns and inflated every subsequent API request payload.

Providers enforce limits on byte size, not token count. Sessions with a handful of browser_vision calls produced HTTP 413 errors that survived every compression pass even when the token estimate was well within the model's context limit (issue #47339: 101 → 13 messages, still 413).

Fix

Add _strip_all_multimodal_tool_results(), called inside compress() after _sanitize_tool_pairs() and before _strip_historical_media().

It replaces every _multimodal envelope with its text_summary string:

{"_multimodal": True, "content": [...3MB base64...], "text_summary": "login page with a form"}
→ "[screenshot removed] login page with a form"

Why it's safe to strip all of them (including the most recent): compress() only runs between turns, never mid-turn. By the time compression triggers, the model always has an assistant response for every tool result in the conversation — the base64 has already been processed and retaining it serves no purpose.

Tests

6 new tests in TestStripAllMultimodalToolResults:

  • single envelope stripped (no anchor exception)
  • all envelopes stripped across multiple calls
  • text_summary used as replacement text
  • fallback placeholder when text_summary absent
  • non-_multimodal tool results unchanged
  • user/assistant messages unchanged

All 102 tests pass.

Fixes #47339.

browser_vision and computer_use return screenshots in a _multimodal
envelope {_multimodal: True, content: [...base64...], text_summary: "..."}.
Each screenshot can be several megabytes.  _strip_historical_media only
stripped user-attached images, so _multimodal tool results accumulated
across turns and inflated every subsequent API request.  Because providers
enforce payload limits on byte size rather than token count, sessions with
a handful of browser_vision calls produced HTTP 413 errors that survived
every compression pass even after the token estimate was well within limit.

Add _strip_all_multimodal_tool_results(), called inside compress() after
_sanitize_tool_pairs() and before _strip_historical_media(). It replaces
every _multimodal envelope with its text_summary string.  Calling it post-
compress() is safe because the model always has an assistant response for
every tool result before compression runs — the base64 has already been
processed and keeping it serves no purpose.

Fixes NousResearch#47339.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Jun 16, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the clear root-cause analysis and focused regression coverage. This is already implemented on current main; this is an automated hermes-sweeper review.

@teknium1 teknium1 closed this Jul 14, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

413 Request Entity Too Large: context compression doesn't evict image/vision payloads — fails even at ~44k tokens on Copilot

3 participants