Skip to content

fix(compressor): strip _multimodal tool-result images in _strip_historical_media - #47398

Closed
srojk34 wants to merge 1 commit into
NousResearch:mainfrom
srojk34:fix/strip-historical-media-tool-results
Closed

fix(compressor): strip _multimodal tool-result images in _strip_historical_media#47398
srojk34 wants to merge 1 commit into
NousResearch:mainfrom
srojk34:fix/strip-historical-media-tool-results

Conversation

@srojk34

@srojk34 srojk34 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Problem

_strip_historical_media only searched for image-bearing user messages when
selecting the anchor. Sessions driven by browser_vision or computer_use
(native-vision fast path) accumulate screenshots exclusively in tool result
messages as _multimodal envelopes — their user turns carry plain text. Because
the anchor search never matched those tool results, no base-64 payload was ever
stripped. After two compression passes (e.g. 101 → 13 messages) the screenshots
were still present in the tail, pushing the outgoing payload above the provider's
HTTP 413 limit even though the estimated token count looked safe. Reported in #47339.

Fix

  • Add _tool_result_has_media() to recognise _multimodal envelopes and
    OpenAI-style list content in tool-result messages.
  • Add _strip_tool_result_media() to replace a _multimodal envelope with its
    text_summary string and to strip image parts from list-style tool results.
  • Extend the anchor search in _strip_historical_media to also consider
    tool-result messages (user-image anchoring is preserved as-is).
  • Strip tool-result images for all messages before the anchor alongside the
    existing user-image stripping.

Tests

TestStripHistoricalMediaToolResults (5 cases):

  • _multimodal envelope in tool result before anchor → stripped to text_summary
  • Most-recent tool result (the anchor itself) → left untouched
  • Only one screenshot in session → nothing stripped (no-op, returns original list)
  • Legacy user-image stripping still works unchanged
  • OpenAI-style list content in tool results → image parts stripped
uv run --frozen python -m pytest tests/agent/test_context_compressor.py -x -q
101 passed in 6.11s

…rical_media

browser_vision and computer_use (native-vision fast path) return a
``_multimodal`` envelope as the tool result.  ``_strip_historical_media``
only searched for image-bearing *user* messages when choosing the anchor,
so sessions that accumulated multiple browser_vision screenshots never
stripped any base-64 payload from tool results.  After two compression
passes (e.g. 101 → 13 messages) those screenshots were still present in
the tail and kept the outgoing payload above Copilot's HTTP 413 limit
even though the estimated token count appeared safe.

Fix:
- Add ``_tool_result_has_media`` to recognise ``_multimodal`` envelopes
  and OpenAI-style list content in tool-result messages.
- Add ``_strip_tool_result_media`` to replace a ``_multimodal`` envelope
  with its ``text_summary`` and to strip image parts from list-style
  tool results.
- Extend the anchor search in ``_strip_historical_media`` to also consider
  tool-result messages (user-image anchoring is preserved as-is).
- Strip tool-result images for all messages before the anchor alongside
  the existing user-image stripping.
- Add ``TestStripHistoricalMediaToolResults`` with five cases covering
  ``_multimodal`` envelopes, list-style images, legacy user-image
  stripping, single-screenshot no-op, and mixed scenarios.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround labels Jun 16, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related: #47397, #42509, #37412 (all fix #47339 — context compression not evicting image/vision payloads, leading to HTTP 413).

This PR extends the merged _strip_historical_media fix from #27189, which only anchored on user-image messages and missed tool-result _multimodal envelopes accumulated by browser_vision / computer_use. Verified the gap is still live on main (no _tool_result_has_media/_strip_tool_result_media). Gap-fill of the incomplete merged fix → related, not duplicate. #47397 addresses the same issue at the 413 retry-recovery level instead of the compression root cause.

@srojk34

srojk34 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favor of a more complete fix that strips all _multimodal tool-result images after compress(), not just pre-anchor ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants