Skip to content

fix(compressor): age out stale tool-result images during compaction - #90001

Open
jackulau wants to merge 1 commit into
NousResearch:mainfrom
jackulau:fix/89938-strip-stale-tool-result-images
Open

fix(compressor): age out stale tool-result images during compaction#90001
jackulau wants to merge 1 commit into
NousResearch:mainfrom
jackulau:fix/89938-strip-stale-tool-result-images

Conversation

@jackulau

Copy link
Copy Markdown
Contributor

Relationship to #89965 - please read this first

#89965 and this PR fix different halves of #89938 and do not overlap by a single line. #89965 bounds the image bytes on every outgoing request so the 413 stops happening. This PR fixes the reason the session could not recover once it did.

In one sentence: #89965 prevents the 413; this makes the 413 recovery actually recover.

_strip_historical_media is what the 413 handler's compaction ends up calling, and in the reported session it is a no-op - so every recovery pass returned a body that was still multi-MB and the provider answered 413 again. That is the "7 compactions in 13 minutes, all below 200K tokens" in the report. #89965 does not touch that function, so with only #89965 merged the wedge is unreachable in the common case but still there: any single window that exceeds the provider's body limit (a run of images larger than keep_recent, or a provider whose limit is smaller than three screenshots) drops into the same loop with no way out.

I am not proposing this as an alternative to #89965. If you would rather have one change, merge theirs - it is the larger fix and it is the one that stops the 413. This is Refs #89938, not Fixes, for exactly that reason.

What does this PR do?

agent/context_compressor.py:_strip_historical_media anchors on the newest image-bearing user message and strips images from everything before it:

anchor = -1
for i in range(len(messages) - 1, -1, -1):
    ...
    if msg.get("role") == "user" and _content_has_images(msg.get("content")):
        anchor = i
        break

if anchor <= 0:
    return messages          # <-- the whole pass, skipped

A session whose images arrive from tools rather than attachments has nothing to be "before". The reported reproduction is the worst case of this and trips both exits at once:

  • one attachment on the first user message, so anchor == 0 and the early return fires; and
  • twenty vision_analyze results after it, which the user anchor would not have protected anyway.

So the function returns the list untouched, the ~4MB of base64 rides along on every request, and the compaction the 413 handler invokes as recovery frees nothing.

This ages tool-result images on their own timeline: keep the newest one, strip every older one wherever it sits.

Related Issue

Refs #89938

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Security fix
  • Documentation update
  • Tests (adding or improving test coverage)
  • Refactor (no behavior change)
  • New skill (bundled or hub)

Changes Made

  • agent/context_compressor.py - _strip_historical_media gains a second anchor, tool_anchor, the newest image-bearing role: "tool" message. A message is now stripped if it is before the user anchor (rule 1, unchanged) or it is an image-bearing tool message that is not the newest one (rule 2, new). The early return now fires only when neither rule can do anything.
  • tests/agent/test_compressor_historical_media.py - 9 tests.

Three decisions worth a maintainer's eye

1. Rule 1 keeps precedence. Where the two rules disagree - a tool result that is the newest of its kind but sits before the user anchor - rule 1 wins and it is stripped, which is exactly today's behaviour. test_tool_image_before_the_user_anchor_is_still_stripped pins that, and it is the test to look at if you think the ordering should go the other way.

2. The newest tool image survives inside the protected tail; older ones do not. protect_last_n exists to preserve conversational continuity, not to pin bytes the model has already moved past, and a stale screenshot inside the tail is the single largest thing in the payload. This is the one behavioural change to the tail, and it is deliberate. If you would rather the tail stay byte-exact, rule 2 needs an index >= tail_start exclusion - and the fix then stops working for the reported session, because all twenty results were in the tail.

3. User images are untouched. I deliberately did not unify the two anchors into "newest image-bearing message of any role", which would have been the smaller diff. It would let a tool screenshot evict the user's most recent attachment, and agent/context_compressor.py:7503-7510 documents a live dependency on the user anchor being kept byte-for-byte (the _force_user_leading guard reasons about an image-only user message having no text placeholder). Two anchors is more code and fewer surprises.

How to Test

uv run pytest tests/agent/test_compressor_historical_media.py -q     # 19 passed
uv run pytest tests/agent/test_compressor_zero_user_guard.py -q      #  6 passed
uv run pytest tests/agent -q -k compress                             # 436 passed, 1 pre-existing failure
uv run ruff check agent/context_compressor.py tests/agent/test_compressor_historical_media.py

To see the old behaviour, restore the if anchor <= 0: return messages early return and re-run: test_compress_frees_stale_vision_tool_results fails with both vision_analyze results still carrying their base64.

Verification

Mutation proof - 6 mutations, 6 caught:

# Mutation Caught by
1 restore the original early return + i >= anchor loop (the bug) 6 tests, including the end-to-end compress() one
2 tool_anchor scans forward, so the OLDEST tool image is kept newest-survives-in-tail, sidecar, end-to-end
3 rule 2 strips every tool image, including the newest newest-survives-in-tail, single-tool-image, first-message-user-image
4 rule 2 drops the role == "tool" test and applies to all roles newest-survives-in-tail, rule-1-precedence, the pre-existing user-path integration test
5 rule 1 removed, rule 2 checked first rule-1-precedence, non-dict-passthrough, the pre-existing user-path integration test
6 drop_stale_api_content removed from the rewrite sidecar test

Mutations 4 and 5 being caught by test_compress_strips_historical_images - a test that predates this PR - is the useful signal: the existing user-path contract is still pinned, not merely believed.

Pre-existing failures, named rather than hidden. tests/agent/test_compression_review_76354.py::TestF6ExecutorSaturation::test_cancelled_fence_skips_summary_work_before_start fails on a pristine checkout of 13ce0c5c67 with this branch stashed, and fails identically with it applied. Same for four in the -k "image or vision" sweep (test_image_routing.py x2, test_save_url_image.py, test_vision_routing_31179.py), all of which fail on the clean tree here. This PR touches none of them.

Platform: Windows 11, Python 3.12, uv. The change is pure list/dict manipulation - no paths, no processes, no platform-specific behaviour.

Checklist

Code

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) or N/A: the docstring now states both rules and why the tool rule exists
  • I've updated cli-config.yaml.example if I added/changed config keys or N/A: no config keys
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide or N/A: pure data transformation
  • I've updated tool descriptions/schemas if I changed tool behavior or N/A

Screenshots / Logs

The loop this removes, from the report:

09:41:36  API call failed: HTTP 413: request body exceeds configured limit
09:41:36  context compression started: tokens=~183,487
09:44:02  context compression done: messages=185->168 tokens=~165,384
09:44:05  API call failed: HTTP 413: request body exceeds configured limit
...
09:46:02  Compression made no progress, skipping boundary rewrite

The token count falls on every pass and the body does not, because the bytes that matter are base64 in tool results and nothing was stripping them.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation area/compression Context compression and continuation sessions sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Aug 19, 2026
@jackulau

Copy link
Copy Markdown
Contributor Author

The one red check on this PR is not this PR's.

FAILED tests/tools/test_image_generation.py::TestFalCatalog::test_upscale_defaults_are_all_off
  AssertionError: xai/grok-imagine-image/v2.0/text-to-image must not default upscale on
  assert True is False
=== 257 files, 4135 tests passed, 1 failed, 49 skipped ===

That is Run tests slice 12/12, which runs on every Python PR. It fails on a pristine 13ce0c5c67 with this branch stashed, and it fails identically with it applied. This branch touches agent/context_compressor.py and one test file and does not import tools/image_generation_tool.py.

Cause is on main: ceabb030fb set "upscale": True for the new Grok Imagine 2.0 catalog entry, which the policy test added by f06c41522e forbids for every entry. Fixes are already open as #89933, #89935, #89939 and #89946, so there is nothing to do here; I have left the timeline on #89933. Everything else on this PR is green, and the compression suites it does cover are listed in the body.

@jackulau
jackulau force-pushed the fix/89938-strip-stale-tool-result-images branch from 2499a77 to a92a783 Compare August 19, 2026 18:44
_strip_historical_media anchors on the newest image-bearing USER message and
returns the list untouched when that anchor is index 0 or does not exist. A
session whose images arrive from tools rather than attachments therefore has
nothing to be "before": twenty vision_analyze results keep multi-MB of base64
in every request body, the provider answers 413, and the 413 handler's
recovery compaction lands right back in this function and frees nothing. The
reporter saw seven compactions in thirteen minutes, all below 200K tokens.

Age tool-result images on their own timeline: keep the newest one, since that
is the image the model is reasoning about, and strip every older one wherever
it sits, including inside the protected tail. The tail exists to preserve
conversational continuity, not to pin bytes the model has already moved past.

User-message images keep today's treatment exactly. The user anchor is
checked first, so a tool result that is the newest of its kind but still sits
before that anchor is stripped as it always has been, and the anchor message
itself is still kept byte-for-byte - test_compressor_zero_user_guard depends
on that.

Refs NousResearch#89938
@jackulau
jackulau force-pushed the fix/89938-strip-stale-tool-result-images branch from a92a783 to 749f378 Compare August 19, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/compression Context compression and continuation sessions comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants