fix(agent): bound native vision history payloads

[JoaoMarcos44]

Summary

Native-vision tool results could make long Hermes sessions exceed provider context, quota, or request-body limits even when the model-token estimate remained low.

This PR adds a shared, copy-on-write projection for inline image payloads, bounds native vision images before they enter reusable history, and applies the same policy to every request path that can resend conversation history.

Fixes #92699

User-visible problem

When vision_analyze or native browser_vision handled a screenshot, Hermes could embed the image as a large base64/data URL inside a tool result. That tool result remained in the conversation history and was replayed on every later model request.

The previous history-entry limits were:

• approximately 4 MiB for the reusable image payload;
• approximately 7900 pixels on the longest edge.

Those values were reasonable as one-shot provider limits, but not as history-reuse limits. A multi-megabyte screenshot could therefore be transmitted again and again throughout a session.

The issue reported sessions where:

• a single native embed was hundreds of thousands of characters;
• real request usage grew from roughly 150K to 550K tokens in one turn;
• the rough preflight estimate stayed much lower because images used a flat token heuristic;
• compression appeared to run but could not remove images inside the protected tail;
• anti-thrash eventually disabled further compression after repeated low-savings passes;
• the provider ultimately returned request-size or usage-limit errors such as HTTP 413 and HTTP 429.

Root cause

Five independent boundaries allowed the payload to escape protection:

1. History admission was too permissive. Native images entered reusable history at up to 4 MiB / 7900 px.
2. Model-token estimation was not a wire-size metric. Image parts were charged with a flat model-oriented heuristic, so multi-megabyte base64 payloads were almost invisible to preflight compression.
3. Historical-media pruning was role- and position-dependent. The old logic was anchored on user-role images and could miss native tool results, including images in the protected tail.
4. The max-iteration summary built its own API payload. That path could resend historical image payloads without going through the main request projection.
5. Provider-specific representations could bypass a content-only guard. OpenAI image_url, Responses input_image, Anthropic source.data, _multimodal envelopes, Anthropic sidecars, and native browser screenshots all needed coverage.

The core failure was not only that one image could be large. The deeper failure was allowing large visual data to become a persistent part of every future request without a shared aggregate byte policy.

Implementation

1. Bound images before they enter reusable history

tools/vision_tools.py now uses history-specific limits:

_EMBED_TARGET_BYTES = 256 KiB
_EMBED_MAX_DIMENSION = 1568 px

These are intentionally smaller than one-shot provider ceilings. The goal is to keep a native result cheap enough to reuse in a long conversation, not merely small enough to pass one individual API call.

A shared _prepare_native_vision_embed() function now owns this policy for both vision_analyze and native browser_vision.

The preparation flow is:

1. inspect the original payload size;
2. verify the original image dimensions with Pillow;
3. resize when the byte or dimension cap is exceeded;
4. verify the resized payload size;
5. decode the resized data URL;
6. verify the resized image dimensions again;
7. only then allow the image to be inserted into the native tool result.

If resizing cannot produce a payload within the history cap, the tool returns a structured error instead of inserting an unsafe image into reusable history.

If dimensions cannot be verified, the operation now fails closed. _image_exceeds_dimension() distinguishes:

• True: the image exceeds the dimension limit;
• False: the image was verified and is within the limit;
• None: the dimensions could not be verified.

This prevents malformed, corrupt, or undecodable images from bypassing the history guard.

2. Keep model-token estimation separate from wire-byte budgeting

agent/model_metadata.py adds:

• _inline_image_part_payload_bytes();
• estimate_messages_inline_image_bytes().

The byte metric understands:

• OpenAI Chat Completions image_url parts;
• OpenAI Responses input_image parts;
• Anthropic image blocks with source.data;
• transient _multimodal envelopes;
• _anthropic_content_blocks sidecars.

Remote HTTP image references do not contribute inline bytes because their binary contents are not serialized into the request.

The existing model-token estimator remains intentionally unchanged. Base64 transport bytes and model tokens are different currencies:

• model tokens approximate semantic context and model usage;
• inline bytes measure request-body pressure and transport limits.

The production protection uses the actual inline-byte measurement for projection instead of pretending that base64 length is a universal model-token count.

3. Add a shared copy-on-write request projection

agent/context_compressor.py adds _bound_inline_image_payloads() with these internal limits:

_INLINE_IMAGE_REQUEST_BUDGET_BYTES = 512 KiB
_INLINE_IMAGE_HISTORY_MAX_BYTES = 128 KiB

The projection:

1. discovers image parts across all supported representations;
2. measures each inline payload;
3. identifies the current user image and the active tool exchange;
4. preserves the complete active exchange when it fits the budget;
5. retains the newest active tool result as the continuity floor when the active exchange is too large;
6. retains older images newest-first while the aggregate budget allows them;
7. does not retain historical images larger than 128 KiB solely because aggregate room remains;
8. replaces removed images with provider-compatible text placeholders;
9. preserves message rows, ordering, roles, and tool_call_id pairing;
10. removes stale api_content sidecars from rewritten copies.

The current user image has continuity priority. The 512 KiB budget is therefore a soft aggregate safety bound around the active exchange, rather than a rule that can delete the input currently being answered.

The projection never mutates the canonical transcript. Only the API-bound copy is rewritten, so persistence, UI replay, and session history retain the original logical messages.

4. Make compression remove images from the protected tail

_strip_historical_media() now delegates to the same projection with:

max_inline_bytes=0

At a compression boundary, this explicitly ages out historical media even when it sits inside protect_last_n.

The compression projection retains the current user image and the newest active tool result for continuity, while replacing older image parts with text placeholders. This makes compression reclaim real payload bytes instead of reporting low savings while large screenshots remain protected.

Historical remote image references are also removed at this compression boundary even though they do not count as inline base64 bytes during ordinary request projection.

5. Apply the projection before prompt-cache planning

agent/conversation_loop.py applies _bound_inline_image_payloads() to the normal API message copy before prompt-cache planning and before the provider call.

This ordering is intentional:

conversation history
    -> API-bound copy
    -> sanitization and normalization
    -> inline-image projection
    -> prompt-cache planning
    -> provider request

Cache planning therefore sees the same bounded message representation that will be sent to the provider.

A necessary image eviction may change the provider prefix once, but subsequent projections are deterministic and stable. The persisted transcript remains unchanged.

6. Apply the same policy to max-iteration summaries

agent/chat_completion_helpers.py applies the same projection to the independently assembled max-iteration summary request.

This closes a separate request path that previously bypassed the main conversation-loop projection and could reintroduce historical native-vision payloads at the point where the session was already under the most context pressure.

7. Enforce the same cap for native browser screenshots

tools/browser_tool.py now sends native browser screenshots through _prepare_native_vision_embed() before building the native tool result.

This prevents browser_vision from bypassing the history cap by constructing its own data URL and calling the result builder directly.

Placeholder and compatibility behavior

When an image is removed from an outbound copy, the message itself remains present.

The replacement is a short text part:

[image omitted from older context to save request bytes]

The replacement type is adapted to the input format:

• OpenAI Chat-style image parts become text;
• Responses-style input_image parts become input_text;
• Anthropic sidecar image blocks become text blocks.

This preserves:

• message ordering;
• role alternation;
• tool-result rows;
• tool_call_id pairing;
• enough textual context to explain that visual data was intentionally omitted.

When a message is rewritten, its stale api_content sidecar is dropped so an older exact-wire representation cannot restore the removed image later in the request pipeline.

Tests added and updated

Byte metric and projection tests

tests/agent/test_inline_image_payload_budget.py covers:

• large inline payloads producing a larger byte metric while the model-token estimate remains heuristic;
• remote URLs not being counted as inline payloads;
• _multimodal envelopes;
• Anthropic content sidecars;
• aggregate request budgeting;
• preservation of the active exchange when it fits;
• newest-first historical retention;
• copy-on-write behavior;
• stale api_content removal;
• Responses input_image placeholders;
• text follow-ups aging out old tool images;
• protected-tail image removal during compression;
• historical remote-image removal;
• user-role images remaining protected during the active turn.

Production-path tests

tests/run_agent/test_413_compression.py verifies the real conversation path:

• the provider receives the bounded request copy;
• tool rows remain present;
• old image data is absent from the outgoing payload;
• the logical transcript returned by the agent still retains the original image parts.

tests/agent/test_api_content_sidecar.py verifies that the separately assembled summary request applies the same image projection.

Native vision and browser tests

tests/tools/test_vision_native_fast_path.py verifies:

• resized embeds stay within the byte cap;
• post-resize dimensions stay within the long-edge cap;
• a resize that remains oversized is rejected;
• unverifiable resized dimensions are rejected.

tests/tools/test_vision_tools.py updates the dimension-validation contract so unknown dimensions return None instead of being treated as safe.

tests/tools/test_browser_console.py now uses a valid decodable 1x1 PNG fixture because the native history guard verifies real image dimensions.

Verification

Focused canonical test runs recorded on Windows 11 / Python 3.11:

scripts/run_tests.sh tests/agent/test_inline_image_payload_budget.py tests/agent/test_compressor_image_tokens.py tests/agent/test_compressor_historical_media.py tests/agent/test_api_content_sidecar.py tests/agent/test_model_metadata.py tests/tools/test_vision_native_fast_path.py tests/tools/test_vision_tools.py tests/run_agent/test_image_shrink_recovery.py tests/run_agent/test_413_compression.py -q
-> 236 passed, 0 failed

scripts/run_tests.sh tests/agent/test_context_compressor.py tests/agent/test_compressor_media_stripping.py tests/agent/test_compressor_zero_user_guard.py tests/agent/test_compressor_tool_call_budget.py tests/agent/test_compression_progress.py tests/agent/test_compression_anti_thrash_persistence.py tests/agent/test_compression_anti_thrash_recovery.py tests/agent/test_preflight_compression_gate.py tests/run_agent/test_multimodal_tool_content_recovery.py tests/run_agent/test_vision_aware_preprocessing.py -q
-> 189 passed, 0 failed

scripts/run_tests.sh tests/tools/test_browser_console.py -k browser_vision -q
-> 3 passed, 0 failed

uv run ruff check <changed files>
-> passed

uv run python -m py_compile <changed files>
-> passed

uv run python scripts/check-windows-footguns.py --all
-> passed

The following local runs were not reported as green because they were blocked by unrelated environment issues:

• tests/tools/test_browser_use_cli.py: 70 passed and 22 failed because the Windows fixture attempted to launch POSIX shell scripts and received [WinError 193].
• tests/run_agent/test_run_agent.py: 274 passed and one pre-existing test failed because the optional anthropic package was not installed locally.

The first remote CI run identified an invalid browser PNG fixture after the new fail-closed dimension validation. The follow-up commit replaces that header-only fixture with a valid decodable PNG.

The current PR head is 28fef2dcbce8c258d4ef1ba27f63a959a3f5a2df.

Remote GitHub status for the current head:

• All required checks pass;
• Python tests passed;
• E2E passed;
• Windows-only tests passed;
• macOS-only tests passed;
• Ruff and type checks passed;
• Windows footgun checks passed;
• Docker builds passed;
• Nix checks passed;
• supply-chain checks passed;
• no blocking check failed.

Behavioral trade-offs

• The byte budgets are internal safety bounds; no new user configuration key was added.
• The current user image is protected for continuity.
• A complete active tool exchange is protected when it fits the budget.
• If the active exchange is too large, the newest result is retained as the continuity floor and older active screenshots are degraded deterministically.
• Historical images larger than 128 KiB are not retained merely because aggregate room remains.
• Native new embeds are capped at 256 KiB / 1568 px before they enter reusable history.
• The original conversation history remains available for persistence and UI replay; only outbound request copies are projected.
• Model-token estimation remains separate from wire-byte accounting by design.

Risk and rollback

The change consists of in-memory request projection plus the existing image resize machinery. The canonical conversation transcript is not destructively rewritten by normal request projection, and placeholders preserve message ordering and tool-call pairing in the outbound copy.

Reverting the two commits restores the previous image retention and native history behavior.

Infographic

[kshitijk4poor]

Thanks for the thorough work here — closing in favor of #92783, which salvages the narrower #92725 plus a follow-up that adopts this PR's key insight (one shared cap policy covering BOTH vision_analyze and browser_vision native embeds). You're credited with Co-authored-by on that commit.

Why the narrower fix was taken (both findings probe-confirmed against your branch):

1. Per-request projection vs the prompt-cache invariant. _bound_inline_image_payloads on every API call in conversation_loop rewrites already-sent history outside compression. In the realistic screenshot-QA shape (tool exchange → user follow-up each turn), every new user turn makes the previous exchange historical, so its just-sent inline image flips to a placeholder — the cached prefix diverges at its tail every turn. The repo's hardest invariant (AGENTS.md) is that compression is the only sanctioned context mutation; a standing per-request rewrite is a design fork that would need maintainer sign-off regardless of code quality.

2. 128 KiB history cap vs 256 KiB embed cap. A screenshot in the 128–256 KiB band is dropped from the outbound copy the moment the user sends a text follow-up — probe: "what color was the button?" right after a 200 KiB screenshot gets a model that cannot see the screenshot it just took. That's the exact follow-up workflow the issue is about.

Also noted during review: the fail-closed Pillow dimension check hard-fails native vision for any image Pillow can't decode (exotic formats/truncated files), where main degraded gracefully to byte-checks.

The mechanism itself (copy-on-write projection, sidecar handling, max-iteration-summary parity) is genuinely well-built — if the maintainers ever want request-time byte budgeting as a policy decision, this PR is the reference implementation. Thanks again!