fix(agent): recover from image-dominated 413 payloads

[brianbaldock]

What does this PR do?

A session with plenty of context left dies permanently on 413:

Request payload too large: max compression attempts (3) reached.

Status bar at the time: 122k/936k │ 13%. Once hit, the session is unrecoverable — /compress and /retry both fail the same way, because the oversized content is re-sent from stored history on every subsequent turn.

Root cause. A 413 is a byte-size error, but the recovery path decides whether it made progress using the token estimate, which is deliberately blind to image bytes.

agent/model_metadata.py:

_IMAGE_TOKEN_COST = 1500   # flat, per image

That flat cost is correct and intentional for context budgeting — without it a 1MB screenshot would estimate at ~250K tokens and trigger premature compaction. But agent/conversation_loop.py reuses that same estimate to score 413 recovery:

new_tokens = estimate_messages_tokens_rough(messages)
if len(messages) < original_len or (new_tokens > 0 and new_tokens < original_tokens * 0.95):
    # retry with compressed messages
else:
    # -> "cannot compress further", turn is dead

When the payload is image-dominated, that test can never be satisfied. Real numbers from an affected session:

	bytes	contribution to estimate
2 × vision_analyze results	5,627,202	~3,000 tokens
all 193 other messages	~193,000	~77,000 tokens

The images are 96.6% of the request body and ~3.7% of the token estimate. Compression can summarize away every text turn and still not move the estimate by 5%, so it reports no_progress, burns all three attempts, and gives up. Telemetry from the failing session:

{"failure_class":"no_progress","commit_status":"aborted",
 "current_estimated_tokens":80453,"effective_threshold":234000,
 "main_context_limit":936000}

Note 80453 < 234000 — compression doesn't even believe it should be running.

There is an existing image-strip fallback in the else branch, but it doesn't resolve this for two reasons:

1. it only mutates api_messages (the transient per-call copy), so the megabytes return from stored history on the very next turn
2. it's reached only after the attempt budget is spent

Why this approach. The fix measures the thing the provider is actually rejecting (bytes) rather than a proxy that is designed to ignore it, and it mutates messages so the reduction survives into the next turn. It is additive and scoped to the no-progress branch, so every non-image-dominated 413 keeps its current behavior exactly.

Related Issue

This PR was opened before I searched the tracker properly — that was my mistake, and the search turns up substantial prior art. Recording it honestly rather than quietly claiming the ground:

• 413 Request Entity Too Large: context compression doesn't evict image/vision payloads — fails even at ~44k tokens on Copilot #47339 (closed) — the original report of this exact symptom on Copilot: 413 at ~44k tokens, "cannot compress further", browser_vision screenshots. Same bug class.
• 413 payload-too-large recovery loops: text compression runs before image stripping, so base64 vision payloads are never dropped #89286 (open) — same branch in conversation_loop.py, proposes reordering it so the image strip runs before text compression rather than after the budget is spent.
• HTTP 413 loop: base64 images in tool results re-ship on every turn until provider rejects #89938 (open) — the upstream cause: _strip_historical_media misses tool-result images and first-message images, so base64 accumulates in messages and re-ships every turn.
• Screenshots are re-uploaded on every turn: evict stale vision payloads before send instead of after a 413 #89296 (open) — the proactive complement: evict stale vision payloads on the outbound path so the oversized body is never serialized at all.

Fixes #47339

Overlapping open PRs (maintainers: this needs a consolidation decision, not four parallel merges):

• fix(413): recover from image-payload 413s instead of bricking the session #52444 (@Caoxuyang, 2026-06-25, +186/-1) — predates this PR by two months and edits the same branch, but with a different mechanism: it shrinks, this one strips. fix(413): recover from image-payload 413s instead of bricking the session #52444 re-encodes every embedded image progressively smaller via Pillow (≤512KB, then ≤256KB) so the images survive at lower fidelity; this PR removes oversized image parts entirely and leaves a placeholder. Two substantive differences beyond that: (a) fix(413): recover from image-payload 413s instead of bricking the session #52444 mutates api_messages only, so the full-size base64 remains in stored history and the next turn rebuilds an oversized body — recovering the turn but paying a 413 round-trip plus a re-encode pass on every subsequent turn, whereas this PR mutates messages so the reduction persists; (b) fix(413): recover from image-payload 413s instead of bricking the session #52444 leaves the new_tokens < original_tokens * 0.95 progress check unfixed and instead tries never to reach it, so if the shrink no-ops (both passes spent, Pillow unavailable, or images already small) it still falls through to the same false-negative that bricks the session. These are complementary, not competing — shrink first to preserve fidelity, strip as the correctness floor when shrinking can't get under the limit. Where a screenshot is still decision-relevant, fix(413): recover from image-payload 413s instead of bricking the session #52444's behavior is clearly better than mine.
• fix(agent): drop vision payloads before compressing on a 413 #89776 (@AP3X-Dev, +147/-20) — implements the 413 payload-too-large recovery loops: text compression runs before image stripping, so base64 vision payloads are never dropped #89286 reordering.
• fix(agent): bound historical image payloads per request #89965 (@Kewe63, +390/-0) — implements the Screenshots are re-uploaded on every turn: evict stale vision payloads before send instead of after a 413 #89296 per-request eviction.

These are not all mutually exclusive, and none of them is a strict duplicate of another. #89965 is proactive (don't send it), #89776 is ordering (try the cheap fix first), #52444 is fidelity-preserving reduction (shrink the images), and this PR is the correctness floor (make the progress check measure bytes so recovery can't false-negative even when the others no-op). But all four touch the is_payload_too_large branch in conversation_loop.py and will conflict textually, so this wants a consolidation decision rather than four independent merges. The coherent stack, in my view, is #89965 → #52444 → this PR (avoid the oversized body; if it happens, shrink; if shrinking can't get under the limit, strip and make the progress check honest). I'm happy to rebase this into a narrower change on top of whichever lands first, or to close it if a maintainer would rather fold strip_oversized_image_parts() into one of the others — the byte-measurement tests are useful under any ordering. Say the word and I'll do the work.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• agent/message_sanitization.py — new strip_oversized_image_parts(). Measures actual payload bytes of inline data: URLs and removes only image parts above the byte budget.
  • Preserves alternation invariants: a tool-role message stripped to nothing becomes a plaintext placeholder rather than being deleted, keeping the paired tool_call_id on the prior assistant message matched. Deleting it would produce a 400. Same contract as the neighbouring _strip_images_from_messages.
  • protect_last_n=4 — an image the user just attached isn't removed out from under the current turn; recovery reaches for older history first.
  • Text is never touched — only oversized image parts are removed, so the model keeps the surrounding reasoning.
  • Threshold 256KB — above a normal screenshot's base64 size, low enough to catch full-resolution captures before they dominate a request.
• agent/conversation_loop.py — call it in the 413 no-progress branch before declaring the turn dead, mutating messages (not just api_messages) so the reduction persists across turns.
• tests/agent/test_413_image_payload_recovery.py — 11 tests, new file.

Behavior is unchanged for every non-image-dominated 413: if nothing exceeds the byte budget, the function reports no change and the existing fallback and terminal error fire exactly as before.

How to Test

Reproduction:

1. Start a session with a vision-capable model on a provider that enforces a request body limit (reproduced on copilot).
2. Call vision_analyze on two full-resolution screenshots (~2MB PNGs, ~2.7MB each as base64).
3. Continue the conversation for a few turns.
4. Before this PR: the turn dies with Request payload too large: max compression attempts (3) reached while the status bar still reads ~13% context, and /compress and /retry both re-fail because history re-ships the base64.
5. After this PR: the no-progress branch strips the oversized image parts from messages, the retry succeeds, and the session continues with its text intact.

Automated:

PYTHONPATH="$PWD:$PWD/venv/lib/python3.11/site-packages" \
  pytest tests/agent/test_413_image_payload_recovery.py -q
# 11 passed

The tests assert invariants rather than snapshots:

• TestTokenEstimateIsBlindToImageBytes pins the root cause directly: a ~3000x increase in image bytes leaves the token estimate essentially unchanged, which is why text compression can't clear this class of 413.
• Alternation safety: a tool message stripped to nothing keeps its slot and tool_call_id.
• protect_last_n shields the current turn.
• Text survives image stripping.
• Realistic end-to-end sizing based on the session that produced this report (>5MB reclaimed).
• Degenerate inputs (empty list, zero budget, non-list) are no-ops.

Verified these tests actually fail without the fix — sabotage run with the byte-measurement neutered gave 5 failed / 6 passed — rather than trusting a green run on code I just wrote.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(agent):)
• I searched for existing PRs to make sure this isn't a duplicate — done late; it surfaced three overlapping PRs, analyzed in Related Issue above. Not a duplicate of any of them, but it does need a consolidation decision.
• My PR contains only changes related to this fix (3 files, +333/-0, no unrelated commits)
• I've run pytest tests/ -q and all tests pass — partial, see note below
• I've added tests for my changes
• I've tested on my platform: Debian GNU/Linux 13 (trixie), Python 3.13.5

Note on the full suite. I ran the 11 new tests (pass) and a clean upstream baseline of tests/run_agent/ (1694/1694 in 224s). I have not got a clean full-repo pytest tests/ -q on the branch: my first attempt was run while switching branches underneath the process, which produced 254 uninterpretable failures for a tree that didn't even contain the fix. I discarded that result rather than report it. Flagging the gap honestly instead of checking the box. The branch is also currently based ~35 commits behind main and I'll rebase on request.

Documentation & Housekeeping

• I've updated relevant documentation — N/A, internal recovery-path fix with no user-facing config or docs surface
• I've updated cli-config.yaml.example if I added/changed config keys — N/A, no new config keys (threshold and protect_last_n are internal constants; happy to promote them to config.yaml under compression. if maintainers prefer)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A, no architecture change
• I've considered cross-platform impact — N/A for OS specifics: pure in-memory string/dict manipulation, no file I/O, no process or path handling. Provider-specific though, since body limits differ; tested against copilot.
• I've updated tool descriptions/schemas if I changed tool behavior — N/A, no tool schema change

Screenshots / Logs

Failing session, before the fix:

⚠️  API call failed (attempt 1/3): APIStatusError [HTTP 413]
   🔌 Provider: copilot  Model: claude-opus-5
   📝 Error: HTTP 413: Request Entity Too Large
   ⏱️  Context: 195 msgs, ~80,453 tokens
⚠️  Request payload too large (413) — compression attempt 3/3...
❌ Payload too large and cannot compress further.

Measured payload composition of that same session, from ~/.hermes/state.db:

2 vision_analyze tool results : 5,627,202 bytes  (96.6% of body)
193 other messages            :   ~193,000 bytes ( 3.4% of body)

[Enough1122]

AI code review — automated review for reference; please use your judgment.

Excellent root-cause analysis: the estimator's flat per-image pricing is exactly why text compression structurally cannot clear an image-dominated 413, and measuring actual payload bytes is the correct recovery driver. Persisting the strip into stored history (so the fix survives future turns), protecting the most recent N messages, and preserving tool_call_id linkage via placeholder are all thoughtful. One correctness gap:

1. agent/message_sanitization.py:strip_oversized_image_parts (~470) — when a non-tool message (typically role:"user") loses all its parts, it's deleted outright rather than replaced with a placeholder like the tool case. A [user(image), assistant(reply)] pair then becomes [assistant(reply)]: possible double-assistant adjacency with earlier history, or a transcript that starts with an assistant row after later truncation — several providers hard-reject both. Mirror the tool treatment (plaintext "[image removed]" placeholder) for user/system roles too, or verify every target provider tolerates the reshaped alternation.
2. The recovery fires only on "compression scored no progress"; if compression does make progress but the remaining payload still exceeds the limit, each attempt burns budget before reaching here. Consider checking measured image bytes up front when a 413 arrives, so the first retry already strips. (nit)
3. protect_last_n shields messages, not images — a user's just-attached screenshot within the protected window makes the strip a no-op (changed=False), correctly falling through to the existing tool-message stripper below. Worth one comment noting that fall-through is intentional. (nit)
4. The tests asserting the bytes-driven invariant (flat-estimate scenario must still recover) are exactly right. (positive)

No blocking issues found beyond item 1's alternation risk.