fix(agent): shrink oversized inline images before compressing on 413

[mcull]

Problem

When a user posts a large photo (e.g. 15–25 MB iPhone HEIC/JPEG, common via Slack/Telegram), the gateway downloads it and the agent inlines it as a base64 data: URL on the user turn. With Anthropic, this triggers a 413 "payload too large" because the whole request body blows past the cap.

The existing image-shrink recovery (_try_shrink_image_parts_in_messages) only fires on FailoverReason.image_too_large, which is matched against Anthropic's 400 "image exceeds 5 MB maximum" wording. A whole-request 413 routes to payload_too_large → compression-only path.

Compression drops old messages but leaves the huge inline image on the current turn intact, so:

413 → compress (no change) → 413 → compress (no change) → … → max attempts → session auto-reset

This is the exact failure mode Marc hit on the items-for-sale household workflow — drop a photo in the grove Slack channel, get a session reset before the agent can analyze it.

Fix

In the 413 branch of the recovery handler, try _try_shrink_image_parts_in_messages once before counting a compression attempt. If it changes anything, retry immediately. Otherwise fall through to existing compression behavior.

• Symmetric with the existing 400/image_too_large path (same helper, same flag)
• Gated by image_shrink_retry_attempted so a second 413 still falls through to compression rather than looping on shrink
• Zero behavior change when there are no oversized data-URL image parts to shrink

Tests

Added two tests to tests/run_agent/test_413_compression.py:

• test_413_with_oversized_image_shrinks_before_compression — 413 + shrinkable image → shrink fires, compression does not run
• test_413_falls_back_to_compression_when_no_image_to_shrink — 413 + no image → existing compression path still works

All 36 tests in test_413_compression.py + test_image_shrink_recovery.py pass.

[AlexBevan]

Confirming this on a second provider – I hit the identical failure on GitHub Copilot, not just Anthropic. A single ~4 MB phone JPEG (base64 ≈ 5.2 MB on the wire) trips Copilot's request-body cap as a bare HTTP 413 Request Entity Too Large, which classifies to payload_too_large and lands in the compression-only path. Same loop you describe: 413 → compress (no change) → 413 → …, surfaced to the user as a misleading "context window exceeded" even though the conversation is tiny. So this is a provider-agnostic whole-request 413, and your fix (shrink oversized image parts before counting a compression attempt, gated by image_shrink_retry_attempted) covers it cleanly.

One optional suggestion to strengthen the regression coverage. The two tests here mock _try_shrink_image_parts_in_messages to return True, which proves the routing (shrink runs, compression doesn't) but not that the shrinker actually reduces the payload. I wrote a variant that exercises the real Pillow path end-to-end: it builds a genuine >4 MB noise JPEG as a data: URL, lets the real shrinker run, and asserts the image part was actually shrunk in place (len(new_url) < original_len) with no text compression. I ran it against this PR's HEAD and both cases pass (and your existing 22 in test_413_compression.py still pass alongside it).

Happy to drop it in as a suggested addition if useful – either you cherry-pick it or I can push it to your branch / open it as a follow-up, whatever you prefer. Either way, thanks for the fix – this bug's been biting real photo workflows.

[teknium1]

Thanks for identifying a real 413 recovery gap. Current main still sends FailoverReason.payload_too_large directly into compression at agent/conversation_loop.py:3337-3401, so shrinking inline images first remains useful.

Problems

• The proposed call at agent/conversation_loop.py:2868 changes only api_messages. On current main, that list is derived from canonical messages at agent/conversation_loop.py:792-835 and prompt caching deep-copies it at agent/prompt_caching.py:84-99. A recovered retry can therefore succeed while the persisted history still contains the oversized image, causing a later turn to rebuild and resend it.
• The new test mocks the shrinker (tests/run_agent/test_413_compression.py:429-431), so it does not cover canonical persistence, the actual reduced wire payload, or the one-shot guard after a second 413.

Suggested changes

• Salvage this into the current retry loop with _retry.image_shrink_retry_attempted, and mirror successful image replacements into canonical messages before _persist_session() and retry. PR #62005 contains the current-main replacement/persistence pattern.
• Add a detached canonical/API regression that verifies request → repaired persistence → repaired retry ordering and verifies a second 413 falls through to compression.

Automated hermes-sweeper review.

[teknium1]

On current main, api_messages can be a deep-copied cache request while canonical messages remains unchanged (agent/conversation_loop.py:792-899, agent/prompt_caching.py:84-99). Please record and mirror successful URL replacements into canonical history and persist before retrying, or a later turn can resend the original oversized image.