fix(agent): validate and downscale oversized images at ingestion to prevent infinite shrink-recovery loops (#61994) - #62051
Conversation
…revent infinite shrink-recovery loops (NousResearch#61994)
Related to #61994 (the issue this fixes) and competing with #62005, which also fixes #61994 but via a different mechanism: #62005 persists the repaired post-shrink parts back to canonical session history, whereas this PR validates and downscales oversized images at ingestion time in |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for tracing the recurrence to immutable history. The persistence gap is real on current main, but this implementation needs a design and correctness adjustment before it is ready.
Problems
agent/image_routing.py:453-465deliberately keeps native attachments at full size until a provider rejects them;_file_to_data_urlatagent/image_routing.py:613receives no provider identity, so this change globally applies Anthropic-oriented quality limits to local attachments.- The added truthiness return does not prove the image fits.
tools/vision_tools.py:737-742returns its best candidate even after it cannot meet the requested limit, so an over-cap candidate can still enter history. - The PR has no regression test for that non-fitting resize result or for canonical-history recurrence; current native-input coverage documents the opposite full-size/reactive contract at
tests/agent/test_image_routing.py:483-500.
Suggested changes
- Choose between preserving the existing reactive/provider-specific policy and persisting successful repair into canonical history (the competing #62005 approach), versus approving a global ingestion cap.
- If keeping ingestion resizing, verify both output bytes and decoded dimensions before returning the resized URL, and add regression coverage for failed-to-fit resizing plus the repeated-turn history case.
Automated hermes-sweeper review.
| "image_routing: proactively downscaled %s " | ||
| "(dimension/byte cap) before embedding into history", | ||
| path.name, | ||
| ) |
There was a problem hiding this comment.
_resize_image_for_vision() can return a non-empty best candidate even when it still exceeds the requested cap (tools/vision_tools.py:737-742). Returning it solely because it is truthy can still bake an oversized image into history; verify output bytes and decoded dimensions before accepting it.
Summary
Oversized images (e.g. >8000px for Anthropic) trigger image-shrink recovery on EVERY API call because shrunken images are never persisted to session history. 101 recurrences in ~24h observed in one session.
Change
Instead of fixing the retry-recovery path, this fix validates and downscales images at INGESTION time — when they first enter the conversation — so oversized originals never reach session history.
Modified
_file_to_data_url()inagent/image_routing.pyto proactively check both pixel dimensions and estimated base64 payload size against embed-time caps (7900px max dimension, 4MB max base64 size — headroom under Anthropic's limits). Images exceeding these caps are downscaled once at ingestion.Verification
147 tests pass across 4 test files (image_routing, image_shrink_recovery, compressor_image_tokens, image_rejection_fallback).