Skip to content

fix(agent): validate and downscale oversized images at ingestion to prevent infinite shrink-recovery loops (#61994) - #62051

Open
webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/61994-image-shrink-ingestion
Open

fix(agent): validate and downscale oversized images at ingestion to prevent infinite shrink-recovery loops (#61994)#62051
webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/61994-image-shrink-ingestion

Conversation

@webtecnica

Copy link
Copy Markdown
Contributor

Summary

Oversized images (e.g. >8000px for Anthropic) trigger image-shrink recovery on EVERY API call because shrunken images are never persisted to session history. 101 recurrences in ~24h observed in one session.

Change

Instead of fixing the retry-recovery path, this fix validates and downscales images at INGESTION time — when they first enter the conversation — so oversized originals never reach session history.

Modified _file_to_data_url() in agent/image_routing.py to proactively check both pixel dimensions and estimated base64 payload size against embed-time caps (7900px max dimension, 4MB max base64 size — headroom under Anthropic's limits). Images exceeding these caps are downscaled once at ingestion.

Verification

147 tests pass across 4 test files (image_routing, image_shrink_recovery, compressor_image_tokens, image_rejection_fallback).

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Jul 10, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #61994 (the issue this fixes) and competing with #62005, which also fixes #61994 but via a different mechanism: #62005 persists the repaired post-shrink parts back to canonical session history, whereas this PR validates and downscales oversized images at ingestion time in _file_to_data_url() so they never reach history. Same goal, different approach — flagging for a maintainer to pick. Both are distinct from the shrink-accept-gate family (open canonical #48035, merged #49140), which decides which downscale is accepted.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the recurrence to immutable history. The persistence gap is real on current main, but this implementation needs a design and correctness adjustment before it is ready.

Problems

  • agent/image_routing.py:453-465 deliberately keeps native attachments at full size until a provider rejects them; _file_to_data_url at agent/image_routing.py:613 receives no provider identity, so this change globally applies Anthropic-oriented quality limits to local attachments.
  • The added truthiness return does not prove the image fits. tools/vision_tools.py:737-742 returns its best candidate even after it cannot meet the requested limit, so an over-cap candidate can still enter history.
  • The PR has no regression test for that non-fitting resize result or for canonical-history recurrence; current native-input coverage documents the opposite full-size/reactive contract at tests/agent/test_image_routing.py:483-500.

Suggested changes

  • Choose between preserving the existing reactive/provider-specific policy and persisting successful repair into canonical history (the competing #62005 approach), versus approving a global ingestion cap.
  • If keeping ingestion resizing, verify both output bytes and decoded dimensions before returning the resized URL, and add regression coverage for failed-to-fit resizing plus the repeated-turn history case.

Automated hermes-sweeper review.

Comment thread agent/image_routing.py
"image_routing: proactively downscaled %s "
"(dimension/byte cap) before embedding into history",
path.name,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_resize_image_for_vision() can return a non-empty best candidate even when it still exceeds the requested cap (tools/vision_tools.py:737-742). Returning it solely because it is truthy can still bake an oversized image into history; verify output bytes and decoded dimensions before accepting it.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants