Skip to content

fix(agent): stop old user images from bloating every model request - #87555

Open
fangliquanflq wants to merge 1 commit into
NousResearch:mainfrom
fangliquanflq:fix/agent-history-user-image-eviction
Open

fix(agent): stop old user images from bloating every model request#87555
fangliquanflq wants to merge 1 commit into
NousResearch:mainfrom
fangliquanflq:fix/agent-history-user-image-eviction

Conversation

@fangliquanflq

Copy link
Copy Markdown
Contributor

What does this PR do?

Historical user-uploaded images can remain in every provider request indefinitely when a session stays below the compression threshold. A reported three-image turn added about 2.9 MB to every request, producing about 58 MB of redundant daily upload at 20 calls per day. This PR bounds older user-image payloads at request time while preserving the current user turn and the original persisted session history.

Symptom

After a user uploads images and continues the conversation, the original base64 image data is sent again with every later model request. Large-context models may never trigger compression, so the repeated payload has no practical eviction point.

Impact

Users with image-bearing sessions can incur repeated multi-megabyte request bodies, additional upload latency, and unnecessary provider traffic on every later turn. The reported production example retained 4,047,677 base64 characters from one three-image message for 19 days.

Bug Cause

Trigger: agent/conversation_loop.py:2281 / outbound request assembly with historical user image parts

Causal chain:

  1. A user uploads one or more images, then sends later follow-up messages.
  2. Request assembly copies the persisted multimodal user turn into each outbound request without a request-time user-image limit.
  3. The same base64 payload is uploaded on every request until compression eventually strips it, which may not happen on large-context models.

Why it is wrong: Historical user images had no bounded request-time lifecycle independent of compression. The compressor's historical-media pass only runs when compression is triggered and deliberately preserves the newest image-bearing user turn as its anchor.

Working sibling / contrast: Non-vision models strip images for compatibility, but vision-capable models retain them. Tool-result screenshot pruning is a separate role-specific path and does not cap user uploads.

Ruled out: Token-threshold tuning does not solve the request payload lifecycle because large-window sessions can remain below the compression threshold indefinitely, and the newest image-bearing anchor remains preserved when compression does run.

Fix

Add a request-only newest-first cap for image parts in older user turns. The newest user turn is always preserved in full, evicted parts become stable text placeholders, and copy-on-write handling keeps persisted history unchanged. Apply the same policy to the main conversation loop and the max-iterations summary request before prompt-cache planning. Expose agent.max_history_user_images with a default of 3 and allow 0 to preserve the previous behavior.

Related Issue

Closes #87513

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • Documentation update
  • Tests (adding or improving test coverage)

Changes Made

  • agent/context_compressor.py - add copy-on-write historical user-image eviction with provider-compatible placeholders.
  • agent/conversation_loop.py - apply the cap before prompt-cache planning and provider dispatch.
  • agent/chat_completion_helpers.py - apply the same cap to max-iterations summary requests.
  • agent/agent_init.py and hermes_cli/config_defaults.py - load and default the new agent setting.
  • cli-config.yaml.example and website/docs/user-guide/configuration.md - document behavior and disable semantics.
  • tests/agent/test_history_user_image_eviction.py and tests/run_agent/test_history_user_image_eviction.py - cover retention order, current-turn preservation, provider shapes, request-path parity, and history immutability.

How to Test

  1. Configure agent.max_history_user_images: 1.
  2. Send a user turn with two images, then send a text-only follow-up.
  3. Verify the outbound provider request retains only the newest historical image, the older image becomes a placeholder, and the persisted history still contains both originals.
  4. Run the focused request and transformation coverage:
scripts/run_tests.sh tests/agent/test_history_user_image_eviction.py tests/run_agent/test_history_user_image_eviction.py

Result: 8 tests passed. The broader related suite also passed 117 tests across 7 files on the reviewed commit.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run the repository test entry on the relevant tests and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 11

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings)
  • I've updated cli-config.yaml.example because this PR adds a config key
  • N/A - no contributor workflow or architecture guide changes are required
  • I've considered cross-platform impact per the compatibility guide; the transformation is platform-independent
  • N/A - no tool descriptions or schemas changed

Screenshots / Logs

2 files, 8 tests passed, 0 failed
Related suite: 7 files, 117 tests passed, 0 failed

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation area/config Config system, migrations, profiles P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 16, 2026
@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference, author can ignore or act on any point.

fix(agent): stop old user images from bloating every model request

  1. Placeholder type mapping may produce invalid parts. Only input_image parts become input_text placeholders; any other part type in _IMAGE_PART_TYPES (e.g. "image") is replaced with a plain text part (agent/context_compressor.py _evict_historical_user_images). If "image"-typed parts appear in a content format where bare text parts aren't valid, the placeholder itself fails the provider's content schema — the exact failure class this feature is meant to avoid. Confirm which formats emit "image" parts, or map placeholder type per-format.
  2. Mid-turn re-transmission isn't bounded. Eviction only engages when the newest user turn isn't at index 0; during a long tool-calling run the original user images are re-sent on every iteration request until the next user turn. That's consistent with "current turn protected," but if the goal is payload bounding for long runs, note this explicitly in the docs/config comment so it isn't mistaken for a hard cap.
  3. Cache interaction is handled well — the eviction runs before cache planning and is deterministic for a given history, so the evicted prefix stays byte-stable across requests. Good.
  4. Config plumbing is completeagent_init clamps invalid values to 3, and the key is documented in cli-config.yaml.example, config_defaults.py, and the website. Just confirm the gateway's config loader also surfaces agent.max_history_user_images (the three-loader split in AGENTS.md), so a CLI-set value is honored by gateway runs too.

No blocking issues; (1) is the one to verify against the actual "image" part formats.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

User-message images have no request-time eviction — one 3-image message ships 2.9 MB in every request forever on large-window models

3 participants