Skip to content

fix(agent): drop the api_content sidecar when stripping images from history - #68811

Open
Frowtek wants to merge 1 commit into
NousResearch:mainfrom
Frowtek:fix/image-strip-stale-api-content
Open

fix(agent): drop the api_content sidecar when stripping images from history#68811
Frowtek wants to merge 1 commit into
NousResearch:mainfrom
Frowtek:fix/image-strip-stale-api-content

Conversation

@Frowtek

@Frowtek Frowtek commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

api_content is the byte-stability sidecar from #67274: it holds the exact
bytes previously sent for a message, and every turn substitutes it back into
content when building api_messages. drop_stale_api_content exists so a
content rewrite cannot be replayed from it — its own docstring states the
contract, and names the historical image strip as one of the callers:

Replaying the pre-rewrite sidecar would resend exactly what the rewrite
removed, so it must be dropped — the cost is one cache boundary miss,
never wrong content.

_strip_images_from_messages never drops it. The image-rejection recovery in
conversation_loop runs it over the persistent history, not just the
per-call copy:

agent._vision_supported = False
_imgs_removed = _strip_images_from_messages(messages)      # history
if isinstance(api_messages, list):
    _strip_images_from_messages(api_messages)

and api_messages are copies (api_msg = msg.copy()), so the history message
keeps its sidecar. The strip is undone on the very next turn.

Reproduction

With the real functions:

history content after strip : [{'type': 'text', 'text': 'look'}]
sidecar still present       : True
NEXT TURN sends             : 'look<IMAGE BYTES SENT LAST TURN>'

Why this is session-fatal, not a one-turn glitch

The recovery cannot fire again: it is gated on
getattr(agent, "_vision_supported", True) and just set that False. So on
every subsequent turn the sidecar re-injects the images, the text-only endpoint
rejects them again, and the branch that would strip them is disabled — the
session stays wedged on a 4xx it already knew how to fix.

Fix

Drop the sidecar on each message the strip rewrites, inside the function so
every caller is covered. Messages with no images keep theirs, so only the
rewritten message pays a cache boundary — the tradeoff the invariant prescribes.

The two sibling recovery paths, _sanitize_messages_surrogates and
_sanitize_messages_non_ascii, are already safe: both walk every string field
on the message and so scrub the sidecar in passing. This one only touches
content. (The sibling rewrite in replay_cleanup does it explicitly, with the
same rationale spelled out.)

Test plan

tests/run_agent/test_image_rejection_fallback.py — new
TestStripImagesDropsStaleApiContent:

  • the rewritten message loses its sidecar
  • the next turn does not resend the stripped images
  • the tool-placeholder rewrite is covered too
  • untouched messages keep their sidecar

All four fail on main. 436 passed in test_run_agent.py; 53 passed
across the image-rejection + api_content-sidecar suites; 307 passed across the
sanitization/image/sidecar/replay agent tests. Remaining failures
(test_image_routing.py, test_save_url_image.py, TestPathCanonicalization)
are pre-existing — verified by re-running them against the pre-fix code.

Checklist

  • Tested on Ubuntu 24.04
  • New tests fail without the fix and pass with it
  • No regressions (every remaining failure reproduced on pre-fix code)
  • Honors the drop_stale_api_content contract the other rewrite paths follow

…istory

`api_content` is the byte-stability sidecar from NousResearch#67274: it holds the exact
bytes previously sent for a message, and every turn substitutes it back into
`content` when building `api_messages`. `drop_stale_api_content` exists so a
content rewrite cannot be replayed from it — its own docstring states the
contract, and names the historical image strip as one of the callers:

    Replaying the pre-rewrite sidecar would resend exactly what the rewrite
    removed, so it must be dropped — the cost is one cache boundary miss,
    never wrong content.

`_strip_images_from_messages` never drops it. The image-rejection recovery in
`conversation_loop` runs it over the persistent history, not just the per-call
copy:

    agent._vision_supported = False
    _imgs_removed = _strip_images_from_messages(messages)      # history
    if isinstance(api_messages, list):
        _strip_images_from_messages(api_messages)

and `api_messages` are copies (`api_msg = msg.copy()`), so the history message
keeps its sidecar. The strip is therefore undone on the very next turn.

Reproduced with the real functions:

    history content after strip : [{'type': 'text', 'text': 'look'}]
    sidecar still present       : True
    NEXT TURN sends             : 'look<IMAGE BYTES SENT LAST TURN>'

This is worse than a one-turn glitch, because the recovery cannot fire again:
it is gated on `getattr(agent, "_vision_supported", True)` and just set that
False. So on every subsequent turn the sidecar re-injects the images, the
text-only endpoint rejects them again, and the branch that would strip them is
disabled — the session stays wedged on a 4xx it already knew how to fix.

Drop the sidecar on each message the strip rewrites, inside the function so
every caller is covered. Messages with no images keep theirs, so only the
rewritten message pays a cache boundary — the tradeoff the invariant
prescribes. The two sibling recovery paths, `_sanitize_messages_surrogates`
and `_sanitize_messages_non_ascii`, are already safe: both walk every string
field on the message and so scrub the sidecar in passing. This one only
touches `content`.

tests/run_agent/test_image_rejection_fallback.py: new
TestStripImagesDropsStaleApiContent — the rewritten message loses its sidecar,
the next turn does not resend the stripped images, the tool-placeholder rewrite
is covered too, and untouched messages keep their sidecar. All four fail on
main. 53 passed across the image-rejection and api_content-sidecar suites; 307
passed across the sanitization/image/sidecar/replay agent tests (8 failures in
test_image_routing.py / test_save_url_image.py are pre-existing and fail
identically on clean main).
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Jul 21, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused sidecar-invalidation hardening.

Problems

  • The added tests construct api_content on a multimodal message, but current normal sidecar production does not establish that state: compose_user_api_content() returns None for non-string content (agent/turn_context.py:73-84), and persistence retains api_content only as a string (hermes_state.py:5270). _strip_images_from_messages() only rewrites list content (agent/message_sanitization.py:410-429). The tests therefore do not demonstrate the claimed production retry wedge.

Suggested changes

  • Please reproduce through a real producer plus the image-rejection recovery path, or identify the supported ingress that creates this sidecar/content combination. If this is defensive hardening for externally supplied history, scope the test and rationale to that invariant instead.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants