Skip to content

fix(context): preserve multimodal content during compression - #7857

Closed
qWaitCrypto wants to merge 1 commit into
NousResearch:mainfrom
qWaitCrypto:fix-context-compressor-multimodal-content-clean
Closed

qWaitCrypto wants to merge 1 commit into
NousResearch:mainfrom
qWaitCrypto:fix-context-compressor-multimodal-content-clean

Conversation

@qWaitCrypto

@qWaitCrypto qWaitCrypto commented Apr 11, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes a ContextCompressor bug when conversation history contains structured or multimodal content (e.g. list-valued message content with text/image blocks).

Before this change, ContextCompressor assumed msg["content"] was always a string in several compression paths. That caused two problems:

  1. Summary serialization emitted Python reprs for multimodal content instead of readable text, degrading summary quality
  2. merge-summary-into-tail raised TypeError: can only concatenate str (not "list") to str when the first tail message had list-valued content

This patch adds a _coerce_content_to_text() helper that normalizes structured content to readable text where text is needed, while preserving multimodal list structure when merging a summary into the tail message.

Manually verified against real multimodal request shapes from OpenAI-style chat completions, OpenAI-style responses, and an Anthropic-compatible messages endpoint.

Related Issue

No existing issue — discovered during code review of the compression paths.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Added _coerce_content_to_text() in agent/context_compressor.py to normalize structured / multimodal message content into readable text for compression logic.
  • Updated compression paths in agent/context_compressor.py to use normalized text for summary serialization, tail token estimation, old tool-result pruning, and system-note appending.
  • Fixed merge-summary-into-tail in agent/context_compressor.py so list-valued multimodal content keeps its original structure by prepending a new text block instead of flattening the whole message into a string.
  • Added a compatibility fallback for input_image parts that use url instead of image_url.
  • Added regression coverage in tests/agent/test_context_compressor.py for:
    • readable multimodal serialization (including input_image.url fallback)
    • pruning non-string tool content
    • merge-summary-into-tail with multimodal content while preserving structure

How to Test

python3 -m pytest tests/agent/test_context_compressor.py -q
41 passed, 41 warnings in 20.94s

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 11 (WSL2 Ubuntu)

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for the thorough write-up and the tests! The bugs you identified are real — but the same fix landed on main independently before this PR could be merged.

Automated hermes-sweeper review — closing as implemented on main.

  • The two helpers that address these crashes — _content_text_for_contains() and _append_text_to_content() — were added in commit 1e8254e59 (fix(agent): guard context compressor against structured message content).
  • That commit fixes both the system-note injection path (agent/context_compressor.py ~line 1218) and the summary-merge-into-tail path (~line 1235) using the same safe-prepend/append approach you proposed.
  • Regression tests for both crash paths were also added in the same commit to tests/agent/test_context_compressor.py.
  • PR fix(agent): handle structured and multimodal content in ContextCompressor #14143, which you cross-referenced, was the salvaged version of this fix — it was closed but its commit is reachable from main.

If you find any remaining edge cases not covered by the current implementation, feel free to open a new focused issue or PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants