Skip to content

fix: handle multimodal content in context compression summarization - #917

Closed
teknium1 wants to merge 1 commit into
mainfrom
fix/multimodal-compress-content
Closed

fix: handle multimodal content in context compression summarization#917
teknium1 wants to merge 1 commit into
mainfrom
fix/multimodal-compress-content

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

Fixes a bug where multimodal user messages (containing images, audio, etc.) would produce garbled output during context compression summarization.

The bug

In _generate_summary(), message content was accessed as:

content = msg.get("content") or ""

When content is a multimodal list like:

[
    {"type": "text", "text": "What is in this image?"},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
]

The `or """ fallback doesn't trigger (non-empty lists are truthy), and then:

  • len(content) returns 2 (list length) instead of character count
  • content[:1000] slices list items instead of characters
  • The resulting summary prompt contains raw dict representations instead of readable text

The fix

New _content_to_text() static method on ContextCompressor

Safely converts any content format to plain text:

Input type Output
str Returned as-is
None Empty string
list (multimodal) Text parts joined with newlines; image_url[image]; other types → [type_name]
dict / other JSON serialization, str() fallback

Changed line in _generate_summary()

# Before:
content = msg.get("content") or ""

# After:
content = self._content_to_text(msg.get("content"))

Tests added

  • TestContentToText class with 6 test cases:
    • String passthrough
    • None → empty string
    • Multimodal text + image parts
    • Mixed content types (text, audio, etc.)
    • Dict content JSON serialization
    • End-to-end: multimodal messages through _generate_summary() — verifies the summarization prompt contains "What is in this image?" and "[image]" instead of raw dicts

What does NOT change

  • The compression algorithm, role alternation, or trigger logic
  • How string content is handled (identical behavior for the common case)
  • The content=None handling that already worked via the existing TestGenerateSummaryNoneContent tests

Inspired by PR #776 by @kshitijk4poor.

The _generate_summary() method assumed message content is always a
string (msg.get('content') or ''). When content is a multimodal list
(e.g. [{type: 'text', text: '...'}, {type: 'image_url', ...}]), this
produced mangled output: len() returned the list length instead of
character count, and slicing produced list items instead of substrings.

Add _content_to_text() helper that safely converts any content format
to plain text:
- str → returned as-is
- None → empty string
- list (multimodal) → text parts joined, images replaced with [image]
- dict/other → JSON serialization with str() fallback

This ensures multimodal conversations compress correctly instead of
producing garbled summaries.

Inspired by PR #776 by @kshitijk4poor.
@teknium1

Copy link
Copy Markdown
Contributor Author

Thanks for the PR. I looked into this, but it doesn’t correspond to a real Hermes message-flow path today. We don’t keep raw multimodal image blocks in the conversation history that context compression summarizes; image handling goes through the vision tool, and the resulting text description is what gets stored in chat history. Because of that, this edge case isn’t currently actionable in the live architecture, so I’m closing this for now. If we later add a path that stores raw multimodal content directly in chat history, we can revisit it then.

@teknium1 teknium1 closed this Mar 14, 2026
kiddhu added a commit to kiddhu/hermes-agent that referenced this pull request Aug 21, 2026
fix(kanban): pin PR NousResearch#917 proof-module authority / 固定 PR NousResearch#917 证明模块权威
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant