Skip to content

fix(memory): flatten multimodal content before provider sync - #44517

Closed
erosika wants to merge 1 commit into
NousResearch:mainfrom
erosika:fix/memory-sync-multimodal-content
Closed

fix(memory): flatten multimodal content before provider sync#44517
erosika wants to merge 1 commit into
NousResearch:mainfrom
erosika:fix/memory-sync-multimodal-content

Conversation

@erosika

@erosika erosika commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Problem

Turns with an attached image log this and silently never reach the memory backend:

WARNING agent.memory_manager: Memory provider 'honcho' sync_turn failed: expected string or bytes-like object, got 'list'

Multimodal user messages carry content as a list of typed parts ({type: "text"|"image_url", ...}). _sync_external_memory_for_turn passed that list straight into MemoryManager.sync_all, and providers feed it to regexes — Honcho's sync_turn calls sanitize_context in its synchronous prologue, where re.sub raises the TypeError before the sync thread ever spawns.

The chain:

  1. agent/turn_context.pyoriginal_user_message is the raw content (Any), a list for multimodal turns
  2. agent/turn_finalizer.pyrun_agent.py: _sync_external_memory_for_turn — no string guard (other consumers of original_user_message in turn_context.py do guard with isinstance(..., str))
  3. plugins/memory/honcho/__init__.py: sync_turnsanitize_context(user_content or ""): a non-empty list is truthy, so the list hits re.sub

Other plugins (mem0, hindsight, supermemory, …) would choke on a list the same way.

Fix

Flatten content to plain text at the boundary in _sync_external_memory_for_turn, before sync_all/queue_prefetch_all — one fix covers every provider instead of patching each plugin. New helper flatten_message_content in agent/memory_manager.py (alongside sanitize_context, its main consumer):

  • strings pass through, None""
  • text parts joined with newlines
  • images become a [N image(s)] marker so the attachment isn't erased from recall
  • turns with no recoverable text (e.g. unknown part types only) are skipped rather than synced empty

Both original_user_message and final_response are guarded — both are typed Any.

Tests

  • tests/agent/test_memory_provider.py — unit coverage for flatten_message_content, including a regression case asserting the flattened output survives sanitize_context
  • tests/run_agent/test_memory_sync_interrupted.py — boundary coverage: multimodal user message, multimodal response, and no-recoverable-text skip

All 141 tests across the touched suites pass.

Multimodal turns carry message content as a list of typed parts
({type: "text"|"image_url", ...}). _sync_external_memory_for_turn
passed that list straight into MemoryManager.sync_all, and providers
feed it to regexes — Honcho's sync_turn calls sanitize_context, where
re.sub raised 'expected string or bytes-like object, got list'. Every
turn with an attached image silently never synced.

Flatten to plain text at the boundary: text parts joined, images noted
as an [N image(s)] marker so the attachment isn't erased from recall.
Fixing here covers all providers instead of patching each plugin.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/memory Memory tool and memory providers P3 Low — cosmetic, nice to have labels Jun 11, 2026
@liuhao1024

Copy link
Copy Markdown
Contributor

Verified flatten_message_content — the multimodal content→string boundary is correct:

  1. Type dispatch: None→"", str→passthrough, list→joined text + image markers, fallback str(content).
  2. Image counting: pluralization handled ([1 image] vs [2 images]), image-only messages produce marker-only output.
  3. Guard in caller: _sync_external_memory_for_turn checks if not (user_text and response_text): return — skips sync when flattening yields empty strings, preventing vacuous memory writes.
  4. Content type coverage: text, input_text, output_text, image_url, input_image all handled; unknown types silently skipped.

Test suite covers all branches including the original crash case (sanitize_context(list)). CI green. Clean fix.

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Thanks for this @erosika — the diagnosis is exactly right: multimodal content reaches the provider sync as a list and crashes sanitize_context's re.sub before the sync thread spawns, and flattening at the _sync_external_memory_for_turn boundary is the right place to fix it once for every provider.

I've opened #44738 as a salvage that cherry-picks your commit verbatim (authorship preserved) and adds one follow-up: the new flatten_message_content turned out to be a near-duplicate of the existing agent/codex_responses_adapter.py:_summarize_user_message_for_log (same dispatch, same part-type sets, identical [N image(s)] marker), which is already imported into turn_finalizer.py — the same file your call site lives in. The follow-up parameterizes that helper with a sep arg (sep="\n" for memory, default sep=" " keeps the logging callers unchanged) and drops the fork, so there's a single source of truth.

Closing this in favor of #44738 — full credit to you for the fix. Happy to fold in any feedback there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have tool/memory Memory tool and memory providers type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants