Skip to content

fix(memory): flatten multimodal content before provider sync - #44738

Merged
kshitijk4poor merged 2 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/memory-sync-multimodal-content
Jun 12, 2026
Merged

fix(memory): flatten multimodal content before provider sync#44738
kshitijk4poor merged 2 commits into
NousResearch:mainfrom
kshitijk4poor:salvage/memory-sync-multimodal-content

Conversation

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Salvage of #44517 by @erosika — original fix cherry-picked verbatim (authorship preserved), plus one follow-up commit that dedupes the helper. Based on current main.

Problem (commit 1, @erosika)

Turns with an attached image log this and silently never reach the memory backend:

WARNING agent.memory_manager: Memory provider 'honcho' sync_turn failed: expected string or bytes-like object, got 'list'

Multimodal user messages carry content as a list of typed parts ({type: "text"|"image_url", ...}). _sync_external_memory_for_turn passed that list straight into MemoryManager.sync_all, and providers feed it to regexes — Honcho's sync_turn calls sanitize_context in its synchronous prologue, where re.sub raises the TypeError before the sync thread spawns (caught by sync_all, surfaced as the warning above). mem0, supermemory, and the other providers expect a string the same way.

The fix flattens content to plain text at the boundary in _sync_external_memory_for_turn, before sync_all/queue_prefetch_all — one fix covers every provider instead of patching each plugin:

  • strings pass through, None""
  • text parts joined, images become a [N image(s)] marker so the attachment isn't erased from recall
  • turns with no recoverable text are skipped rather than synced empty

Both original_user_message and final_response are guarded — both are typed Any.

Follow-up (commit 2): reuse the existing helper instead of forking it

The original commit added flatten_message_content to agent/memory_manager.py, but it was a near-exact duplicate of agent/codex_responses_adapter.py:_summarize_user_message_for_log — same None/str/list dispatch, same {text,input_text,output_text} / {image_url,input_image} part sets, the identical [N image(s)] marker, and the same scalar fallback. The only difference was the join separator (newline for memory vs the space-join the existing helper already uses for log/trajectory previews), and that helper is already imported into agent/turn_finalizer.py — the same file whose call site the memory fix touches.

This commit parameterizes the existing helper with sep=" " (default preserves every current logging/trajectory caller byte-for-byte) and calls it with sep="\n" at the memory boundary; the forked flatten_message_content is dropped. Single source of truth for multimodal-content flattening, no behavior change for the fix or for existing callers.

Tests

  • tests/agent/test_memory_provider.py — unit coverage for the consolidated helper (newline-join for memory, plus a case locking the default space-join), including a regression case asserting the flattened output survives sanitize_context
  • tests/run_agent/test_memory_sync_interrupted.py — boundary coverage: multimodal user message, multimodal response, and no-recoverable-text skip

All touched suites pass locally (110 tests), ruff clean. Verified end-to-end on the branch: a multimodal list flattens to [1 image] <text> and survives sanitize_context; a raw list still crashes it (the bug); logging callers keep the space-join.

Salvage of #44517.

erosika and others added 2 commits June 12, 2026 12:46
Multimodal turns carry message content as a list of typed parts
({type: "text"|"image_url", ...}). _sync_external_memory_for_turn
passed that list straight into MemoryManager.sync_all, and providers
feed it to regexes — Honcho's sync_turn calls sanitize_context, where
re.sub raised 'expected string or bytes-like object, got list'. Every
turn with an attached image silently never synced.

Flatten to plain text at the boundary: text parts joined, images noted
as an [N image(s)] marker so the attachment isn't erased from recall.
Fixing here covers all providers instead of patching each plugin.

(cherry picked from commit 705bdb6)
…rking it

The original fix added agent/memory_manager.py:flatten_message_content, but
that helper was a near-exact duplicate of
agent/codex_responses_adapter.py:_summarize_user_message_for_log — same
None/str/list dispatch, same {text,input_text,output_text}/{image_url,input_image}
part sets, the identical [N image(s)] marker, and the same str() fallback. The
only difference was the join separator (newline for memory vs space for the
log/trajectory previews the existing helper already serves), and that helper is
already imported into agent/turn_finalizer.py — the same file whose call site the
memory fix touches.

Parameterize the existing helper with sep=' ' (default preserves every current
logging/trajectory caller byte-for-byte) and call it with sep='\n' at the memory
boundary; drop the forked flatten_message_content. Repoints the unit tests to the
consolidated helper and adds a case locking the default space-join.

Single source of truth for multimodal-content flattening; no behavior change for
the fix or for existing callers.
@kshitijk4poor
kshitijk4poor merged commit 046f444 into NousResearch:main Jun 12, 2026
28 checks passed
AIalliAI pushed a commit to AIalliAI/Hermes that referenced this pull request Jun 14, 2026
…ry-sync-multimodal-content

fix(memory): flatten multimodal content before provider sync
T02200059 pushed a commit to T02200059/hermes-agent that referenced this pull request Jun 18, 2026
…ry-sync-multimodal-content

fix(memory): flatten multimodal content before provider sync
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…ry-sync-multimodal-content

fix(memory): flatten multimodal content before provider sync
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
…ry-sync-multimodal-content

fix(memory): flatten multimodal content before provider sync
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…ry-sync-multimodal-content

fix(memory): flatten multimodal content before provider sync
mgcstudios86 pushed a commit to mgcstudios/hermes-agent that referenced this pull request Jul 20, 2026
Per Bryntly's review (PR NousResearch#68072):
- Use `Any` type hint instead of bare `text` (preserves type info)
- Replace broad `except Exception` with `logger.warning(...)` so
  failures are observable instead of silently swallowed
- Add 8 unit tests covering: string passthrough, empty, None,
  list of text parts, list with fence tags, list with image,
  scalar fallback, and a direct guard against the original
  TypeError-on-list regression

Per alt-glitch comment (PR NousResearch#68072):
- Docstring now references PR NousResearch#44738 (boundary normalization
  already merged in upstream) and frames this PR as a defensive
  guard for callers that forget the boundary normalization
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…ry-sync-multimodal-content

fix(memory): flatten multimodal content before provider sync
@kshitijk4poor
kshitijk4poor deleted the salvage/memory-sync-multimodal-content branch August 5, 2026 07:09
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…ry-sync-multimodal-content

fix(memory): flatten multimodal content before provider sync
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants