Skip to content

fix(moa): preserve prompt text of content-block messages in reference view - #61426

Closed
jrfbch wants to merge 1 commit into
NousResearch:mainfrom
jrfbch:fix/moa-content-blocks-text
Closed

fix(moa): preserve prompt text of content-block messages in reference view#61426
jrfbch wants to merge 1 commit into
NousResearch:mainfrom
jrfbch:fix/moa-content-blocks-text

Conversation

@jrfbch

@jrfbch jrfbch commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Problem

_reference_messages (agent/moa_loop.py) builds the advisory view MoA sends to
each reference model. It flattens every message's content with:

text = content if isinstance(content, str) else ""

When a user turn's content is a list of content blocks — the standard
OpenAI/Anthropic multimodal shape produced whenever an image/file is attached,
or any multi-part turn — this drops the prompt text to an empty string.

Consequences:

  • Lax providers (deepseek, MiniMax) tolerate the empty turn, but the reference
    still loses the actual instruction, silently degrading its advice.
  • Strict providers reject an empty user turn outright. Z.AI GLM returns
    400 ... 未正常接收到prompt参数 ("prompt param not received"), so that
    reference fails every turn.

Either way this contradicts the function's own documented contract that
"no context is lost" and the reference "still has the full picture".

Fix

Add _flatten_content_text() which extracts and joins the text of each block
and drops non-text blocks (images, etc.), handling str / list-of-blocks /
None. Apply it at the main render path and at the degenerate-case fallback
(same bug class). This mirrors the text-extraction pattern already used in
context_compressor and the provider adapters.

Tests

Adds two regression tests to tests/run_agent/test_moa_loop_mode.py:

  • test_reference_messages_flattens_content_blocks_to_text — a content-block
    user turn preserves its text and drops the image block.
  • test_flatten_content_text_shapes — helper handles str / block list /
    image-only / None.

Full tests/run_agent/test_moa_loop_mode.py passes (29/29) against current main.

Repro

With a MoA preset using a Z.AI GLM reference model, any first-turn CLI query
(hermes chat -q '...', whose content is content-block form) makes the GLM
reference fail with 400 未正常接收到prompt参数, while deepseek/MiniMax
references "succeed" on a blanked prompt. After the fix all references receive
the real prompt text.

… view

`_reference_messages` flattened every message's `content` with
`content if isinstance(content, str) else ""`, so a user turn whose content
is a list of blocks (the OpenAI/Anthropic multimodal shape produced whenever
an image/file is attached, or any multi-part turn) had its text silently
dropped to an empty string. Lax providers (deepseek, MiniMax) tolerated the
resulting empty turn but the reference still lost the actual instruction;
strict providers (Z.AI GLM) reject it outright with HTTP 400. Either way this
contradicts the function's own documented contract that 'no context is lost'
and the reference 'still has the full picture'.

Add `_flatten_content_text()` which extracts and joins the text blocks and
drops non-text blocks (images, etc.), handling str / list-of-blocks / None.
Apply it at the main render path and the degenerate-case fallback (same bug
class). Mirrors the text-extraction pattern already used in context_compressor
and the provider adapters.

Adds regression tests for the content-block flattening and the helper's shapes.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Jul 9, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing this to the advisory transcript. The premise is confirmed on current main: agent/moa_loop.py:473 drops all non-string content before agent/moa_loop.py:480 emits the user turn.

Problems

  • The new helper duplicates agent.message_content.flatten_message_text (agent/message_content.py:34-49), which already supports the common Chat/Responses text fields and object parts while excluding media blocks. Reusing it avoids two normalizers drifting apart.
  • The new image-only test locks in an empty result. That still reaches _reference_messages as an empty user turn (agent/moa_loop.py:477-480), so the strict-reference empty-prompt case remains for image-only requests.

Suggested changes

  • Use flatten_message_text in both extraction paths and keep the MoA regression coverage.
  • Make the image-only advisory representation non-empty (or omit the empty turn) and test that strict-safe transcript shape.

Automated hermes-sweeper review.

Comment thread agent/moa_loop.py
return "\n".join(lines)


def _flatten_content_text(content: Any) -> str:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please reuse agent.message_content.flatten_message_text here rather than adding a second content normalizer. The existing helper already covers Chat/Responses text fields, mapping/object parts, and media filtering; this narrower copy will drift from those supported shapes.

Comment thread agent/moa_loop.py
parts.append(text)
elif isinstance(block, str) and block:
parts.append(block)
return "\n".join(parts)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For an image-only user message this returns "", and _reference_messages will still send an empty user turn to the reference. That preserves the strict-provider empty-prompt failure described by this PR for that valid input shape; use a non-empty advisory placeholder or omit the empty turn.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 11, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks @jrfbch — you had the correct root-cause diagnosis and fix direction 5 days before anyone else, and your analysis (str-only read flattening content-block turns to "", strict providers 400ing while lax ones silently lose the instruction) was exactly right.

This landed on main via #64319 (commit 8582f35), which carries you as Co-authored-by. That PR covers the same _reference_messages flatten (reusing the shared agent/message_content.flatten_message_text helper instead of a local one) plus the sibling sites: the cache-decorated-turn case (Anthropic cache_control decoration converts string content to part lists before MoA runs — the trigger for a live incident on Jul 14), _attach_reference_guidance for decorated/multimodal trailing user turns, the conversation_loop MoA context injection, and an image-only-turn placeholder. Live-verified on the wire with advisor + aggregator prompt caching intact.

Closing as landed-with-credit. Appreciate the contribution.

@teknium1 teknium1 closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants