feat(moa): stream the aggregator response to the user - #55625
Merged
Conversation
MoA sessions could not stream: the gateway streaming toggle was a no-op for
provider "moa", so users saw nothing until the entire response finished — minutes
of silence on long turns. The aggregator's reply was always fetched whole.
Root cause was twofold:
1. conversation_loop hard-disabled streaming for provider in {"copilot-acp",
"moa"} (MoA grouped with the ACP client, whose facade isn't a stream).
2. MoAChatCompletions.create() fetched the aggregator response whole via
call_llm(), which had no streaming mode.
For provider "moa", _create_request_openai_client() returns the MoAClient facade
itself, so the existing streaming consumer already calls
MoAChatCompletions.create(stream=True). We reuse that battle-tested consumer
(text-delta delivery, tool_call reassembly, stale-stream detection, non-streaming
fallback) instead of adding a parallel streaming path.
Changes:
- call_llm() gains stream/stream_options. When streaming it returns the raw SDK
stream iterator directly, bypassing _validate_llm_response and the
temperature/max_tokens/payment fallback chain (which assume a complete
response). The caller owns reassembly and fallback.
- MoAChatCompletions.create() runs the references first (unchanged), then when
stream=True returns the aggregator's raw stream, forwarding stream_options and
the consumer's per-request read timeout. stream=False is byte-identical to
before (no stream/stream_options/timeout forwarded).
- conversation_loop streams MoA only when a display/TTS consumer is present;
quiet/subagent/health-check paths keep the complete-response path.
Tests: tests/run_agent/test_moa_streaming.py — create() stream/non-stream
branches, stream_options + timeout forwarding, call_llm raw-stream return vs
validated non-stream. Existing MoA tests unchanged (20 passed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
alt-glitch
added
type/feature
Collaborator
Salvage of #53848 (by @lEWFkRAD) — same fix (MoA aggregator streaming) rebased with authorship credited via the |
tonydwb
reviewed
Jun 30, 2026
tonydwb
left a comment
tonydwb
left a comment
There was a problem hiding this comment.
Code Review Summary
Verdict: LGTM
Clean streaming support for the MoA aggregator. When the agent has a display/TTS consumer, the aggregator's raw token stream is returned directly; without consumers, the complete-response path is preserved byte-identically.
Looks Good
- Stream path is well-isolated:
stream=Truereturns the raw SDK iterator, skips_validate_llm_response(which assumes complete responses) - Non-streaming path is explicitly preserved — no stream/stream_options/timeout keys leak through
- MoA facade's
create()correctly extracts stream/stream_options fromapi_kwargsand forwards timeout - 6 comprehensive tests in
test_moa_streaming.pycover: stream path, non-stream invariance, timeout forwarding, caller stream_options passthrough, and both call_llm paths conversation_loop.pygating is correct: MoA only streams when consumers are present
Reviewed by Hermes Agent
19 tasks
1 task
This was referenced Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MoA sessions now stream the aggregator's response to the user live, instead of fetching it whole and showing nothing until the entire turn finishes.
Changes
agent/moa_loop.py:MoAChatCompletions.create()honorsstream=True— runs the references first, then returns the aggregator's raw token stream. Non-streaming path (stream=False) is byte-identical to before.agent/auxiliary_client.py:call_llm()gainsstream/stream_options; when streaming it returns the raw SDK iterator and skips_validate_llm_response+ the fallback chain (the caller owns reassembly/fallback).agent/conversation_loop.py: stop hard-disabling streaming forprovider == "moa"— stream only when a display/TTS consumer is present (_has_stream_consumers()).scripts/release.py: AUTHOR_MAP entry for @lEWFkRAD.Why
conversation_loophard-disabled streaming forprovider in {"copilot-acp", "moa"}, andMoAChatCompletions.create()fetched the aggregator reply whole. On long turns the user saw minutes of silence before any output.The fix reuses the existing streaming consumer (which already handles text-delta delivery, tool-call reassembly, stale-stream detection, and non-streaming fallback) rather than adding a parallel streaming path — because for
provider: moa,_create_request_openai_clientreturns theMoAClientfacade itself, so the consumer already invokesMoAChatCompletions.create(stream=True).Cache-safe: streaming changes only how the final assistant text is delivered — no message-history mutation, no prompt-cache invalidation. References still run first.
Validation
stream=Falseaggregator callstream/stream_options/timeoutkeys (byte-identical)Salvaged from #53848 (@lEWFkRAD), cherry-picked onto current main with authorship preserved.
Infographic
Nous Research