Skip to content

feat(moa): stream the aggregator response to the user - #53848

Closed
lEWFkRAD wants to merge 1 commit into
NousResearch:mainfrom
lEWFkRAD:feat/moa-streaming
Closed

feat(moa): stream the aggregator response to the user#53848
lEWFkRAD wants to merge 1 commit into
NousResearch:mainfrom
lEWFkRAD:feat/moa-streaming

Conversation

@lEWFkRAD

Copy link
Copy Markdown
Contributor

What does this PR do?

Makes Mixture-of-Agents (MoA) sessions stream the aggregator's response to the
user
. Today MoA can't stream: the gateway streaming toggle is a no-op for
provider: moa, so the user sees nothing until the entire response finishes —
minutes of silence on long turns — because the aggregator's reply is always
fetched whole.

Two causes: (1) conversation_loop hard-disables streaming for
provider in {"copilot-acp", "moa"}, and (2) MoAChatCompletions.create()
fetches the aggregator response whole via call_llm(), which had no streaming
mode.

For provider: moa, _create_request_openai_client() returns the MoAClient
facade itself, so the existing streaming consumer already invokes
MoAChatCompletions.create(stream=True). This PR reuses that battle-tested
consumer (text-delta delivery, tool-call reassembly, stale-stream detection,
non-streaming fallback) rather than adding a parallel streaming path — so the
change is small and low-risk.

Related Issue

No existing issue — happy to open one if preferred.

Fixes #

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • agent/auxiliary_client.pycall_llm() gains stream/stream_options.
    When streaming it returns the raw SDK stream iterator directly, bypassing
    _validate_llm_response and the temperature/max_tokens/payment fallback chain
    (which assume a complete response). The caller owns reassembly and fallback.
  • agent/moa_loop.pyMoAChatCompletions.create() runs the references first
    (unchanged), then when stream=True returns the aggregator's raw stream,
    forwarding stream_options and the consumer's per-request read timeout.
    stream=False is byte-identical to before (no stream/stream_options/timeout
    forwarded).
  • agent/conversation_loop.py — MoA streams only when a display/TTS consumer is
    present; quiet/subagent/health-check paths keep the complete-response path,
    preserving prior behavior (and the existing _create_request_openai_client
    guard test).
  • tests/run_agent/test_moa_streaming.py — new tests (below).

How to Test

  1. Configure a MoA preset and run an interactive turn (provider: moa) with a
    display consumer attached — the aggregator's output now appears
    incrementally instead of all at once when the turn completes.
  2. Confirm the references still run before the aggregator (MoA not bypassed), and
    that tool-calling still works under streaming (the reused consumer reassembles
    tool_calls).
  3. Quiet mode / subagents are unchanged (no consumer → complete-response path).
  4. Run the tests: pytest tests/run_agent/test_moa_streaming.py tests/run_agent/test_moa_loop_mode.py -q

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (feat(moa): ...)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass (the MoA suites; 20 passed)
  • I've added tests for my changes
  • I've tested on my platform: Windows 11

Documentation & Housekeeping

  • I've updated relevant documentation (docstrings on call_llm and the
    changed methods) — no user-facing docs reference MoA streaming
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A
    (no new config keys; reuses existing streaming behavior)
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or
    workflows — N/A
  • I've considered cross-platform impact — N/A (pure Python, no platform-specific code)
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

Verified live on a local llama.cpp MoA stack: a plain turn streamed 74 incremental
content chunks (previously one buffered block); references still fired; a
tool-calling turn streamed and executed the tool with no errors.

MoA sessions could not stream: the gateway streaming toggle was a no-op for
provider "moa", so users saw nothing until the entire response finished — minutes
of silence on long turns. The aggregator's reply was always fetched whole.

Root cause was twofold:
  1. conversation_loop hard-disabled streaming for provider in {"copilot-acp",
     "moa"} (MoA grouped with the ACP client, whose facade isn't a stream).
  2. MoAChatCompletions.create() fetched the aggregator response whole via
     call_llm(), which had no streaming mode.

For provider "moa", _create_request_openai_client() returns the MoAClient facade
itself, so the existing streaming consumer already calls
MoAChatCompletions.create(stream=True). We reuse that battle-tested consumer
(text-delta delivery, tool_call reassembly, stale-stream detection, non-streaming
fallback) instead of adding a parallel streaming path.

Changes:
  - call_llm() gains stream/stream_options. When streaming it returns the raw SDK
    stream iterator directly, bypassing _validate_llm_response and the
    temperature/max_tokens/payment fallback chain (which assume a complete
    response). The caller owns reassembly and fallback.
  - MoAChatCompletions.create() runs the references first (unchanged), then when
    stream=True returns the aggregator's raw stream, forwarding stream_options and
    the consumer's per-request read timeout. stream=False is byte-identical to
    before (no stream/stream_options/timeout forwarded).
  - conversation_loop streams MoA only when a display/TTS consumer is present;
    quiet/subagent/health-check paths keep the complete-response path.

Tests: tests/run_agent/test_moa_streaming.py — create() stream/non-stream
branches, stream_options + timeout forwarding, call_llm raw-stream return vs
validated non-stream. Existing MoA tests unchanged (20 passed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have labels Jun 27, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

MoA aggregator streaming response (4 files). Clean feature extension:

  • New stream and stream_options parameters on call_llm
  • Streaming path returns raw SDK iterator directly (bypasses response validation)
  • Conversation loop integration for streaming MoA aggregator output
  • Clear docstring explaining the streaming contract and fallback ownership

Well-scoped streaming extension. The deliberate skip of validation on the streaming path is correct — the caller owns chunk reassembly.

@teknium1

Copy link
Copy Markdown
Contributor

Merged via #55625 — your commit was cherry-picked onto current main with your authorship preserved in git log (commit on main under your name). One small conflict resolved during salvage: the call_llm signature also gained an api_mode kwarg from an earlier MoA fix (#55579), so both kwargs now coexist. Thanks for the clean, low-risk approach — reusing the existing streaming consumer was exactly right.

@teknium1 teknium1 closed this Jun 30, 2026
dtera pushed a commit to dtera/hermes-agent that referenced this pull request Jul 1, 2026
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
Jasper6439 pushed a commit to Jasper6439/hermes-agent that referenced this pull request Jul 5, 2026
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants