MoA: Mixture-of-Agents multi-model strategies - #9
Closed
michaelneale wants to merge 7 commits into
Closed
michaelneale wants to merge 7 commits into
michaelneale wants to merge 7 commits into
Conversation
3-model ensemble (MiniMax-253B, Qwen3-8B, Qwen3-30B-A3B) on Studio. NSED protocol: parallel proposals, anonymous cross-evaluation with diagonal mask, quadratic voting, up to 3 deliberation rounds. Key findings: - Tool use: ensemble correctly picks model with best tool coverage - Reasoning: multi-round deliberation improves structure/rigor - Math/code: no improvement when all models solve correctly - Latency: 3-10x overhead, too much for interactive use - Weak models (8B) become pure evaluators, never win proposals Promising direction: single-round MoM as a mesh router strategy for model=auto — try all models, vote on best answer.
Direct head-to-head: Claude Sonnet 4 vs Qwen3-8B (local) on finding bugs in a statistics module. Claude: 34s, found 10 issues, excellent quality, idiomatic Python Qwen3-8B: 29s, found 7 issues, good quality, missed subtle edge cases Claude meaningfully better but gap isn't enormous. MoM ensemble could close it by combining diverse model perspectives.
Proper agentic comparison using pi -p harness (same as Open Model Gym). Task: add display_name() to Rust User struct. Claude Sonnet 4: 12s, read file, edited correctly, compiles. 5/5. Qwen3-8B: 84s, hallucinated method already exists, no edit. 3/5. The gap is massive for agentic work. Single-turn reasoning is comparable, but tool use (read/edit/verify) is where small models fall apart. Need to test MiniMax-253B and Qwen2.5-72B next.
New module moa.rs with three strategies, selected by model name: - moa: Fan-out to all models → aggregator synthesizes (MoA paper) - best-of-n: Fan-out → aggregator picks best response verbatim - moa-2: Two-layer fan-out → all refine → aggregator synthesizes All run client-side via reqwest loopback through local proxy. Hosts are unaware. Virtual models appear in /v1/models when 2+ models. Handles thinking models (reasoning_content fallback). Tested end-to-end: 3 mesh models (MiniMax-253B, Qwen2.5-72B, Qwen3.5-9B). 98/98 tests pass.
Key findings: - MiniMax-253B via mesh: 5/5, 24s — identical edit to Claude - Claude Sonnet 4: 5/5, 15s — baseline ceiling - Qwen3-8B: 3/5 — hallucinated, no edit (confirmed again) - MoA: failed (0 output) — fan-out latency exceeds pi timeout Local algo comparison (3 models): - MoA synthesis amplifies hallucinations from weaker models - Best-of-N better than synthesis for correctness - All strategies 3.5x slower than solo on convergent tasks Model scale is the answer for agentic work. MiniMax-253B (free) matches Claude (paid) on the same task.
…l-use - Filter MoA fan-out to tier 3+ models only (drops Qwen3.5-9B) - Fan-out time: 80s → 3s - Increase connect_timeout to 30s (QUIC tunnel setup needs time) Eval results (confirmed across 2 runs): - Claude Sonnet 4: 13-15s, 5/5 - MiniMax-253B solo: 18-24s, 5/5 — MATCHES CLAUDE - MoA: 45s, 3/5 — aggregation prompt breaks tool-use protocol MoA is good for chat quality, bad for agentic work. Solo strong model is the right strategy for agents.
- Move superseded Python prototypes to mom/historical/ - Add REFERENCE.md (MoA/NSED/RouteMoA literature notes) - .gitignore Mixture.pdf - Document MoA tool-call strategy as key next step: option (a) MoA for reasoning → separate tool extraction pass option (b) Best-of-N for tool calls → pick best verbatim (simpler) - Updated file index in PLAN.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Client-side Mixture-of-Agents (MoA) implementation — fan out requests to multiple models, aggregate responses.
Based on:
Strategies
moa— synthesize all responses into one (best for chat quality)best-of-n— pick the best response verbatim (best for correctness)moa-2— two-layer: fan-out → all refine → aggregate (highest quality, 3x latency)All three appear as virtual models in
/v1/modelswhen 2+ real models are available. Selectable by model name.How it works
pick_aggregator())reasoning_contentfallback)Eval results
Known limitation: tool calls
MoA synthesis injects a prompt that breaks agent tool-use flows. Two strategies under consideration:
Files
mesh-llm/src/moa.rs— strategies, fan-out, aggregation (~300 LOC)mesh-llm/src/proxy.rs— MoA interception inhandle_mesh_requestmesh-llm/src/main.rs— MoA interception inapi_proxymom/— experiment data, eval scripts, resultsNSED experiment (also in this branch)
Implemented N-Way Self-Evaluating Deliberation from the MoM paper. Multi-round deliberation with cross-evaluation and quadratic voting. Findings: