MoA: mesh answers improve as capable nodes join - #1116
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughMoA now supports single-model fallback routing, structured worker-pool assembly, truncation-aware text arbitration, optional refinement, asymmetric tool handling, and expanded replay and OpenRouter evaluation coverage. ChangesMoA routing and worker-pool assembly
Mixture-of-agents orchestration
Validation and evaluation
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review. |
michaelneale
changed the title
michaelneale
force-pushed
the
feat/moa-synthesis
branch
from
August 1, 2026 01:50
43d1f05 to
44ef1d7
Compare
michaelneale
changed the title
michaelneale
changed the title
michaelneale
changed the title
michaelneale
force-pushed
the
feat/moa-synthesis
branch
from
August 4, 2026 03:42
28b94b5 to
228a1ef
Compare
michaelneale
marked this pull request as ready for review
August 4, 2026 04:14
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 19
🧹 Nitpick comments (18)
evals/moa-openrouter/probe_tools.py (1)
65-70: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueManage the output file handles in both eval scripts. Both scripts open a JSONL output file without a context manager, so no exit path guarantees a close. The shared fix is to scope each handle with
with open(...).
evals/moa-openrouter/probe_tools.py#L65-L70: move the module-levelopen("fanout.jsonl", "a")intomainand scope it withwith, so importing the module no longer creates the file.evals/moa-openrouter/record_agentic.py#L236-L293: replaceout = open("agentic.jsonl", "w")plusout.close()with awith open(...)block, so an exception during a scenario still closes the file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@evals/moa-openrouter/probe_tools.py` around lines 65 - 70, Manage both output file handles with context managers: in evals/moa-openrouter/probe_tools.py lines 65-70, move the module-level open call into main and scope it with with; in evals/moa-openrouter/record_agentic.py lines 236-293, replace the out handle and explicit close with a with open block covering scenario processing. Ensure importing probe_tools.py does not create the output file and exceptions still close both handles.evals/moa-openrouter/record_agentic.py (1)
275-291: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse iterable unpacking for the message list.
Ruff reports RUF005 here. Replace the list concatenation with unpacking.
♻️ Proposed refactor
- messages = messages + [ + messages = [ + *messages, { "role": "assistant",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@evals/moa-openrouter/record_agentic.py` around lines 275 - 291, Update the message accumulation expression in the agentic recording flow to use iterable unpacking for the existing messages and the two new message dictionaries, replacing list concatenation while preserving their order and contents.Source: Linters/SAST tools
evals/moa-openrouter/make_fixture.py (2)
23-23: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the redundant
intcall.
round()with a single argument already returns anint. Ruff reports this as RUF046.♻️ Proposed change
- "elapsed_ms": int(round((w["elapsed"] or 0) * 1000)), + "elapsed_ms": round((w["elapsed"] or 0) * 1000),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@evals/moa-openrouter/make_fixture.py` at line 23, Update the elapsed_ms expression in the fixture-building logic to remove the redundant int wrapper and rely on round((w["elapsed"] or 0) * 1000) returning an integer, preserving the existing elapsed fallback and rounding behavior.Source: Linters/SAST tools
14-16: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winResolve the input and output paths relative to the script.
OUTand bothopencalls use paths relative to the current working directory. The script writes the fixture to the wrong location, or fails, when it runs from the repository root. The docstring does not state the required working directory. The two input handles also stay open until interpreter exit.Anchor the paths to
__file__and use context managers.♻️ Proposed change
import json +from pathlib import Path -OUT = "../../crates/mesh-mixture-of-agents/tests/fixtures/real_traces.json" +HERE = Path(__file__).resolve().parent +OUT = HERE.parents[1] / "crates/mesh-mixture-of-agents/tests/fixtures/real_traces.json"-for line in open("agentic.jsonl"): - line = line.strip() - if not line: - continue - r = json.loads(line) - cases.append( +with open(HERE / "agentic.jsonl") as fh: + agentic_lines = fh.read().splitlines() +for line in agentic_lines: + line = line.strip() + if not line: + continue + r = json.loads(line) + cases.append(Apply the same change to the
corpus.jsonlloop at line 53.Also applies to: 34-34, 53-53, 71-72
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@evals/moa-openrouter/make_fixture.py` around lines 14 - 16, Update the fixture path setup in the script to resolve OUT and both input paths relative to the script’s __file__, so execution is independent of the current working directory. Wrap every input and output open call, including the corpus.jsonl loop, in context managers to close handles promptly.crates/mesh-mixture-of-agents/tests/eval_openrouter.rs (1)
1453-1459: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winArm C can get one candidate while arm B gets none.
diverse.len().max(1)forces at least one homogeneous candidate. If every diverse peer call fails, arm B runs with zero candidates and is identical to arm A, while arm C still receives one candidate. The reportedB-Cdifferential then measures candidate presence, not family diversity. Match the counts exactly.♻️ Proposed change
- for _ in 0..diverse.len().max(1) { + for _ in 0..diverse.len() { if let Some(c) = structured_proposal(&backend, &finalizer, task).await { homo.push(c); } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/tests/eval_openrouter.rs` around lines 1453 - 1459, Update the homogeneous candidate loop in the arm C setup to generate exactly diverse.len() candidates, removing the max(1) fallback. Preserve the existing structured_proposal failure handling so zero diverse candidates produces zero homogeneous candidates and the B-C comparison measures family diversity.crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs (1)
213-217: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUpdate the stale comment on
enable_thinking.The comment says
try_handle_moaoverrides this value "when the caller has expressed a preference". That is no longer true.effective_enable_thinking_for_moanow always returnsSome(false)(lines 17-46), andtry_handle_moaassigns it unconditionally. A reader following this comment would look for caller-driven behavior that no longer exists.📝 Proposed wording
- // Defaults to leaving each model's thinking behavior alone. - // `try_handle_moa` overrides this from the inbound request body - // when the caller has expressed a preference - // (`reasoning_effort: "none"`, `enable_thinking: false`, etc.). - enable_thinking: None, + // Placeholder: `try_handle_moa` always overwrites this with + // `effective_enable_thinking_for_moa`, which is `Some(false)` as a + // policy. Caller reasoning knobs are parsed for logging only. + enable_thinking: None,🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs` around lines 213 - 217, Update the comment above enable_thinking to reflect that effective_enable_thinking_for_moa always returns Some(false) and try_handle_moa assigns that value unconditionally; remove the outdated claim that inbound caller preferences control the override.crates/mesh-mixture-of-agents/src/normalize.rs (1)
55-60: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider passing truncation into the normalizer.
Every path in this file hardcodes
truncated: false, and each caller must remember to stamp the real value after the call (fanout::gather_workers_incremental,fanout::gather_references,refinement::refine_round). A new call site that omits the stamp silently reports a truncated answer as complete, which is the exact failure this field exists to prevent. Accepting the flag as a parameter ofnormalize_worker_outputwould make the compiler enforce it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/normalize.rs` around lines 55 - 60, Update normalize_worker_output to accept a truncation flag parameter and use it when constructing WorkerOutput instead of hardcoding truncated: false. Pass the actual flag from every caller, including fanout::gather_workers_incremental, fanout::gather_references, and refinement::refine_round, then remove their post-call stamping.crates/mesh-mixture-of-agents/src/backend.rs (1)
155-159: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueThe retry removes caller-supplied
chat_template_kwargscontent.
apply_enable_thinkingmergesenable_thinkinginto any existingchat_template_kwargsobject. The retry removes the wholechat_template_kwargskey, so template options the caller set for other purposes are lost on the retried request. Removing only the thinking keys keeps the rest intact.♻️ Proposed change
let mut retry_body = body.clone(); if let Some(obj) = retry_body.as_object_mut() { obj.remove("reasoning_effort"); - obj.remove("chat_template_kwargs"); + if let Some(kwargs) = obj + .get_mut("chat_template_kwargs") + .and_then(Value::as_object_mut) + { + kwargs.remove("enable_thinking"); + if kwargs.is_empty() { + obj.remove("chat_template_kwargs"); + } + } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/backend.rs` around lines 155 - 159, The retry preparation in the backend request flow must preserve caller-supplied chat_template_kwargs. Update the retry_body cleanup to remove only the thinking-related entries added by apply_enable_thinking, while retaining the rest of the chat_template_kwargs object; keep removal of reasoning_effort unchanged.crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs (1)
34-42: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConsider the token requirement and a deterministic pick when degrading.
degrade_to_single_modeltakes the first model returned bymodels_being_served(). Two effects follow:
- The picked model may not satisfy
required_tokens.build_moa_configappliescontext_selection::context_can_satisfyper worker, but this fallback path applies no context check, so a request that needs a large context can be routed to a small-context model.- The iteration order of
models_being_served()decides the answer. If that order is not stable, the same lone node can answer with different models across requests.A deterministic selection that prefers locally served models with sufficient context would make the degraded path predictable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs` around lines 34 - 42, Update the fallback model selection around degrade_to_single_model to filter out the virtual model and retain only models whose context satisfies required_tokens, preferring locally served models when available. Sort or otherwise select from the remaining candidates deterministically before choosing the target, while preserving the existing 503 response when no eligible model exists.crates/mesh-mixture-of-agents/src/fanout.rs (1)
350-363: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSimplify the reference stop condition.
Line 352 checks
!outputs.is_empty() && outputs.len() >= min_references. The first clause only matters whenmin_references == 0, and the only caller passesdispatched.len().div_ceil(2).max(1). Either drop the redundant clause or clampmin_referenceswith.max(1)inside this function so the intent is local.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/fanout.rs` around lines 350 - 363, In the reference collection loop, simplify the stop condition around outputs.len() and min_references: remove the redundant outputs non-empty check, or normalize min_references to at least one within the function before the loop. Preserve the behavior that collection stops once the configured minimum number of references is reached.crates/mesh-llm-host-runtime/src/network/openai/ingress.rs (1)
842-857: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSet the degraded flag in the match arm instead of comparing model values.
Line 856 infers
degradedfromrouting_model != decision.effective_model. The branch that produced the value is already known at line 847. The comparison holds today only becausedegrade_to_single_modelexcludesmoa::VIRTUAL_MODEL_NAME, so the rewritten name can never equal"mesh". If that exclusion changes, the pipeline classifier would run against a stale decision. A flag set in the arm removes the dependency.♻️ Proposed refactor
- let mut routing_model = decision.effective_model.clone(); - let tcp_stream = match try_handle_moa_intercept(tcp_stream, &mut request, &ctx, &decision).await - { - MoaInterceptResult::Handled => return, - MoaInterceptResult::NotMoa(stream) => stream, - MoaInterceptResult::Degraded { stream, model } => { - routing_model = model; - stream - } - }; - - let mut tcp_stream = tcp_stream; - // A degraded turn is a plain single-model request; skip the pipeline - // classifier (computed against "mesh") and route it directly. - let degraded = routing_model != decision.effective_model; + let mut routing_model = decision.effective_model.clone(); + let mut degraded = false; + let mut tcp_stream = + match try_handle_moa_intercept(tcp_stream, &mut request, &ctx, &decision).await { + MoaInterceptResult::Handled => return, + MoaInterceptResult::NotMoa(stream) => stream, + MoaInterceptResult::Degraded { stream, model } => { + routing_model = model; + degraded = true; + stream + } + }; + + // A degraded turn is a plain single-model request; skip the pipeline + // classifier (computed against "mesh") and route it directly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-llm-host-runtime/src/network/openai/ingress.rs` around lines 842 - 857, Update the request-routing flow around try_handle_moa_intercept to initialize a degraded flag before the match and set it explicitly in the MoaInterceptResult::Degraded arm. Replace the routing_model != decision.effective_model comparison with that flag, while leaving the Handled and NotMoa paths unchanged so pipeline classification is skipped whenever the intercept produced a degraded single-model request.crates/mesh-mixture-of-agents/src/context.rs (1)
532-545: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueName the advisor payload bound.
reducer_payload_budgetandREFINEMENT_DRAFT_BUDGETexpress their limits as named items with the reasoning attached. This loop hardcodes500and the derived497. Introduce a constant, for exampleADVICE_PAYLOAD_BUDGET, and derive the truncation length from it so the two values cannot drift.♻️ Proposed refactor
+/// How much of each advisor's prose the actor may see. +const ADVICE_PAYLOAD_BUDGET: usize = 500; + @@ - let payload = if r.payload.len() > 500 { - format!("{}...", crate::worker::truncate_chars(&r.payload, 497)) + let payload = if r.payload.len() > ADVICE_PAYLOAD_BUDGET { + format!( + "{}...", + crate::worker::truncate_chars(&r.payload, ADVICE_PAYLOAD_BUDGET - 3) + ) } else { r.payload.clone() };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/context.rs` around lines 532 - 545, In the advisor payload loop over references, replace the hardcoded 500-byte/character bound and derived 497 truncation length with a named constant such as ADVICE_PAYLOAD_BUDGET, documenting the limit’s purpose consistently with reducer_payload_budget and REFINEMENT_DRAFT_BUDGET. Derive the truncation length from that constant so the displayed payload plus ellipsis always stays within the same budget.crates/mesh-mixture-of-agents/src/tool_turn.rs (2)
255-336: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd tests for
actor_body.The policy tests are thorough.
actor_bodyis untested, and it encodes the response contract for the whole tool path: aToolProposalmust emittool_calls, anUncertaintywith a forced tool must emit the forced call, and anUncertaintywith no forced tool and no references must emitMOA_ERR_NO_USABLE_ANSWER. It is a pure function overWorkerOutput, so it needs no backend. Add cases for those three branches.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/tool_turn.rs` around lines 255 - 336, Add unit tests for the pure actor_body function covering three response-contract branches: ToolProposal produces tool_calls, Uncertainty with a forced tool produces that forced call, and Uncertainty without a forced tool or references returns MOA_ERR_NO_USABLE_ANSWER. Construct WorkerOutput fixtures directly without backend setup, and keep the existing policy tests unchanged.
143-147: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winBoth new fan-out modules rebuild an identical prompt inside the per-worker loop.
dispatch_and_gather_referencesandrefine_roundeach call a context packer once per spawned worker with arguments that do not vary by worker, so the samePackedContextis built N times. Hoist the packing above the loop and clone the messages into each task.
crates/mesh-mixture-of-agents/src/tool_turn.rs#L143-L147: movecontext::pack_for_reference(session, REFERENCE_HISTORY_MESSAGES)above thefor a in &assignmentsloop and clonepacked.messagesper task.crates/mesh-mixture-of-agents/src/refinement.rs#L100-L102: movecontext::pack_for_refinement(session, &texts)above the loop; this call also concatenates and truncates every draft, so the repeated work is larger here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/tool_turn.rs` around lines 143 - 147, The fan-out paths rebuild identical packed context for every worker. In crates/mesh-mixture-of-agents/src/tool_turn.rs lines 143-147, update dispatch_and_gather_references to call context::pack_for_reference once before the for a in &assignments loop and clone packed.messages into each task; in crates/mesh-mixture-of-agents/src/refinement.rs lines 100-102, update refine_round to call context::pack_for_refinement once before its worker loop and clone the resulting messages per task.crates/mesh-mixture-of-agents/src/refinement.rs (2)
194-291: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a
refine_roundtest for the shortfall path.The tests cover policy and budget arithmetic well. They do not exercise
refine_round, whose central safety claim is that fewer thanMIN_DRAFTSrefinements returnNoneso the caller keeps the round-1 outputs.reducer.rsalready has aFakeBackendhelper that can drive this. Add one test where a single worker succeeds and assertrefine_roundreturnsNone.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/refinement.rs` around lines 194 - 291, Add a unit test for refine_round using the existing reducer.rs FakeBackend helper, configuring one worker to succeed and produce fewer than MIN_DRAFTS refinement results. Assert that refine_round returns None, preserving the caller’s round-1 outputs; keep the test focused on this shortfall path.
141-148: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRecord a summary when a refinement reply is empty.
Line 142 skips a reply whose text is blank and continues without pushing a
WorkerSummary. The worker ran and consumed tokens, but it appears in no accounting. Every other outcome in this loop produces a summary, and the round-1 gather reconciles all dispatched workers. Add a failed summary so refinement attempts stay observable.♻️ Proposed change
Ok((model, role, Ok(reply), elapsed)) => { if reply.text.trim().is_empty() { + summaries.push(WorkerSummary { + model, + role, + succeeded: false, + elapsed_ms: elapsed, + output_kind: None, + confidence: None, + }); continue; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/refinement.rs` around lines 141 - 148, Update the empty-reply branch in the refinement result loop before continue so it pushes a failed WorkerSummary for the completed worker, preserving the worker identity and relevant timing/token accounting; then continue without normalizing output. Ensure every dispatched refinement attempt remains represented consistently with the other result branches and round-1 reconciliation.crates/mesh-mixture-of-agents/src/lib.rs (2)
1309-1309: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffConsider extracting the response builders into their own module.
These six builders are now
pub(crate)becausetool_turnconsumes them. They form a separable responsibility: OpenAI wire-response construction.lib.rsis over 2,000 lines and this PR modifies it.Move
best_answer,fallback_worker_response,tool_proposal_response,error_response,chat_response, andtool_call_responseplusresponse_builder_testsinto aresponsemodule. That keepslib.rsshrinking as the crate grows and moves the tests with the behavior.As per coding guidelines: "When modifying a Rust source file over 1,000 lines, extract any separable responsibility into a semantically named module, keep the new file under 1,000 lines, and move relevant tests with the extracted behavior."
Also applies to: 1326-1326, 1338-1338, 1374-1374, 1420-1420, 1434-1434
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/lib.rs` at line 1309, Extract the OpenAI wire-response builders best_answer, fallback_worker_response, tool_proposal_response, error_response, chat_response, and tool_call_response from lib.rs into a semantically named response module, preserving their pub(crate) visibility and existing behavior. Move response_builder_tests into the same module, update module declarations and call sites such as tool_turn, and keep the new module under 1,000 lines.Source: Coding guidelines
316-350: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSimplify the now-unreachable tool bookkeeping on the text path.
The guard at line 310 returns for every turn where
has_toolsis true orforced_toolis set. After that guard,query_uses_toolsat line 343 is alwaysfalse,selected_tool_namesis always empty, andgrace_mode_for_turn(session, has_tools)is always called withhas_tools == false, soGraceMode::Toolis unreachable here. The comment at lines 318-342 explains a decision that this path no longer makes.Replace the computed values with constants and move the historical rationale to
tool_turn. The change keeps behavior identical and removes a misleading contract for the next reader.♻️ Proposed simplification
- let query_uses_tools = forced_tool.is_some() || has_tools; - let selected_tool_names = if let Some(tool) = forced_tool { - vec![tool.name.clone()] - } else if query_uses_tools { - selected_tool_names_for_turn(session, allowed_tools) - } else { - Vec::new() - }; + // Tool-bearing turns returned above, so this path is always tool-free. + let query_uses_tools = false; + let selected_tool_names: Vec<String> = Vec::new();🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-mixture-of-agents/src/lib.rs` around lines 316 - 350, In the post-guard text path, replace the computed tool state with constant no-tools values and call grace_mode_for_turn with tools unavailable, since forced_tool and has_tools cannot reach this branch. Move the historical tool-availability rationale from this block to tool_turn, retaining only comments relevant to the text path. Preserve the existing behavior while removing the misleading query_uses_tools and selected_tool_names contract.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs`:
- Around line 399-470: Extract the separable pool-assembly responsibilities from
this oversized module into semantically named modules, such as admission and
patience, keeping each new file under 1,000 lines. Move apply_admission_control
and its tests into the admission module; move cap_committee,
self_fill_from_extra_instances, assemble_worker_pool, and their tests into an
appropriate pool/committee module, and move PatienceProfile, patience_profile,
and related tests into the patience module. Update module declarations and call
sites while preserving existing behavior.
- Around line 522-548: Update assemble_worker_pool and
self_fill_from_extra_instances to carry the initial remote peer selected by
add_worker_backend, and skip that peer when adding extra instances to the MoA
committee. Pass the tracked peer through the call chain, remove the unused _http
parameter from self_fill_from_extra_instances and its callers, and preserve the
existing worker-count limit.
In `@crates/mesh-mixture-of-agents/src/arbiter.rs`:
- Around line 455-463: Update the match in single_output_decision so
OutputKind::ToolProposal returns Decision::NeedsReducer with the appropriate
reducer reason, alongside OutputKind::Uncertainty. Keep non-tool, non-uncertain
outputs on the existing Decision::Answer path.
In `@crates/mesh-mixture-of-agents/src/backend.rs`:
- Around line 356-361: Update the tool-proposal path around
BackendReply::complete to validate the raw args as JSON and set the reply’s
truncated state when parsing fails specifically with finish_reason == "length";
preserve the complete status for valid JSON or non-length finish reasons, and
avoid treating the unwrap_or("{}") fallback as proof that the original arguments
were complete.
In `@crates/mesh-mixture-of-agents/src/context.rs`:
- Around line 347-453: Extract the packing responsibilities from context.rs into
semantically named modules: move REFERENCE_PREAMBLE and pack_for_reference with
their tests to a reference module, pack_for_refinement and
REFINEMENT_DRAFT_BUDGET with related tests to refinement_context, and
pack_for_actor with its tests to actor. Update module declarations, imports, and
call sites so the existing behavior and APIs remain unchanged while keeping each
source file under 1,000 lines.
- Around line 475-497: Verify the refinement configuration in the code and
measurement setup around the PackedContext construction and its max_tokens
value. If the measured refinement configuration used the full worker token
budget rather than 1024 tokens, increase max_tokens to that budget so synthesis
is not unnecessarily truncated; otherwise preserve the current value.
In `@crates/mesh-mixture-of-agents/src/fanout.rs`:
- Around line 383-397: Update reconcile_dispatched, used by gather_references
and gather_workers_incremental, to match returned summaries to dispatched
workers by slot index or another unique per-dispatch identifier instead of model
name. Propagate that identifier through the dispatch/result flow so duplicate
same-model slots reconcile independently and failed or cancelled slots remain in
TurnResult.worker_summaries.
In `@crates/mesh-mixture-of-agents/src/reducer.rs`:
- Around line 154-158: Make reducer truncation observable in the output handling
around the result mapping that returns `(name, result.map(|reply| reply.text))`.
Preserve the full reducer text, but inspect `reply.truncated` and log when the
final synthesis response was truncated, using the existing logging mechanism and
context available in the reducer flow.
In `@crates/mesh-mixture-of-agents/src/tool_turn.rs`:
- Around line 178-187: Cap the optional advisory wait in the tool-turn flow
before calling gather_references, using the same worker-budget fraction
established by refinement.rs rather than the full config.worker_timeout. Keep
min_references unchanged and preserve pack_for_actor’s empty-reference fallback
when the bounded advisory round expires.
- Around line 79-92: Update the TurnResult construction in finalize_actor_output
so reducer_used reflects actor_ok instead of always being true. Preserve the
existing reducer_attempts value and response handling, ensuring failed reducer
attempts that return fallback_worker_response or error_response report
reducer_used as false.
- Around line 54-55: Update handle_tool_query to accept and propagate the
caller’s has_tools flag, passing it to context::pack_for_actor instead of
hardcoding true. Ensure tool_proposal_response and forced tool_call_response
paths use the same flag so tool-call responses are omitted when the request has
no tools array.
In `@crates/mesh-mixture-of-agents/src/worker.rs`:
- Around line 144-181: Align the documentation for canonical_base_name and
pool_is_homogeneous with the actual behavior: quant-tagged names remain distinct
from untagged names, so mixed quant variants are not considered homogeneous and
refinement::refinement_expected will not enable refinement for them. Update the
comments to remove the claim that quant variants share a base, and add coverage
for a plain name versus a quant-tagged alias to pin this contract.
In `@crates/mesh-mixture-of-agents/tests/eval_openrouter.rs`:
- Around line 2212-2215: Update the solo-baseline setup around small_mesh_pool
so it does not claim pool[0] is the strongest member: either select the
strongest model explicitly or revise the comment and output wording to describe
the first declaration-order entry. Also handle an empty pool before indexing,
using the test’s existing failure or early-return convention instead of allowing
pool[0] to panic.
- Around line 460-466: Replace the local truncate helper and its call sites with
the exported truncate_chars behavior, ensuring truncation never slices through a
UTF-8 character. Update crates/mesh-mixture-of-agents/src/lib.rs::truncate_chars
if necessary so its limit semantics match the required character-count boundary,
and apply the safe helper to model output, HTTP error bodies, and advisor
errors.
In `@crates/mesh-mixture-of-agents/tests/sim_real_traces.rs`:
- Around line 541-557: Update
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L541-L557 to remove the
result.reducer_used skip for tool-bearing turns so the majority-argument
assertions execute; update is_reducer_call at
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L146-L154 to recognize
the pack_for_actor system marker “## Advice from other models” and return
SYNTHESIZED for the actor call instead of replaying a worker body.
In `@crates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rs`:
- Around line 3-7: Update the module-level rationale near the refinement
description to remove the withdrawn “42/66/12, p=5.2e-05” result or replace it
with the controlled evaluation result from evals/moa-openrouter/RESULTS.md;
retain the explanation that refinement is best-effort and may introduce another
hanging fan-out.
In `@evals/moa-openrouter/probe_tools.py`:
- Around line 100-105: Update the tool-call formatting comprehension in the
result-printing block to read arguments defensively via the nested function
mapping, defaulting missing or falsy values to "{}" as in record_agentic.py,
while preserving the existing call-name formatting and probe flow.
In `@evals/moa-openrouter/record.py`:
- Around line 31-41: Update the worker record construction around the response
handling to persist the complete OpenAI response, including usage and unselected
fields, by storing resp directly in each record while retaining the existing
derived fields as needed.
In `@evals/moa-openrouter/RESULTS.md`:
- Around line 487-493: Update the documented merge-blocker status around the
harness-versus-production parity claim in moa::handle_turn: either provide
evidence that the orchestration gap is resolved, or explicitly record an
approved waiver. Do not present the production reasoning-quality improvement as
validated while this blocker remains unresolved.
---
Nitpick comments:
In `@crates/mesh-llm-host-runtime/src/network/openai/ingress.rs`:
- Around line 842-857: Update the request-routing flow around
try_handle_moa_intercept to initialize a degraded flag before the match and set
it explicitly in the MoaInterceptResult::Degraded arm. Replace the routing_model
!= decision.effective_model comparison with that flag, while leaving the Handled
and NotMoa paths unchanged so pipeline classification is skipped whenever the
intercept produced a degraded single-model request.
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs`:
- Around line 34-42: Update the fallback model selection around
degrade_to_single_model to filter out the virtual model and retain only models
whose context satisfies required_tokens, preferring locally served models when
available. Sort or otherwise select from the remaining candidates
deterministically before choosing the target, while preserving the existing 503
response when no eligible model exists.
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs`:
- Around line 213-217: Update the comment above enable_thinking to reflect that
effective_enable_thinking_for_moa always returns Some(false) and try_handle_moa
assigns that value unconditionally; remove the outdated claim that inbound
caller preferences control the override.
In `@crates/mesh-mixture-of-agents/src/backend.rs`:
- Around line 155-159: The retry preparation in the backend request flow must
preserve caller-supplied chat_template_kwargs. Update the retry_body cleanup to
remove only the thinking-related entries added by apply_enable_thinking, while
retaining the rest of the chat_template_kwargs object; keep removal of
reasoning_effort unchanged.
In `@crates/mesh-mixture-of-agents/src/context.rs`:
- Around line 532-545: In the advisor payload loop over references, replace the
hardcoded 500-byte/character bound and derived 497 truncation length with a
named constant such as ADVICE_PAYLOAD_BUDGET, documenting the limit’s purpose
consistently with reducer_payload_budget and REFINEMENT_DRAFT_BUDGET. Derive the
truncation length from that constant so the displayed payload plus ellipsis
always stays within the same budget.
In `@crates/mesh-mixture-of-agents/src/fanout.rs`:
- Around line 350-363: In the reference collection loop, simplify the stop
condition around outputs.len() and min_references: remove the redundant outputs
non-empty check, or normalize min_references to at least one within the function
before the loop. Preserve the behavior that collection stops once the configured
minimum number of references is reached.
In `@crates/mesh-mixture-of-agents/src/lib.rs`:
- Line 1309: Extract the OpenAI wire-response builders best_answer,
fallback_worker_response, tool_proposal_response, error_response, chat_response,
and tool_call_response from lib.rs into a semantically named response module,
preserving their pub(crate) visibility and existing behavior. Move
response_builder_tests into the same module, update module declarations and call
sites such as tool_turn, and keep the new module under 1,000 lines.
- Around line 316-350: In the post-guard text path, replace the computed tool
state with constant no-tools values and call grace_mode_for_turn with tools
unavailable, since forced_tool and has_tools cannot reach this branch. Move the
historical tool-availability rationale from this block to tool_turn, retaining
only comments relevant to the text path. Preserve the existing behavior while
removing the misleading query_uses_tools and selected_tool_names contract.
In `@crates/mesh-mixture-of-agents/src/normalize.rs`:
- Around line 55-60: Update normalize_worker_output to accept a truncation flag
parameter and use it when constructing WorkerOutput instead of hardcoding
truncated: false. Pass the actual flag from every caller, including
fanout::gather_workers_incremental, fanout::gather_references, and
refinement::refine_round, then remove their post-call stamping.
In `@crates/mesh-mixture-of-agents/src/refinement.rs`:
- Around line 194-291: Add a unit test for refine_round using the existing
reducer.rs FakeBackend helper, configuring one worker to succeed and produce
fewer than MIN_DRAFTS refinement results. Assert that refine_round returns None,
preserving the caller’s round-1 outputs; keep the test focused on this shortfall
path.
- Around line 141-148: Update the empty-reply branch in the refinement result
loop before continue so it pushes a failed WorkerSummary for the completed
worker, preserving the worker identity and relevant timing/token accounting;
then continue without normalizing output. Ensure every dispatched refinement
attempt remains represented consistently with the other result branches and
round-1 reconciliation.
In `@crates/mesh-mixture-of-agents/src/tool_turn.rs`:
- Around line 255-336: Add unit tests for the pure actor_body function covering
three response-contract branches: ToolProposal produces tool_calls, Uncertainty
with a forced tool produces that forced call, and Uncertainty without a forced
tool or references returns MOA_ERR_NO_USABLE_ANSWER. Construct WorkerOutput
fixtures directly without backend setup, and keep the existing policy tests
unchanged.
- Around line 143-147: The fan-out paths rebuild identical packed context for
every worker. In crates/mesh-mixture-of-agents/src/tool_turn.rs lines 143-147,
update dispatch_and_gather_references to call context::pack_for_reference once
before the for a in &assignments loop and clone packed.messages into each task;
in crates/mesh-mixture-of-agents/src/refinement.rs lines 100-102, update
refine_round to call context::pack_for_refinement once before its worker loop
and clone the resulting messages per task.
In `@crates/mesh-mixture-of-agents/tests/eval_openrouter.rs`:
- Around line 1453-1459: Update the homogeneous candidate loop in the arm C
setup to generate exactly diverse.len() candidates, removing the max(1)
fallback. Preserve the existing structured_proposal failure handling so zero
diverse candidates produces zero homogeneous candidates and the B-C comparison
measures family diversity.
In `@evals/moa-openrouter/make_fixture.py`:
- Line 23: Update the elapsed_ms expression in the fixture-building logic to
remove the redundant int wrapper and rely on round((w["elapsed"] or 0) * 1000)
returning an integer, preserving the existing elapsed fallback and rounding
behavior.
- Around line 14-16: Update the fixture path setup in the script to resolve OUT
and both input paths relative to the script’s __file__, so execution is
independent of the current working directory. Wrap every input and output open
call, including the corpus.jsonl loop, in context managers to close handles
promptly.
In `@evals/moa-openrouter/probe_tools.py`:
- Around line 65-70: Manage both output file handles with context managers: in
evals/moa-openrouter/probe_tools.py lines 65-70, move the module-level open call
into main and scope it with with; in evals/moa-openrouter/record_agentic.py
lines 236-293, replace the out handle and explicit close with a with open block
covering scenario processing. Ensure importing probe_tools.py does not create
the output file and exceptions still close both handles.
In `@evals/moa-openrouter/record_agentic.py`:
- Around line 275-291: Update the message accumulation expression in the agentic
recording flow to use iterable unpacking for the existing messages and the two
new message dictionaries, replacing list concatenation while preserving their
order and contents.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1f1bf079-4ba5-464a-8bc2-9d17fd49c867
📒 Files selected for processing (39)
crates/mesh-llm-guardrails/src/rescue.rscrates/mesh-llm-host-runtime/src/network/openai/ingress.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rscrates/mesh-mixture-of-agents/src/arbiter.rscrates/mesh-mixture-of-agents/src/backend.rscrates/mesh-mixture-of-agents/src/context.rscrates/mesh-mixture-of-agents/src/fanout.rscrates/mesh-mixture-of-agents/src/lib.rscrates/mesh-mixture-of-agents/src/normalize.rscrates/mesh-mixture-of-agents/src/reducer.rscrates/mesh-mixture-of-agents/src/refinement.rscrates/mesh-mixture-of-agents/src/tool_guard.rscrates/mesh-mixture-of-agents/src/tool_turn.rscrates/mesh-mixture-of-agents/src/worker.rscrates/mesh-mixture-of-agents/tests/eval_openrouter.rscrates/mesh-mixture-of-agents/tests/fixtures/ablation_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/committee_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/real_traces.jsoncrates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rscrates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rscrates/mesh-mixture-of-agents/tests/sim_partial_worker_survival.rscrates/mesh-mixture-of-agents/tests/sim_real_traces.rscrates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rscrates/mesh-mixture-of-agents/tests/sim_strong_patience.rscrates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rscrates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rscrates/mesh-mixture-of-agents/tests/sim_worker_accounting.rsevals/moa-openrouter/README.mdevals/moa-openrouter/RESULTS.mdevals/moa-openrouter/agentic.jsonlevals/moa-openrouter/analyze_ablation.pyevals/moa-openrouter/corpus.jsonlevals/moa-openrouter/fanout.jsonlevals/moa-openrouter/make_fixture.pyevals/moa-openrouter/orclient.pyevals/moa-openrouter/probe_tools.pyevals/moa-openrouter/record.pyevals/moa-openrouter/record_agentic.py
Comment on lines
+455
to
463
| // Tool turns take the asymmetric actor path, so this text-path arbiter | ||
| // never sees an executable ToolProposal (tool-shaped text is demoted to | ||
| // Uncertainty by `enforce_tool_call_contract` when tools are disabled). | ||
| match output.kind { | ||
| OutputKind::ToolProposal if has_tools => { | ||
| if let Some(ref name) = output.tool_name { | ||
| Decision::ToolCall { | ||
| name: name.clone(), | ||
| arguments: output | ||
| .tool_arguments | ||
| .clone() | ||
| .unwrap_or(Value::Object(Default::default())), | ||
| } | ||
| } else { | ||
| Decision::Answer(output.payload.clone()) | ||
| } | ||
| } | ||
| OutputKind::Uncertainty => Decision::NeedsReducer { | ||
| reason: "single worker uncertain".into(), | ||
| }, | ||
| _ => Decision::Answer(output.payload.clone()), | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Confirm empty-allow-list behaviour of enforce_tool_call_contract.
fd -t f 'tool_guard.rs' crates -x sed -n '1,80p'Repository: Mesh-LLM/mesh-llm
Length of output: 2911
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the relevant arbiter implementation and nearby tests without running repo code.
fd -t f 'arbiter.rs' crates/mesh-mixture-of-agents/src -x sh -c '
echo "=== $1 ==="
wc -l "$1"
echo "--- outline ---"
ast-grep outline "$1" || true
echo "--- relevant sections ---"
sed -n "420,490p" "$1"
echo "--- empty_allowed_list references in arbiter ---"
rg -n "enforce_tool_call_contract|single_output_decision|arbitrate|ToolProposal|OutputKind::" "$1" || true
' sh {}
echo "=== tool_guard tests ==="
fd -t f 'tool_guard.rs' crates -x sh -c '
sed -n "60,160p" "$1"
' sh {}Repository: Mesh-LLM/mesh-llm
Length of output: 13581
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the multi-output branch and make sure ToolProposal is absent from answer/critique/uncertainty collections.
fd -t f 'arbiter.rs' crates/mesh-mixture-of-agents/src -x sh -c '
echo "=== $1 relevant branch ==="
sed -n "33,75p" "$1"
echo "--- ToolProposal/Arbitrate tests ---"
rg -n "ToolProposal|empty_allowed_list_is_noop|enforce_tool_call_contract|arbitrate" "$1" || true
' sh {}
# Read-only behavioral probe of the relevant guard condition and decision arms from the source text.
python3 - <<'PY'
from pathlib import Path
arbiter = Path("crates/mesh-mixture-of-agents/src/arbiter.rs").read_text()
for pattern in [
"let allowed_tools = enforce_tool_call_contract",
"let mut allowed_tools = allowed_tools.clone()",
"enforce_tool_call_contract(&mut out, allowed_tools",
]:
print(f"{pattern!r}: {'present' if pattern in arbiter else 'absent'}")
print("single_output_decision guard references allowed_tools:", "allowed_tools" in arbiter[arbiter.find("fn single_output_decision"):arbiter.find("\n}", arbiter.find("fn single_output_decision"))])
PYRepository: Mesh-LLM/mesh-llm
Length of output: 2795
Route lone ToolProposal outputs to the reducer.
enforce_tool_call_contract leaves an empty allow-list unchanged, and this arbiter has no allowed_tools guard. A single tool-shaped payload reaches single_output_decision as OutputKind::ToolProposal and falls into the _ arm as Decision::Answer(payload), returning proposal prose to a tool-free request. Send ToolProposal to NeedsReducer so synthesis produces real text.
🐛 Proposed fix
- // Tool turns take the asymmetric actor path, so this text-path arbiter
- // never sees an executable ToolProposal (tool-shaped text is demoted to
- // Uncertainty by `enforce_tool_call_contract` when tools are disabled).
+ // Tool turns take the asymmetric actor path, so a `ToolProposal` here is
+ // tool-shaped text on a tools-disabled turn. Its payload is not an answer,
+ // so synthesize instead of returning the proposal prose verbatim.
match output.kind {
- OutputKind::Uncertainty => Decision::NeedsReducer {
+ OutputKind::Uncertainty => Decision::NeedsReducer {
reason: "single worker uncertain".into(),
},
+ OutputKind::ToolProposal => Decision::NeedsReducer {
+ reason: "single worker proposed a tool on a tool-free turn".into(),
+ },
_ => Decision::Answer(output.payload.clone()),
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| // Tool turns take the asymmetric actor path, so this text-path arbiter | |
| // never sees an executable ToolProposal (tool-shaped text is demoted to | |
| // Uncertainty by `enforce_tool_call_contract` when tools are disabled). | |
| match output.kind { | |
| OutputKind::ToolProposal if has_tools => { | |
| if let Some(ref name) = output.tool_name { | |
| Decision::ToolCall { | |
| name: name.clone(), | |
| arguments: output | |
| .tool_arguments | |
| .clone() | |
| .unwrap_or(Value::Object(Default::default())), | |
| } | |
| } else { | |
| Decision::Answer(output.payload.clone()) | |
| } | |
| } | |
| OutputKind::Uncertainty => Decision::NeedsReducer { | |
| reason: "single worker uncertain".into(), | |
| }, | |
| _ => Decision::Answer(output.payload.clone()), | |
| } | |
| // Tool turns take the asymmetric actor path, so a `ToolProposal` here is | |
| // tool-shaped text on a tools-disabled turn. Its payload is not an answer, | |
| // so synthesize instead of returning the proposal prose verbatim. | |
| match output.kind { | |
| OutputKind::Uncertainty => Decision::NeedsReducer { | |
| reason: "single worker uncertain".into(), | |
| }, | |
| OutputKind::ToolProposal => Decision::NeedsReducer { | |
| reason: "single worker proposed a tool on a tool-free turn".into(), | |
| }, | |
| _ => Decision::Answer(output.payload.clone()), | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/mesh-mixture-of-agents/src/arbiter.rs` around lines 455 - 463, Update
the match in single_output_decision so OutputKind::ToolProposal returns
Decision::NeedsReducer with the appropriate reducer reason, alongside
OutputKind::Uncertainty. Keep non-tool, non-uncertain outputs on the existing
Decision::Answer path.
Comment on lines
+541
to
+557
| // The reducer may legitimately rewrite the turn into prose; only | ||
| // assert when a tool call was emitted from worker proposals. | ||
| if tools.is_empty() || result.reducer_used { | ||
| continue; | ||
| } | ||
|
|
||
| let majority_val: Value = serde_json::from_str(&majority).unwrap_or(Value::Null); | ||
| for (name, args) in &tools { | ||
| let got: Value = serde_json::from_str(args).unwrap_or(Value::Null); | ||
| assert_eq!( | ||
| got, majority_val, | ||
| "case `{}`: tool `{name}` should use the majority arguments \ | ||
| ({majority_n} workers proposed {majority}), not a minority variant. \ | ||
| Got {args}", | ||
| case.id | ||
| ); | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
The replay harness still models the symmetric reducer path for tool turns. Tool-bearing turns now route to handle_tool_query, which packs the final request with pack_for_actor and always reports reducer_used: true. Both sites assume the old reducer fan-out shape, so the tool assertions no longer measure the shipped path.
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L541-L557: replace theresult.reducer_usedskip, which is alwaystrueforhas_toolscases and makes the majority-argumentassert_eq!unreachable.crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L146-L154: extendis_reducer_callto recognize thepack_for_actorsystem marker (## Advice from other models), so the actor call is answered withSYNTHESIZEDinstead of a replayed worker body.
📍 Affects 1 file
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L541-L557(this comment)crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L146-L154
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/mesh-mixture-of-agents/tests/sim_real_traces.rs` around lines 541 -
557, Update crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L541-L557 to
remove the result.reducer_used skip for tool-bearing turns so the
majority-argument assertions execute; update is_reducer_call at
crates/mesh-mixture-of-agents/tests/sim_real_traces.rs#L146-L154 to recognize
the pack_for_actor system marker “## Advice from other models” and return
SYNTHESIZED for the actor call instead of replaying a worker body.
Comment on lines
+31
to
+41
| text, tcs = oc.first_choice(resp) | ||
| ch = (resp.get("choices") or [{}])[0] | ||
| return { | ||
| "model": model, | ||
| "tier": oc.tier(model), | ||
| "elapsed": round(elapsed, 2), | ||
| "error": resp.get("error"), | ||
| "finish_reason": ch.get("finish_reason"), | ||
| "text": text, | ||
| "tool_calls": tcs, | ||
| } |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Persist the complete worker response.
The recorder discards usage and all unselected OpenAI response fields. This conflicts with evals/moa-openrouter/README.md, which requires full response preservation, including usage. Store resp in each worker record, or add the omitted fields and narrow the documented contract.
Proposed fix
return {
"model": model,
"tier": oc.tier(model),
"elapsed": round(elapsed, 2),
+ "response": resp,
"error": resp.get("error"),
"finish_reason": ch.get("finish_reason"),
"text": text,
"tool_calls": tcs,
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| text, tcs = oc.first_choice(resp) | |
| ch = (resp.get("choices") or [{}])[0] | |
| return { | |
| "model": model, | |
| "tier": oc.tier(model), | |
| "elapsed": round(elapsed, 2), | |
| "error": resp.get("error"), | |
| "finish_reason": ch.get("finish_reason"), | |
| "text": text, | |
| "tool_calls": tcs, | |
| } | |
| text, tcs = oc.first_choice(resp) | |
| ch = (resp.get("choices") or [{}])[0] | |
| return { | |
| "model": model, | |
| "tier": oc.tier(model), | |
| "elapsed": round(elapsed, 2), | |
| "response": resp, | |
| "error": resp.get("error"), | |
| "finish_reason": ch.get("finish_reason"), | |
| "text": text, | |
| "tool_calls": tcs, | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/record.py` around lines 31 - 41, Update the worker
record construction around the response handling to persist the complete OpenAI
response, including usage and unselected fields, by storing resp directly in
each record while retaining the existing derived fields as needed.
i386
reviewed
Aug 4, 2026
i386
left a comment
i386
left a comment
Collaborator
There was a problem hiding this comment.
One functional issue found in the committee-cap path.
i386
reviewed
Aug 4, 2026
i386
left a comment
i386
left a comment
Collaborator
There was a problem hiding this comment.
A second issue concerns the capacity signal that gates admission.
Collaborator
|
Follow-up on the tiering signal: parsing The better source is GGUF inspection: enumerate the GGUF tensors and sum their |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rs`:
- Around line 242-244: Update assemble_worker_pool and
self_fill_from_extra_instances to retain and pass the resolved alias group and
required_tokens. Query all aliases in that group when self-filling, and apply
the same remote-host context eligibility check as add_worker_backend before
adding each endpoint, so insufficient-context hosts are excluded. Add
regressions covering equivalent canonical aliases and hosts below the required
context capacity.
- Around line 19-29: Update resolve_one_worker_from_aliases to accept and use
the existing WorkerBackendResolution value for immutable resolution inputs,
removing its separate targets, aliases, and required_tokens parameters as
appropriate. Remove the unused local_count parameter and eliminate the
#[allow(clippy::too_many_arguments)] suppression, updating all call sites to
match.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: ba7e182b-4708-4854-986a-b213df4ea9cb
📒 Files selected for processing (3)
crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs
michaelneale
force-pushed
the
feat/moa-synthesis
branch
from
August 4, 2026 08:26
365c470 to
01d717d
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rs`:
- Around line 314-333: Ensure self-fill never duplicates an existing peer or
re-adds the local backend. Update add_worker_backend to return or store the
selected remote mesh::PeerId, collect peer identities already represented by the
current backends, and filter hosts_for_model(&name) in the self-fill loop before
creating each RemoteModelBackend; use the actual peer-id type from
RemoteModelBackend.
In `@evals/moa-openrouter/analyze_ablation.py`:
- Around line 175-196: Update the verdict logic in the analysis flow after the
B/C estimates to bootstrap a paired B−C differential using the same task
resamples, producing its own confidence interval. Base the “gain is CONTENT”
classification on that differential interval being entirely above zero, rather
than on b_lo; retain the existing B−A verdict and non-content-specific
comparison separately.
In `@evals/moa-openrouter/orclient.py`:
- Around line 74-76: Update the HTTPError handling in chat to decode the
response body defensively, falling back safely when the body is not valid UTF-8
so UnicodeDecodeError cannot escape the handler. Preserve the existing HTTP
status/detail formatting and structured error return behavior.
In `@evals/moa-openrouter/probe_tools.py`:
- Around line 27-31: Update each function schema in probe_tools.py, including
the schemas around the shown properties and the additional locations noted in
the review, to set additionalProperties to false alongside type, properties, and
required. Ensure every probe tool schema matches the production constraint
enforced by tool_guard.
- Line 79: Update the oc.chat call in the probe flow, including the
corresponding calls around the additionally affected lines, to pass
no_think=True. Preserve the existing model, messages, tools, token, and
temperature arguments so all workers and aggregators use the production MoA
thinking configuration.
- Line 199: Update the fanout_tool_result record call to store the complete
tool-result conversation by passing msgs_b as messages alongside the existing
results payload. Preserve the current record name and result data.
In `@evals/moa-openrouter/record_agentic.py`:
- Around line 131-140: Update consensus_tool to normalize valid JSON arguments
before counting tool-call proposals: parse each argument string, re-serialize
parsed objects with stable key ordering, and use that canonical form in the
consensus key. Preserve malformed arguments under a separate raw-string fallback
key, while keeping tool names and existing consensus selection behavior
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 80edd6cb-badd-48c4-9245-c592678da89a
📒 Files selected for processing (40)
crates/mesh-llm-guardrails/src/rescue.rscrates/mesh-llm-host-runtime/src/network/openai/ingress.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/pool.rscrates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rscrates/mesh-mixture-of-agents/src/arbiter.rscrates/mesh-mixture-of-agents/src/backend.rscrates/mesh-mixture-of-agents/src/context.rscrates/mesh-mixture-of-agents/src/fanout.rscrates/mesh-mixture-of-agents/src/lib.rscrates/mesh-mixture-of-agents/src/normalize.rscrates/mesh-mixture-of-agents/src/reducer.rscrates/mesh-mixture-of-agents/src/refinement.rscrates/mesh-mixture-of-agents/src/tool_guard.rscrates/mesh-mixture-of-agents/src/tool_turn.rscrates/mesh-mixture-of-agents/src/worker.rscrates/mesh-mixture-of-agents/tests/eval_openrouter.rscrates/mesh-mixture-of-agents/tests/fixtures/ablation_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/committee_tasks.jsoncrates/mesh-mixture-of-agents/tests/fixtures/real_traces.jsoncrates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rscrates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rscrates/mesh-mixture-of-agents/tests/sim_partial_worker_survival.rscrates/mesh-mixture-of-agents/tests/sim_real_traces.rscrates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rscrates/mesh-mixture-of-agents/tests/sim_strong_patience.rscrates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rscrates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rscrates/mesh-mixture-of-agents/tests/sim_worker_accounting.rsevals/moa-openrouter/README.mdevals/moa-openrouter/RESULTS.mdevals/moa-openrouter/agentic.jsonlevals/moa-openrouter/analyze_ablation.pyevals/moa-openrouter/corpus.jsonlevals/moa-openrouter/fanout.jsonlevals/moa-openrouter/make_fixture.pyevals/moa-openrouter/orclient.pyevals/moa-openrouter/probe_tools.pyevals/moa-openrouter/record.pyevals/moa-openrouter/record_agentic.py
🚧 Files skipped from review as they are similar to previous changes (31)
- crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs
- crates/mesh-mixture-of-agents/src/tool_guard.rs
- crates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rs
- crates/mesh-mixture-of-agents/src/reducer.rs
- crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs
- evals/moa-openrouter/fanout.jsonl
- crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs
- crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs
- crates/mesh-mixture-of-agents/src/normalize.rs
- crates/mesh-mixture-of-agents/tests/sim_partial_worker_survival.rs
- crates/mesh-mixture-of-agents/tests/fixtures/committee_tasks.json
- crates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs
- crates/mesh-mixture-of-agents/tests/fixtures/ablation_tasks.json
- crates/mesh-mixture-of-agents/src/worker.rs
- crates/mesh-mixture-of-agents/src/backend.rs
- crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs
- crates/mesh-llm-host-runtime/src/network/openai/ingress.rs
- crates/mesh-mixture-of-agents/tests/fixtures/real_traces.json
- evals/moa-openrouter/agentic.jsonl
- crates/mesh-mixture-of-agents/tests/sim_real_traces.rs
- crates/mesh-mixture-of-agents/src/refinement.rs
- evals/moa-openrouter/corpus.jsonl
- crates/mesh-mixture-of-agents/tests/sim_refinement_mesh_conditions.rs
- crates/mesh-mixture-of-agents/src/tool_turn.rs
- crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/workers.rs
- crates/mesh-mixture-of-agents/src/context.rs
- crates/mesh-llm-guardrails/src/rescue.rs
- crates/mesh-mixture-of-agents/tests/eval_openrouter.rs
- crates/mesh-mixture-of-agents/src/lib.rs
- crates/mesh-mixture-of-agents/src/fanout.rs
- crates/mesh-mixture-of-agents/src/arbiter.rs
Comment on lines
+175
to
+196
| b_pt, b_k = point_estimate(tasks, "B") | ||
| c_pt, c_k = point_estimate(tasks, "C") | ||
| b_lo, b_hi = bootstrap_ci(tasks, "B", args.iters, args.seed) | ||
| c_lo, c_hi = bootstrap_ci(tasks, "C", args.iters, args.seed) | ||
|
|
||
| print(" net uplift = P(rescue) - P(harm), equal-weight mean over tasks") | ||
| print(f" B (real) uplift {b_pt:+.3f} 95% CI [{b_lo:+.3f}, {b_hi:+.3f}] (tasks n={b_k})") | ||
| print(f" C (shuffled) uplift {c_pt:+.3f} 95% CI [{c_lo:+.3f}, {c_hi:+.3f}] (tasks n={c_k})") | ||
| print(f" differential B-C: {b_pt - c_pt:+.3f} (content effect beyond token/prompt effect)") | ||
| print() | ||
|
|
||
| # Verdicts (directional; the CI is what matters for a claim). | ||
| if b_lo > 0: | ||
| print(" => references HELP: B net uplift CI is entirely > 0") | ||
| elif b_hi < 0: | ||
| print(" => references HARM: B net uplift CI is entirely < 0") | ||
| else: | ||
| print(" => inconclusive: B net uplift CI spans 0") | ||
| if b_pt - c_pt > 0 and b_lo > 0: | ||
| print(" => and the gain is CONTENT (B > C), not just extra tokens/prompt") | ||
| elif abs(b_pt - c_pt) < 0.02: | ||
| print(" => gain (if any) is NOT content-specific (B ~ C)") |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Bootstrap the B−C differential before classifying a content effect.
Line 193 tests b_lo > 0, but b_lo is the lower confidence bound for B−A. It is not a confidence bound for B−C. This condition can report a content-specific gain when B−C is statistically uncertain.
Use the same paired task and draw resamples to calculate a B−C confidence interval. Base the content verdict on that interval.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/analyze_ablation.py` around lines 175 - 196, Update the
verdict logic in the analysis flow after the B/C estimates to bootstrap a paired
B−C differential using the same task resamples, producing its own confidence
interval. Base the “gain is CONTENT” classification on that differential
interval being entirely above zero, rather than on b_lo; retain the existing B−A
verdict and non-content-specific comparison separately.
| """ | ||
|
|
||
| def one(model): | ||
| resp, elapsed = oc.chat(model, messages, tools=tools, max_tokens=max_tokens, temperature=0.8) |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Use the production MoA thinking setting.
oc.chat defaults no_think to False. Its contract states that no_think=True mirrors the MoA default. These calls can therefore measure reasoning-enabled workers and aggregators instead of the production configuration. Pass no_think=True; orclient.chat already retries without the flag for providers that require reasoning.
Proposed fix
- resp, elapsed = oc.chat(model, messages, tools=tools, max_tokens=max_tokens, temperature=0.8)
+ resp, elapsed = oc.chat(
+ model, messages, tools=tools, max_tokens=max_tokens, temperature=0.8, no_think=True
+ )
...
- aggregator, msgs, tools=TOOLS if with_tools else None, max_tokens=384, temperature=0.3
+ aggregator, msgs, tools=TOOLS if with_tools else None,
+ max_tokens=384, temperature=0.3, no_think=TrueAlso applies to: 135-137
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/probe_tools.py` at line 79, Update the oc.chat call in
the probe flow, including the corresponding calls around the additionally
affected lines, to pass no_think=True. Preserve the existing model, messages,
tools, token, and temperature arguments so all workers and aggregators use the
production MoA thinking configuration.
| }, | ||
| ] | ||
| res_b = summarize(fan_out(msgs_b, TOOLS), "B. TOOL-RESULT TURN (agentic step 2)") | ||
| record("fanout_tool_result", results=res_b) |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Record the tool-result input conversation.
This record contains only results. It omits the user prompt, assistant tool call, tool-call ID, and tool result in msgs_b. A replay corpus cannot reconstruct this tool-result turn. Store messages=msgs_b with the result.
Proposed fix
- record("fanout_tool_result", results=res_b)
+ record("fanout_tool_result", messages=msgs_b, results=res_b)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| record("fanout_tool_result", results=res_b) | |
| record("fanout_tool_result", messages=msgs_b, results=res_b) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/probe_tools.py` at line 199, Update the
fanout_tool_result record call to store the complete tool-result conversation by
passing msgs_b as messages alongside the existing results payload. Preserve the
current record name and result data.
Comment on lines
+131
to
+140
| def consensus_tool(workers): | ||
| """Most-proposed (name, arguments) across workers, or None.""" | ||
| counts = {} | ||
| for w in workers: | ||
| for c in w.get("tool_calls") or []: | ||
| key = (c["function"]["name"], c["function"].get("arguments") or "{}") | ||
| counts[key] = counts.get(key, 0) + 1 | ||
| if not counts: | ||
| return None | ||
| return max(counts.items(), key=lambda kv: kv[1])[0] |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Normalize JSON arguments before counting tool-call consensus.
Equivalent argument objects can use different key order or whitespace. The current raw-string key splits those votes. A lower-support proposal can then control the canned observation and all later recorded steps.
Parse valid argument JSON and serialize it with stable key ordering before counting proposals. Keep malformed arguments in a separate raw fallback key.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@evals/moa-openrouter/record_agentic.py` around lines 131 - 140, Update
consensus_tool to normalize valid JSON arguments before counting tool-call
proposals: parse each argument string, re-serialize parsed objects with stable
key ordering, and use that canonical form in the consensus key. Preserve
malformed arguments under a separate raw-string fallback key, while keeping tool
names and existing consensus selection behavior unchanged.
…laky peers
Agentic MoA turns now reliably produce tool calls, and diverging answers are
synthesized instead of relaying one arbitrarily-chosen worker's text.
Fixes found by replaying recorded traces from 9 open-weight models:
* Tools were withheld from workers. `query_uses_tools` was derived from
`looks_like_tool_intent`, an English keyword match on the user's text. "The
test suite is failing. Find out which test fails and why" matches nothing, so
workers were dispatched without tool schemas and the arbiter ran with
has_tools=false — a unanimous tool proposal then fell through to the answer
path and leaked "calling search" as prose. 5 of 10 recorded scenarios hit
this. Tool availability is now the caller's declaration.
* Argument outliers could win a unanimous tool call. Native tool calls are all
normalized to a fixed 0.9 confidence, so the confidence-only tiebreak could
not separate 8 workers proposing {"path":"src"} from one hallucinating
{"path":"rust_project/src"}. Added argument clustering (key-order
independent); the largest cluster wins and argument-free calls never outvote
filled-in ones.
* Early exit committed on tied arguments. Early exit runs on whoever has
arrived, so two fast models agreeing on a tool name with different arguments
was a coin flip that also aborted the workers who would have broken the tie.
It now waits.
* Truncation was invisible. `finish_reason` was never read; 39/140 recorded
responses came back "length" and 24 carried partial text, which parsed as a
normal answer and could be returned verbatim. Plumbed through as
BackendReply.truncated -> WorkerOutput.truncated; such answers are barred
from consensus and verbatim output but still feed synthesis, labelled
incomplete.
* A strict endpoint could kill a worker permanently. minimax-m2.5 returns
HTTP 400 "Reasoning is mandatory for this endpoint" and failed 12/12
requests; HttpBackend now drops the thinking flags and retries once.
Policy changes:
* Thinking is always off for MoA workers, not merely defaulted off. The
previous escape hatch only let callers request the broken configuration:
qwen3-32b spent 408 reasoning tokens against a 384-token cap and returned
null content. Ignored overrides are logged.
* Diverging answers go to the reducer. Returning the top-confidence answer was
near-arbitrary because models rarely emit our confidence envelope, so
everything defaults to 0.5 and max_by returns whichever worker finished
first. Agreement still short-circuits without paying for synthesis.
* Reducer prompt adopts Together's aggregator framing (synthesize, critically
evaluate, agreement is not proof), keeping our per-worker attribution,
structured tool proposals, and 500-char payload bound.
Tests: new sim_real_traces.rs replays 56 recorded cases from 9 models through
handle_turn, plus regression unit tests for each bug. Recorders and corpora
live in evals/moa-openrouter/.
…r acts Tool-bearing turns now use the Hermes/Nous-style asymmetric shape instead of a majority vote across workers: - references run TOOL-FREE and only advise in prose - the single best tool-caller (the "actor") acts on that advice with the real tools and emits the tool call Tool authority now tracks capability, not popularity. The old path let several weak models proposing a popular-but-wrong tool outvote the one strong tool-caller that picked correctly (observed: qwen3-32b alone chose run_command for a failing-test triage while the smaller models chose list_dir, and the vote shipped list_dir). An actor model removes that failure class instead of patching the vote arithmetic. Stays a pure stateless /v1/chat/completions turn — references are regenerated from the caller's transcript each request, and the external client still owns tool execution. Text-only turns are unchanged (symmetric fan-out + synthesis-on-divergence). Actor selection is capability-first: the host ranks candidates by gossiped `tool_use` level (Supported > Likely > None), then model size tier, then stable order, and passes the ordering to the engine via the new `GatewayConfig.actor_candidates`. Empty ordering falls back to the engine's name-derived size tier, so existing callers and tests are unaffected. Mesh guardrails for mixed/public meshes: references are gathered with a bounded wait (proceed at a majority of advisors, never block on the slow tail), and the actor is called through the existing hedged ladder so a slow/broken best candidate falls through to the next tool-capable peer. New: - crates/mesh-mixture-of-agents/src/tool_turn.rs — asymmetric tool-turn handler - context::pack_for_actor — "advise, then act" framing (vs synthesis framing) - fanout::gather_references — bounded, tool-free reference collection - GatewayConfig.actor_candidates + reducer_candidates honours it - host compute_actor_candidates from gossiped tool_use + size The dead majority-vote code in the arbiter (now unreachable — tool turns bypass it) is removed in the follow-up commit. Tests: eval_openrouter.rs adds a live OpenRouter harness (ignored by default) that runs the real handle_turn over 6 open-weight models with mesh-realism latency/failure injection, for benchmarking asymmetric MoA vs single models. 192 engine tests pass; clippy -D warnings and fmt clean on both crates.
Tool turns now take the asymmetric actor path (previous commit), so the arbiter's tool-proposal voting is unreachable — and it *was* the majority-of-weakness bug: a popular-but-wrong tool choice from several weak models could outvote the one strong tool-caller that picked correctly. Removes it rather than leaving dead code: - delete best_tool_proposal, best_tool_proposal_by_consensus, and decisive_argument_cluster - drop the tool-arbitration and tool-vs-answer branches from arbitrate, and the tool-consensus branch from try_early_decision - drop the now-unused has_tools parameter from arbitrate, try_early_decision, single_output_decision, and gather_workers_incremental - the arbiter is now purely an answer/critique/uncertainty arbiter; tool-shaped text on the text path is already demoted to Uncertainty by enforce_tool_call_contract (tools disabled ⇒ empty allow-list) - drop the arbiter tests that pinned the removed tool-vote behavior; keep and re-point every answer-consensus, truncation, synthesis, and tier-gate test No behavior change for text turns. 178 engine tests pass; host-runtime lib tests (1851) pass; clippy -D warnings and fmt clean on mesh-mixture-of-agents, mesh-llm-host-runtime, and mesh-llm.
Replaces the confounded "MoA vs best single model" comparison with a within-actor ablation that isolates the one variable that matters: the references. One pinned actor, identical sampling / token budget / system prompt across all arms; only the advice changes: A. actor alone (no references) B. actor + real references (the production tool path) C. actor + shuffled references (advice from a different task, length-similar) Metric: rescue (A✗→B✓) minus harm (A✓→B✗). Arm C is the key control — it separates "useful information" from "extra tokens + a think-carefully prompt". Why the change: the earlier best-single comparison ran solo models at different sampling (0.8/512) than the actor (0.3/2048) and with a different system prompt, and scored transient 429/502/504 and tool-unsupported endpoints as capability failures. Those numbers measured sampling+prompt+infra, not the design, and are not citable. This ablation removes all of that by construction: the same scorer hits all three arms of the same actor, so an imperfect label largely cancels in the rescue-minus-harm delta. Also: - chat_completion_retrying: predeclared retry on transient infra errors, which are then excluded (∅) from the capability analysis rather than scored as wrong answers. - MOA_ABLATION_ACTOR / MOA_ABLATION_DRAWS env overrides. - the old best-single test is kept but documented as a confounded smoke test, not evidence. Pilot findings (5 draws, 4 tasks, ignored/live): - Strong actor (qwen3-32b): 20/20 all arms — aces every task alone, so references are inert-but-harmless (no headroom to measure rescue). - Weak actor (qwen3-8b): A 15/20, B 20/20, C 15/20 — net uplift +5, all on the one task with headroom (triage). Real advice rescued the weak actor every draw; SHUFFLED advice did not (C=A), so the gain is advice CONTENT, not token count or the decision prompt. Zero harm. Reading: reference value tracks actor-alone headroom — a safety net when the actor is weak, dead weight (not damage) when strong. Suggests gating the reference phase on actor strength. This is a directional pilot, not the merge-blocking study (that needs ~40 preregistered stratified tasks, k>=10 draws, a paired hierarchical bootstrap CI, and the production-selected actor).
…yzer The defensible version of the pilot, committed BEFORE running so the labels are preregistered and can't be seen to chase results. - tests/fixtures/ablation_tasks.json: 40 preregistered tasks, 4 strata x 10 (inspect / search / execute / no_tool). Set-valued accept labels (accept_tools is a SET; empty = "no tool call"), with optional arg substring constraints — addresses the brittle single-tool label from the pilot. - ablation_scaled_study: loads the fixture, runs A/B/C arms at k draws (default 10) with bounded concurrency, writes one JSONL trial per (draw,task,arm) to MOA_ABLATION_OUT. Live measurement only; no stats here. - evals/moa-openrouter/analyze_ablation.py: deterministic paired HIERARCHICAL bootstrap — resample tasks within stratum, then draws within task — for a 95% CI on net uplift = P(rescue) - P(harm), plus the shuffled-arm control (content effect = B_uplift - C_uplift). Validated on synthetic data: 3/40 rescue tasks -> +0.075 point estimate, shuffled +0.000, and the CI honestly spans 0 when rescue is sparse (no over-claiming). Arms (one pinned actor, identical sampling/prompt; only references vary): A actor alone · B actor + real references · C actor + shuffled references. Env: MOA_ABLATION_ACTOR, MOA_ABLATION_DRAWS, MOA_ABLATION_CONCURRENCY, MOA_ABLATION_OUT. Ignored by default (live network + cost).
Cut the essay-length doc/inline comments added with the actor design down to the non-obvious "why". No code change. - tool_turn.rs: 26-line module preamble -> 6 lines; inline comments tightened (20% -> 10% comment density) - context.rs: pack_for_actor docstring 16 -> 5 lines; reducer synthesis-framing block 14 -> 6 lines - workers.rs: compute_actor_candidates docstring 21 -> 6 lines (kept the terse sort-key comments, which are load-bearing)
Two more ablation arms to answer "can peers help tool selection at all?", reusing the A/B/C JSONL schema + analyze_ablation.py. matched_peer_structured_study — do similar-strength, different-family peers help a fixed finalizer via STRUCTURED candidate tool calls (not prose)? A solo · B diverse (2 different-family peers) · C homogeneous (2 resamples of the finalizer's own model). B-C isolates cross-family diversity from extra sampling; also records oracle-union. correction_rescues_weak_tool_caller — the mesh scenario neither Hermes nor Together handles: a weak tool-caller with no strong peer. Tests correction of the CONCRETE drafted call instead of pre-hoc advice. A draft-alone · B deterministic (schema-validate + re-prompt on structural failure) · C semantic (different-family critic reviews the concrete call, finalizer revises once). Findings (live, directional): every tool-selection intervention was inert-to-harmful vs routing to a capable model. Pre-hoc structured proposals were flat (37-39/40 all arms). Deterministic correction fired ~never (qwen3-8b already drafts structurally valid calls ~95%); the residual failures are semantic (wrong tool choice), which neither validation nor a strong critic fixed because the revision still runs through the weak actor. Conclusion: tool selection is a "best capable model acts" task; MoA's value is on the answer/chat path (untested).
Tests where MoA's value should live per Together's validated claim: open-ended answer QUALITY on realistic agent-session turns (reason-over-tool-output, planning, explanation) — not tool selection. Fixed aggregator; only its input varies: A alone · B committee (synthesize 3 diverse-peer drafts, 1 round) C layered (peers refine seeing each other first, then synthesize — Together's `layers`) Judged pairwise by an out-of-pool different-family judge (gpt-4o-mini), position-swapped (win only if consistent both orders), output lengths logged. - tests/fixtures/committee_tasks.json: 15 realistic reasoning/answer turns - committee_beats_solo_on_reasoning: writes per-trial JSONL Pilot findings (2 draws, live; DIRECTIONAL, underpowered): - committee(B) vs solo(A): win 6 / tie 2 / loss 2 — positive but sign-test p=0.29, NOT significant at n=10. Every B win was also the longer answer, so length is not cleanly ruled out. - layered(C) vs committee(B): loss 6 / tie 2 / win 2 — Together's extra refinement round is negative value on these tasks, at extra cost. - 20/30 trials skipped because the aggregator (qwen3-32b) returned empty content (reasoning-budget exhaustion — the content:null bug Hermes' troubleshooting.md documents). A flaky aggregator degrades the committee; the instrument needs empty-output handling before a real run. Net: first directionally-positive result for MoA in this investigation, but unproven. Contrast with 5 tool-selection experiments that were all null-to-harmful. Supports the task-split: route tools to one caller, convene a (single-round) committee on reasoning turns.
…l transcript Our references were packed very differently from Hermes', and it was measurably costly. Adds `pack_for_reference`, which gives advisors only the conversation's user/assistant prose: - strips the agent system prompt (an advisor told "you are a coding agent, run the tests" role-plays the actor instead of advising it) - strips the tool transcript (prior tool_calls + results anchored every advisor on the trajectory already taken, collapsing the error-independence that makes aggregation worth anything) - drops the "respond with your best answer or tool call" instruction: advisors hold no schemas, so asking for a tool call yields tool-shaped prose — exactly the advice that pulled the actor off its own better choice - uniform view across advisors (no per-role trimming), so the packing is a stable function of history and caches across iterations - caps advisor output at 600 tokens (advisor generation dominates turn latency; the turn waits for the slowest advisor) Head-to-head on the same preregistered study (strong actor qwen3-32b, 40 tasks x 10 draws, identical everything except packing): packing B pass net uplift 95% CI original 359/400 -0.102 [-0.170, -0.045] hermes 385/400 -0.037 [-0.090, -0.003] Harm cut by ~64%. Per-category: search -14 -> -2, execute -25 -> -13, inspect -2 -> 0. The content-specific component (B-C differential) fell from -0.075 to -0.015, i.e. with correct packing the residual harm is no longer mostly "bad advice content". Honest read: most of the harm I previously attributed to "references" was an artifact of how WE packed them, not a property of reference-based MoA. The direction still stands though — even correctly packed, references remain a small but statistically real regression for a STRONG actor on tool selection (CI still entirely below zero). Adds 4 unit tests pinning the packing contract (no tool-call request, no system prompt leak, no tool-transcript leak, prose preserved), and MOA_REFERENCE_PACKING=hermes to select the style in the eval harness.
…t failed Writes up every live study in one place (evals/moa-openrouter/RESULTS.md) so the conclusions and their caveats survive the investigation. Headline 2x2 (40 preregistered tasks x 10 draws, paired hierarchical bootstrap): actor packing B pass net uplift 95% CI strong original 359/400 -0.102 [-0.170, -0.045] <- the bug strong hermes 385/400 -0.037 [-0.090, -0.003] weak original 365/400 -0.013 [-0.090, +0.070] weak hermes 377/400 +0.017 [-0.053, +0.100] Two monotonic effects: fixing the packing helps in both actor conditions, and references are worth more to a weaker actor. The only significant cell is the original-packing strong-actor harm — i.e. our bug, not a property of MoA. Per-stratum (weak + hermes) shows references help exactly where the actor has headroom (search +10, execute +4) and hurt where it was already perfect (inspect -7). That is a gating signal. Also records what did NOT work for tool selection (pre-hoc structured proposals: flat; deterministic correction: never fires, the weak actor already emits valid calls ~95%; semantic correction: negative, since the revision still runs through the weak actor), and the committee pilot on reasoning turns (directionally positive, n=10, not significant; Together's layering loses to single-round).
The Hermes-style packing was measured and unit-tested but only wired into the eval harness — the production tool path still used pack_for_worker_selected, the exact packing measured at -0.102 net uplift. tool_turn now calls pack_for_reference: conversation prose only, no agent system prompt, no tool transcript, no request for a tool call. Same head-to-head (strong actor, 40 tasks x 10 draws) puts this at -0.037 vs -0.102, and it is the only configuration where references show a positive point estimate for a weak actor (+0.017). See evals/moa-openrouter/RESULTS.md.
Collaborator
Author
|
@i386 thanks - have it reading real data now |
Collaborator
Author
|
@i386 — fixed the tiering P1s you flagged (name-based tier / unparseable-name-as-big driving destructive admission). Now exactly your prescription: authoritative size from GGUF tensor sum, gossiped via the existing Commit: 1. Producer computes the real size — pub fn scan_gguf_total_parameters(path: &Path) -> Option<u64> {
let GgufHeader { file: mut f, n_tensors, n_kv } = open_gguf_header(path)?;
skip_all_kv_pairs(&mut f, n_kv)?;
let mut total: u64 = 0;
for _ in 0..n_tensors {
let _name = read_gguf_string(&mut f).ok()?;
let n_dims = read_u32(&mut f).ok()?;
if n_dims > MAX_GGUF_TENSOR_DIMS { return None; }
let mut elements: u64 = 1;
for _ in 0..n_dims {
let dim = read_u64(&mut f).ok()?;
elements = elements.checked_mul(dim)?;
}
let _ggml_type = read_u32(&mut f).ok()?;
let _offset = read_u64(&mut f).ok()?;
total = total.checked_add(elements)?;
}
Some(total)
}2. Producer prefers it, falls back to name — 3. Consumer tiers off the gossiped value — enum SizeTier { Small, Big, Unknown }
fn tier_for(name: &str, sizes: &HashMap<String, f64>) -> SizeTier {
if let Some(b) = sizes.get(&canonical_base_name(name)) {
return if *b < SMALL_TIER_MAX_B { SizeTier::Small } else { SizeTier::Big };
}
match mesh_llm_guardrails::model_param_size_b(name) { // name-parse fallback
Some(b) if (b as f64) < SMALL_TIER_MAX_B => SizeTier::Small,
Some(_) => SizeTier::Big,
None => SizeTier::Unknown, // missing = Unknown, not big
}
}Admission excludes only verified Verified live on a 2-node mesh (MacBook 3B + mini Qwen3.5-4B): gossiped sizes went from a bogus name-parsed 4910 to real 3.21 / 4.21, and admission now no-ops on unknown-size workers. Two new unit tests: Note deliberately deferred to follow-up: MoE active-parameter signal (total tensor count is the safety gate for now, as you said). Does this match what you had in mind? |
Collaborator
I think the fallback to name should be removed. if you cant get this info from gguf, the model must be ranked as the lowest param model. As would be worth chucking anything in your cache at this and checking if that tensor counts matches the name to validate. |
Per i386: parsing NNb from a served name is brittle and a destructive admission decision must not rest on an unverified label. Removed the name fallback entirely. Producer (profile.rs): parameter_count_b comes ONLY from the GGUF tensor sum. No local GGUF to sum ⇒ None (no guessed count). Deleted the now-dead parameter_count_b_from_text + its test. Consumer (pool.rs): tier_for uses only the gossiped verified size. Dropped SizeTier::Unknown — a model with no verified size is Small (weakest), so an unverifiable label can never masquerade as big and displace a real strong worker. cap_committee ranking updated (no-size ranks last). Admission excludes only verified-small; with no verified big there is nothing to protect so an all-unsized pool is untouched. Tests updated: unsized-worker-is-weakest (excluded next to verified big), no-verified-sizes keeps all. 71 moa_gateway + 28 profile tests pass; clippy -D warnings and fmt clean.
Collaborator
MeshLLM PR 1116 Homelab Validation Report
General FindingsPR 1116's MoA machinery works through the shipped OpenAI HTTP path on a real The PR is nevertheless NOT_READY for its headline claim that the shipped Two additional release defects were found: repository Scope And ExclusionsValidated:
Canonical Change Inventory And Validation Ledger
Environment And Build ProvenanceAll worktrees were created at
Product Validation Results
Private Mesh EvidenceFinal topology (left running):
API, Logs, UI, And Inference FindingsThe same-model early-exit request produced 2 worker summaries, matching the The tool request selected one actor and returned structurally correct arguments. No final log contained a structured Defects And Anomalies
Risk And Follow-Up Register
Final Gate Assessment
Sign-OffDecision: NOT_READY for merge/release on the stated quality-improvement |
Self-fill could re-add the same physical node (or an endpoint already backing the sole worker), turning ONE box into a fake 2-worker committee that hits it twice for near-identical drafts. Catastrophic for mesh mode: a lone node must degrade to single-model serving, never pretend to be a committee. Rewrote self_fill_from_extra_instances to build the pool from DISTINCT physical endpoints only: the local skippy port (if this node serves the model and context fits) plus each distinct remote peer from hosts_for_model. If fewer than two distinct endpoints serve the model, the pool stays the single worker and MoA degrades to single-model. A single endpoint can no longer appear twice. Also fixes the CodeRabbit self-fill bugs (re-adds same peer; skipped context eligibility) — context fit is now checked on the local endpoint, and endpoints are distinct by construction. 71 moa_gateway tests pass; clippy -D warnings and fmt clean.
A `model=mesh` request with no `messages`, a non-array `messages`, or an empty
array fell through to the workers and fabricated a 200 answer from nothing
(homelab validation API-1). try_handle_moa now requires a present, non-empty
`messages` array and returns 400 otherwise, before any model call.
Verified live: {"model":"mesh"}, string messages, and [] all now 400; a valid
request still returns 200. 71 moa_gateway tests pass; clippy -D warnings + fmt
clean.
Collaborator
Author
|
@ndizazzo the cuda thing seems unrelated, and can you try that again with non trivial model as i can't see that model yielding much at all, so probably results are too noisy. |
Collaborator
Author
|
@ndizazzo yeah the claim is only for larger models combining (20B ish) unfortunately, so need to test with that (still probing myself). Also want to make sure this improves over the main branch. but this reminds me to try it with larger combos... so standby.. (in the mean time feel free to try it with larger - and some subsequent fixes) |
Width sprint (evals/moa-openrouter, aggregator qwen3-8b, 8B peers, shipped committee path) shows the fan-out cap was throttling the pools that benefit most: 6× diverse 8B vs best member: 12W/65T/2L, p=0.013 (wins) 4× diverse 8B: 5W/73T/0L, p=0.06 (marginal) 2× diverse 8B: 2W/77T/1L, p=1.0 (null) Small, weak drafts need WIDTH — more independent proposals — before aggregation clears the best member. A verified big model, by contrast, wins at 2 and gains nothing past ~4. - committee_cap is now tier-aware: COMMITTEE_CAP_SMALL=6 for all-small pools, COMMITTEE_CAP_BIG=4 when a verified big model is present (replaces the flat MAX_COMMITTEE_WORKERS=4). - eval harness: the three per-trial judge comparisons now run concurrently (tokio::join!) instead of three serial awaits — ~3x faster judging phase, no behaviour change. Also measured (recorded in RESULTS.md): the refine round never beats single aggregation at 8B (refine-vs-single null in every cell) — Hermes' cheaper single-synth cadence matches Together's layered shape here; and at 8B diversity matters (6 diverse >> 6 same), unlike mid-scale. 72 moa_gateway tests (new: committee_cap_is_wide_for_small_pools_tight_for_big); clippy -D warnings and fmt clean.
Collaborator
Author
Update: width sprint + review fixes (since
|
| pool | single-agg vs best member | refine vs single-agg |
|---|---|---|
| 2× 8B diverse | 2W/77T/1L, p=1.0 | null |
| 4× 8B diverse | 5W/73T/0L, p=0.06 | null |
| 6× 8B diverse | 12W/65T/2L, p=0.013 | null |
| 6× 8B same | 4W/75T/1L, p=0.38 | null |
Three results:
- Six diverse 8B models beat the best single member (p=0.013). The old flat
MAX_COMMITTEE_WORKERS=4throttled exactly the small pools that need width. Cap is now tier-aware: 6 for all-small, 4 when a verified big model is present (a 24–32B pair already wins at 2). - The refine round never earns its serial cost —
refine vs single-aggis null in every cell. Hermes' single-aggregation cadence matches Together's layered shape here at half the latency. (Noted for follow-up; not changed in this PR.) - At 8B, diversity matters (6 diverse 12W/2L ≫ 6 same 4W/1L) — unlike mid-scale where Self ≈ Mixed.
Review fixes pushed
- i386 P1 ×2: tiering now uses verified GGUF tensor-sum size gossiped via
parameter_count_b(not name parsing);cap_committeeranks by size, nottool_use. Name fallback removed per follow-up — no verified size ⇒ ranked weakest. Validated: cached 3B→3.21, 4B→4.21;gemma-4-E4Bname says 4B but sums to 7.5B (proves the name heuristic was unsafe). - Iron law: self-fill rebuilt to use distinct physical endpoints only — a single node can never fake a 2-worker committee; it degrades to single-model. (Also closes the two CodeRabbit self-fill bugs.)
- API-1:
model=meshwith missing/non-array/emptymessagesnow returns 400, not a fabricated 200. Verified live. - CodeRabbit:
pool.rsextracted; eval judges parallelised.
Still open (agreed)
- Capable-model (24–32B) shipped-path run for the headline quality claim — length-controlled single judge (AlpacaEval-style), needs paid runs, separate effort.
- aarch64
check-releaseparity + CUDA-toolkit-default: pre-existing, not touched by this branch (check-releasepasses on this HEAD; branch changes no release/packaging code).
Full data + method in evals/moa-openrouter/RESULTS.md.
…is dead cost The width sprint measured refine-vs-single-aggregation as null in every 8B cell (2/4/6 workers, diverse and same). Small pools win by WIDTH under a single aggregation, not by the extra serial refine pass. Previously RefinementPolicy:: Auto ran the round for ALL-small pools — paying two synthesis passes for no measured gain. Auto now refines only for a homogeneous pool that is NOT all-small (i.e. same-model at real scale, where correlated drafts + the round measured 48/2 vs 35/10). All-small and diverse pools skip it — matching Hermes' cheaper single-synth cadence, where refine buys nothing. Effect: an all-small mesh turn drops from 2 serial synthesis passes to 1, roughly halving added latency, with no measured quality loss. Tests: auto_skips_an_all_small_pool (was auto_refines_...); the 5 all-small mechanics sim tests (straggler/grace/degradation) pinned with Always so they still exercise the round; big-pool gate tests keep Auto. 178 + sim tests pass; clippy -D warnings and fmt clean.
Collaborator
Author
|
@ndizazzo — thanks for the thorough homelab validation. Re-review requested; here's what changed since API-1 (S2, schema too permissive) — fixed. MOA-6 (admission only unit-tested) — the tiering it depends on is now hardened: verified GGUF tensor-sum size gossiped via MOA-8 (S1, shipped quality parity) — the stale On the 0.6B topology: that model is an order of magnitude below the smallest in our data (8B), and our committed table shows an 8B floor (2–3× is null, 4× marginal, 6× wins). So the null you observed at 0.6B is the expected result, not a design signal — a fresh width sprint (aggregator qwen3-8b, 8B peers, shipped committee path) shows 6× diverse 8B beats the best member, 12W/2L p=0.013; the fan-out cap is now tier-aware (6 small / 4 big) so small meshes aren't throttled, and the refine round is skipped where it measured null. aarch64 Branch is merged current with main and CI is green. Full data/method in |
Three shipped-path bugs meant `handle_turn` never ran the committee it was
supposed to. Found by measuring the shipped entrypoint at capable scale.
1. Grace pre-empted synthesis. `first_answer_grace` armed on the FIRST answer
and finalized the turn, so on fast backends it fired ~every turn and shipped
one worker's text. Measured 80/80 EarlyExit at 32B scale. Now grace
finalizes on TOOL turns only; on answer turns it is a collection deadline —
stop waiting for the tail, then synthesize what arrived. Private-mesh window
widened 3s -> 10s so a normal committee completes before it arms.
2. Role-tiered draft budgets (Fast 256 / Specialist 512) truncated the drafts
that synthesis consumes. Those tiers existed only to make the grace
fast-path cheap; grace no longer finalizes answer turns, so every answer-turn
worker now gets the full budget.
3. Answer turns shipped one worker verbatim when drafts agreed. Agreeing drafts
are the best input to synthesis, not a reason to skip it. Answer turns with
>=2 workers always synthesize now. Also drops the worker preamble from the
reducer prompt — the synthesizer is not one of the parallel answerers, and
the contradictory framing cost a weak aggregator.
Measured through `moa::handle_turn` at production defaults, 40 prompts x 2
draws, out-of-family judge, length-controlled:
32B/24B/35B-MoE/14B pool vs best member: 71W / 8T / 1L p<0.0001
(was 26W/14T/40L before these fixes; wins 28/29 even when MoA is SHORTER,
so it is not a length artifact)
Robustness contracts unchanged and still verified: slow worker cannot stall a
turn, lone survivor answers, hanging refiner cannot hold the turn, patience
expiry releases held consensus.
Measured through the shipped path (OpenRouter, 8B-class peers with an 8B reducer, 40 preregistered prompts x 2 draws, out-of-family position-swapped judge, vs the pool's best member alone): 2x 8B: 0W/43T/37L p<0.0001 6x 8B: 5W/52T/23L p=0.0009 The committee never won and lost about a third of decided trials, with consistently shorter answers (3236-3372 chars vs ~4070 solo). A capable pool is the opposite (71W/8T/1L, p<0.0001), so this is a statement about *this* configuration -- a weak reducer synthesizing weak drafts -- not about small-model MoA in general. The untested cell is small peers with a strong reducer; if a mesh gains a big-tier model the pool is no longer all-small and MoA engages again. Rather than ship a measured regression, an all-small pool now collapses to its single strongest member, so the caller degrades to serving that model directly. Also corrects the private-mesh grace assertion to the widened 10s default. The earlier "6x 8B beats its best member (12W/2L, p=0.013)" harness result did not replicate: the same rig on the same 40 tasks now gives 3W/76T/1L (p=0.63), with 11 of 40 tasks flipping verdict. It rested on ~14 decided trials out of 80 (the rest ties) and was a single unreplicated run. Withdrawn.
…ed results The width-sprint headline (12W/2L, p=0.013) did not replicate: same rig, same tasks, same pool now gives 3W/76T/1L (p=0.63), 11 of 40 tasks flipping. It rested on ~14 decided trials from a single unreplicated run. Records the shipped-path all-small measurements that motivated the best-member gate (2x: 0W/37L, 4x: 5W/12L, 6x: 5W/23L), scopes the claim to weak-reducer configurations, and documents known methodology limitations (tie bucket conflates failures, per-draw vs per-prompt significance, unverified best member, single judge).
…imeout Public-mesh comparison against released v0.74.0 (same 3B model, same prompts, `model=mesh`) exposed a liveness regression I introduced earlier today: released v0.74.0: 4.0s / 3.2s / 11.3s (reducer never ran) branch (N-1 gate): 61.0s / 6.7s / 61.7s (reducer ran, SHORTER answers) `min_grace_answers` gates whether grace can ARM at all, so any value above 1 is a liveness hazard. With N-1 on a 6-worker public-mesh pool where two peers never returned, only 4 answers ever arrived, grace could never arm, and the turn rode `worker_timeout` (60s) instead. The two 61s turns had 6 workers/4 ok and 6/3; the fast 6.7s turn had 4 workers/3 ok, where N-1=3 was satisfiable. 3/3 consistent. Width comes from the grace WINDOW (10s), not a count gate: healthy peers land inside it and are all synthesized, while a dead peer costs 10s rather than 60s. This also restores the exact configuration the capable-pool win was measured under (71W/8T/1L, p<0.0001) — that run predates the N-1 gate — and the gate bought nothing on small pools either (8W/17L with it, 9W/17L without).
…claim MOA_GATEWAY.md described behaviour that no longer holds: - "requires >=2 distinct models, returns 503 if fewer" -> a single model (or an all-small pool collapsed to its best member) now degrades: the virtual `mesh` name is rewritten to a real served model and routed normally. Only a node serving nothing returns 503. - name-derived tiering -> tiering now comes from verified GGUF tensor sums gossiped as `parameter_count_b`, split at 10B, with unverified sizes ranking small so an unparseable alias cannot pose as big-tier. Name parsing mis-tiered real models (gemma-4-E4B stores 7.5B, not 4B). - documents pool shaping in order (admission, all-small best-member fallback, committee cap) with the measured numbers behind each. - records that answer turns pack every worker at the full budget. - stale peer-timeout row (15s) -> 60s worker timeout with the 10s grace window. RESULTS.md: the width-sprint section now carries an explicit WITHDRAWN banner pointing at the replication failure, so the 6x8B number cannot be cited from the middle of the file.
michaelneale
changed the title
The tool path is deliberately asymmetric — route to the single best tool-caller
rather than fan out and vote, because voting on tool calls measured
null-to-harmful. That is only safe if structured `tool_calls` still come back
intact when several models are on the wire, so this asserts the contract through
the shipped `handle_turn` with a real pool: a tool prompt yields exactly one
well-formed call whose arguments parse as JSON and whose name is an offered
tool, and a no-tool prompt does not invent one.
Measured live (OpenRouter, both cells pass):
4x 8B pool -> dispatched=4 ok=3 reducer=true
list_dir({"path":"src"})
search({"path":".","pattern":"MeshError::Timeout\\(.*?\\)"})
run_command({"cmd":"pytest --verbose"})
no_tool_concept: no call invented
4x 24-35B pool -> dispatched=1 ok=1 reducer=true
same shapes, single actor
The dispatch difference is `ReferencePolicy::Auto` working as measured: a
small-tier actor gets tool-free advisors (+0.017 uplift), a big-tier actor acts
alone (advisors measured -0.037 there). The 4-worker cell also survived a worker
flaking (ok=3 of 4) and still emitted a valid call, which is the ensemble-active
case that was previously only covered by replay fixtures.
Asserts the contract, not a quality delta — the tool chosen may legitimately
differ from a single model's first move, so only a non-offered tool or malformed
arguments fail.
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ready for review. Every number below is measured through the shipped entrypoint (
moa::handle_turn) against real open-weight models; method, raw numbers, and withdrawn results are inevals/moa-openrouter/RESULTS.md.Closes part of #1115.
What this delivers
The virtual
meshmodel gets stronger as capable nodes join, degrades gracefully when there aren't any, and never convenes a committee that measured worse than serving one model.The core result
Capable pool (
qwen3-32b,mistral-small-24b,qwen3.5-35b-a3b,qwen3-14b), vs the pool's best single member answering alone. 40 preregistered prompts × 2 draws, out-of-family position-swapped judge, length logged:71W / 8T / 1L, p<0.0001 — and it wins 28 of 29 trials where the MoA answer was shorter than solo, so this is not a length artifact. Reproduced across three runs (65W/0L, 62W/1L, 71W/1L).
Before the fixes below, the same pool through the same entrypoint lost 26W/14T/40L.
Three shipped-path bugs
The mechanism was fine; the wiring never ran it.
first_answer_gracearmed on the first answer and finalized the turn, so on fast backends it fired on ~every turn and shipped one worker's text — measured 80/80 early-exit at capable scale. Grace now finalizes on tool turns only; on answer turns it is a collection deadline (stop waiting for the tail, then synthesize what arrived). Private-mesh window 3s → 10s.Pool shaping (all measured)
parameter_count_b, not name parsing. Name parsing mis-tiered real models (gemma-4-E4Bstores 7.5B, not 4B). No verified size ⇒ ranks small, so an unparseable alias can't pose as big-tier. (i386 P1s.)meshto a real served model and routes normally. Only a node serving nothing returns 503. Onmainthis is a 503.Withdrawn
The width-sprint headline — "6× diverse 8B beats its best member, 12W/65T/2L, p=0.013" — did not replicate. The same rig on the same 40 tasks and pool now gives 3W/76T/1L (p=0.63), with 11 of 40 tasks flipping verdict. It rested on ~14 decided trials out of 80 (the rest ties) from a single unreplicated run selected out of a width sweep. Withdrawn in
RESULTS.mdand it motivated the all-small gate above.Also fixed
Nine eval-vs-production divergences found by measuring the shipped path (grace finalizing before synthesis; reducer/refiner truncation discarding most of each answer; prompts telling workers to "be concise"/"be direct"; named vs anonymous reducer inputs; the worker preamble on the reducer; role-tier token starvation). Plus a production panic in
mesh-llm-guardrails(byte-slicing a multi-byte character), with regression tests. Two hypotheses were tested, rejected by measurement, and reverted rather than rationalised — including one of my own (min_grace_answers = N-1) that stalled public-mesh turns to the 60s worker timeout.Protocol / compatibility
No wire, gossip, plugin, or skippy-ABI changes. Sizing reads the existing
ServedModelMetadata.parameter_count_b(now populated from GGUF tensor sums instead of a name regex). New behaviour is in-memory config (ReferencePolicy,RefinementPolicy). One caller-visible change:reasoning_effort/enable_thinkingon amodel=meshrequest is ignored — that override only selected a broken config where reasoning models returncontent: null.Validation
Robustness contracts verified by simulation: a slow worker cannot stall a turn, a lone survivor still answers, a hanging refiner cannot hold the turn, patience expiry releases held consensus, partial worker death is fully accounted.
Live release check on this branch (built here, real node): local private inference,
model=meshon a single node (200, degrades), console index + assets +/api/status+/api/models+/v1passthrough chat + SSE streaming, joined the public mesh (13–16 peers, mixed 0.65.1/0.72.1/0.74.0, 12-model union), MoA fanout there with 2 of 6 workers failing and the turn still completing, and model-by-name routing to a remote peer.Public-mesh contrast vs released v0.74.0 (same 3B model, same prompts,
model=mesh): released never synthesizes (reducer=falseon all three probes, two early-exiting at 3–4s); this branch always synthesizes. An earlier revision of this branch rode the 60s worker timeout on two of three probes — root-caused tomin_grace_answers, reverted, and the grace window now bounds a dead peer at 10s.Honest limits (for the reviewer)
handle_turn, not a live capable-node mesh. The mesh-config paths (admission, cap, all-small gate, self-fill) are unit-tested and the plumbing is live-verified, but the capable-pool quality lift itself has not been reproduced on real mesh hardware.judge_paircollapses genuine ties, position disagreements, and API/parse failures into the same0, so tie counts cannot be read as agreement.+14.8kline count is almost entirely recorded traces and eval data; the code surface is 13 files.