MoA: don't let small-model consensus pre-empt a still-running large model

[michaelneale]

Summary

When model: "mesh" fans out across a mixed mesh — one large model (e.g. MiniMax on studio) plus several small models — the large model's answer could be silently discarded: two fast small workers agreeing (or the 3s answer-grace timer) finalized the turn while the large model was still prefilling. Users got small-model consensus even when a much stronger answer was seconds away.

After this change, MoA holds small-tier-only answers for a bounded patience window (20s) when a big-tier Strong worker is still running:

• If the strong worker lands in time, its answer participates (and consensus that includes the strong answer ships immediately — that's agreement with the strong model, not against it).
• If the strong worker is stuck or slow past the window, the held small-tier consensus ships — the gate switches hard off at expiry, so a hung peer can never hold a turn hostage (the failure mode that sank Stabilize mesh MoA context and tool loops #820). A dedicated wake-up re-evaluates held outputs at expiry rather than waiting for worker_timeout.
• Tool proposals are never held — they're schema-verified by tool_guard, and agent loops (goose / OpenClaw) keep their current latency.
• Same-tier pools are unaffected: has_quality_gap only arms the gate when the Strong worker is big-tier and small-tier workers are present. Many-small-models meshes keep today's early-exit latency, preserving the consensus-lift behavior.
• Answer grace now prefers the Strong worker's qualifying answer over marginally-higher self-reported confidence from a smaller model (confidence is not comparable across models).

Why

Self-MoA (arXiv:2502.00674) shows MoA output quality is far more sensitive to proposer quality than diversity — mixing weak proposers into a strong model's pool actively hurts. "Are More LLM Calls All You Need?" (arXiv:2403.02419) shows majority voting degrades on hard queries. The goal of mesh MoA is to lift intelligence when there are many comparable small models, not to dumb down a large one; this change separates those two regimes.

Architecture

• worker::has_quality_gap — tier analysis over (model_name, role) pairs, reusing the existing single-digit-B name heuristic shared with the router.
• arbiter::StrongGate — explicit gate state passed into try_early_decision; Off reproduces pre-change behavior exactly (all existing arbiter tests pass unchanged with Off).
• fanout::GatherPolicy — groups the timing knobs (first_answer_grace, grace_mode, new strong_patience).
• GatewayConfig.strong_patience — zero disables the gate entirely; sim tests run with zero except the new gate-specific sims.

Protocol

No wire/protocol changes. Purely host-local decision logic inside the MoA gateway; mixed-version meshes unaffected.

Validation

• cargo test -p mesh-mixture-of-agents — 158 lib tests + all sims pass, including new sim_strong_patience.rs pinning: held consensus until strong lands, hard release at patience expiry with a hung strong worker (anti-Stabilize mesh MoA context and tool loops #820), and same-tier pools keeping early-exit.
• cargo test -p mesh-llm-host-runtime --lib — 1439 passed.
• Clippy -D warnings clean on both crates; fmt clean.
• Live multi-node validation (lab mini + public mesh, goose + claw vs baseline) to follow in PR comments.

Summary by CodeRabbit

• New Features

  • Added configurable "strong patience" gating to hold small-tier consensus while higher-quality workers are still producing, with a patience timeout to release decisions.
  • Detects quality gaps between worker tiers to enable tier-aware arbitration and prefer a qualifying strong answer when it disagrees with a small-tier consensus.

• Refactor

  • Consolidated grace/timing into a GatherPolicy for clearer tier-gating behavior.

• Tests

  • Expanded tests covering gating, patience expiry, preference for strong answers, and all-small pools.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b3d3ad7e-aa3a-4814-ab9a-bda44aa72f13

📥 Commits

Reviewing files that changed from the base of the PR and between 102c2d4 and a9a9abf.

📒 Files selected for processing (3)

• crates/mesh-mixture-of-agents/src/arbiter.rs
• crates/mesh-mixture-of-agents/src/fanout.rs
• crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs

🚧 Files skipped from review as they are similar to previous changes (3)

• crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs
• crates/mesh-mixture-of-agents/src/fanout.rs
• crates/mesh-mixture-of-agents/src/arbiter.rs

---

📝 Walkthrough

Walkthrough

This PR introduces a configurable "strong patience" timeout that gates early arbitration decisions in mixed-tier worker pools, preventing small-tier consensus from finalizing while a Strong worker is still active within the patience window, with hard expiry releasing held consensus.

Changes

Strong-Worker Patience Tier Gating

Layer / File(s)	Summary
StrongGate contract and early-arbitration logic crates/mesh-mixture-of-agents/src/arbiter.rs	StrongGate enum and updated try_early_decision signature enable tier-aware gating; tier_aware_consensus_decision helper withholds small-tier-only answers when Strong is pending unless the strong tier agrees; new tests validate hold/release behavior, strong-tier agreement acceptance, sole-survivor suppression, and tool-proposal exemption.
GatherPolicy and fan-out patience-aware gathering crates/mesh-mixture-of-agents/src/fanout.rs	GatherPolicy struct bundles grace timing with strong_patience; gather_workers_incremental dynamically computes StrongGate based on strong-worker completion state and elapsed time; patience-expiry timeout fires to re-evaluate and release held decisions even without new worker completions; Answer-grace suppressed during strong holds while Tool-grace remains exempt; Strong-tier answers preferred in grace selection; existing unit tests updated to pass GatherPolicy instead of separate parameters.
Gateway configuration and query-path wiring crates/mesh-mixture-of-agents/src/lib.rs, crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs	GatewayConfig gains strong_patience: Duration field (default 20 seconds in the host runtime); handle_query threads the config value into fanout::GatherPolicy, exposing the feature to query execution.
Quality-gap detection helper crates/mesh-mixture-of-agents/src/worker.rs	New has_quality_gap function identifies mixed-tier pools (big-tier Strong + small-tier non-Strong) using existing is_single_digit_b_name heuristic; tests validate gap detection for MiniMax+Qwens (gap), all-small (no gap), all-big (no gap), and no-Strong (no gap) cases.
Strong-patience integration test suite crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs	DelayedAnswerBackend test double with configurable latency; mixed_pool helper wires two fast small-tier and one slow strong-tier backend; three tests validate: small consensus held until strong lands within patience, patience expiry releasing held consensus (issue #820 anti-regression), and same-tier pools retaining early-exit without patience delays.
Existing test configuration updates crates/mesh-mixture-of-agents/tests/sim_*.rs	Test helper configurations across sim_all_workers_fail.rs, sim_enable_thinking_propagation.rs, sim_tool_call_text_not_passed_as_content.rs, sim_tool_result_routes_to_reducer.rs, sim_worker_accounting.rs now explicitly set strong_patience: Duration::ZERO for backward-compatible behavior.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• Mesh-LLM/mesh-llm#820: Addresses the anti-pattern this PR prevents—early exit before strong workers complete—by introducing patience-based timeout-driven consensus release.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: preventing small-model consensus from pre-empting a still-running large model, which is the core objective of this PR.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch moa-tier-aware-consensus

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (3)

crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs (1)

161-165: ⚡ Quick win

Tighten the latency bound to the configured patience window.

elapsed < 5s will still pass if the release regresses from ~500ms to several seconds, so it does not really pin the “promptly at patience expiry” contract described above. A much tighter upper bound with some scheduler slack would catch that regression.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs` around lines 161
- 165, The assertion currently allows up to 5s (assert!(elapsed <
Duration::from_secs(5), ...)) which is too loose; replace that upper bound with
a tighter timeout near the configured patience window (~500ms) plus scheduler
slack (e.g., ~750ms or 1s) so regressions are caught—update the assertion
comparing elapsed to a Duration reflecting that tightened bound and keep the
existing error message referencing elapsed.

crates/mesh-mixture-of-agents/src/fanout.rs (1)

257-264: 💤 Low value

Panicked/cancelled Strong worker leaves strong_finished false, gate holds until patience expires.

When a task produces a JoinError, we have no (model, role) payload to identify the worker. If the Strong worker panics, strong_finished stays false and the gate continues holding until the patience window expires rather than releasing immediately.

This is a minor edge case (panicking workers are rare), and patience expiry is the bounded fallback. No action required unless this becomes a real latency issue.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mesh-mixture-of-agents/src/fanout.rs` around lines 257 - 264, The
JoinError handling block in fanout.rs currently increments total_finished and
logs the error but does not mark the Strong worker as finished, so a
panicked/cancelled Strong leaves strong_finished false and the gate waits until
patience expires; update the Err(e) arm that logs "moa: worker task panicked or
was cancelled" to also detect if the missing worker was the Strong worker (by
correlating the JoinError to the dispatched slot or checking the worker identity
used when spawning) and set strong_finished = true (or otherwise mark that slot
as completed) so the gate can release immediately; ensure this change integrates
with reconcile_dispatched logic that expects dispatched summaries by name.

crates/mesh-mixture-of-agents/src/arbiter.rs (1)

347-380: 💤 Low value

Variable strong_agrees is misleading — it checks whether Strong has finished with any answer, not cluster agreement.

The docstring says "Consensus that includes the strong worker's answer passes through" but strong_agrees only checks if Strong produced any usable answer, not whether Strong's answer is part of the agreeing cluster. If Strong answered "Berlin" while two small models agreed on "Paris", strong_agrees would be true and the "Paris" consensus would ship.

If this is intentional (Strong finishing lifts the gate regardless of agreement), consider renaming to strong_has_answered and adjusting the docstring to clarify that Strong's mere presence (not agreement) releases the hold.

Suggested clarification

-    let strong_agrees = answers
+    // Strong has produced a usable answer — it had its chance to weigh in.
+    // Whether Strong agrees with the cluster or not, the gate releases.
+    let strong_has_answered = answers
         .iter()
-        .any(|a| a.role == WorkerRole::Strong && is_usable_answer(a));
-    if strong_pending && !strong_agrees {
+        .any(|a| a.role == WorkerRole::Strong);
+    if strong_pending && !strong_has_answered {

Note: is_usable_answer(a) is redundant since answers is already filtered by that predicate.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/mesh-mixture-of-agents/src/arbiter.rs` around lines 347 - 380, In
tier_aware_consensus_decision change the misleading strong_agrees semantics:
either (A) if intent is "strong finishing lifts the gate regardless of content"
rename strong_agrees to strong_has_answered and update the function/doc comment
to state that any usable Strong answer releases the hold (also remove redundant
is_usable_answer check since answers is already filtered), or (B) if intent is
"strong must agree with the cluster" change the check to compare Strong's
payload to the agreeing cluster (i.e., find the Strong WorkerOutput in answers
and verify its payload matches the best consensus payload) before treating the
gate as lifted; update variable names (e.g., strong_matches_consensus) and the
docstring accordingly to reflect the chosen behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs`:
- Around line 118-145: The test only asserts that the strong worker (model
"MiniMax-M2.5") completed but does not assert that its answer was selected;
update the test to assert the turn response equals the strong worker's answer by
calling response_text(&result) and comparing it to the expected strong-worker
text (the answer produced by the strong model in this scenario), ensuring the
contract that "when the strong worker lands with a usable answer, it wins" is
enforced; locate the test function small_consensus_is_held_until_strong_lands,
the result variable from moa::handle_turn(&config, &user_turn(...)), and add an
assertion that response_text(&result) == "<expected strong answer>" (or match
the known strong summary output) after verifying strong_summary.succeeded.

---

Nitpick comments:
In `@crates/mesh-mixture-of-agents/src/arbiter.rs`:
- Around line 347-380: In tier_aware_consensus_decision change the misleading
strong_agrees semantics: either (A) if intent is "strong finishing lifts the
gate regardless of content" rename strong_agrees to strong_has_answered and
update the function/doc comment to state that any usable Strong answer releases
the hold (also remove redundant is_usable_answer check since answers is already
filtered), or (B) if intent is "strong must agree with the cluster" change the
check to compare Strong's payload to the agreeing cluster (i.e., find the Strong
WorkerOutput in answers and verify its payload matches the best consensus
payload) before treating the gate as lifted; update variable names (e.g.,
strong_matches_consensus) and the docstring accordingly to reflect the chosen
behavior.

In `@crates/mesh-mixture-of-agents/src/fanout.rs`:
- Around line 257-264: The JoinError handling block in fanout.rs currently
increments total_finished and logs the error but does not mark the Strong worker
as finished, so a panicked/cancelled Strong leaves strong_finished false and the
gate waits until patience expires; update the Err(e) arm that logs "moa: worker
task panicked or was cancelled" to also detect if the missing worker was the
Strong worker (by correlating the JoinError to the dispatched slot or checking
the worker identity used when spawning) and set strong_finished = true (or
otherwise mark that slot as completed) so the gate can release immediately;
ensure this change integrates with reconcile_dispatched logic that expects
dispatched summaries by name.

In `@crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs`:
- Around line 161-165: The assertion currently allows up to 5s (assert!(elapsed
< Duration::from_secs(5), ...)) which is too loose; replace that upper bound
with a tighter timeout near the configured patience window (~500ms) plus
scheduler slack (e.g., ~750ms or 1s) so regressions are caught—update the
assertion comparing elapsed to a Duration reflecting that tightened bound and
keep the existing error message referencing elapsed.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 076da41c-6532-4c86-8b8f-e0030df34e90

📥 Commits

Reviewing files that changed from the base of the PR and between 226d1c6 and 102c2d4.

📒 Files selected for processing (11)

• crates/mesh-llm-host-runtime/src/network/openai/moa_gateway/mod.rs
• crates/mesh-mixture-of-agents/src/arbiter.rs
• crates/mesh-mixture-of-agents/src/fanout.rs
• crates/mesh-mixture-of-agents/src/lib.rs
• crates/mesh-mixture-of-agents/src/worker.rs
• crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs
• crates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs
• crates/mesh-mixture-of-agents/tests/sim_strong_patience.rs
• crates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rs
• crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs
• crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs