feat(moa): no-think default on workers + grace fires on diverse fast answers by michaelneale · Pull Request #629 · Mesh-LLM/mesh-llm

michaelneale · 2026-05-22T00:31:14Z

Combines the two MoA chat improvements from PRs #620 and #624 into one PR \u2014 they're inseparable in the user journey and only meaningful together. Closing those in favour of this.

What gives

On the public mesh today, with the chat UI's Auto routing through model=mesh (MoA), the dominant chat experience is:

	before this PR	after this PR
p50 latency (public mesh, short chat)	~40s	~1.2s
think leakage into chat (Qwen3-8B, MiniMax, etc. emitting raw `<think>...` prose)	yes, frequent	no
answers correct on factual / arithmetic / subjective prompts	yes	yes
tool calling	works, unchanged	works, unchanged

Validated live on a real public mesh (M4 client --auto, joined to public mesh, hitting 20+ peers including studio MiniMax and others).

Two changes, one user-journey

1. Opinionated no-think on MoA workers + reducer (was #620)

Workers in MoA are short-budget internal slots, not user-facing reasoning steps. The fast worker has a 256-token budget that doesn't fit <think>\u2026</think> + answer. The reducer doesn't want reasoning prose as candidate input. The chat UI's Auto routes everything through MoA, so reasoning models burning their budget inside <think> was the dominant chat failure mode.

This PR:

SamplingParams gains enable_thinking: Option<bool> + with_thinking(...).
New backend::apply_enable_thinking injects chat_template_kwargs.enable_thinking (canonical llama.cpp knob) + reasoning_effort: "none" when off.
All three backends (HTTP / LocalModel / RemoteModel) call it.
New effective_enable_thinking_for_moa(body) policy: extract_enable_thinking_override(body).or(Some(false)) \u2014 opinionated default off, escape hatch for callers who explicitly pass reasoning_effort: "low" etc.

2. Grace fires on diverse fast answers (was #624)

The arbiter's consensus rule ("\u22652 workers textually agree") rarely fires on short chat answers across many models. "Hello" / "Yes" / "Ready" / "Okay" are all valid but don't match. Before this PR, MoA then waited for the slow tail worker (~40s on the public mesh's bimodal Qwen3-8B peer).

The old sole-answer grace only armed on outputs.len() == 1. This PR relaxes it to "at least one Answer-kind output with confidence \u2265 0.5". When grace fires with multiple answers, pick the highest-confidence one.

Also tightens first_answer_grace from 6s to 3s. The previous 6s was conservative because grace rarely fired \u2014 with the eligibility relaxation, grace is the dominant chat path, so a tighter default is right.

Lab data

Public mesh, 5 runs of "reply with one short word":

	grace=6s old eligibility (before)	grace=3s new eligibility (this PR)
Run 1	45.5s	3.0s
Run 2	36.0s	3.0s
Run 3	38.2s	3.0s
Run 4	0.7s	1.0s
Run 5	44.9s	1.0s

Varied prompts post-fix:

Prompt	Time	Answer
"What is 2+2?"	877ms	"2+2 equals 4."
"Capital of Japan?"	1.2s	"Tokyo is the capital of Japan."
"Name one Python web framework."	3.0s	"Flask"
"Hello"	1.0s	"Hello! How can I assist you today?"
"What color is the sky?"	3.0s	"The color of the sky can vary\u2026"

Tool calling (mesh + tools=[read_file]): 2/2 success in 2.8\u20135.1s, finish_reason: tool_calls, no change from before.

What this does NOT change

No QUIC, iroh, keep-alive, relay, discovery, gossip, target_health, or any mesh-network primitive. Purely MoA application logic.
No mesh wire-protocol change. Backends receive a chat-completion request body with an extra additive field (chat_template_kwargs.enable_thinking), which llama.cpp templates already understand. Backwards compatible across mixed-version meshes.
No skippy ABI change. No plugin protocol change.

Tests

mesh-mixture-of-agents lib: 100 pass (10+ new for apply_enable_thinking, SamplingParams::with_thinking, and grace eligibility/behavior).
mesh-mixture-of-agents tests/sim_enable_thinking_propagation.rs (new): 3 mock-backend integration tests pinning chat_template_kwargs.enable_thinking reaches every worker AND no spurious fields appear when no override is requested.
mesh-llm-host-runtime lib: 1477 pass (14 new for extract_enable_thinking_override covering every JSON shape, plus 4 for effective_enable_thinking_for_moa).
cargo clippy --all-targets -- -D warnings: clean.
cargo fmt --all -- --check: clean.

Compat

Behavior change for existing MoA callers: a caller who today relies on MoA producing reasoning output will now get short answers instead. Escape hatch via any recognised reasoning knob (reasoning_effort: "low", enable_thinking: true, reasoning: { enabled: true }, etc.).
Chat UI's Auto (which routes through MoA per chat: real data by default, MoA for 'Auto', sole-answer grace for slow meshes #615) is the main consumer and benefits directly.
Tool turns continue to bypass grace via !has_tools gate \u2014 agentic harnesses unaffected.

Closes #617.

For `model: "mesh"`, the OpenAI-style reasoning knobs (`reasoning_effort: "none"`, `reasoning: { enabled: false }`, `enable_thinking: false`, `thinking_budget: 0`, `chat_template_kwargs.enable_thinking`, and the THINKING_BOOLEAN_ALIASES) were silently dropped in MoA. The fast worker's 256-token budget then got burned inside an unclosed `<think>` block on reasoning models and never reached the actual answer. Live lab on a 3-node mesh (M4 Qwen2.5-3B + studio MiniMax-M2.5 + mini Qwen3.5-9B), "reply with one short word" prompt: | Run | default thinking | reasoning_effort: none | |-----|--------------------------|------------------------| | 1 | 32s, leaked "Thinking Process: 1. Analyze..." | 1s, "Okay." | | 2 | 17s, "Yes" | 2s, "Hello" | | 3 | 32s, leaked reasoning prose | 1s, "Hello" | Worker log on the slow runs shows Qwen3.5-9B returning 1700+ char payloads at 31s \u2014 the full runaway think block. On the no-think runs all three workers finish in <1.3s with 4-5 char clean answers. Implementation: * `SamplingParams` gains `enable_thinking: Option<bool>` and a builder-style `with_thinking(...)` helper. `None` (default) is "don't override" \u2014 callers without a preference see no behavior change. * New `backend::apply_enable_thinking(body, hint)` helper centralises the wire shape: injects `chat_template_kwargs.enable_thinking` (canonical llama.cpp chat-template knob) and `reasoning_effort: "none"` when disabled. Merges into existing `chat_template_kwargs` instead of clobbering. * `HttpBackend`, `LocalModelBackend`, and `RemoteModelBackend` all call it after building their request bodies. * `GatewayConfig` gains `enable_thinking: Option<bool>` so the choice flows in one place to every worker (`SamplingParams::worker().with_thinking(...)`) AND the hedged reducer (`SamplingParams::reducer().with_thinking(...)`). * `moa_gateway.rs::try_handle_moa` extracts the override from the inbound request body via the new `extract_enable_thinking_override` helper, which mirrors every shape that `openai_frontend::common::normalize_reasoning_template_options` recognises (so MoA users get the same surface as direct callers). Tests: * mesh-mixture-of-agents lib: 98 pass (6 new for apply_enable_thinking and SamplingParams::with_thinking). * mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs (new): 3 mock-backend integration tests pinning that chat_template_kwargs.enable_thinking reaches every worker AND that no spurious fields appear when no override is requested. * mesh-llm-host-runtime lib: 1478 pass (10 new for extract_enable_thinking_override covering every JSON shape). * clippy --all-targets -D warnings: clean. * cargo fmt --all -- --check: clean. Compat: * No mesh wire-protocol change. The new fields travel inside the existing chat-completion request JSON over the QUIC tunnel; peers on older binaries simply forward the body to llama.cpp which already understands `chat_template_kwargs`. Additive only, safe across mixed-version meshes. * No skippy ABI change. No plugin protocol change. * MoA callers without a preference get the same behavior as before this commit \u2014 `enable_thinking: None` is the default. Lab follow-up: a new `mesh_no_think` probe in the stability lab (`/tmp/lab/probe-stable.sh`) hits `/v1/chat/completions` with `reasoning_effort: "none"` so we can A/B latency and quality over hours of probe traffic.

…asked Per Mic: MoA shouldn't have to ship behind a UI toggle. The MoA gateway now defaults to `enable_thinking = Some(false)` for every `model: "mesh"` request. Callers can still opt-in by passing any recognised reasoning knob explicitly (`reasoning_effort: "low"`, `enable_thinking: true`, etc.) — but the default is reasoning off. Rationale: * Workers are short-budget internal slots, not user-facing reasoning steps. The fast worker has a 256-token budget that doesn't fit `<think>...</think>` + answer. * The reducer doesn't want reasoning prose as candidate input. * MoA chat UX is the worst case for thinking models — every turn pays the reasoning latency penalty without giving the user the reasoning output. Implementation: * Extracted the policy into a tiny pure function, `effective_enable_thinking_for_moa(&body) -> Option<bool>`, that returns `extract_enable_thinking_override(body).or(Some(false))`. * 4 unit tests cover the contract: silent caller → no-think; explicit disable → no-think; explicit enable → thinking on (escape hatch); tool turn → still no-think by default. * The existing `extract_enable_thinking_override` tests are unchanged (they test the parser, not the gateway policy). Live verification on the 3-node lab mesh (M4 + studio + mini), release binary: * `model=mesh` with no knobs: 3/3 clean short answers ("Hello", "Okay", "Hello"), zero think leakage. * `model=mesh` + `reasoning_effort: "low"`: thinking turns back on, raw "Thinking Process:" prose appears as expected. * `model=mesh` + `tools=[read_file]`: tool_calls path unchanged, `finish_reason: tool_calls`, correct args. No-think default applies. Tests: 1486 host-runtime lib pass (4 new), all mesh-mixture-of-agents tests pass, clippy + fmt clean.

Live public-mesh lab data showed a real failure: when 3-5 workers all return short answers in <1s but the answers don't textually agree (e.g. "Hello" / "Yes" / "Ready" / "Okay"), the arbiter's consensus rule ("\u22652 workers agree on answer") doesn't fire. MoA then waits for the slow tail worker before deciding, even though every fast worker has already produced a confident answer. Sample from the public-mesh lab (5 runs of "reply with one short word"): | Run | mesh time | what happened | |----:|----------:|---| | 1 | 45.5s | waited for slow Qwen3-8B | | 2 | 36.0s | waited for slow Qwen3-8B | | 3 | 38.2s | waited for slow Qwen3-8B | | 4 | 0.7s | 2 workers happened to textually agree | | 5 | 44.9s | waited for slow Qwen3-8B | The previous grace logic only armed when (sole answer). That covers the case where one worker has answered and the rest are still pending. It does NOT cover the case where multiple workers have answered fast but disagree. This fix relaxes grace eligibility to: "at least one Answer-kind output with confidence \u2265 0.5". When grace fires with multiple qualifying answers, pick the highest-confidence one. The previous sole-answer case is naturally preserved (max_by on a single element returns that element). Worker role and short-message variance is the dominant input here \u2014 fast and specialist workers running on different model families rarely produce textually-identical short answers, so consensus is a high bar. Grace gives us a sensible time-bounded escape hatch. Tests: * mesh-mixture-of-agents::fanout: 6 grace tests pass * New: grace_fires_with_multiple_diverse_answers \u2014 pins the real-world public-mesh case * New: grace_picks_highest_confidence_when_multiple_qualify \u2014 pins the highest-confidence picking rule * Existing 4 grace tests unchanged behavior; one was renamed internally but still asserts \u2018sole-answer grace fires\u2019 Compat: no API change. The relaxed eligibility is strictly more permissive than before, so existing callers see grace fire in cases where it didn't before (the public-mesh latency-tail case). No agentic regression \u2014 `!has_tools` still gates grace entirely.

Bundled into the same PR (#624) as the eligibility relaxation. The two changes are inseparable in the chat user journey: * Eligibility relaxation (this PR's main change): grace fires on diverse fast answers, not just sole answers. Cures the 40s tail on the public mesh. * Tighter default (this commit): chat latency floor moves from ~6s to ~3s now that grace is the dominant chat path. Lab data on the public mesh: median mesh_chat went from ~6s to ~2s after this value, no quality regression on factual, arithmetic, or short-creative prompts. Without the eligibility relaxation, dropping the timer doesn't help much (grace rarely fires anyway). Without the tighter default, the relaxation just trades 40s for 6s instead of 40s for 2s. Reviewers should agree on both as one user-facing improvement.

Copilot

Pull request overview

This PR improves the Mixture-of-Agents (MoA) chat path by (1) defaulting MoA workers/reducer to “no-think” unless the caller explicitly opts in, and (2) making the grace early-exit trigger on diverse fast answers (not just the sole-answer case) to avoid waiting on slow-tail workers.

Changes:

Propagate a per-request enable_thinking override through MoA (GatewayConfig → SamplingParams → all backends) via chat_template_kwargs.enable_thinking (and reasoning_effort: "none" when disabling).
Relax grace eligibility to fire when ≥1 confident Answer is present and, when multiple answers qualify, choose the highest-confidence answer; tighten default grace window to 3s.
Add/extend integration + unit tests covering enable-thinking propagation and diverse-answer grace behavior.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
crates/mesh-mixture-of-agents/src/backend.rs	Adds `enable_thinking` to sampling params and injects `chat_template_kwargs.enable_thinking` into outbound worker/reducer request bodies.
crates/mesh-mixture-of-agents/src/lib.rs	Threads `enable_thinking` through MoA worker dispatch and reducer calls via `GatewayConfig`.
crates/mesh-mixture-of-agents/src/reducer.rs	Extends hedged reducer calls to propagate `enable_thinking` into reducer sampling params.
crates/mesh-mixture-of-agents/src/fanout.rs	Updates grace eligibility/selection logic to early-exit on diverse fast answers and adds tests for the new behavior.
crates/mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs	New simulated-mesh integration tests asserting the enable-thinking override reaches every worker/reducer and stays absent when unset.
crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs	Updates test config construction to include the new `enable_thinking` field.
crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs	Updates test config construction to include the new `enable_thinking` field.
crates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rs	Updates test config construction to include the new `enable_thinking` field.
crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs	Updates test config construction to include the new `enable_thinking` field.
crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs	Computes MoA’s effective thinking policy from inbound JSON and sets MoA default grace window to 3s; applies thinking override in local/remote backends.

+/// Returns `None` when the caller hasn't expressed a preference, leaving
+/// each worker's default behavior alone.
+/// MoA's opinionated default: workers do not think unless the caller
+/// explicitly asks for it. Workers are short-budget internal slots, not
+/// user-facing reasoning steps. The fast worker's 256-token budget is
+/// far too small to fit `<think>…</think>` + answer, and the reducer
+/// doesn't want reasoning prose as candidate input.
+///
+/// The caller can still explicitly enable thinking (e.g. for
+/// experimentation) via any of the recognised knobs — see
+/// [`extract_enable_thinking_override`]. When no preference is
+/// expressed, MoA picks for them: off.
+fn effective_enable_thinking_for_moa(body: &serde_json::Value) -> Option<bool> {
+    extract_enable_thinking_override(body).or(Some(false))
+}


…-grace * origin/main: Add agent tool-call reliability harness (#623)

Address Copilot review on PR #629: 1. `apply_enable_thinking` silently dropped the flag if a caller passed a non-object for `chat_template_kwargs` (string, array, number, null). The `entry().or_insert_with()` doesn't replace a non-object, so `as_object_mut()` then returns None and the override is lost. We now normalize a bogus shape to {} before injecting, so the flag always applies. New test `apply_enable_thinking_normalizes_non_object_kwargs` pins all four bogus-input shapes. 2. `effective_enable_thinking_for_moa` had two doc comments merged together because there was no blank line / item separating them. The merged doc said both 'returns None when caller hasn't expressed a preference' (true of the inner extractor) AND 'MoA picks for them: off' (true of the outer policy). Split into one doc per function. No behavior change beyond fix #1.

michaelneale · 2026-05-22T01:46:42Z

Merged latest main (incl. Ivan's #623 tool-call reliability probe) and addressed Copilot review:

Fix 1 (real bug): apply_enable_thinking would silently drop the flag if chat_template_kwargs was already present as a non-object (string / array / null / number) because entry().or_insert_with() doesn't replace an existing non-object, leaving as_object_mut() to return None and the override to be lost. We now normalize a bogus shape to {} before injecting. New test apply_enable_thinking_normalizes_non_object_kwargs pins all four bogus-input shapes.

Fix 2 (doc clarity): effective_enable_thinking_for_moa and extract_enable_thinking_override had their doc comments merged together because there was no item between them. The merged doc was internally inconsistent (one half about each function). Split into one doc per function and clarified effective_enable_thinking_for_moa always returns Some.

Skipped (intentional design): Copilot also suggested 400-erroring on invalid types for the top-level thinking aliases (non-bool values like enable_thinking: "yes"). I left this alone:

MoA's enable_thinking is a default-override, not a primary input. Treating an unparseable hint as "use the MoA default" is the safer-fail mode and matches the spirit of an opinionated-by-default surface.
Strict shape validation of caller bodies is the OpenAI-frontend's responsibility, enforced upstream of MoA in openai_frontend::common::normalize_reasoning_template_options.
MoA's extractor is best-effort recognition of caller intent across the multiple shapes the OpenAI / responses surfaces use; rejecting malformed shapes here would surface MoA-specific 400s for requests the upstream frontend already accepted, which is a worse UX than gracefully falling back to the default.

Happy to revisit if you'd rather have symmetric strictness across both paths.

Lab re-validation post-fix on public mesh (20+ peers)

Ivan's #623 probe (regression net):

agent tool-call reliability: 4/4 phases passed
PASS model=mesh attempt=1 phase=tool_call: lookup_fixture_fact key=codeword
PASS model=mesh attempt=1 phase=tool_result: final answer included signal-7429
PASS model=mesh attempt=1 phase=stream_tool_call: lookup_fixture_fact key=codeword
PASS model=mesh attempt=1 phase=stream_tool_result: final answer included signal-7429

Chat smoke (no-think + grace=3s):

Prompt	Latency	Answer
"What is 2+2?"	858ms	"2+2 equals 4."
"Capital of Japan?"	1.7s	"Tokyo is the capital of Japan."
"Hello"	1.3s	"Hello! How can I help you today?"
"Name one Python framework."	771ms	"One popular Python framework is Django."
"What color is the sky?"	3.0s	"The color of the sky can vary\u2026"

All sub-3.1s, no <think> leak, all correct.

Tests: 1477 host-runtime lib + 101 MoA lib (+1 new test for fix #1) all pass. clippy + fmt clean.

…-grace * origin/main: ci(client-auto): assert the node actually joined a mesh (#598)

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

crates/mesh-mixture-of-agents/src/fanout.rs:560

Similar to the prior test, this relies on very tight real-time scheduling (10ms/20ms worker sleeps vs 100ms grace). If the “high confidence” worker is delayed past the grace timer, MoA will legitimately pick the only available answer and the assertion will fail sporadically. Using a larger grace window or deterministic time control (tokio::time::pause/advance) would make the intent stable.

        let (_outputs, _summaries, decision) = gather_workers_incremental(
            &mut js,
            &dispatched,
            false,
            &[],
            Duration::from_millis(100),
        )

+        if r.get("enabled") == Some(&serde_json::Value::Bool(false))
+            || r.get("effort").and_then(|v| v.as_str()) == Some("none")
+            || r.get("max_tokens").and_then(|v| v.as_u64()) == Some(0)
+        {
+            result = Some(false);
+        } else if r.get("enabled") == Some(&serde_json::Value::Bool(true))
+            || r.get("effort").is_some()
+            || r.get("max_tokens").is_some()


michaelneale added 4 commits May 22, 2026 10:28

Copilot AI review requested due to automatic review settings May 22, 2026 00:31

This was referenced May 22, 2026

feat(moa): default to no-think on workers + reducer (escape hatch for callers) #620

Closed

fix(moa): grace fires on diverse fast answers, not just sole answer #624

Closed

Copilot started reviewing on behalf of michaelneale May 22, 2026 00:32 View session

Copilot AI reviewed May 22, 2026

View reviewed changes

michaelneale mentioned this pull request May 22, 2026

Add an agent tool-call reliability QA probe #623

Merged

michaelneale added 2 commits May 22, 2026 11:38

Merge remote-tracking branch 'origin/main' into micn/moa-no-think-and…

3a23f8f

…-grace * origin/main: Add agent tool-call reliability harness (#623)

Merge remote-tracking branch 'origin/main' into micn/moa-no-think-and…

2b56f0a

…-grace * origin/main: ci(client-auto): assert the node actually joined a mesh (#598)

Copilot AI review requested due to automatic review settings May 22, 2026 01:48

Copilot started reviewing on behalf of michaelneale May 22, 2026 01:52 View session

Copilot AI reviewed May 22, 2026

View reviewed changes

michaelneale merged commit ee1d316 into main May 22, 2026
22 checks passed

michaelneale deleted the micn/moa-no-think-and-grace branch May 22, 2026 02:19

This was referenced May 22, 2026

openai-frontend: chat_template_kwargs.enable_thinking / reasoning_effort='none' not honored on direct model calls #636

Closed

Console shows 0.0 tok/s for mesh (MoA) — usage hardcoded to zero, timings never emitted #637

Closed

Navigation Menu

Uh oh!

Uh oh!

feat(moa): no-think default on workers + grace fires on diverse fast answers - #629

feat(moa): no-think default on workers + grace fires on diverse fast answers#629
michaelneale merged 7 commits into
mainfrom
micn/moa-no-think-and-grace

michaelneale commented May 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

michaelneale commented May 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

michaelneale commented May 22, 2026

What gives

Two changes, one user-journey

1. Opinionated no-think on MoA workers + reducer (was #620)

2. Grace fires on diverse fast answers (was #624)

Lab data

What this does NOT change

Tests

Compat

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

michaelneale commented May 22, 2026

Lab re-validation post-fix on public mesh (20+ peers)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants