fix(moa): grace fires on diverse fast answers, not just sole answer - #624
Closed
michaelneale wants to merge 16 commits into
Closed
fix(moa): grace fires on diverse fast answers, not just sole answer#624michaelneale wants to merge 16 commits into
michaelneale wants to merge 16 commits into
Conversation
When the gateway forwards a request to a remote peer via QUIC and the upstream pre-commit phase fails (tunnel open / request forward / response probe), the failure becomes `RouteAttemptResult::RetryableUnavailable`. Today the outer routing loop retries on the next available target, which fixes the case where multiple peers serve the same model but does nothing when only one peer serves it: a single transient QUIC path teardown kills the request, even though iroh typically reconnects within ~1s on a fresh connection. This is the dominant failure mode on asymmetric / lossy direct paths (corporate firewalls, hotel wifi, Tailscale interfering with NAT traversal, M4-style incoming filter rules). Real-world meshes will see this constantly. Implementation: * Splits `route_remote_attempt` into a `_once` helper plus a wrapper that retries once with a 750ms backoff if the first attempt returned `RetryableUnavailable`. 750ms is below user-perceptible-stall and matches the typical iroh reconnect window. * Retry is only safe on `RetryableUnavailable` because that variant is exclusively set BEFORE any bytes are written to the client TCP stream (probe failed, tunnel open failed, buffered-request forward failed). Any post-commit error or non-network outcome falls through unchanged. * Pure-function `should_retry_remote_attempt` extracted for unit testing; 5 tests pin the retry policy (yes on unavailable; no on delivered / timeout / context-overflow / client-disconnected). Live lab validation pending on a 3-node mesh where the M4 \u2194 mini path is consistently asymmetric. Baseline before the fix was direct_mini 27% success over 1h40m. Expect the curl_fail subset (pre-commit drops) to recover to ~100% with this change; http_err 429 subset (mini's own admission control) is unaffected by design.
For `model: "mesh"`, the OpenAI-style reasoning knobs
(`reasoning_effort: "none"`, `reasoning: { enabled: false }`,
`enable_thinking: false`, `thinking_budget: 0`, `chat_template_kwargs.enable_thinking`,
and the THINKING_BOOLEAN_ALIASES) were silently dropped in MoA. The fast
worker's 256-token budget then got burned inside an unclosed `<think>`
block on reasoning models and never reached the actual answer.
Live lab on a 3-node mesh (M4 Qwen2.5-3B + studio MiniMax-M2.5 + mini
Qwen3.5-9B), "reply with one short word" prompt:
| Run | default thinking | reasoning_effort: none |
|-----|--------------------------|------------------------|
| 1 | 32s, leaked "Thinking Process: 1. Analyze..." | 1s, "Okay." |
| 2 | 17s, "Yes" | 2s, "Hello" |
| 3 | 32s, leaked reasoning prose | 1s, "Hello" |
Worker log on the slow runs shows Qwen3.5-9B returning 1700+ char
payloads at 31s \u2014 the full runaway think block. On the no-think runs
all three workers finish in <1.3s with 4-5 char clean answers.
Implementation:
* `SamplingParams` gains `enable_thinking: Option<bool>` and a
builder-style `with_thinking(...)` helper. `None` (default) is
"don't override" \u2014 callers without a preference see no behavior
change.
* New `backend::apply_enable_thinking(body, hint)` helper centralises
the wire shape: injects `chat_template_kwargs.enable_thinking`
(canonical llama.cpp chat-template knob) and `reasoning_effort:
"none"` when disabled. Merges into existing
`chat_template_kwargs` instead of clobbering.
* `HttpBackend`, `LocalModelBackend`, and `RemoteModelBackend` all
call it after building their request bodies.
* `GatewayConfig` gains `enable_thinking: Option<bool>` so the choice
flows in one place to every worker (`SamplingParams::worker().with_thinking(...)`)
AND the hedged reducer (`SamplingParams::reducer().with_thinking(...)`).
* `moa_gateway.rs::try_handle_moa` extracts the override from the
inbound request body via the new `extract_enable_thinking_override`
helper, which mirrors every shape that
`openai_frontend::common::normalize_reasoning_template_options`
recognises (so MoA users get the same surface as direct callers).
Tests:
* mesh-mixture-of-agents lib: 98 pass (6 new for apply_enable_thinking
and SamplingParams::with_thinking).
* mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs (new):
3 mock-backend integration tests pinning that
chat_template_kwargs.enable_thinking reaches every worker AND that
no spurious fields appear when no override is requested.
* mesh-llm-host-runtime lib: 1478 pass (10 new for
extract_enable_thinking_override covering every JSON shape).
* clippy --all-targets -D warnings: clean.
* cargo fmt --all -- --check: clean.
Compat:
* No mesh wire-protocol change. The new fields travel inside the
existing chat-completion request JSON over the QUIC tunnel; peers
on older binaries simply forward the body to llama.cpp which
already understands `chat_template_kwargs`. Additive only, safe
across mixed-version meshes.
* No skippy ABI change. No plugin protocol change.
* MoA callers without a preference get the same behavior as before
this commit \u2014 `enable_thinking: None` is the default.
Lab follow-up: a new `mesh_no_think` probe in the stability lab
(`/tmp/lab/probe-stable.sh`) hits `/v1/chat/completions` with
`reasoning_effort: "none"` so we can A/B latency and quality over
hours of probe traffic.
…asked
Per Mic: MoA shouldn't have to ship behind a UI toggle. The MoA gateway
now defaults to `enable_thinking = Some(false)` for every `model: "mesh"`
request. Callers can still opt-in by passing any recognised reasoning
knob explicitly (`reasoning_effort: "low"`, `enable_thinking: true`,
etc.) — but the default is reasoning off.
Rationale:
* Workers are short-budget internal slots, not user-facing reasoning
steps. The fast worker has a 256-token budget that doesn't fit
`<think>...</think>` + answer.
* The reducer doesn't want reasoning prose as candidate input.
* MoA chat UX is the worst case for thinking models — every turn pays
the reasoning latency penalty without giving the user the reasoning
output.
Implementation:
* Extracted the policy into a tiny pure function,
`effective_enable_thinking_for_moa(&body) -> Option<bool>`, that
returns `extract_enable_thinking_override(body).or(Some(false))`.
* 4 unit tests cover the contract: silent caller → no-think; explicit
disable → no-think; explicit enable → thinking on (escape hatch);
tool turn → still no-think by default.
* The existing `extract_enable_thinking_override` tests are unchanged
(they test the parser, not the gateway policy).
Live verification on the 3-node lab mesh (M4 + studio + mini), release
binary:
* `model=mesh` with no knobs: 3/3 clean short answers ("Hello", "Okay",
"Hello"), zero think leakage.
* `model=mesh` + `reasoning_effort: "low"`: thinking turns back on, raw
"Thinking Process:" prose appears as expected.
* `model=mesh` + `tools=[read_file]`: tool_calls path unchanged,
`finish_reason: tool_calls`, correct args. No-think default applies.
Tests: 1486 host-runtime lib pass (4 new), all mesh-mixture-of-agents
tests pass, clippy + fmt clean.
…able-thinking * origin/main: Cover asymmetric KV cache width pricing (#594)
Live public-mesh lab data showed a real failure: when 3-5 workers all
return short answers in <1s but the answers don't textually agree (e.g.
"Hello" / "Yes" / "Ready" / "Okay"), the arbiter's consensus rule
("\u22652 workers agree on answer") doesn't fire. MoA then waits for the
slow tail worker before deciding, even though every fast worker has
already produced a confident answer.
Sample from the public-mesh lab (5 runs of "reply with one short word"):
| Run | mesh time | what happened |
|----:|----------:|---|
| 1 | 45.5s | waited for slow Qwen3-8B |
| 2 | 36.0s | waited for slow Qwen3-8B |
| 3 | 38.2s | waited for slow Qwen3-8B |
| 4 | 0.7s | 2 workers happened to textually agree |
| 5 | 44.9s | waited for slow Qwen3-8B |
The previous grace logic only armed when (sole
answer). That covers the case where one worker has answered and the
rest are still pending. It does NOT cover the case where multiple
workers have answered fast but disagree.
This fix relaxes grace eligibility to: "at least one Answer-kind
output with confidence \u2265 0.5". When grace fires with multiple
qualifying answers, pick the highest-confidence one. The previous
sole-answer case is naturally preserved (max_by on a single element
returns that element).
Worker role and short-message variance is the dominant input here \u2014
fast and specialist workers running on different model families
rarely produce textually-identical short answers, so consensus is a
high bar. Grace gives us a sensible time-bounded escape hatch.
Tests:
* mesh-mixture-of-agents::fanout: 6 grace tests pass
* New: grace_fires_with_multiple_diverse_answers \u2014 pins the
real-world public-mesh case
* New: grace_picks_highest_confidence_when_multiple_qualify \u2014
pins the highest-confidence picking rule
* Existing 4 grace tests unchanged behavior; one was renamed
internally but still asserts \u2018sole-answer grace fires\u2019
Compat: no API change. The relaxed eligibility is strictly more
permissive than before, so existing callers see grace fire in cases
where it didn't before (the public-mesh latency-tail case). No
agentic regression \u2014 `!has_tools` still gates grace entirely.
…ng' into micn/moa-grace-on-diverse-answers * origin/micn/moa-propagate-enable-thinking: lab: update TODO after public-mesh probe + overnight report lab: overnight report + opinionated no-think A/B + public-mesh tok/s feat(moa): opinionated no-think default — workers don't think unless asked feat(moa): propagate enable_thinking to every worker and the reducer lab notes: rotation experiment + studio cross-validation wip: lab notes updates — VPN extension, retry impl, A/B baseline feat(network): one-shot same-target retry on pre-commit QUIC failures
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adjusts the MoA “grace” early-exit logic so that it can trigger when multiple workers respond quickly with diverse (non-textually-identical) answers, avoiding long tail latency caused by waiting for slow workers when consensus won’t form.
Changes:
- Relax grace eligibility from “exactly one qualifying answer” to “at least one qualifying Answer output (confidence ≥ 0.5)”.
- When grace fires, select and return the highest-confidence Answer rather than assuming a sole Answer.
- Add tests covering the real-world diverse-fast-answers scenario and the highest-confidence selection rule.
Comment on lines
+85
to
+89
| .max_by(|a, b| { | ||
| a.confidence | ||
| .partial_cmp(&b.confidence) | ||
| .unwrap_or(std::cmp::Ordering::Equal) | ||
| }) |
Comment on lines
+518
to
+522
| other => panic!("expected Decision::Answer, got {other:?}"), | ||
| } | ||
| // We had at least 2 answers when grace fired. | ||
| assert!(outputs.len() >= 2); | ||
| } |
Collaborator
Author
Live result on public mesh after the fixSame 5-run test, same prompt (
All 5 now sub-7s. The 6s ceiling is the grace timer firing on the first qualifying diverse answer; the 4.8s run probably had textual consensus fire first. Grace fired 4/5 times in the gateway log. This is the single biggest user-visible win in the lab so far. The chat UI on |
Bundled into the same PR (#624) as the eligibility relaxation. The two changes are inseparable in the chat user journey: * Eligibility relaxation (this PR's main change): grace fires on diverse fast answers, not just sole answers. Cures the 40s tail on the public mesh. * Tighter default (this commit): chat latency floor moves from ~6s to ~3s now that grace is the dominant chat path. Lab data on the public mesh: median mesh_chat went from ~6s to ~2s after this value, no quality regression on factual, arithmetic, or short-creative prompts. Without the eligibility relaxation, dropping the timer doesn't help much (grace rarely fires anyway). Without the tighter default, the relaxation just trades 40s for 6s instead of 40s for 2s. Reviewers should agree on both as one user-facing improvement.
Collaborator
Author
Update: also bundling the 6s\u00a0\u2192\u00a03s default change in this PRClosing #626 and rolling its one-line change in here. Reasoning: the two changes are inseparable in the chat user journey \u2014 the eligibility relaxation fixes the 40s tail, the tighter default takes us from 6s to ~2s. Either alone is a partial win; together they're the whole story (40s \u2192 ~2s). Latest commit ( Lab data updated:
Quality across 10 varied prompts (factual / arithmetic / subjective / creative-short) at grace=3s: all correct or reasonable answers. Same kind of "5-word constraint loose" behavior MoA shows at any grace value. |
…erse-answers * origin/main: Rebase llama.cpp patch queue for upstream canary (#609)
Comment on lines
82
to
85
| /// Chat-only grace: after this long since dispatch, if a single answer | ||
| /// (conf >= 0.5) is in, accept it instead of waiting for consensus. | ||
| /// Disabled for tool turns. Zero disables entirely. | ||
| pub first_answer_grace: Duration, |
Comment on lines
+127
to
+131
| // chat_template_kwargs.{enable_thinking, ...} | ||
| if let Some(kwargs) = obj.get("chat_template_kwargs").and_then(|v| v.as_object()) { | ||
| for alias in openai_frontend::common::THINKING_BOOLEAN_ALIASES { | ||
| if let Some(b) = kwargs.get(*alias).and_then(|v| v.as_bool()) { | ||
| result = Some(b); |
Comment on lines
+379
to
+385
| // 3 seconds is empirically good across the public mesh today. | ||
| // Long enough that slow-but-good workers (studio MiniMax | ||
| // landing at ~1s, mini Qwen3.5 at ~700ms) finish before the | ||
| // timer; short enough that chat doesn't sit on a multi-second | ||
| // ceiling on every turn. Lab data: median mesh_chat dropped | ||
| // from ~6s (old default) to ~2s with this value, no quality | ||
| // regression measured on factual / arithmetic / short-creative |
Comment on lines
+395
to
+397
| // Defaults to leaving each model's thinking behavior alone. | ||
| // `try_handle_moa` overrides this from the inbound request body | ||
| // when the caller has expressed a preference |
…ng' into micn/moa-grace-on-diverse-answers * origin/micn/moa-propagate-enable-thinking:
These were tracking files I wrote during the lab investigation. The MoA / grace code change stands on its own \u2014 the lab notes don't belong in the canonical repo. Moved to a private archive locally. (The accidental inclusion in the PR diff was making it look like I was modifying QUIC / keep-alive / discovery code, which I am not.)
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs:399
build_moa_configcomments sayenable_thinkingdefaults to leaving each model's behavior alone and is overridden only when the caller expresses a preference. Howevertry_handle_moaunconditionally setsconfig.enable_thinking = effective_enable_thinking_for_moa(&body_json), which currently defaults toSome(false)even when the caller is silent. Please update these comments (or the wiring) so the documented default matches the actual runtime behavior.
// Defaults to leaving each model's thinking behavior alone.
// `try_handle_moa` overrides this from the inbound request body
// when the caller has expressed a preference
// (`reasoning_effort: "none"`, `enable_thinking: false`, etc.).
enable_thinking: None,
Comment on lines
82
to
85
| /// Chat-only grace: after this long since dispatch, if a single answer | ||
| /// (conf >= 0.5) is in, accept it instead of waiting for consensus. | ||
| /// Disabled for tool turns. Zero disables entirely. | ||
| pub first_answer_grace: Duration, |
Comment on lines
+76
to
+90
| /// Returns `None` when the caller hasn't expressed a preference, leaving | ||
| /// each worker's default behavior alone. | ||
| /// MoA's opinionated default: workers do not think unless the caller | ||
| /// explicitly asks for it. Workers are short-budget internal slots, not | ||
| /// user-facing reasoning steps. The fast worker's 256-token budget is | ||
| /// far too small to fit `<think>…</think>` + answer, and the reducer | ||
| /// doesn't want reasoning prose as candidate input. | ||
| /// | ||
| /// The caller can still explicitly enable thinking (e.g. for | ||
| /// experimentation) via any of the recognised knobs — see | ||
| /// [`extract_enable_thinking_override`]. When no preference is | ||
| /// expressed, MoA picks for them: off. | ||
| fn effective_enable_thinking_for_moa(body: &serde_json::Value) -> Option<bool> { | ||
| extract_enable_thinking_override(body).or(Some(false)) | ||
| } |
Comment on lines
+3
to
+6
| //! When the MoA gateway is called with `reasoning_effort: "none"` (or any | ||
| //! other recognized "don't think" knob), every worker AND the reducer | ||
| //! must receive `chat_template_kwargs.enable_thinking: false` in their | ||
| //! outbound request body. That's how the reasoning-template flag reaches |
This was referenced May 22, 2026
Collaborator
Author
michaelneale
added a commit
that referenced
this pull request
May 22, 2026
…answers (#629) * feat(moa): propagate enable_thinking to every worker and the reducer For `model: "mesh"`, the OpenAI-style reasoning knobs (`reasoning_effort: "none"`, `reasoning: { enabled: false }`, `enable_thinking: false`, `thinking_budget: 0`, `chat_template_kwargs.enable_thinking`, and the THINKING_BOOLEAN_ALIASES) were silently dropped in MoA. The fast worker's 256-token budget then got burned inside an unclosed `<think>` block on reasoning models and never reached the actual answer. Live lab on a 3-node mesh (M4 Qwen2.5-3B + studio MiniMax-M2.5 + mini Qwen3.5-9B), "reply with one short word" prompt: | Run | default thinking | reasoning_effort: none | |-----|--------------------------|------------------------| | 1 | 32s, leaked "Thinking Process: 1. Analyze..." | 1s, "Okay." | | 2 | 17s, "Yes" | 2s, "Hello" | | 3 | 32s, leaked reasoning prose | 1s, "Hello" | Worker log on the slow runs shows Qwen3.5-9B returning 1700+ char payloads at 31s \u2014 the full runaway think block. On the no-think runs all three workers finish in <1.3s with 4-5 char clean answers. Implementation: * `SamplingParams` gains `enable_thinking: Option<bool>` and a builder-style `with_thinking(...)` helper. `None` (default) is "don't override" \u2014 callers without a preference see no behavior change. * New `backend::apply_enable_thinking(body, hint)` helper centralises the wire shape: injects `chat_template_kwargs.enable_thinking` (canonical llama.cpp chat-template knob) and `reasoning_effort: "none"` when disabled. Merges into existing `chat_template_kwargs` instead of clobbering. * `HttpBackend`, `LocalModelBackend`, and `RemoteModelBackend` all call it after building their request bodies. * `GatewayConfig` gains `enable_thinking: Option<bool>` so the choice flows in one place to every worker (`SamplingParams::worker().with_thinking(...)`) AND the hedged reducer (`SamplingParams::reducer().with_thinking(...)`). * `moa_gateway.rs::try_handle_moa` extracts the override from the inbound request body via the new `extract_enable_thinking_override` helper, which mirrors every shape that `openai_frontend::common::normalize_reasoning_template_options` recognises (so MoA users get the same surface as direct callers). Tests: * mesh-mixture-of-agents lib: 98 pass (6 new for apply_enable_thinking and SamplingParams::with_thinking). * mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs (new): 3 mock-backend integration tests pinning that chat_template_kwargs.enable_thinking reaches every worker AND that no spurious fields appear when no override is requested. * mesh-llm-host-runtime lib: 1478 pass (10 new for extract_enable_thinking_override covering every JSON shape). * clippy --all-targets -D warnings: clean. * cargo fmt --all -- --check: clean. Compat: * No mesh wire-protocol change. The new fields travel inside the existing chat-completion request JSON over the QUIC tunnel; peers on older binaries simply forward the body to llama.cpp which already understands `chat_template_kwargs`. Additive only, safe across mixed-version meshes. * No skippy ABI change. No plugin protocol change. * MoA callers without a preference get the same behavior as before this commit \u2014 `enable_thinking: None` is the default. Lab follow-up: a new `mesh_no_think` probe in the stability lab (`/tmp/lab/probe-stable.sh`) hits `/v1/chat/completions` with `reasoning_effort: "none"` so we can A/B latency and quality over hours of probe traffic. * feat(moa): opinionated no-think default — workers don't think unless asked Per Mic: MoA shouldn't have to ship behind a UI toggle. The MoA gateway now defaults to `enable_thinking = Some(false)` for every `model: "mesh"` request. Callers can still opt-in by passing any recognised reasoning knob explicitly (`reasoning_effort: "low"`, `enable_thinking: true`, etc.) — but the default is reasoning off. Rationale: * Workers are short-budget internal slots, not user-facing reasoning steps. The fast worker has a 256-token budget that doesn't fit `<think>...</think>` + answer. * The reducer doesn't want reasoning prose as candidate input. * MoA chat UX is the worst case for thinking models — every turn pays the reasoning latency penalty without giving the user the reasoning output. Implementation: * Extracted the policy into a tiny pure function, `effective_enable_thinking_for_moa(&body) -> Option<bool>`, that returns `extract_enable_thinking_override(body).or(Some(false))`. * 4 unit tests cover the contract: silent caller → no-think; explicit disable → no-think; explicit enable → thinking on (escape hatch); tool turn → still no-think by default. * The existing `extract_enable_thinking_override` tests are unchanged (they test the parser, not the gateway policy). Live verification on the 3-node lab mesh (M4 + studio + mini), release binary: * `model=mesh` with no knobs: 3/3 clean short answers ("Hello", "Okay", "Hello"), zero think leakage. * `model=mesh` + `reasoning_effort: "low"`: thinking turns back on, raw "Thinking Process:" prose appears as expected. * `model=mesh` + `tools=[read_file]`: tool_calls path unchanged, `finish_reason: tool_calls`, correct args. No-think default applies. Tests: 1486 host-runtime lib pass (4 new), all mesh-mixture-of-agents tests pass, clippy + fmt clean. * fix(moa): grace fires on diverse fast answers, not just sole answer Live public-mesh lab data showed a real failure: when 3-5 workers all return short answers in <1s but the answers don't textually agree (e.g. "Hello" / "Yes" / "Ready" / "Okay"), the arbiter's consensus rule ("\u22652 workers agree on answer") doesn't fire. MoA then waits for the slow tail worker before deciding, even though every fast worker has already produced a confident answer. Sample from the public-mesh lab (5 runs of "reply with one short word"): | Run | mesh time | what happened | |----:|----------:|---| | 1 | 45.5s | waited for slow Qwen3-8B | | 2 | 36.0s | waited for slow Qwen3-8B | | 3 | 38.2s | waited for slow Qwen3-8B | | 4 | 0.7s | 2 workers happened to textually agree | | 5 | 44.9s | waited for slow Qwen3-8B | The previous grace logic only armed when (sole answer). That covers the case where one worker has answered and the rest are still pending. It does NOT cover the case where multiple workers have answered fast but disagree. This fix relaxes grace eligibility to: "at least one Answer-kind output with confidence \u2265 0.5". When grace fires with multiple qualifying answers, pick the highest-confidence one. The previous sole-answer case is naturally preserved (max_by on a single element returns that element). Worker role and short-message variance is the dominant input here \u2014 fast and specialist workers running on different model families rarely produce textually-identical short answers, so consensus is a high bar. Grace gives us a sensible time-bounded escape hatch. Tests: * mesh-mixture-of-agents::fanout: 6 grace tests pass * New: grace_fires_with_multiple_diverse_answers \u2014 pins the real-world public-mesh case * New: grace_picks_highest_confidence_when_multiple_qualify \u2014 pins the highest-confidence picking rule * Existing 4 grace tests unchanged behavior; one was renamed internally but still asserts \u2018sole-answer grace fires\u2019 Compat: no API change. The relaxed eligibility is strictly more permissive than before, so existing callers see grace fire in cases where it didn't before (the public-mesh latency-tail case). No agentic regression \u2014 `!has_tools` still gates grace entirely. * fix(moa): also tighten chat grace default 6s -> 3s Bundled into the same PR (#624) as the eligibility relaxation. The two changes are inseparable in the chat user journey: * Eligibility relaxation (this PR's main change): grace fires on diverse fast answers, not just sole answers. Cures the 40s tail on the public mesh. * Tighter default (this commit): chat latency floor moves from ~6s to ~3s now that grace is the dominant chat path. Lab data on the public mesh: median mesh_chat went from ~6s to ~2s after this value, no quality regression on factual, arithmetic, or short-creative prompts. Without the eligibility relaxation, dropping the timer doesn't help much (grace rarely fires anyway). Without the tighter default, the relaxation just trades 40s for 6s instead of 40s for 2s. Reviewers should agree on both as one user-facing improvement. * fix(moa): normalize non-object chat_template_kwargs + clarify doc Address Copilot review on PR #629: 1. `apply_enable_thinking` silently dropped the flag if a caller passed a non-object for `chat_template_kwargs` (string, array, number, null). The `entry().or_insert_with()` doesn't replace a non-object, so `as_object_mut()` then returns None and the override is lost. We now normalize a bogus shape to {} before injecting, so the flag always applies. New test `apply_enable_thinking_normalizes_non_object_kwargs` pins all four bogus-input shapes. 2. `effective_enable_thinking_for_moa` had two doc comments merged together because there was no blank line / item separating them. The merged doc said both 'returns None when caller hasn't expressed a preference' (true of the inner extractor) AND 'MoA picks for them: off' (true of the outer policy). Split into one doc per function. No behavior change beyond fix #1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Real lab failure
When studio AND mini both joined the public mesh today (so MoA now had 5 workers: Qwen2.5-3B, Qwen3-8B, Qwen3.5-9B, MiniMax, Qwen3-32B),
model=meshlatency DEGRADED:Looking at the gateway log: Qwen2.5-3B answered in 84ms, Qwen3.5-9B in 670ms, MiniMax in 1011ms. All three fast answers in hand within 1 second. But because their content didn't textually agree ("Hello" / "Yes" / "Ready" / "Okay"), the arbiter's consensus rule ("\u22652 workers agree on answer") didn't fire. MoA then sat there for 30-45 seconds waiting for the tail worker (Qwen3-8B \u2014 the known-bimodal public-mesh peer).
The bug
The previous grace eligibility was:
That covers the case where one worker has answered and the rest are still pending. It does NOT cover the case where multiple workers have answered fast but their answers disagree.
Fix
Relax eligibility to "\u22651 Answer-kind output with confidence \u2265 0.5". When grace fires with multiple qualifying answers, pick the highest-confidence one. The previous sole-answer case is preserved trivially (max_by on a single element returns that element).
When grace fires:
Tests
mesh-mixture-of-agents::fanout: 6 grace tests pass.
New:
grace_fires_with_multiple_diverse_answers\u2014 pins the real-world public-mesh case (3 workers, fast diverse answers, slow tail).grace_picks_highest_confidence_when_multiple_qualify\u2014 pins the highest-confidence picking rule (mix of 0.5/0.7/0.9 answers \u2014 must pick 0.9).Existing 4 grace tests pass unchanged. The sole-answer case is now expressed via the relaxed eligibility, but its semantics are identical.
cargo test -p mesh-mixture-of-agentsclean.cargo clippy --all-targets -- -D warningsclean.cargo fmtclean.Compat
mesh_chatget slower.!has_toolsstill gates grace entirely. Tool turns continue to wait for consensus.Lab follow-up
The 3-node + public-mesh labs are running. Once this PR's binary is deployed I'll re-run the 5x
model=meshprobe and expect to see all 5 sub-2s, not just the 1/5 lucky-consensus case.Closes a real failure observed on the public mesh today; data in MIC_LAB_REPORT.md.