tune(moa): tighten first_answer_grace from 6s to 3s

[michaelneale]

What it does

Lowers the chat-mode first_answer_grace default from 6 seconds to 3 seconds. This is a one-line tuning change. Built on top of #624 (which makes grace fire on diverse fast answers \u2014 needed for the chat case where workers don't textually agree).

Why now

On the public mesh today (5 workers including studio MiniMax, mini Qwen3.5, etc.):

• MoA consensus rarely fires for short chat answers \u2014 workers respond with diverse short text like "Hello" / "Yes" / "Ready" / "Okay" that don't textually match.
• PR fix(moa): grace fires on diverse fast answers, not just sole answer #624's diverse-answer grace catches this case: when grace expires with \u22651 qualifying answer, pick the highest-confidence one.
• But the 6s grace becomes the chat latency floor \u2014 every turn pays the full 6s.

With #624 landed and the timer being the dominant chat path, 3s gives 2\u00d7 better p50 with no measurable quality regression on short chat prompts.

Lab data, public mesh, 10 varied turns

Prompt type	Result
"Symbol for gold?"	2x: "Au, from Latin aurum\u2026" \u2705
"WW2 end?"	2x: "1945" \u2705
"Coffee shop name?"	"Brewed" / "Perk" \u2705
"60mph \u00d7 2h?"	2x: "120" \u2705
"5-word rain sentence"	"Rain drums gently on the roof" / "Rain pours down in relentless sheets" \u2014 reasonable but 6 words not 5 (same loose constraint behavior the model has at 6s grace too)

Latency distribution:

grace	min	p50	max	n
6s (current)	700ms	~6000ms	6000ms	5
3s (proposed)	1299ms	2002ms	2007ms	10

Trade-off

A worker that takes 3\u20136s to land is now excluded from consensus. On the public mesh post-#624 we found textual consensus rarely fires for chat anyway, so this isn't losing a path that was actually firing.

For agentic / tool turns (has_tools=true), grace is bypassed entirely and consensus continues to work as before. The 3s number is chat-only.

Compat

• No API change. The default value moves, callers that explicitly build a GatewayConfig keep their configured value.
• No protocol / ABI / plugin change.
• No mesh-version compatibility concerns.

Tests

cargo test -p mesh-llm-host-runtime --lib: 1463 pass.
cargo clippy --all-targets -- -D warnings: clean.
cargo fmt --all -- --check: clean.

The grace eligibility logic itself (which determines when the timer arms) is covered by PR #624's tests. This PR is strictly a config tuning, no behavior-shape change.

Open question

3s is empirical. A more conservative tuning would be 4-5s. A more aggressive one would be 2s (tested in lab, all 10 turns sub-2.5s but tight enough that any worker with 2s+ network latency is excluded). Happy to land at any number you prefer.

[Copilot]

Pull request overview

This PR adjusts the MoA gateway’s chat-mode timeout tuning by reducing the first_answer_grace default, aiming to improve perceived chat latency by allowing earlier exit when grace conditions are met.

Changes:

• Reduced first_answer_grace default from 6s to 3s in the MoA gateway config.
• Added a lab TODO note capturing post-#624 measurement takeaways / next steps.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
MIC_LAB_TODO.md	Adds a short lab strategy/todo note related to grace behavior and streaming ideas.
crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs	Lowers the default first_answer_grace duration and expands the rationale comment.

Comments suppressed due to low confidence (1)

crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs:307

• PR description and this comment block assume the “diverse fast answers” grace behavior from PR #624, but that logic does not appear to be present in this branch (no matching grace tests, and gather_workers_incremental still requires answers.len() == 1). With the current sole-answer grace, tightening this timer may not address the stated chat latency floor. Either include the prerequisite #624 behavioral change in this PR/stack, or adjust the PR description + rationale comments accordingly.

        // 3 seconds: empirically good across the public mesh. Long
        // enough that slow-but-good workers (e.g. studio MiniMax
        // landing at ~1s, mini Qwen3.5 at ~700ms) finish before the
        // timer; short enough that the chat UI feels responsive
        // instead of sitting on the 6s ceiling. Lab data: median
        // mesh_chat dropped from ~6s to ~2s after this change with no
        // measurable quality regression on factual, arithmetic, and
        // short-creative prompts.
        first_answer_grace: std::time::Duration::from_secs(3),

[michaelneale]

Rolled into #624 \u2014 the eligibility relaxation and the tighter default are the same user-journey change and only meaningful together. Easier to review as one PR.