chat: real data by default, MoA for 'Auto', sole-answer grace for slow meshes by michaelneale · Pull Request #615 · Mesh-LLM/mesh-llm

michaelneale · 2026-05-21T06:05:01Z

What this gives you

The chat experience on the web console (local mesh-llm client --auto AND the fly app) was unusable in two distinct ways. This PR fixes both, on top of the already-merged MoA hardening (#612).

1. Real mesh data on first visit

A fresh visitor to http://localhost:3131/ (or fly) used to see fixture providers (Vast.ai, RunPod) on the Reserves page and harness data on Chat. The DataModeProvider defaulted initialMode to 'harness', persisted per-origin in localStorage. The harness/live split shipped recently (c3592b2b + #560) and the fly redeploy yesterday was what surfaced it broadly.

Flipped the default to 'live'. Harness is now an explicit opt-in (via the in-app data-mode toggle for designers, or initialMode='harness' for tests / the dev playground).

2. Chat with `Auto` actually returns something

Two layered fixes:

Sole-answer grace for MoA fan-out. Previously the arbiter waited for at least two answers so it could detect disagreement. On the public mesh that meant a perfectly good first answer at t=5\u201310s would sit unused while we waited 30\u201360s on slow peers. New first_answer_grace config (default 6s): once a single Answer-kind output with confidence \u2265 0.5 is in AND the grace window has elapsed, take it and abort the rest. Strictly chat-only \u2014 has_tools=true falls through to the existing consensus rule, so agentic harnesses (Goose, OpenCode, mini-agents) are unaffected.

Responses-API adapter for MoA. MoA's SSE writer always emitted chat.completion.chunk events, but the web console's chat path hits /api/responses (Responses API), which expects response.output_text.delta / response.completed etc. The Responses parser ignored MoA's chunks \u2014 so chat would say "streaming response\u2026" forever with model=mesh. Plumbed ResponseAdapter through try_handle_moa and added a Responses-shape SSE writer plus a non-streaming JSON converter (reuses the existing translate_chat_completion_to_responses helper).

Chat UI: Auto routes through MoA. With the above two fixes in place, Auto in the chat dropdown sends model: "mesh" to the backend. Users still see the "Auto" label \u2014 the value sent on the wire is the only thing changing. Direct API model=auto (curl, Goose, OpenCode, SDKs) is unchanged and still hits the single-model router.

Validation

cargo test -p mesh-mixture-of-agents: 91 pass (4 new grace tests, +1 integration suite tests in each sim file pinning grace=ZERO)
cargo test -p mesh-llm-host-runtime --lib: 1460 pass (2 new adapter tests)
cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime --all-targets -- -D warnings: clean
cargo fmt --all -- --check: clean
just build: clean

Live validation on public mesh

5 runs of the same prompt against a mesh-llm client --auto joined to the public mesh:

Path	Successes	Wall time on success
mesh (MoA + grace 6s)	5/5	17.1, 17.9, 18.1, 18.7, 19.0s
auto (single-model)	0/5	empty-content or 80s timeout
Qwen3-8B direct	1/5	1\u00d7 10.2s; 4\u00d7 80s timeout
Qwen2.5-3B direct	5/5	15.6\u201323.7s
Qwen3-32B direct	0/5	all 502 or sub-second empty

Tool-calling sanity: model=mesh + tools=[{read_file}] returns a real tool_calls reply with {"path":"\u2026"} in 8.6s, grace correctly bypassed, no early-exit log.

Chat UI manually verified end-to-end: pick Auto, type, get an answer in 10\u201320s with streaming text rendering. No more spinner-of-doom.

Architecture

mesh-mixture-of-agents crate is unchanged in shape \u2014 it still produces chat-completion JSON. The Responses-API translation lives entirely in moa_gateway.rs (host runtime), so MoA itself stays surface-agnostic. The grace knob is a new field on GatewayConfig, default 6s, zero disables.

Protocol

None changed. New SSE event shapes for MoA \u2014 same shapes the existing translator emits for non-MoA Responses-API responses. /v1/chat/completions with model=mesh still emits the same chat.completion.chunk SSE as before for agent harnesses.

… mesh The DataModeProvider previously defaulted initialMode to 'harness', which means every first-time visitor to a production console (local `mesh-llm`, the fly app, anywhere) sees fixture providers (Vast.ai, RunPod, etc.) on the Reserves page and harness data on Chat / Configuration until they toggle to 'live' \u2014 with the toggle then persisted in localStorage per-origin. That was the right default while the harness/live split lived in the ui-preview tree (designers iterating on mockups), but the swap-ui-preview commit (c3592b2) and the recent Reserves mockup PR made it user-visible on production deployments. The fly redeploy that finally picked these changes up was what surfaced the regression. Flip the provider default to 'live'. The 'harness' mode is now an explicit opt-in via the in-app data-mode toggle (persisted, so designers who flip it stay in harness) or by passing `initialMode=\"harness\"` from tests / the developer playground. Also flip the React `createContext` fallback to match, so a component reading useDataMode outside a provider (should never happen in production, but tests / Storybook) doesn't silently land in fixture mode. Tests: existing default-mode test rewritten to assert 'live'; new test pins that an explicit initialMode='harness' still works for designers and the developer playground. cargo & UI noop \u2014 no Rust changes.

Pulls in PR #612 (MoA hardening from review) so the local chat improvements (data-mode default flip, MoA fan-out grace, auto->mesh UI hack) sit on top of the latest gateway code. * origin/main: moa: harden gateway from PR #566 review (panic surface, error propagation, dedup race, dead code) (#612)

… turn On a slow or public mesh the arbiter can sit waiting for a second answer to detect disagreement while a perfectly good first answer is already in hand. Adds a time-based early-exit: * New `GatewayConfig::first_answer_grace` field (Duration). * fanout::gather_workers_incremental now runs a `tokio::select!` between the next worker event and a grace-expiry sleep. The sleep is only armed when we have exactly one Answer-kind output with confidence >= 0.5; otherwise the timer arm is effectively asleep. * Grace is disabled when has_tools=true OR when grace == 0, so agentic turns ALWAYS fall through to the existing consensus rule. Live result on the public mesh, 'how are you' prompt: * Before: 33.9s wall (waited for slow strong worker to 502). * After: 16-19s wall (grace early-exit on first sole Answer). Existing integration tests opt out (Duration::ZERO) so they preserve their event-driven assertions.

The chat UI's streaming client (`/api/responses` -> `/v1/responses`) expects OpenAI Responses-API events (`response.output_text.delta`, `response.completed`). MoA's SSE writer always emitted `chat.completion.chunk` events, which the Responses-API parser ignores -- so chat with model=mesh looked like it stalled. * run_moa_turn now takes `response_adapter` and forwards it to write_moa_response. * write_moa_response branches on adapter: - Streaming + OpenAiResponsesStream -> send_moa_as_responses_sse (new) emits response.created / output_text.delta / output_text.done / completed + [DONE]. - Streaming + chat-completions / None -> existing send_moa_as_sse emits chat.completion.chunk events. - Non-streaming + OpenAiResponsesJson -> single Responses-shape JSON via the existing response_adapter::translate_chat_completion_to_responses helper. - Failure path always returns non-streaming HTTP 502, unchanged. * Wires `first_answer_grace: Duration::from_secs(6)` into the gateway config so chat-only sole-answer grace ships with sane defaults. The MoA crate itself is unchanged -- it still produces chat-completion JSON. The adapter translation is all in moa_gateway. Live verified against /api/responses on a real mesh: event types `response.created`, `response.output_text.delta`, `response.output_text.done`, `response.completed`, plus `[DONE]`, chat UI renders the response. Tool-calling path unchanged. cargo test -p mesh-llm-host-runtime --lib: 1458/1458 pass cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings: clean

Locks in the behavioural guarantees of the two preceding commits: fanout::tests * grace_fires_when_lone_answer_qualifies_and_grace_elapsed * grace_does_not_fire_when_tools_present (agentic protection) * grace_zero_disables_the_check * grace_does_not_fire_below_confidence_threshold moa_gateway::tests * chat_completion_to_responses_json_returns_response_object * chat_completion_to_responses_json_passes_through_on_malformed mesh-mixture-of-agents lib: 91 pass (4 new) mesh-llm-host-runtime lib: 1460 pass (2 new)

The dropdown still shows 'Auto' but the model id sent on chat requests is the virtual `mesh` model, so the backend fans out across the mesh, arbitrates, and returns one answer. Why: on the public mesh `auto` routes to a single peer that often crashes mid-think or 502s, producing a stalled 'streaming response' spinner with no visible content. MoA in chat mode (with the new sole-answer grace + Responses-API adapter from the previous two commits) returns a real answer in ~10-20s consistently while gracefully handling peer failures. Scope: chat UI only. Direct API `model=auto` (curl, Goose, OpenCode, SDKs) is unchanged and still hits the single-model router.

Copilot

Pull request overview

This PR improves the web console chat and data defaults by switching the UI to live mesh data by default, routing the UI’s “Auto” chat selection through MoA (model="mesh"), and adding a chat-only sole-answer grace window so slow peers don’t block returning a good first answer. It also adds a Responses-API–compatible rendering path for MoA responses so /v1/responses streaming clients (the web UI) can display output correctly.

Changes:

Default UI data mode to live (fixtures/harness become explicit opt-in) and update related tests/fallback context.
Add first_answer_grace to MoA gateway config and implement a chat-only early-exit path when a single confident answer arrives.
Add MoA Responses-API adapter support (SSE + JSON conversion), and route Chat UI “Auto” to MoA on the wire.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs	Pins `first_answer_grace` to zero for deterministic sim behavior.
crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs	Pins `first_answer_grace` to zero for deterministic sim behavior.
crates/mesh-mixture-of-agents/tests/sim_tool_call_text_not_passed_as_content.rs	Pins `first_answer_grace` to zero for deterministic sim behavior.
crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs	Pins `first_answer_grace` to zero for deterministic sim behavior.
crates/mesh-mixture-of-agents/src/lib.rs	Adds `first_answer_grace` to `GatewayConfig` and threads it into worker gathering.
crates/mesh-mixture-of-agents/src/fanout.rs	Implements grace-timer early exit + adds unit tests for grace behavior.
crates/mesh-llm-ui/src/lib/data-mode/DataModeContext.tsx	Switches `DataModeProvider` default from `harness` to `live`.
crates/mesh-llm-ui/src/lib/data-mode/DataModeContext.test.tsx	Updates tests for new `live` default and explicit harness opt-in.
crates/mesh-llm-ui/src/lib/data-mode/data-mode-context.ts	Aligns unwrapped context fallback default to `live`.
crates/mesh-llm-ui/src/features/chat/pages/ChatPage.tsx	Routes UI “Auto” to backend `model="mesh"` (MoA).
crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs	Plumbs response adapter into MoA and adds Responses-API SSE/JSON rendering paths.

CI from the previous commits caught two things: * prettier reformatted the activeModelName ternary to one line. * 5 ChatPage tests asserted the wire model id was 'auto'; with the redirect it's now 'mesh'. Tests updated to match. Validation: * pnpm run format:check: clean * pnpm vitest run src/features/chat/pages/ChatPage.test.tsx: 45/45 pass

… comment Three issues from PR #615 Copilot review, all fixed with failing tests first then code: * ChatPage.tsx + ModelSelect.tsx: the chat dropdown was showing the wrong (or no) selected option when 'Auto' was picked, because the controlled value was the wire model id ('mesh') which Radix Select couldn't find in . Now there are two derived values: - selectedModelValue (UI): always one of the option values. - activeModelName (wire): 'mesh' when Auto is selected, real model name otherwise. Also: dropdown entry is now just 'Mesh — automatic' with no meta description, and ModelSelect's special-case label for value='auto' is gone. * moa_gateway.rs: send_moa_as_responses_sse was forwarding MoA's chat-shape ({prompt_tokens, completion_tokens}) straight into the Responses-API completed event, which expects {input_tokens, output_tokens}. Now converted via openai_frontend::responses::chat_usage_to_responses_usage. * fanout.rs: test helper comment said 'chat-completion JSON' but was actually the MoA normalization envelope. Clarified. Tests added: * keeps Auto highlighted in the dropdown even when other live models are available — proves the controlled-value bug is gone. * responses_sse_emits_responses_shape_usage_not_chat_shape — proves input_tokens/output_tokens/total_tokens are emitted, and chat-shape keys do NOT leak. Validation: * cargo test -p mesh-llm-host-runtime --lib: 1461/1461 * cargo test -p mesh-mixture-of-agents --lib: 91/91 * pnpm vitest run (UI): 656/656 (3 skipped) * clippy + fmt + prettier: clean

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Three nits from Copilot's second review pass, all addressed, plus a regression test that catches a class of streaming regressions we were worried about: * moa_gateway.rs: response.created and response.completed events were emitting different response.id values inside the same SSE stream (one auto-generated 'resp_<created_at>', one from the chat body). Clients that correlate events by response.id were seeing what looked like an orphaned 'created' event. The created event now uses the same response_id as completed. * fanout.rs: test helper answer_text was interpolating the payload directly into a JSON template, which would break on payloads containing quotes, backslashes, or newlines. Switched to serde_json::json! so the envelope is always valid. * moa_gateway.rs: doc comment on capture_responses_sse_body said it 'strips HTTP/chunked framing' but it returns raw bytes. Updated the comment to describe the actual behavior (callers use .contains() which is framing-robust). New regression test in transport.rs: * relay_translated_responses_stream_emits_one_delta_per_upstream_chunk Drives relay_translated_responses_stream with a fake upstream that emits 3 separate chat.completion.chunk frames, asserts the relay emits >=3 response.output_text.delta events. Locks in the invariant that direct-model /v1/responses streams token-by-token and that a future refactor of the relay can't accidentally buffer the whole upstream body into a single delta. New regression test in moa_gateway.rs: * responses_sse_uses_same_response_id_for_created_and_completed Pins the response.id correlation invariant fixed above. Validation: * cargo test -p mesh-mixture-of-agents --lib: 91/91 pass * cargo test -p mesh-llm-host-runtime --lib: 1463/1463 pass (2 new) * cargo clippy + fmt: clean * Live curl against rebuilt release binary: - created/completed share response.id - usage emits input_tokens/output_tokens, no chat-shape leak - direct-model /v1/responses emits 26 deltas for 'count to ten' - mesh path returns clean answer

Resolves a single conflict in crates/mesh-llm-ui/src/lib/data-mode/DataModeContext.tsx where both sides changed the DataModeProvider default initialMode: * origin/main (#615) hard-coded 'live' so production consoles (fly app, embedded UI in shipped mesh-llm binary) stop leaking fixture providers labelled Vast.ai / RunPod onto the Reserves page. * This branch introduced a defaultInitialMode() helper that returns 'harness' in dev builds and 'live' in production builds. The helper is a strict superset of #615's intent: production bundles still default to 'live', while 'npm run dev' keeps the harness default so designers iterating on mockups don't have to flip the in-app toggle every reload. Kept the helper, expanded the rationale comment to cite the #615 regression, and updated DataModeContext.test.tsx to pin env.isDevelopment explicitly for both production ('live') and dev ('harness') defaults (mirroring the env-mutation pattern already used in app-tabs.test.tsx).

michaelneale added 6 commits May 21, 2026 14:32

Copilot AI review requested due to automatic review settings May 21, 2026 06:05

Copilot started reviewing on behalf of michaelneale May 21, 2026 06:05 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

Comment thread crates/mesh-llm-ui/src/features/chat/pages/ChatPage.tsx Outdated

Comment thread crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs

Comment thread crates/mesh-mixture-of-agents/src/fanout.rs Outdated

michaelneale added 2 commits May 21, 2026 16:11

Copilot AI review requested due to automatic review settings May 21, 2026 06:29

Copilot started reviewing on behalf of michaelneale May 21, 2026 06:30 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

Comment thread crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs

Comment thread crates/mesh-mixture-of-agents/src/fanout.rs Outdated

Comment thread crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs Outdated

This was referenced May 21, 2026

MoA: fast worker shouldn't think — thinking models can poison fan-out and leak reasoning to chat #617

Closed

MoA: stream the winning worker's tokens instead of one-shot delta #618

Closed

michaelneale merged commit 3665c93 into main May 21, 2026
18 checks passed

michaelneale deleted the micn/data-mode-default-live branch May 21, 2026 07:35

michaelneale mentioned this pull request May 21, 2026

doc(chat): comment AUTO_BACKEND_MODEL as the one-line flip point #619

Merged

michaelneale mentioned this pull request May 22, 2026

feat(moa): no-think default on workers + grace fires on diverse fast answers #629

Merged

michaelneale mentioned this pull request Aug 13, 2026

Unify auto and mesh into one auto directive (MoA with modes) #1289

Open

Navigation Menu

Uh oh!

Uh oh!

chat: real data by default, MoA for 'Auto', sole-answer grace for slow meshes - #615

chat: real data by default, MoA for 'Auto', sole-answer grace for slow meshes#615
michaelneale merged 9 commits into
mainfrom
micn/data-mode-default-live

michaelneale commented May 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

michaelneale commented May 21, 2026

What this gives you

1. Real mesh data on first visit

2. Chat with Auto actually returns something

Validation

Live validation on public mesh

Architecture

Protocol

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

2. Chat with `Auto` actually returns something