feat(moa): chunked output_text.delta so /v1/responses streams smoothly by michaelneale · Pull Request #621 · Mesh-LLM/mesh-llm

michaelneale · 2026-05-21T17:07:20Z

Closes #618.

MoA was emitting a single response.output_text.delta event with the entire arbitrated answer, making the chat UI feel like a long spinner followed by a hard pop-in. This PR splits the buffered content into ~24-char chunks (roughly 5 tokens of English text) and emits one delta per chunk so the UI renders incrementally.

This is presentation-layer chunking, not real worker-stream splicing. By the time MoA decides on a winner the content is already buffered, so the underlying timing is unchanged \u2014 the user-perceived feel just changes from "long spinner then pop" to "text flowing in". Issue #618's option C (cheapest). The bigger option A (splice the winning worker's chat.completion.chunk stream into the Responses-API SSE during arbitration) is left as a separate follow-up.

Implementation

New chunk_for_streaming(&str) -> Vec<&str> helper. Walks UTF-8 character boundaries, prefers to break at whitespace when that produces a reasonably-sized chunk, falls back to the full window when no whitespace is close. Lossless reassembly is a tested invariant.
send_moa_as_responses_sse now emits one delta per chunk plus the existing created / output_text.done / completed / [DONE] events.
Empty content still produces a well-formed end-of-stream so clients don't hang.

Tests

7 new in crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs:

5 unit tests on chunk_for_streaming:
- empty input \u2192 no chunks
- short content \u2192 single chunk
- long content \u2192 multiple chunks with lossless reassembly
- multi-byte UTF-8 (\u00e9, \u4f60\u597d, \ud83d\udc4b, \u3053\u3093\u306b\u3061\u306f) \u2014 no codepoint splits, lossless reassembly
- no-whitespace runs \u2014 falls back to char window
2 end-to-end SSE-shape tests via the existing TCP-loopback harness:
- long content emits \u22653 deltas + exactly 1 done + 1 completed
- empty content emits 0 deltas + still 1 done + 1 completed + [DONE]

cargo test -p mesh-llm-host-runtime --lib: 1477 pass (7 new).
cargo clippy --all-targets -- -D warnings: clean.
cargo fmt --all -- --check: clean.

Compat

No wire-protocol change. Same SSE event types (response.output_text.delta, etc.), just more of them.
No mesh / skippy / plugin protocol change.
Clients that already parse multi-delta streams (any spec-compliant Responses-API client) handle this correctly without modification.

Live validation

To follow once the binary is rebuilt and pushed to the lab. The release-build is wired into the same lab that already validated PR #620 (opinionated MoA no-think). Once swapped in, the chat UI on M4 should feel like real streaming for model=mesh instead of one-shot pop.

When the gateway forwards a request to a remote peer via QUIC and the upstream pre-commit phase fails (tunnel open / request forward / response probe), the failure becomes `RouteAttemptResult::RetryableUnavailable`. Today the outer routing loop retries on the next available target, which fixes the case where multiple peers serve the same model but does nothing when only one peer serves it: a single transient QUIC path teardown kills the request, even though iroh typically reconnects within ~1s on a fresh connection. This is the dominant failure mode on asymmetric / lossy direct paths (corporate firewalls, hotel wifi, Tailscale interfering with NAT traversal, M4-style incoming filter rules). Real-world meshes will see this constantly. Implementation: * Splits `route_remote_attempt` into a `_once` helper plus a wrapper that retries once with a 750ms backoff if the first attempt returned `RetryableUnavailable`. 750ms is below user-perceptible-stall and matches the typical iroh reconnect window. * Retry is only safe on `RetryableUnavailable` because that variant is exclusively set BEFORE any bytes are written to the client TCP stream (probe failed, tunnel open failed, buffered-request forward failed). Any post-commit error or non-network outcome falls through unchanged. * Pure-function `should_retry_remote_attempt` extracted for unit testing; 5 tests pin the retry policy (yes on unavailable; no on delivered / timeout / context-overflow / client-disconnected). Live lab validation pending on a 3-node mesh where the M4 \u2194 mini path is consistently asymmetric. Baseline before the fix was direct_mini 27% success over 1h40m. Expect the curl_fail subset (pre-commit drops) to recover to ~100% with this change; http_err 429 subset (mini's own admission control) is unaffected by design.

For `model: "mesh"`, the OpenAI-style reasoning knobs (`reasoning_effort: "none"`, `reasoning: { enabled: false }`, `enable_thinking: false`, `thinking_budget: 0`, `chat_template_kwargs.enable_thinking`, and the THINKING_BOOLEAN_ALIASES) were silently dropped in MoA. The fast worker's 256-token budget then got burned inside an unclosed `<think>` block on reasoning models and never reached the actual answer. Live lab on a 3-node mesh (M4 Qwen2.5-3B + studio MiniMax-M2.5 + mini Qwen3.5-9B), "reply with one short word" prompt: | Run | default thinking | reasoning_effort: none | |-----|--------------------------|------------------------| | 1 | 32s, leaked "Thinking Process: 1. Analyze..." | 1s, "Okay." | | 2 | 17s, "Yes" | 2s, "Hello" | | 3 | 32s, leaked reasoning prose | 1s, "Hello" | Worker log on the slow runs shows Qwen3.5-9B returning 1700+ char payloads at 31s \u2014 the full runaway think block. On the no-think runs all three workers finish in <1.3s with 4-5 char clean answers. Implementation: * `SamplingParams` gains `enable_thinking: Option<bool>` and a builder-style `with_thinking(...)` helper. `None` (default) is "don't override" \u2014 callers without a preference see no behavior change. * New `backend::apply_enable_thinking(body, hint)` helper centralises the wire shape: injects `chat_template_kwargs.enable_thinking` (canonical llama.cpp chat-template knob) and `reasoning_effort: "none"` when disabled. Merges into existing `chat_template_kwargs` instead of clobbering. * `HttpBackend`, `LocalModelBackend`, and `RemoteModelBackend` all call it after building their request bodies. * `GatewayConfig` gains `enable_thinking: Option<bool>` so the choice flows in one place to every worker (`SamplingParams::worker().with_thinking(...)`) AND the hedged reducer (`SamplingParams::reducer().with_thinking(...)`). * `moa_gateway.rs::try_handle_moa` extracts the override from the inbound request body via the new `extract_enable_thinking_override` helper, which mirrors every shape that `openai_frontend::common::normalize_reasoning_template_options` recognises (so MoA users get the same surface as direct callers). Tests: * mesh-mixture-of-agents lib: 98 pass (6 new for apply_enable_thinking and SamplingParams::with_thinking). * mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs (new): 3 mock-backend integration tests pinning that chat_template_kwargs.enable_thinking reaches every worker AND that no spurious fields appear when no override is requested. * mesh-llm-host-runtime lib: 1478 pass (10 new for extract_enable_thinking_override covering every JSON shape). * clippy --all-targets -D warnings: clean. * cargo fmt --all -- --check: clean. Compat: * No mesh wire-protocol change. The new fields travel inside the existing chat-completion request JSON over the QUIC tunnel; peers on older binaries simply forward the body to llama.cpp which already understands `chat_template_kwargs`. Additive only, safe across mixed-version meshes. * No skippy ABI change. No plugin protocol change. * MoA callers without a preference get the same behavior as before this commit \u2014 `enable_thinking: None` is the default. Lab follow-up: a new `mesh_no_think` probe in the stability lab (`/tmp/lab/probe-stable.sh`) hits `/v1/chat/completions` with `reasoning_effort: "none"` so we can A/B latency and quality over hours of probe traffic.

…asked Per Mic: MoA shouldn't have to ship behind a UI toggle. The MoA gateway now defaults to `enable_thinking = Some(false)` for every `model: "mesh"` request. Callers can still opt-in by passing any recognised reasoning knob explicitly (`reasoning_effort: "low"`, `enable_thinking: true`, etc.) — but the default is reasoning off. Rationale: * Workers are short-budget internal slots, not user-facing reasoning steps. The fast worker has a 256-token budget that doesn't fit `<think>...</think>` + answer. * The reducer doesn't want reasoning prose as candidate input. * MoA chat UX is the worst case for thinking models — every turn pays the reasoning latency penalty without giving the user the reasoning output. Implementation: * Extracted the policy into a tiny pure function, `effective_enable_thinking_for_moa(&body) -> Option<bool>`, that returns `extract_enable_thinking_override(body).or(Some(false))`. * 4 unit tests cover the contract: silent caller → no-think; explicit disable → no-think; explicit enable → thinking on (escape hatch); tool turn → still no-think by default. * The existing `extract_enable_thinking_override` tests are unchanged (they test the parser, not the gateway policy). Live verification on the 3-node lab mesh (M4 + studio + mini), release binary: * `model=mesh` with no knobs: 3/3 clean short answers ("Hello", "Okay", "Hello"), zero think leakage. * `model=mesh` + `reasoning_effort: "low"`: thinking turns back on, raw "Thinking Process:" prose appears as expected. * `model=mesh` + `tools=[read_file]`: tool_calls path unchanged, `finish_reason: tool_calls`, correct args. No-think default applies. Tests: 1486 host-runtime lib pass (4 new), all mesh-mixture-of-agents tests pass, clippy + fmt clean.

Closes #618. MoA was emitting a single `response.output_text.delta` event with the entire arbitrated answer, making the chat UI feel like a long spinner followed by a hard pop-in. This commit splits the buffered content into ~24-char chunks (roughly 5 tokens of English text) and emits one delta per chunk so the UI renders incrementally. This is presentation-layer chunking \u2014 by the time MoA decides on a winner the content is already buffered, so the underlying timing is unchanged. Real worker-stream splicing (option A from #618) is a bigger change tracked separately. Implementation: * New `chunk_for_streaming(&str) -> Vec<&str>` helper. Walks UTF-8 character boundaries, prefers to break at whitespace when that produces a reasonably-sized chunk, falls back to the full window when no whitespace is close. Lossless reassembly under tests. * `send_moa_as_responses_sse` now emits one delta per chunk plus the existing created / output_text.done / completed / [DONE] events. * Empty content still produces a well-formed end-of-stream (done + completed + [DONE]) so clients don't hang. Tests (7 new): * 5 unit tests on `chunk_for_streaming`: - empty input \u2192 no chunks - short content \u2192 single chunk - long content \u2192 multiple chunks with lossless reassembly - multi-byte UTF-8 (\u00e9, \u4f60\u597d, \ud83d\udc4b, \u3053\u3093\u306b\u3061\u306f) \u2014 no codepoint splits, lossless - no-whitespace runs \u2014 falls back to char window * 2 end-to-end SSE-shape tests via the TCP-loopback harness: - long content emits \u22653 deltas + exactly 1 done + 1 completed - empty content emits 0 deltas + still 1 done + 1 completed + [DONE] Validation: * cargo test -p mesh-llm-host-runtime --lib: 1477 pass (7 new) * cargo clippy --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * Live smoke pending on the lab once binary is rebuilt.

…able-thinking * origin/main: Cover asymmetric KV cache width pricing (#594)

Resolved a non-substantive test-list conflict between #620 and #621 \u2014 both branches appended new tests at the end of moa_gateway.rs's `mod tests`. Both blocks retained. No behavioural overlap; the changes are independent. This branch now carries both for combined lab validation. 1489 host-runtime lib tests pass, clippy + fmt clean.

Copilot

Pull request overview

This PR improves the perceived streaming behavior of MoA responses served via /v1/responses?stream=true by splitting the already-buffered arbitrated answer into multiple response.output_text.delta SSE events, so the chat UI renders text incrementally rather than as a single pop-in.

Changes:

Emit one response.output_text.delta per chunk (instead of a single full-content delta) in send_moa_as_responses_sse.
Add chunk_for_streaming(&str) -> Vec<&str> to chunk content on UTF-8 character boundaries with a whitespace-preferred split heuristic.
Add unit tests for chunking behavior and loopback SSE-shape tests to pin the multi-delta stream contract (including empty-content behavior).

+        // Prefer to break at the last whitespace boundary we saw,
+        // unless that would produce a tiny chunk (< half the budget),
+        // in which case just take the full window.
+        let end = match last_ws_byte {
+            Some(ws) if ws - start >= MOA_STREAM_CHUNK_CHARS / 2 => ws,
+            _ => probe,
+        };


+            // SAFETY/correctness: `content` is &str so we can find the
+            // next char boundary by checking utf8_char_width on the
+            // first byte. Use core::str primitives instead.
+            let ch = content[ch_start..].chars().next().expect("char");


Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

+        // Prefer to break at the last whitespace boundary we saw,
+        // unless that would produce a tiny chunk (< half the budget),
+        // in which case just take the full window.
+        let end = match last_ws_byte {
+            Some(ws) if ws - start >= MOA_STREAM_CHUNK_CHARS / 2 => ws,
+            _ => probe,
+        };


+            // SAFETY/correctness: `content` is &str so we can find the
+            // next char boundary by checking utf8_char_width on the
+            // first byte. Use core::str primitives instead.
+            let ch = content[ch_start..].chars().next().expect("char");


+/// Single attempt against a remote host: open tunnel, forward request,
+/// probe response. Returns `RouteAttemptResult` describing what happened.
+///
+/// `route_remote_attempt` (below) wraps this with a one-shot same-target
+/// retry on `RetryableUnavailable`, because real-world meshes have
+/// asymmetric/lossy direct paths and a single transient QUIC path drop
+/// shouldn't doom a request when iroh typically reconnects within ~1s.


+# Lab / overnight TODO (running list)
+
+Captured from the conversation so I don't goldfish them. Updated as
+items are done.


+    let enable_thinking = effective_enable_thinking_for_moa(&body_json);
+
+    let Some(mut config) = build_moa_config(node, targets).await else {
        let _ = proxy::send_503(tcp_stream, "MoA requires ≥2 models available in the mesh").await;
        return None;
    };
+    config.enable_thinking = enable_thinking;



michaelneale · 2026-05-21T20:32:00Z

Honest result on the lab: this PR does NOT improve TTFT

I just timed each delta arrival from a real lab MoA stream:

[  6122ms] response.created
[  6122ms] delta: 'The ocean stretches '
[  6122ms] delta: 'like a vast, heaving '
[  6122ms] delta: 'emerald sea, its '
[  6122ms] delta: 'surface fractured by '
\u2026 21 more deltas all at the same 6122-6123ms timestamp \u2026
[  6123ms] response.output_text.done
[  6123ms] response.completed

All 22 deltas arrive in a 1ms window. That's because MoA still buffers the entire answer before chunking begins (option C from #618 is presentation chunking, not real streaming).

So this PR's actual benefit is wire-shape correctness \u2014 the SSE stream looks like a normal streaming response with multiple deltas instead of a degenerate single-delta stream \u2014 not perceived latency. The chat UI rendering them as they arrive will still see a single frame's worth of content arrive at t=6.1s.

For real TTFT improvement we need option A from #618: splice the winning worker's upstream chat.completion.chunk stream into the Responses-API SSE during arbitration, so the user sees tokens as the model produces them.

Recommendation: keep this PR as a small wire-shape correctness fix; open a follow-up for the proper option-A streaming work. Or close this PR and go straight to option A. Your call.

michaelneale · 2026-05-21T21:46:14Z

Recommendation: close this and do option A

After lab measurement and Mic's note ("how do we stream from worker when we don't know what we'll do with it?") I'm convinced this PR is wire-shape correctness only \u2014 zero perceived TTFT win because the chunks all arrive in a 1ms window at the client end (content is buffered before chunking).

The real fix is option A from #618: stream from the reducer (when arbiter routes to it) or from the winning worker (when arbiter picks one) by holding the upstream chat.completion.chunk stream open through arbitration. That's a ~100 LoC change in moa_gateway, and it's actually what users would feel.

I'll leave this PR open for now in case you want the wire-shape correctness anyway (multi-delta is the canonical Responses-API shape and some clients may render better with it). But I'd recommend closing and doing option A. Will start option A on a fresh branch unless you'd rather skip it.

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs:380

build_moa_config comments say enable_thinking defaults to leaving model behavior alone and is only overridden when the caller expresses a preference, but try_handle_moa now unconditionally sets config.enable_thinking (defaulting to Some(false)). Please align this comment with the actual behavior.

        // Defaults to leaving each model's thinking behavior alone.
        // `try_handle_moa` overrides this from the inbound request body
        // when the caller has expressed a preference
        // (`reasoning_effort: "none"`, `enable_thinking: false`, etc.).
        enable_thinking: None,

+/// Returns `None` when the caller hasn't expressed a preference, leaving
+/// each worker's default behavior alone.
+/// MoA's opinionated default: workers do not think unless the caller
+/// explicitly asks for it. Workers are short-budget internal slots, not
+/// user-facing reasoning steps. The fast worker's 256-token budget is


+        // Prefer to break at the last whitespace boundary we saw,
+        // unless that would produce a tiny chunk (< half the budget),
+        // in which case just take the full window.
+        let end = match last_ws_byte {
+            Some(ws) if ws - start >= MOA_STREAM_CHUNK_CHARS / 2 => ws,
+            _ => probe,


+/// Single attempt against a remote host: open tunnel, forward request,
+/// probe response. Returns `RouteAttemptResult` describing what happened.
+///
+/// `route_remote_attempt` (below) wraps this with a one-shot same-target
+/// retry on `RetryableUnavailable`, because real-world meshes have
+/// asymmetric/lossy direct paths and a single transient QUIC path drop


michaelneale · 2026-05-21T22:04:45Z

Closing per discussion: this is wire-shape-only and doesn't help TTFT \u2014 chunks all arrive in 1ms at the client end because content is buffered before chunking. Going to do option A (real worker-stream splicing from the reducer / winning worker) on a fresh branch instead. Issue #618 stays open for that work.

michaelneale added 9 commits May 21, 2026 19:31

wip: lab notes updates — VPN extension, retry impl, A/B baseline

af0fde6

lab notes: rotation experiment + studio cross-validation

e589a4b

lab: overnight report + opinionated no-think A/B + public-mesh tok/s

5d64d97

lab: update TODO after public-mesh probe + overnight report

9bb481d

Merge remote-tracking branch 'origin/main' into micn/moa-propagate-en…

43717d8

…able-thinking * origin/main: Cover asymmetric KV cache width pricing (#594)

Copilot AI review requested due to automatic review settings May 21, 2026 17:07

Copilot started reviewing on behalf of michaelneale May 21, 2026 17:08 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

lab: same-model lab vs public-mesh A/B; 100x throughput gap

2a846b3

Copilot AI review requested due to automatic review settings May 21, 2026 17:24

lab: update TODO after streaming PR + lab-vs-public A/B

07439be

Copilot started reviewing on behalf of michaelneale May 21, 2026 17:24 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

michaelneale added 2 commits May 22, 2026 07:21

lab: studio on public mesh — 47 tok/s, MoA solid, auto improved

f8a0238

lab: diverse-answer grace bug + mini on public mesh observations

f743f06

Copilot AI review requested due to automatic review settings May 21, 2026 21:46

Copilot started reviewing on behalf of michaelneale May 21, 2026 21:46 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

michaelneale mentioned this pull request May 21, 2026

fix(moa): grace fires on diverse fast answers, not just sole answer #624

Closed

michaelneale closed this May 21, 2026

Navigation Menu

Uh oh!

feat(moa): chunked output_text.delta so /v1/responses streams smoothly - #621

feat(moa): chunked output_text.delta so /v1/responses streams smoothly#621
michaelneale wants to merge 14 commits into
mainfrom
micn/moa-streaming-chunked

michaelneale commented May 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

michaelneale commented May 21, 2026

Uh oh!

michaelneale commented May 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

michaelneale commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

michaelneale commented May 21, 2026

Implementation

Tests

Compat

Live validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

michaelneale commented May 21, 2026

Honest result on the lab: this PR does NOT improve TTFT

Uh oh!

michaelneale commented May 21, 2026

Recommendation: close this and do option A

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

michaelneale commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants