macos menu app by i386 · Pull Request #1 · Mesh-LLM/mesh-llm

i386 · 2026-02-18T03:27:07Z

No description provided.

smart_auto now returns a ranked list of candidates instead of a single best. The caller probes each in order — if #1 is unreachable (stale Nostr listing, dead peer), it falls through to #2, #3, etc. Only starts a new mesh if all candidates fail. Fixes the restart race: old Nostr listing still cached, new process probes it, times out, and used to give up. Now it tries the next mesh.

* docs: update prereqs for RELEASE * docs: remove unnecessary --console option from RELEASE

…didates - extract Node::decode_invite_token() to validate tokens without connecting - reject listings with invalid invite tokens in discover() before ranking - queue all candidates (not just #1) so auto-join falls back if top pick is unreachable - fix integration test to use valid invite tokens

@main

Tested against live mesh: GLM-4.7-Flash (local) + Qwen3-8B (remote peer). Three fixes from live mesh testing: 1. Local reducer: prefer first endpoint (local model) as reducer instead of last. Remote models over QUIC relay are fine as parallel workers but timeout as the sequential reducer. Result: reducer response times dropped from 180s (remote timeout) to 2-18s (local). 2. Model dedup: mesh-llm exposes the same model under multiple aliases (e.g. unsloth/GLM-4.7-Flash-GGUF and @main:Q4_K_M variant). discover_endpoints() now deduplicates by normalized display name. Extracted as shared lib function used by all three test binaries. 3. KV parse fix: models that say 'kind: answer' but also include 'tool: read_file' are now correctly classified as ToolProposal. This was the #1 cause of missed tool calls in the agentic flow.

…#566) * feat: MoA gateway — stateful mixture-of-agents with tool arbitration New standalone crate (moa-gateway) that fans out to N heterogeneous LLM endpoints in parallel, normalizes dirty worker outputs, arbitrates with deterministic logic, and manages the full tool call lifecycle across turns. Tested live against 3 ollama models (llama3.2:3b, qwen3:4b, qwen3.6:27b): - Knowledge/reasoning: picks highest-confidence answer across models - Tool calling: correctly produces tool_calls when workers propose tools - Tool lifecycle: full cycle query → tool_call → result → final answer - Tool results bypass fan-out, go to reducer only (one transcript) The gateway is transport-agnostic — works against any OpenAI-compatible endpoint (ollama, mesh-llm, remote APIs). Integration with mesh model discovery is the next step. * feat: multi-turn context efficiency + mesh endpoint discovery Progressive running summary: workers get compact deterministic summaries instead of raw message history. Tested across 3-turn conversations — workers retain context (Melbourne → restaurant recommendation) through the summary, not through replaying 20k tokens of history. New moa-mesh binary discovers models from any OpenAI-compatible endpoint (mesh-llm proxy, ollama, vLLM) and runs the full MoA test suite against it. Designed to work with 'mesh-llm client --auto' out of the box. Multi-turn tool lifecycle proven end-to-end: Turn 1: weather query → fan-out → tool_call (get_weather) Turn 2: tool result → reducer only → text answer Turn 3: follow-up 'bring jacket?' → fan-out with running summary → contextual answer * fix: increase running summary budget to ~2k tokens Per-fact truncation: 200 → 500 chars Recent fact window: 5 → 15 facts Tool result truncation: 80 → 300 chars Turn outcome capture: first sentence → first 400 chars 200 tokens was too aggressive for real agent sessions where system prompts alone can be 500+ tokens. 2k tokens gives enough room for 15 turns of meaningful context while still being much cheaper than replaying raw history. * feat: agentic workload test + improved prose tool detection New moa-agent binary simulates a multi-step coding agent: read file, analyze bug, edit fix, run tests, diagnose failure, iterate. Exercises 7+ turns with accumulating context, repeated tool use, and loop detection. Improved normalizer: small models that describe tool usage in prose ("I'll use the edit_file tool") are now correctly classified as tool proposals via known-tool-name + action-verb heuristic. Key findings from agentic testing: - Gateway routing, context management, and tool lifecycle are solid through 7 turns / 15 messages / 6 reducer calls - Loop detection catches repeated identical tool calls - Small models (3b/4b) produce correct tool calls for read/search but describe edits in prose instead of calling edit_file - Multi-step plan execution (tool→analyze→tool→fix) needs a stronger model in the reducer role — the arbiter and routing aren't the bottleneck, model capability is * fix: mesh-tested agentic flow — local reducer, model dedup, KV parse fix Tested against live mesh: GLM-4.7-Flash (local) + Qwen3-8B (remote peer). Three fixes from live mesh testing: 1. Local reducer: prefer first endpoint (local model) as reducer instead of last. Remote models over QUIC relay are fine as parallel workers but timeout as the sequential reducer. Result: reducer response times dropped from 180s (remote timeout) to 2-18s (local). 2. Model dedup: mesh-llm exposes the same model under multiple aliases (e.g. unsloth/GLM-4.7-Flash-GGUF and @main:Q4_K_M variant). discover_endpoints() now deduplicates by normalized display name. Extracted as shared lib function used by all three test binaries. 3. KV parse fix: models that say 'kind: answer' but also include 'tool: read_file' are now correctly classified as ToolProposal. This was the #1 cause of missed tool calls in the agentic flow. * moa: integrate into mesh proxy, SSE streaming, Goose support MoA is now available as model="moa" through the mesh proxy on :9337. When ≥2 models are available (local + mesh peers), the MoA virtual model appears in /v1/models automatically. Integration: - mesh-llm-host-runtime depends on moa-gateway - ingress.rs intercepts model="moa" requests, builds endpoints from callable models, calls Gateway::turn(), returns the result - SSE framing: converts the non-streaming MoA response into SSE chunks for streaming clients (Goose, pi, etc.) - Tool calls passed through in SSE format with finish_reason="tool_calls" Passthrough mode (tools present): - When the request includes tools (agentic use via Goose/pi), workers receive the original messages+tools unmodified — no MoA envelope - This avoids conflicting system prompts confusing small models - First successful worker response is returned, providing redundancy Content cleanup: - Think tags (<think>...</think>) stripped from all responses - Orphan </think> tags cleaned up - KV envelope lines (kind:/confidence:/payload:) stripped when they leak into heuristic-classified output - Normalizer pre-cleans think tags before trying JSON/KV parse Tested with: - Direct curl (non-streaming + streaming) - Goose CLI: factual questions, tool execution (shell, execute_typescript) - 22 unit tests passing * fix: add moa-gateway to Docker builds The docker-client CI job failed because crates/moa-gateway/ was missing from both Dockerfile.client and fly/Dockerfile. cargo metadata couldn't resolve the workspace member, breaking the cargo-chef prepare step. * moa: early-exit on worker consensus Instead of waiting for all workers before arbitrating, check for consensus after each worker returns. When 2+ workers agree on an answer or tool call, return immediately and abort remaining workers. This eliminates the 'slowest worker' bottleneck. In testing: - Average latency dropped from 21.0s to 5.8s (single model: 7.1s) - MoA is now 19% faster than querying a single model - Worst case (code-debug) went from 120s timeout to 4.2s The key insight: with parallel fan-out, we only need to wait for the fastest N workers that agree, not all of them. Slow/dead remote workers no longer block the response. Also adds 5 new arbiter tests for early decision logic (27 total). * moa: real context slices, not synthetic envelopes Major architectural change to how MoA packs context for workers. Before: workers got a synthetic system prompt ('You are a fast analysis worker...') that replaced the agent's real system prompt, tool schemas, and conversation history. Workers were asked to respond in a KV envelope format (kind:/confidence:/payload:) that small models followed unreliably. When tools were present, the entire MoA pipeline was bypassed via a 'passthrough mode' that raced identical requests to all workers. After: workers get slices of the REAL context — the agent's actual system prompt and messages — with depth varying by role: - Fast: system prompt + last user msg + tool names only - Specialist: system prompt + last 4 msgs + tool summaries - Strong: system prompt + full recent history + native tool schemas - Reducer: system prompt + worker outputs + full tool schemas The gateway augments with a one-line preamble, not a replacement. The passthrough mode is removed — tool-use goes through the full normalize → arbitrate pipeline. Strong workers get native tool schemas forwarded so they can produce real tool_calls. Also: - Dedup model aliases in ingress (GLM and GLM@main:Q4_K_M are the same) - Early exit handles failed workers (sole survivor returns immediately) - Worker timeout reduced from 120s to 30s - 28 unit tests (up from 27) * moa: rename to mesh-mixture-of-agents, mesh-native transport, model='mesh' Renamed crate from moa-gateway to mesh-mixture-of-agents. Keeps the crate isolated (own tests, own compilation unit) while connecting it to mesh transport via a ModelBackend trait. Transport is now mesh-native instead of HTTP loopback: - LocalModelBackend: direct HTTP to skippy port (bypasses proxy) - RemoteModelBackend: QUIC tunnel to peer (bypasses proxy + tunnel layer) - Both set mesh_hooks: false to prevent recursive consultation The ModelBackend trait keeps the crate testable in isolation — the default HttpBackend works against any OpenAI-compatible endpoint. The mesh backends are implemented in ingress.rs where Node and InferenceTarget are available. Virtual model renamed from 'moa' to 'mesh'. Appears in /v1/models when ≥2 distinct models are available. Test bins removed (used old Gateway API). 29 unit tests remain. * fix: update Dockerfiles for moa-gateway → mesh-mixture-of-agents rename * moa: remove 'mesh' from /v1/models list The 'mesh' virtual model is a routing directive like 'auto', not a real model. It should not appear in the models list. Clients that want MoA fan-out use model: "mesh" explicitly. * fix: clippy warnings in mesh-mixture-of-agents * fix: clippy unnecessary_lazy_evaluations in ingress build_moa_config * docs: update MoA design doc with current architecture and test plan Reflects: mesh-mixture-of-agents crate rename, ModelBackend trait, handle_turn() stateless API, mesh-native transport, model='mesh' virtual routing, early-exit consensus, and eval plan. * moa: fix tool call arguments lost in arbitration Two fixes: - Arbiter now prefers tool proposals with actual arguments over proposals that only have the tool name (from fast workers that don't get native tool schemas). - Specialist workers now receive native tool schemas so they can produce structured tool_calls with arguments, not just mention tool names in text. Before: read_file({}) After: read_file({"path":"/tmp/test.txt"}) * moa: worker diversity sampling, 429 retry, faster timeouts, sole-survivor early exit Three improvements to MoA reliability and response quality: - Workers get high temperature (0.8) + top_p (0.95) for diverse exploration; reducer gets low temperature (0.3) for precise synthesis. SamplingParams flows through the ModelBackend trait. - 429 rate-limit errors trigger one automatic retry after the server's retry-after delay (default 1s). - Worker timeout 30s → 15s, reducer 45s → 30s. - Sole survivor returns immediately when majority of other workers have already failed, instead of waiting for remaining stragglers. 39 unit tests (up from 29). * Fix MoA tool result handling and NaN confidence (PR review feedback) Three issues from Copilot review on PR #534: 1. Tool result turns now include actual tool output content. pack_for_tool_result_turn was reading from pending_tools which is always empty on a fresh session (stateless per request). Now forwards the raw message sequence including assistant tool_call + tool result messages so the reducer sees the full context. Added regression test. 2. NaN confidence no longer panics arbiter comparisons. Replaced partial_cmp().unwrap() with total_cmp() in arbiter, and added a sanitizer in normalize that clamps non-finite confidence to 0.5. Added test. 3. Exclude spec-prefill-poc from workspace members (Docker fix). Moved to Cargo.toml exclude list so Docker builds don't fail on the missing experimental crate. * ci: align WORKSPACE_MEMBERS with current workspace - Add mesh-mixture-of-agents (new crate on this branch) - Rename mesh-llm-client → mesh-client (renamed on main) Fixes the scripts/affected-crates.sh consistency check that gates the Linux CPU CI build. * moa: size-aware role assignment + hallucinated tool name filtering Two surgical fixes from real-world goose testing: 1. Role assignment by capacity tier, not list-order. assign_roles previously used list order: first=fast, last=strong. When a small local model (e.g. Qwen2.5-3B) was loaded last, it got tagged Strong and used as the reducer — exactly when goose needs a capable model for tool arbitration. Now we sort by size tier using the same is_single_digit_b_name heuristic as the main router's pick_model_classified, so MoA's strong worker matches what auto would pick. MiniMax-M2.5 and Qwen3-32B (big tier) become Strong; Qwen3-8B and Qwen2.5-3B (small tier) become Fast. 2. Filter hallucinated tool names from worker proposals. A worker proposing a tool not declared in the request (e.g. local 3B hallucinating 'execute_typescript' when only 'shell' was offered) would bypass arbitration and reach the client, causing silent failures when goose tried to dispatch the unknown tool. gather_workers_incremental and the reducer output paths now demote such proposals to Uncertainty with a tracing warning. The arbiter sees a clean set of valid proposals and resolves correctly. Threading: handle_query, handle_tool_result, gather_workers_incremental, and resolve_decision all now take &[String] allowed_tools derived from session.tool_names(). When allowed_tools is empty (no tools on request) the filter is a no-op. * moa: reducer candidate fallback on 5xx / timeout When the chosen reducer peer is broken (e.g. stale binary returning 502 on tool grammars) or unreachable, the tool-result turn or NeedsReducer arbitration would fail with that single error, even though other strong peers were available. Replace single-pick pick_reducer with reducer_candidates returning all big-tier models (multi-digit B or no size in name) followed by small-tier as last-resort fallback. Both call sites — handle_tool_result and resolve_decision NeedsReducer — now iterate candidates and break on the first success. This rescues the common goose-on-public-mesh case where one strong peer (e.g. Qwen3-32B host) is running a stale binary that 502s on tool calls, while another (e.g. MiniMax) is healthy. Without this, MoA's tool-result turn was as fragile as auto routing. * ci: fix workspace member drift — keep mesh-llm-client package name, add mesh-mixture-of-agents to clippy script Same fix as the prefill-draft branch: - The mesh-client directory rename did not change the package name — the crate is still published as 'mesh-llm-client'. Revert the scripts/affected-crates.sh edit that broke the CI consistency check. - Add mesh-mixture-of-agents to plan-clippy-batches.sh which carries its own WORKSPACE_MEMBERS list with the same drift constraint. * moa: support model:"mesh" on client/standby nodes via forward-to-host Pure --client nodes and standby GPU nodes accept inbound HTTP via handle_mesh_request (in transport.rs) instead of the model-aware api_proxy in ingress.rs. The MoA fan-out intercept lives in api_proxy, so when a client received "model": "mesh" it fell through to the "no host serves this model" branch and 429d. * moa: fix clippy lints surfaced by CI Two pre-existing lint violations in mesh-mixture-of-agents that CI didn't see before because the crate wasn't in the affected-crates / clippy workspace lists. Now that the WORKSPACE_MEMBERS drift is fixed they show up on every PR clippy run. - worker.rs:67 `x == false` -> `!x` (clippy::bool_comparison) - lib.rs:523 `&name` -> `name` (clippy::needless_borrow) No behavior change — tool_call_response takes &str either way, and the sort key inverts identically. * moa: hedge reducer candidates instead of sequential fallback Cut worst-case reducer latency from N×timeout to roughly reducer_timeout + (N-1)·hedge_delay. Big win when a peer is slow or broken; zero cost on the happy path. Before: for candidate in candidates: call(candidate, timeout=30s) # wait up to 30s per stale peer if ok: return # 3 stale big-tier peers ⇒ 90s before falling through to small-tier After: spawn candidate[0] loop: select: a candidate finished: ok → cancel rest, return err → spawn next candidate immediately (no hedge wait) hedge_delay elapsed and more candidates remain: spawn next alongside in-flight ones (race) Cost shape: - Happy path (cand 0 OK in <hedge_delay): exactly 1 backend call. Free. - Slow first (cand 0 takes hedge_delay..reducer_timeout): up to 2 overlapping calls, accept whichever wins, cancel loser. - Fast-fail (cand 0 errors quickly): next candidate immediately, 1 call. - All fail: ≤N calls, capped at reducer_timeout + (N-1)·hedge_delay. Wall-clock improvement for 3 stale big-tier peers (worker_timeout=15s, reducer_timeout=15s, hedge_delay=5s): - Before: 3 × 30s = 90s before reaching small-tier fallback. - After: 15s + 2 × 5s = 25s. Plus reducer_timeout itself drops 30s → 15s now that the hedged ladder makes a single per-attempt cap safe to shorten. Changes: - Add hedged_reducer_call() in mesh-mixture-of-agents/src/lib.rs. - Replace the for-loop in handle_tool_result() with it. - Replace the for-loop in resolve_decision()'s NeedsReducer arm with it. - Add hedge_delay field to GatewayConfig (defaults set at the single construction site, build_moa_config in ingress.rs). - Lower reducer_timeout 30s → 15s in build_moa_config. - 4 new unit tests cover happy path, hedge-on-slow, fast-fail, all-fail. - Refresh stale numbers in docs/design/MOA_GATEWAY.md and replace the "first model wins" reducer paragraph with the hedged-ladder description. Verified: cargo fmt --all -- --check # clean cargo check -p mesh-llm-host-runtime # clean cargo clippy -p mesh-llm-host-runtime --lib # clean cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings # clean cargo test -p mesh-llm-host-runtime --lib # 1381 passed cargo test -p mesh-mixture-of-agents --lib # 47 passed (4 new hedge tests) * evals: add bench-moa.sh — quick wall-clock benchmark for model:"mesh" POSTs N chat completion requests to a running mesh-llm endpoint with model="mesh" and reports p50/p95/p99 wall-clock latency. Probes /v1/models first and warns if fewer than 2 models are present (MoA returns 503). Usage: ./evals/bench-moa.sh # 20 requests, localhost:9337 N=50 ./evals/bench-moa.sh BASE_URL=http://host:9337 ./evals/bench-moa.sh PROMPT="why is the sky blue?" ./evals/bench-moa.sh Output is per-request lines plus a summary block (min/p50/p95/p99/max/ mean/stdev). Failed requests are reported but excluded from percentiles. Raw TSV results are kept in a tmpdir so before/after comparisons are easy. Dependencies: curl, jq, python3 — nothing exotic. Run on the same machine as a serving host so wall-clock is dominated by inference + arbitration. * moa: split lib.rs into backend / reducer / fanout modules lib.rs was past the 1k LoC refactoring threshold and had three separable responsibilities. Extract them into named modules and keep lib.rs as the orchestration entrypoint (handle_turn, GatewayConfig, TurnResult, response builders). - backend.rs: ModelBackend trait, HttpBackend, SamplingParams, ModelEntry, call_backend + retry-after parsing - reducer.rs: reducer_candidates ordering, hedged_reducer_call ladder - fanout.rs: gather_workers_incremental ModelEntry, HttpBackend, ModelBackend, and SamplingParams are re-exported from lib.rs so existing callers (worker.rs, host-runtime ingress) keep working. Tests move with their owner: backend.rs gets the sampling/retry-after tests, reducer.rs gets the 4 hedged-reducer tests plus the FakeBackend helper. No behavior change. lib.rs: 1267 -> 545 LoC 47 existing tests still pass. * moa: cover role-shaped context packing with tests context.rs owned the role-shaped packing logic (fast/specialist/strong/reducer depth contract) without any tests. Pin the design claims: - per-role token budgets (256 / 512 / 1024) - fast: system + last user only, tool names only, no tools field - specialist: tool summaries in system, native tools populated - strong: deep history (>=6 msgs), native tools populated - generalist/reducer roles alias the strong shape - MoA preamble augments rather than replaces the agent's system prompt - reducer context includes the conflict reason and labeled worker payloads - long worker payloads are truncated with an ellipsis to bound context 8 tests, all green. * docs: surface model:"mesh" (MoA) in README Add a workflow-table row pointing at MOA_GATEWAY.md and a short 'Mixture-of-Agents' section with a curl example and the two-models-required gate, so the feature is discoverable from the project entrypoint. * moa: extract tool_guard module, drop dead Endpoint/discover_endpoints Two small cleanups on lib.rs: - Move enforce_allowed_tools into its own tool_guard.rs module. It's a content-policy concern (demote hallucinated tool names to Uncertainty before arbitration), not orchestration, so it doesn't belong in the handle_turn entrypoint file. Comes with 4 unit tests covering allowed pass-through, unknown-tool demotion (incl. confidence drop), empty-allowed-list noop, and non-proposal outputs untouched. - Delete the Endpoint struct and discover_endpoints helper from lib.rs. Their doc comments described them as 'convenience for test harnesses' but they have zero call sites in-tree and no out-of-tree consumers we know of. Dead code from the standalone phase before mesh-native backends landed. lib.rs: 545 -> 454 LoC. Tests: 55 -> 59 (4 new in tool_guard). * docs(moa): drop the speculative hook-integration line MoA and hooks are intentionally independent — worker requests set mesh_hooks: false so the hook pipeline can't re-enter a worker call. The old design doc closed the relationship section with 'they could integrate later (hook signals as arbiter weights)', which makes it look like roadmap. It isn't — keeping them separate is the design. Replace that line with one that states the separation as intentional and points at the mesh_hooks: false invariant that enforces it. * router: weight 'auto' selection by locally observed tok/s Before, 'auto' picked uniformly at random within the multi-digit-B tier. On the public mesh this meant a fast MiniMax on a 4090 and a slow 35B-A3B on an M2 Air were equally likely to be chosen, even though we'd already measured the throughput gap in routing_metrics and were just not reading it. Now: each big-tier candidate is weighted by its locally observed avg_tokens_per_second (clamped to [5, 100] tok/s so nothing fully starves and no outlier monopolizes). Models without enough samples (< 3) get a neutral weight so they compete fairly until data accumulates. A 15% exploration probability ignores weights and picks uniformly, which keeps the system from locking onto stale rankings and guarantees cold peers see traffic. Plumbing: - RoutingMetrics::tps_for_model(name) -> Option<(f64, u64)>: cheap per-model lookup that locks only the relevant shard, avoiding the per-call HashMap allocation model_snapshots() does in the hot path. - Node::routing_metrics() public accessor (Arc-backed, cheap). - RoutingCandidate { name, caps, tps_hint, throughput_samples } replaces the anonymous (&str, f64, ModelCapabilities) tuple whose middle slot was literally always 0.0 at every populated call site. The struct makes the tps hint a real, typed concept rather than a dangling hook. Behaviour preserved: - Single-digit-B partition (smalls stay last-resort) unchanged. - All-cold candidate pool falls back to ~uniform pick (regression test confirms no model is starved when there's no data yet). - Capability filtering for tools / reasoning / vision unchanged. Plumbing per call site: - ingress.rs + transport.rs: live routing path, look up tps_hint from the local RoutingMetrics handle for each candidate. - discovery.rs + integrations.rs: pre-startup paths with no live metrics; build candidates with RoutingCandidate::unscored() so they get the cold-neutral weight. Tests: - weighted_pick_all_cold_is_roughly_uniform — regression safety. - weighted_pick_fast_wins_majority_but_slow_still_gets_some — fast wins by >=1.5x but slow still gets >30/600 picks. - weighted_pick_cold_model_competes_with_hot_fast — newcomer gets >100/600 picks against an established fast peer (so it can actually accumulate samples and earn its score). - weighted_pick_low_sample_count_treated_as_cold — 1-sample measurements don't dominate routing. - candidate_weight_clamps_extremes — weight stays in [5, 100], cold = 25. Removed: - shuffle_in_place (replaced by SplitMix64 + pick_weighted). - The dishonest 0.0 f64 slot in the candidate tuple, everywhere. Validation: cargo fmt --all -- --check # clean cargo check -p mesh-llm-host-runtime # clean cargo clippy -p mesh-llm-host-runtime --lib # clean cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings # clean cargo test -p mesh-llm-host-runtime --lib # 1398 passed (17 in router) cargo test -p mesh-mixture-of-agents --lib # 59 passed (no regression) * docs(moa): clarify topology — N workers + serial 2-call shape - Replace topology diagram with one that shows N workers fanned out in parallel and the serial fan-out → arbiter → reducer path. - Add explicit "how many models" table (2..N) and the worker → role mapping so readers don't have to infer it from worker.rs. - Spell out that a worst-case MoA turn is 2 LLM round-trips serially (fan-out wall-clock = slowest worker, then optional reducer), and that happy paths collapse to 1 (consensus or tool-result turn). - Refresh stale crate-structure table: post-split LoC + test counts for backend / reducer / fanout / tool_guard / arbiter / context / worker / session / normalize / lib. * moa: emit x-moa-* observability headers from gateway Extend TurnResult with turn_kind (Fanout / EarlyExit / ToolResult / Failed) and reducer_attempts (candidates actually spawned). hedged_reducer_call now returns a named HedgedReducerOk struct carrying winner, text, and spawn count so the caller can attribute hedge cost. The ingress MoA intercept reads these and emits: x-moa-elapsed-ms x-moa-turn fanout | early-exit | tool-result | failed x-moa-workers total workers dispatched x-moa-workers-ok workers that returned a usable answer x-moa-reducer true | false x-moa-reducer-attempts 0 on no-reducer path, 1 happy, >=2 hedged Headers are emitted on both the JSON and SSE response paths via a new send_json_ok_with_headers helper and an extra_headers arg to send_moa_as_sse. Normal OpenAI clients ignore unknown headers; benches and ops tooling can read them without parsing the body. Side fix: handle_tool_result previously reported attempts = total candidate pool size rather than candidates actually spawned. Now correctly reports the spawn count from HedgedReducerOk. * bench-moa: aggregate gateway path, reducer, and hedge stats Read x-moa-* response headers per request and roll them up in the summary. New aggregates: Gateway paths histogram of fanout / early-exit / tool-result / failed Reducer invocation rate + hedge rate + avg/max attempts Worker fan-out average width + histogram by N Latency p50 split by gateway path (so 'reducer turns are 4x slower' is visible at a glance) Per-request log line now shows the turn kind, worker count, and reducer status alongside the latency, making it easier to eyeball individual outliers. Older mesh-llm binaries that don't emit x-moa-* headers degrade to the old summary (latency-only) with a note that headers were not seen, so this works against any mesh-llm version. * evals: remove bench-moa.sh — never actually run The script was written but never executed against a live mesh. MoA verification is the manual live-mesh testing called out in the PR body. Real aggregates, if we want them, should come from passive counters fed by real traffic, not a synthetic curl loop. * moa: orchestrate from any node, build worker pool from mesh-wide gossip Before this change MoA only ran when the request hit `api_proxy` on a serving host. A pure `--client` node received `model: "mesh"` in `handle_mesh_request`, fell through to the "forward to any host" fallback, and the receiving host either ran an older binary that ignored the "mesh" name or built a single-model config because its local `ModelTargets` only had its own model. End result: no fan-out happened, MoA was effectively dead from any client node, and the worker pool depended on which node received the request rather than on what was actually in the mesh. Two structural fixes: 1. New `moa_gateway` module owns the intercept. Both `api_proxy` and `handle_mesh_request` now call `try_handle_moa` — the request is handled wherever it lands, whether the node serves models locally or not. 2. `build_moa_config` enumerates `Node::models_being_served()` (the mesh-wide union of local + gossip) instead of the local routing table. Locally-served models are wired directly to the skippy port via the routing table when one is present (host mode); everything else opens a QUIC tunnel to the hash-preferred peer that advertises the model. On a pure client every worker is remote. Side effects: - Canonical-base dedup now strips an `@branch` segment without losing the trailing quant tag, so `unsloth/Qwen3-8B-GGUF@main:Q4_K_M` and `Qwen3-8B-Q4_K_M` collapse to one worker instead of being treated as two distinct models. - The MoA-related backends (`LocalModelBackend`, `RemoteModelBackend`) and the SSE wrapper moved out of `ingress.rs` into the shared module; `ingress.rs` shrinks by ~400 lines. Verified live against the public mesh from a `--client --auto` node: - Chat completion: `x-moa-workers: 4` (all 4 mesh-wide models), early-exit path, correct answer. - Tool-equipped request: 3 tool-capable workers, early-exit path, correct shape. * moa: carry attempts on reducer-failure path; copilot review fixes Two real bugs surfaced by live goose testing against the public mesh from a --client --auto node. Bug 1 — attempts accounting on the failure path. hedged_reducer_call returned Result<HedgedReducerOk, String> where the Ok arm carried 'attempts: u32' but the Err arm dropped it. Both call sites in lib.rs (handle_tool_result, resolve_decision) reported attempts=0 on the all-fail path, producing nonsense like 'Reducer failed (tried 0): remote timeout after 15s'. Replace with Result<HedgedReducerOk, HedgedReducerErr> where Err carries attempts too. Surface the real spawn count to logs and to the user-visible error string. Bug 2 — copilot review issues. - UTF-8-safe truncation: 2 panicking '&text[..len.min(N)]' sites in moa_gateway replaced with new moa::truncate_chars helper that walks back to a char boundary. - Remote-read cap 256 KiB → 4 MiB. Long reasoning + tool synthesis answers can exceed 256 KiB. - CR/LF sanitization on x-moa-* header values. Cheap insurance. - Removed dead ('unknown', 0) fallback in reducer_candidates. Let hedged_reducer_call's empty-input path surface real errors instead of silently dispatching to backend_index=0 with a bogus name. - Consolidated three byte-identical strip_thinking implementations (worker.rs, normalize.rs, moa_gateway.rs) onto one canonical moa::worker::strip_thinking with re-export from moa crate root. - Warn on response-write failure rather than swallowing the error. Tests: 4 new (truncate_chars on UTF-8 boundary, all-fail-reports-attempts), all 63 moa + 1403 host pass. Clippy + fmt clean. Live verified from --client --auto on this Mac, joined to public mesh: - Plain chat: x-moa-workers: 2, early-exit, 1.3s, correct answer. - Goose end-to-end (tool propose → shell exec → tool-result turn → final): full loop completed, server log shows fanout + early-exit on both turns, goose printed DONE and exited 0. * moa: address remaining Copilot review items - context.rs / session.rs / backend.rs: replace byte-index truncation with crate::worker::truncate_chars (UTF-8 safe). Worker payloads, tool outputs, and HTTP error bodies all come from external sources that can contain multi-byte characters. - reducer.rs hedge loop: once `remaining` is exhausted, stop arming the hedge timer and just await join_next() directly. Previously the select! kept rebuilding a fresh hedge_sleep every iteration and firing every hedge_delay just to no-op. Untidy, not a correctness bug — but easier to reason about now. Closes inline review feedback on PR #566. * moa: tighten early-exit content check with subset+negation rule Early-exit previously claimed "workers agree" whenever 2+ outputs were Answer-kind, without comparing payload content. Two workers replying "Paris" and "Berlin" both with confidence ~0.5 (the default for plain prose) would early-exit on whichever was returned first. New rule: two answers agree iff - the smaller content-token set is a subset of the larger, AND - their symmetric difference contains no negation tokens. Tokenization: lowercase, strip punctuation, drop stopwords and tokens <3 chars (digits and negation words always kept). This is biased toward false-negatives: terse-vs-verbose paraphrases like "Paris" / "Paris is the capital of France" cluster correctly, while same-shape disagreements like "...is Paris" / "...is Berlin" do not. When the rule declines to cluster, we just wait for more workers and fall through to arbitrate() — no extra reducer call. Also: - session.rs:341: replace one remaining &first_line[..77] byte-slice with worker::truncate_chars (multi-byte panic risk on tool names containing emoji). - ingress.rs MoA intercept: replace let _ = try_handle_moa(...) with if let Some(...) and a tracing::error! so the impossible "returned unused stream" case is loudly logged instead of silently leaked. Tests: - 4 reworked early-exit tests (terse-vs-verbose, normalized-equivalent, majority cluster, shared-scaffolding-still-blocks) - 3 new negation guard tests (not, don't, "use grep" vs "do not use grep") - 1 numeric agreement test ("42" vs "the answer is 42") 73 moa tests pass, 1426 host-runtime tests pass, both clippies clean. * docs(moa): add pressure-test research plan to MOA_GATEWAY Replace the earlier 'A/B plan' sketch with a research plan that is designed to falsify the mixture hypothesis, not confirm it. - Sharpened hypothesis with three falsifiable corollaries - Pre-committed falsification conditions (so we cannot move goalposts) - Step 1: variance floor measurement as prerequisite for any A/B claim - Adversarial scenarios including failure-mode-amplification cases - Pareto curve as the headline deliverable, not win/tie/loss - Ablations to separate 'mixture' from 'variance reduction' - Composition sweep to test the 'modest models' framing directly - Grader robustness checks (position swap, dual grader, hand spot-check) - Real-task replay as the strongest defense against cherry-picking - Reporting discipline: what must be in a result before calling it a win Documentation only. No crate changes. Worker-set knob noted as a harness-side concern, not a crate change. * docs(moa): reframe pressure test around equal-VRAM split-vs-mix on mesh The earlier pressure-test plan was "is mixture smarter than single best," which is the wrong load-bearing question. The honest question for a mesh is: given fixed aggregate (V)RAM, when does running multiple diverse mid-size models locally beat sharding one large model across the network? Reframes the eval around the equal-VRAM trade between Skippy split-large and MoA mix-diverse, with network conditions (RTT, loss) as the primary axis. Existing scenario/ablation content becomes the quality measurement implementation, not the headline. Adds pre-committed falsification conditions specific to the network-tolerance and scalability claims. Docs-only. * docs(moa): reframe as operating-envelope, not benchmark fight The earlier draft framed MoA vs split-large as a quality competition. The real claim is that split-large has a hard practical ceiling on a real mesh — every cross-node hop is on every token's critical path — and MoA has a much higher ceiling because workers run fully local and the network is only touched at fan-out/collect/reducer. Reframe accordingly: - Headline is *operating-envelope analysis*, not Pareto fight - Define what 'acceptable' means (TTFT, total turn, failure rate, quality floor) before any measurement, so we cannot retrofit it - Deliverable is a *viability map* (config x network condition), not a win/tie/loss table - Quality is demoted to a tertiary axis inside the viable region; its job is to confirm MoA's MoA-only-region answers clear the single-mid floor - Pre-committed falsification conditions are specific to the new claims (envelope shrinkage with mesh size/network, MoA's envelope extending past split's, MoA quality above single-mid floor, mixture vs variance reduction) - single-mid baseline added explicitly so we cannot accidentally ship 'MoA = single-best + overhead' Complementary positioning, not competitive: use split when the network allows; use mix when it doesn't. * docs(moa): promote 'why MoA exists' to top of design doc The opening of MOA_GATEWAY.md described mechanism (fan out, arbitrate) but not purpose. The motivation \u2014 'use the mesh anyway when split-large isn't viable for the current network conditions' \u2014 was buried ~450 lines down inside the operating-envelope section. Add a brief 'Why MoA exists' section at the top that states: * The intended operating region (where split-large stops being viable). * That MoA is not trying to beat split-large on quality. * The complementary, network-conditions-decide-which framing. * A link down to the experimental envelope discussion that already exists. No design or behavior change. Pure framing of existing content. * fix(moa): signal all-workers-fail as a proper error response PR #566 review feedback (Apr 2026): > One concurrency request returned HTTP 200 even though the response > body said all MoA workers failed. That's a bad client contract. > If all workers fail, the API should probably return a proper error, > not a successful-looking response with failure text inside it. The MoA gateway was returning a body shaped identically to a successful `chat.completion` with the error string smuggled into `choices[0].message.content` and `finish_reason: "stop"`. The ingress wrapped that body in an HTTP 200. A client checking either the HTTP status, the top-level `error` field, or `finish_reason` saw "success." ## Test (added first, observed failing) `crates/mesh-mixture-of-agents/tests/sim_all_workers_fail.rs` drives `moa::handle_turn` with three `AlwaysErrBackend`s, asserts the result body is distinguishable from a successful `chat.completion` \u2014 either the top-level `object` is not `chat.completion`, or there is a top-level `error`, or `finish_reason` is one of `error` / `moa_failed`. The test fails against the pre-fix gateway with output > object=Some("chat.completion"), finish_reason=Some("stop"), > has top-level error=false ## Fix * `mesh-mixture-of-agents/src/lib.rs` \u2014 `error_response()` now attaches a top-level OpenAI-shape `error` object and emits `finish_reason: "error"`. The error text stays in `content` for unstructured clients. * `mesh-llm-host-runtime/src/network/openai/transport.rs` \u2014 new `send_json_with_status_and_headers()` helper for sending a custom status code with a full structured body and observability headers. * `mesh-llm-host-runtime/src/network/openai/moa_gateway.rs` \u2014 `write_moa_response` now takes the full `TurnResult` and sends HTTP 502 (Bad Gateway) when `turn_kind == Failed` for non-streaming responses. Streaming SSE stays 200 because we can't change the status after the headers are sent; the failure rides in the chunked body (which now carries the structured error). ## Validation `cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 1 new integration test pass. `cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass. `cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings` \u2014 clean. `cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings` \u2014 clean. `cargo fmt --all -- --check` \u2014 clean. * fix(moa): account for aborted workers in worker_summaries PR #566 review feedback (Apr 2026): > Worker accounting was inconsistent: > - Similar requests reported different x-moa-workers values. > - Similar requests reported different x-moa-workers-ok values. > - Some successful responses used fewer workers than expected. > - Some churn responses still appeared to report stale worker counts. `worker_summaries.len()` is what the `x-moa-workers` header reports. When the arbiter early-exits on consensus, the gateway called `JoinSet::abort_all` and then drained `join_next()` with `if let Ok(...)`. `JoinSet::abort_all` causes aborted tasks to return `Err(JoinError::cancelled)`, with no `(model, role)` payload \u2014 those tasks were silently dropped from `summaries`. A 4-worker fan-out that early-exited from 2 fast workers reported `x-moa-workers: 2`, hiding the fact that 2 workers were cancelled mid-flight. Panicked tasks had the same problem. ## Test (added first, observed failing) `crates/mesh-mixture-of-agents/tests/sim_worker_accounting.rs` sets up 4 mock backends, 2 fast/agreeing and 2 slow, and asserts that `worker_summaries.len() == 4` after early-exit \u2014 i.e. that the header faithfully reflects the dispatched count. The test fails against the pre-fix gateway with output > Got 2 summaries: ["fast-a-3b", "fast-b-3b"]; expected 4. ## Fix * `fanout.rs` \u2014 `gather_workers_incremental` now takes the dispatched-worker list (`&[DispatchedWorker]`) instead of just a count. After fan-out finishes (whether via normal completion or early-exit drain), `reconcile_dispatched` walks the dispatched list and synthesizes a `succeeded: false` summary for any worker whose name does not appear in `summaries`. Aborted tasks and panicked tasks are now both attributed. * `lib.rs` \u2014 builds a `Vec<DispatchedWorker>` alongside the `JoinSet` and threads it through to `gather_workers_incremental`. The header `x-moa-workers` now always equals the worker count we actually dispatched. `x-moa-workers-ok` continues to reflect genuinely-succeeded workers only. ## Validation `cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 2 integration tests pass (new test plus the existing all-workers-fail one). `cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass. `cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings` \u2014 clean. `cargo fmt --all -- --check` \u2014 clean. * fix(moa): route tool-result follow-ups to reducer, not fan-out PR #566 review feedback (Apr 2026): > The tool-result path isn't ready for agent loops: > - A tool-result follow-up was treated like another fanout turn. > - It wasn't handled like a controlled reducer/synthesis turn. > - Tool results should be handled carefully and predictably, not > sprayed back through the whole fanout path. `Session::classify_turn` only routed to `TurnType::ToolResult` when the very last message had `role: "tool"`. Many agent harnesses send the tool result followed by a short `user` nudge ("continue", "what did you find?"). That landed at the very-last-message check as `user`, so the gateway classified the turn as Continuation, fanned out to all workers, and invited a worker to re-propose the same tool call whose result was already in context. The session-state fallback at `last_was_tool_call && has_unprocessed_tool_results` was dead code in production: the gateway never invokes `record_assistant_response` between turns, so `last_was_tool_call` is always false. ## Test (added first, observed failing) `crates/mesh-mixture-of-agents/tests/sim_tool_result_routes_to_reducer.rs` \u2014 three scenarios, all using mock backends that count calls: * OpenAI canonical shape (last msg role=tool) \u2014 must classify as `ToolResult`, exactly one backend call (reducer only). Already passed pre-fix; pinned to prevent regression. * Trailing-user-after-unsynthesised-tool-result \u2014 must also classify as `ToolResult`, exactly one backend call. **Failed pre-fix with `TurnKind::EarlyExit`** (fanned out, multiple worker calls). * Plain fresh user question \u2014 must still fan out. Pins that we don't over-trigger the tool-result path. ## Fix Scan messages from the end in `Session::classify_turn`: * First message we hit with `role: "tool"` \u2192 classify as `ToolResult`. The tool result has not yet been synthesised by an assistant message after it. * First message we hit with `role: "assistant"` \u2192 stop. The assistant has already spoken since the last tool result; the next turn is a normal continuation. * Other roles (`user`, `system`) \u2192 keep scanning. A user nudge after an unsynthesised tool result still belongs in the reducer-only path. If the scan reaches the start without hitting either, fall through to the existing `Fresh`/`Continuation` classification. ## Validation `cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 5 integration tests pass (this PR\u2019s 3 new tests + the two earlier sim files). `cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass. `cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings` \u2014 clean. `cargo fmt --all -- --check` \u2014 clean. * fix(moa): recognise OpenAI-shape inline tool JSON in worker output PR #566 review feedback (Apr 2026): > In the read-tool probe, the model wrote text that looked like a > tool call instead of actually invoking the read tool. Agent harnesses (Goose, OpenCode, pi) only act on real `tool_calls`. If a worker emits inline OpenAI-shape tool JSON \u2014 "I'll read the README. {\"function\": \"read_file\", \"arguments\": {\"path\": \"README.md\"}}" \u2014 today's normalizer's `try_json_parse` requires a `kind` field in the JSON, which the OpenAI tool-call shape never has. `try_json_parse` returns None, the heuristic classifier doesn't have an action verb that matches ("I'll read" isn't on its list), and the worker output falls through to `OutputKind::Answer`. Three workers all return the same text \u2192 arbiter agrees \u2192 `chat_response(text)` \u2014 the agent gets the JSON-bearing prose as `content`, no `tool_calls` field, and silently does nothing. ## Test (added first, observed failing) `tests/sim_tool_call_text_not_passed_as_content.rs` \u2014 two scenarios: * `workers_with_inline_tool_json_emit_real_tool_call` \u2014 workers return prose with embedded `{"function": "read_file", "arguments": {...}}`. The response body must carry a real `tool_calls` array with the proposed function name. **Failed pre-fix**: body had `content` with the prose, no `tool_calls`. * `workers_describing_tool_call_must_emit_structured_tool_call` \u2014 workers describe a tool call in pure prose with no JSON. Today the heuristic catches this and synthesises a `tool_calls` entry (with empty arguments). Pinned so the JSON-shape fix below doesn't regress the pure-prose path. ## Fix `normalize::try_json_parse` now also recognises the OpenAI tool-call shape when no `kind` field is present: {"function": "read_file", "arguments": {...}} {"name": "read_file", "arguments": {...}} {"tool": "read_file", "arguments": {...}} A structurally well-formed inline tool proposal scores confidence 0.75 (above the heuristic's 0.6) so the arbiter prefers it on ties. The rest of the original `kind`-driven envelope path is unchanged. ## Validation `cargo test -p mesh-mixture-of-agents` \u2014 70 unit + 7 integration tests pass (this PR\u2019s 2 new tests + earlier sim files). `cargo test -p mesh-llm-host-runtime --lib` \u2014 1434/1434 pass. `cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings` \u2014 clean. `cargo fmt --all -- --check` \u2014 clean. * fix(moa): /v1/models advertises quant-suffix IDs that route back PR #566 review feedback (Apr 2026): > Some IDs in /v1/models dropped quant suffixes. Other endpoints > used the full model refs. > [...] > Direct calls from Carrack to worker-hosted models didn't work. > Carrack to Lemony 35B returned HTTP 404. Reproduced on a 2-node mesh (Mac M4 Max + Mac Studio M3 Ultra): * M4 served `Qwen/Qwen2.5-3B-Instruct-GGUF:qwen2.5-3b-instruct-q4_k_m`. * Studio served `unsloth/Qwen3-0.6B-GGUF:BF16`. * M4 `/v1/models` listed: Qwen/Qwen2.5-3B-Instruct-GGUF (quant suffix lost) unsloth/Qwen3-0.6B-GGUF:BF16 (full id) * Calling either listed id directly: local short id \u2192 200 (rewritten via internal alias table) remote short id \u2192 404 (no alias for remote models) The natural client flow \u2014 read `/v1/models`, take an id, call `/v1/chat/completions` with it \u2014 was broken for remote-hosted models, and inconsistent for local ones. ## Two root causes 1. **`quant_selector_from_gguf_file` was uppercase-only** when matching markers like `-Q`, `-BF16`. Real GGUF filenames mix cases (`...-Q4_K_M.gguf` vs `...-q4_k_m.gguf`). The matcher on the read side (`gguf_matches_quant_selector`) was already case-insensitive, so emitting a lowercase selector is safe and keeps the public id round-trippable. 2. **`public_huggingface_model_ref` only handled artifact-as-filename.** Locally-built `ServedModelDescriptor`s set `artifact = model_ref.selector` \u2014 e.g. `"qwen2.5-3b-instruct-q4_k_m"`, a quant selector, not a GGUF filename. `quant_selector_from_gguf_file` returned None for anything not ending `.gguf`, so the public id collapsed to just the repo name. The `public_model_id` selection logic also needed tightening so we fall back to local disk only when the descriptor cannot produce a lossless id (e.g. catalog or local-gguf identities without enough metadata). ## Test (added first, observed failing) `models_list_id_preserves_quant_suffix_when_descriptor_has_no_artifact` in `transport.rs` builds a HuggingFace descriptor with no `artifact` field and asserts the resulting public id either matches the internal model_name verbatim or carries a non-empty quant tag. Pre-fix the public id collapsed to bare repo and the test failed. ## Fix * `model-ref/src/lib.rs::quant_selector_from_gguf_file` \u2014 lowercases the stem before matching markers so lowercase-quant filenames like `qwen2.5-3b-instruct-q4_k_m.gguf` extract `q4_k_m` instead of None. Returns the slice from the original stem so display casing is preserved. * `transport.rs::public_huggingface_model_ref` \u2014 accepts artifact values that are already a quant selector (no `.gguf` suffix) in addition to GGUF filenames. The selector now round-trips through the resolver. * `transport.rs::public_model_id` \u2014 prefers the descriptor only when its identity carries enough information to produce a lossless id (HuggingFace needs an artifact; Catalog needs a canonical_ref). Otherwise falls back to the on-disk file, then the model_name itself \u2014 never silently drops information. ## Validation `cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass. `cargo test -p model-ref` \u2014 10/10 pass. Live 2-node mesh (M4 gateway + Studio peer): `/v1/models` now reports both models with their full ids: Qwen/Qwen2.5-3B-Instruct-GGUF:q4_k_m unsloth/Qwen3-0.6B-GGUF:BF16 Direct `/v1/chat/completions` calls with either listed id return 200 and real inference for both local and remote models. `model: "mesh"` (MoA) still works end-to-end on the same setup, returning a real fanout response (`Tokyo` from the capital-of-japan prompt). `cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings` \u2014 clean. `cargo fmt --all -- --check` \u2014 clean. * fix(moa): bump worker/reducer timeouts to 60s for agent-scale prompts PR #566 review feedback (Apr 2026) flagged that MoA worker accounting sometimes reported zero successful workers, especially under churn or load. Investigating from a real 2-node mesh (Mac M4 Max + Mac Studio M3 Ultra) with an OpenCode agent driving `model: "mesh"` showed that the root cause was the 15s worker_timeout being too tight for agent-scale prompts: * OpenCode's default system prompt is ~13.7k tokens. * Large strong-tier models (MiniMax-M2.5 Q4_K_M, Qwen3-32B+) at 13k+ prompts + tool schemas take 20\u201340s for a first useful response \u2014 the reasoning preamble alone often eats the 15s budget. * The MoA gateway killed the strong worker at exactly 15s every turn: moa: worker unsloth/MiniMax-M2.5-GGUF:Q4_K_M (strong) failed after 15001ms: remote timeout after 15s * The arbiter then early-exited on the surviving small worker, never giving the strong worker a chance to land. The strong worker was effectively unreachable for OpenCode/Goose-style flows. Bump both `worker_timeout` and `reducer_timeout` from 15s \u2192 60s in `build_moa_config`. Live verification on the same 2-node mesh: * With 15s: `model: mesh` from OpenCode finished 0 of 3 turns successfully. Every turn returned 1/2 workers, strong worker timeout, no useful response. * With 60s: `model: mesh` from OpenCode finished 2 of 3 turns successfully \u2014 strong worker landed, MoA produced the structured `tool_calls` field, OpenCode invoked the file-read tool correctly. (The 3rd turn hit a separate llama_decode / connection-lost issue in the local stage runtime that is unrelated to MoA timing.) The trade-off is that a single hung remote worker can stall a turn for 60s instead of 15s. That is acceptable for an interactive agent loop where the alternative is consistent failure to land the strong worker at all. The hedged-reducer ladder (`hedge_delay` = 5s) still keeps end-to-end latency bounded when only the *reducer* is slow. `cargo test -p mesh-llm-host-runtime --lib` \u2014 1435/1435 pass. `cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings` \u2014 clean. `cargo fmt --all -- --check` \u2014 clean. * fix(mesh): enable QUIC keep-alive on mesh transport (was: connections dropping mid-inference) PR #566 review feedback flagged MoA returning early with 0/N workers under load. Live debug on a 2-node mesh (M4 Max + Mac Studio M3 Ultra) running an OpenCode agent against `model: "mesh"` showed repeated: WARN noq_proto::connection: failed closing path err=LastOpenPath INFO mesh: Connection to <peer> closed: timed out WARN moa: reducer ... failed: recv: read error: connection lost happening 30-60s into otherwise healthy inference calls, including plain non-MoA `stream: false` requests through `model: "auto"`. Root cause: noq-proto's default `max_idle_timeout` is 30s and `keep_alive_interval` is `None` (the spec, RFC 9000 §10.1.2, makes keep-alive opt-in; quinn / noq follow that). Non-streaming inference requests send no application bytes while the remote model is generating tokens, so the wire is idle. Under concurrent load (parallel MoA workers + reducer + gossip + heartbeats), noq's multipath bookkeeping closes the idle path, and when it is the last open path the entire connection drops mid-stream. The in-flight HTTP tunnel errors with `connection lost` and the caller must retry. This only became visible recently because: * Streaming OpenAI clients (Goose, Claude Code, pi, the web UI) all set `stream: true` by default. SSE chunks flow continuously and reset the idle timer, so the bug never manifests for them. * MoA `RemoteModelBackend` is the first significant non-streaming long-running RPC in the codebase (`stream: false` hardcoded in `crates/mesh-mixture-of-agents/src/backend.rs`). * Reasoning models with big agent prompts (MiniMax-M2.5 on a 13k OpenCode system prompt, Qwen3-32B class reducers) routinely take 30-90s for a first useful response. That is the combination that exceeds the default 30s idle window. Fix: set `keep_alive_interval = 10s` and `max_idle_timeout = 5m` on the mesh QUIC transport config, plus the matching multipath `default_path_keep_alive_interval` and `default_path_max_idle_timeout` so individual paths don't get torn down while the connection-level idle timer is fine. Cost: one QUIC PING (~30-60 bytes) every 10s per connection only when no other application data has been sent for that long. In a typical mesh with periodic gossip and heartbeats this fires rarely. `keep_alive` is opportunistic, not unconditional. Live verification on the same 2-node mesh: * 60s idle test, before fix: 2x `Connection to <peer> closed: timed out`. After fix: 0x. Connection stays healthy. * 75s of mixed non-streaming inference (53s `auto` to MiniMax + 21s `mesh` 2-worker fanout), before fix: multiple `LastOpenPath` + `connection lost` errors. After fix: 0x. Both completed successfully with finish_reason=stop and full content. * OpenCode `model: mesh` agent loop, before fix: 0 of 2 turns landed. After fix: 2 of 3 turns landed (the 3rd hit a separate KV cache exhaustion in the local stage runtime, tracked independently). Validation: `cargo fmt --all -- --check` clean, `cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings` clean, `cargo test -p mesh-llm-host-runtime --lib` 1435/1435 pass. * fix(planner): cap auto lane count to llama-server's 4-lane unified-KV default PR #566 review feedback uncovered a hard 502 from the embedded skippy stage runtime under concurrent agent-style workloads. On a Mac M4 Max serving Qwen3-8B at the model's native 32k context, the auto planner was picking `slots = 16` (MAX_AUTO_PARALLEL_SLOTS). Three concurrent ~14k-token requests \u2014 the exact shape an OpenCode agent loop produces when MoA fans a worker call out next to a reducer call \u2014 fail in the embedded llama with: decode: failed to find a memory slot for batch of size 2048 surfacing as HTTP 502: skippy ABI call failed: RuntimeError: llama_decode failed Root cause: skippy's stage runtime sets `kv_unified = true` whenever `lane_count > 1` (`third_party/llama.cpp/patches/0034-Add-shared-execution-lanes-to-skippy-ABI.patch`). In unified mode llama allocates exactly `n_ctx` cells total, shared across all `n_seq_max` sequences. The previous planner derived `slots` from VRAM as if each lane carved off its own `n_ctx \u00d7 bytes_per_token` allocation \u2014 which is the `kv_unified = false` semantics, not what skippy actually does. On a node with comfortable VRAM the math happily returned the snapped maximum of 16 lanes, even though all 16 raced for the *same* fixed pool of `n_ctx` cells. Fix: drop `MAX_AUTO_PARALLEL_SLOTS` from 16 to 4, matching upstream llama-server's own auto default for the same reason. From `.deps/llama.cpp/tools/server/server.cpp`: LOG_INF("n_parallel is set to auto, using n_parallel = 4 and kv_unified = true"); params.n_parallel = 4; params.kv_unified = true; Lane count is purely a concurrency-policy knob under `kv_unified = true`; it does not change the KV cache allocation. Going from 16 to 4 frees zero RAM; it just gates admission control to a sane number of concurrent in-flight requests for the shared cell pool. Operators who know their workload (short chat turns, low-concurrency hosts, etc.) can still pick a higher value via the existing `parallel_override` plumbing, including `[models.throughput] parallel = N` in the TOML config from PR #564. Live verification on the same 2-node mesh used to find the bug: * M4 + Qwen3-8B at 32k `n_ctx`: planner now picks `slots = 4`, llama logs `n_seq_max = 4`, KV cache stays at 2448 MiB (one shared buffer; no RAM cost change). * Studio + MiniMax-M2.5 at 128k `n_ctx`: planner now picks `slots = 4`, llama logs `n_seq_max = 4`, KV cache stays at 8928 MiB. 4 \u00d7 32k cells per lane on average is plenty of headroom for agent prompts. * Repro that previously 502'd \u2014 3 parallel ~15k-prompt tool-result follow-ups on the M4 \u2014 now all succeed with `finish_reason=stop`, full content, ~20s wall time. Zero `find_slot` failures, zero `llama_decode` errors, zero skippy ABI errors. * Burst test \u2014 5 parallel at the same prompt shape \u2014 the 5th request correctly hits the admission-control queue and returns a clean `{"type":"rate_limit_error","code":"rate_limit_exceeded"}` after the admission timeout, instead of an opaque mid-flight 502. Adds two regression tests in `context_planning::tests`: * `auto_slots_capped_at_llama_server_default` covers the high-VRAM small-model case that used to plan 16. * `explicit_parallel_can_exceed_auto_ceiling` covers the override path so operators retain control. Validation: `cargo fmt --all -- --check` clean, `cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings` clean, `cargo test -p mesh-llm-host-runtime --lib` 1437/1437 pass (includes the two new regression tests). * docs(moa): report on micn/moa branch's critical fixes beyond MoA itself While iterating on PR #566 review feedback, this branch surfaced and fixed several pre-existing host-runtime bugs that were either hard to hit before or silently masked by KV-leakage bugs that have since been fixed. Capture the full picture in one place so PR reviewers and future readers don't have to spelunk through 60+ commits to see what landed. Headline fixes documented: 1. Mesh QUIC keep-alive (f5cf4b86) \u2014 connections were dropping mid-inference at noq-proto's 30s default idle timeout for any long non-streaming RPC across the mesh. Affects every user, not just MoA. 2. Auto-planner lane count capped to llama-server's default (1b901219) \u2014 the planner picked 16 lanes on the high-VRAM box for any model where one lane fit, but skippy's unified-KV mode shares one n_ctx cell pool across lanes; 3 concurrent agent requests would exhaust the pool with cryptic 502s. 3. /v1/models advertises quant-suffix IDs that round-trip (f3355bfd) \u2014 case-insensitive quant marker matching and artifact-as-selector handling, fixes 404s on peer-hosted models for any client browsing /v1/models. 4. The PR #566 review items themselves (5169d120 a396ab1e 000ae50c 64c4e0ec d5279656). Includes end-to-end agent validation results: Goose with GOOSE_MODEL=mesh runs to completion against the 2-node mesh and correctly identifies a fixture bug; a minimal Python agent harness runs MoA over multiple tool-calling turns without KV exhaustion or connection drops; a multi-turn exploration agent exercises the reducer hedge ladder when the remote MiniMax reducer transiently 502s. The remaining open item \u2014 OpenCode's ~14k-token system prompt exceeding a 32k-context local reducer's KV after a few turns \u2014 is documented as a deployment-side concern (use a \u226564k-context local reducer) plus a follow-up code option (clamp pack_for_tool_result_turn to the reducer's effective context budget). It reproduces on both model:"mesh" and model:"auto" routed to the same small-context model, so it is not MoA-specific. * docs(moa): expand branch report — include throughput-weighted router fix, agent harness validation, accurate 'auto' status PR #566 review wanted clearer accounting of what this branch fixes beyond MoA itself. Update the branch report to: * Add commit `25248409` (throughput-weighted auto-router) as generally-applicable fix C. Every `auto`-using client on the public mesh now picks faster peers more often instead of uniformly within the multi-digit-B tier. * Walk through the three pre-existing host-runtime bugs the MoA work surfaced (QUIC keep-alive, unified-KV lane cap, throughput-weighted router) with explicit scope notes for why each affects all mesh-llm users, not just MoA users. * Document agent harness validation: 5/5 Goose `mesh` runs, 3/3 Goose `auto` runs, 5/5 Python-mini-agent `mesh` runs, 3/3 Python-mini-agent `auto` runs, plus a multi-turn exploration agent that exercises the reducer hedge ladder. * Flag the single transient I observed (cold-start MiniMax tool-call parse failure on Goose `auto`) honestly — did not reproduce across subsequent runs, consistent with a lazy-grammar trigger race during model warmup, not a branch regression. * Re-frame the open item (OpenCode's 14k-token system prompt overflowing a 32k-context local reducer's KV) as not-MoA-specific — it reproduces equally on `model: auto` routed to the same local model. Lists three plausible avenues (bigger reducer, prompt trimming in pack_for_tool_result_turn, context-overflow distinguishing in skippy). * fix(family_policy): tighten prefix-cache budget for unified-KV serving Sustained agent traffic against a node running skippy's unified-KV stage runtime exhausts the shared KV cell pool. On a Mac Studio M3 Ultra serving MiniMax-M2.5 at 131072-cell `n_ctx`, running 20 consecutive Goose `model: "auto"` requests against the standard `calc.py` fixture reliably fails 14 of 20 starting at request 7 with: Server error: skippy ABI call failed: RuntimeError: llama_decode failed The embedded skippy native log shows: decode: failed to find a memory slot for batch of size 1805 Root cause: the resident prefix cache pins each recorded prefix onto a dedicated sequence id in the *same* unified KV cell pool the active lanes use. The previous budget had two bugs: * `estimate_stage_cache_max_bytes` multiplied the pool size by…

PR #612 review feedback (Nick) \u2014 two related findings on the streaming failure path. 1. Streaming MoA failures returned HTTP 200 ------------------------------------------- `write_moa_response` previously routed all streaming responses through `send_moa_as_sse`, which always emits `HTTP/1.1 200 OK` because SSE clients expect 200 before they start parsing the event stream. That meant `stream: true` MoA failures (`all_workers_failed`, `all_reducers_failed`, reducer hedge exhaustion) arrived as a 200 SSE that *happened* to carry `finish_reason: "error"` \u2014 dumb HTTP clients saw a "successful" stream and didn't realise inference had failed. The body is fully available before we decide how to write it, so we can collapse failure-shaped streaming responses to a non-streaming HTTP 502 JSON response with the structured error body. This matches the OpenAI API shape (failures on streaming endpoints come back as a single non-streaming JSON error response with the right HTTP status) and means streaming and non-streaming MoA failures are now consistent at the HTTP layer. 2. Error-only SSE chunk could break OpenAI-shape clients -------------------------------------------------------- The previous error chunk emitted on failure was `{ object: chat.completion.chunk, choices: [], error: \u2026 }`. Many OpenAI client SDKs assume every `chat.completion.chunk` has `choices[0]` and index blindly into it, so an empty `choices: []` array crashes those clients with an index error. With finding #1's routing change, failure-shaped bodies never reach `send_moa_as_sse` anymore, so the error-chunk emission is dead code. Drop it. `send_moa_as_sse` now does one clear thing: emit a single delta chunk + a `finish_reason: "stop" | "tool_calls"` stop chunk, both with a real `choices[0]`. A `debug_assert!` pins the invariant in tests. Tests ----- Four new unit tests covering the four corners of the routing decision: * `streaming_success_routes_to_sse` * `streaming_failure_routes_to_json_502_not_sse` * `non_streaming_success_routes_to_json_200` * `non_streaming_failure_routes_to_json_502` Validation ---------- * cargo test -p mesh-llm-host-runtime --lib: 1458/1458 pass (4 new) * cargo test -p mesh-mixture-of-agents --lib: 87/87 pass * cargo clippy ... --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean Live end-to-end validation on 2-node mesh (M4 + Studio MiniMax): * Non-streaming MoA happy path: HTTP 200 + structured response. * Streaming MoA happy path: HTTP 200 + `text/event-stream` + `[DONE]`. * mini-agent.py model=mesh: 2 turns, correct. * mini-agent2.py model=mesh: 4 turns multi-file, correct. * Goose model=mesh: 1 tool call, correct. The failure path is exercised by unit tests; production-triggering streaming failures would require fault injection beyond what the harness covers, but the routing logic and the SSE invariant are locked down at the code level.

…tion, dedup race, dead code) (#612) * docs(README): mark `model: \"mesh\"` MoA as experimental The MoA gateway is new and still being tuned (routing heuristics, error shapes, tuning knobs). Flag it explicitly in the README so readers do not assume the surface is stable. * fix(moa): close panic surface in worker output normalization PR #566 review (Copilot): the MoA crate had several latent panics around untrusted parsed responses. This commit closes them in one pass and adds regression tests for each. Changes ------- normalize.rs * Single sanitize pass in `normalize_worker_output` runs on the result of *every* parse strategy (JSON, KV, heuristic), not just the heuristic path. The previous shape returned early on JSON/KV success and let non-finite confidences leak into the arbiter where `partial_cmp/total_cmp` could panic on `.unwrap()`. The new helper `sanitize_worker_output` clamps NaN/Inf confidence to 0.5 and collapses non-object `tool_arguments` (Null, primitives, arrays) to `Some({})`. * New `extract_tool_arguments` helper replaces two dead `obj.get("arguments").cloned().or_else(\u2026)` chains in `try_json_parse`. The `or_else` branch was unreachable because `.cloned()` on `Some(Value::String(\u2026))` is already `Some`, so string-encoded JSON arguments leaked through unparsed. `extract_tool_arguments` now explicitly branches on String vs Object vs Null and parses the inner JSON when needed. backend.rs * `parse_retry_after` switched from `to_lowercase()` (Unicode-aware, can change UTF-8 byte length) to `to_ascii_lowercase()` (1:1 byte mapping). The earlier shape sliced the original string using the offset from the lowercased one, which could land mid-codepoint and panic for non-ASCII inputs before the marker. * `extract_text_from_response` switched from direct-indexing `resp["choices"][0]["message"]` to `.pointer()` chaining, returning a structured `Err("malformed response: \u2026")` on missing fields instead of silently producing empty content or panicking downstream. lib.rs * `best_answer` now uses `total_cmp` instead of `partial_cmp(\u2026).unwrap()`. `total_cmp` is total over all f32 (NaN/ Inf included), so even if a future caller bypasses `normalize_worker_output`, this site is panic-free. * `tool_call_response` now explicitly handles every input shape callers can construct: object \u2192 serialize, Null \u2192 `"{}"`, primitive / array \u2192 `"{}"`, validated JSON string \u2192 pass through, invalid string \u2192 `"{}"`. Previously `Value::Null` serialized to the literal four-char string "null", which downstream OpenAI tool-call consumers reject. Tests ----- 13 new tests (now 83/83 pass for the crate): * normalize.rs: kv_path_clamps_nan_confidence, kv_path_clamps_inf_confidence, json_string_encoded_arguments_are_parsed_to_object, null_tool_arguments_become_none, primitive_tool_arguments_collapse_to_empty_object. * backend.rs: parse_retry_after_handles_non_ascii_prefix_without_panic, extract_text_returns_err_on_missing_choices, extract_text_returns_err_on_empty_choices. * lib.rs (new `response_builder_tests` mod): best_answer_does_not_panic_on_nan_confidence, tool_call_response_emits_object_args_for_null, tool_call_response_emits_object_args_for_primitive, tool_call_response_passes_through_string_form_when_valid, tool_call_response_rejects_invalid_string_form. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 83/83 pass * cargo check -p mesh-llm: clean * cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): propagate failure signals to HTTP status, SSE, and tool-result reducer PR #566 review (Copilot) \u2014 three related fixes that all touch how MoA failures reach the caller. 1. HTTP status follows the body's failure signal, not TurnKind ------------------------------------------------------------- `write_moa_response` previously only used HTTP 502 when `TurnKind == Failed`. The tool-result reducer path (`handle_tool_result`) can return `error_response(\u2026)` with `TurnKind::ToolResult` when every reducer candidate fails \u2014 that returned HTTP 200 with an in-band error body, so dumb clients that only check the status code saw a "success". New helper `is_moa_failure_body(body)` recognises the two canonical failure signals MoA emits: a top-level `error` field, and / or `choices[0].finish_reason == "error"`. Status decision now uses this helper, so *every* error-shaped MoA response surfaces as 502 regardless of which sub-flow produced it. 2. SSE adapter propagates the original finish_reason ---------------------------------------------------- `send_moa_as_sse` used to hard-code `finish_reason: "stop"` (or `"tool_calls"` if any tool_calls were present). SSE clients keyed on `finish_reason` (Goose, OpenAI SDKs) therefore saw MoA failures as successful completions. Now the adapter reads `choices[0].finish_reason` from the response body and propagates it ("error" wins over the tool_calls heuristic), and when `is_moa_failure_body` is true it emits an explicit error chunk (`{ object: chat.completion.chunk, choices: [], error: \u2026 }`) before the final finish_reason chunk so SSE clients that scan deltas for an `error` field see the failure too. 3. Tool-result reducer emits tool_calls whenever tool_name is set ----------------------------------------------------------------- Both the tool-result path (`handle_tool_result`) and the fanout/arbiter path (`resolve_decision`) had the same bug: a `ToolProposal` from the reducer only became a real `tool_calls` reply when *both* `tool_name` AND `tool_arguments` were present. With `tool_arguments` missing, both paths silently fell back to a `chat_response` carrying the reducer's prose \u2014 which agent harnesses (Goose, OpenCode) ignore because they only act on `tool_calls`. Now both paths emit `tool_calls` whenever `tool_name` is set, and `tool_call_response` already collapses missing / non-object arguments to `"{}"`. Behaviour is consistent across paths and agent harnesses no longer lose tool calls to reducer prose. Tests ----- Four new `is_moa_failure_body` unit tests in moa_gateway.rs: top-level error, finish_reason=error, success body, tool_calls body. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 83/83 pass * cargo test -p mesh-llm-host-runtime --lib: 1446/1446 pass * cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): group aliases by canonical base before resolving backend PR #566 review (Copilot, item #10): the worker pool builder committed to a single alias per canonical model *before* trying to resolve a backend. Two real failure modes: 1. Stale-peer drop. The shortest alias is advertised only by a peer that drops between gossip refresh and orchestration. The peer is gone, `hosts_for_model(alias)` returns empty, the model is silently removed from the worker pool, and longer-form aliases for the same canonical model from still-reachable peers are rejected as duplicates. 2. Forced QUIC hop. The local node advertises a longer convention (e.g. `unsloth/Qwen3-8B-GGUF:Q4_K_M`) while a peer advertises a shorter variant (e.g. `Qwen3-8B-Q4_K_M`). The shortest-name rule picks the peer alias, `add_worker_backend` looks for a local port under that specific string in `targets`, finds nothing, and forces a QUIC tunnel even though the model is locally served. Fix: group all advertised aliases by canonical base first, then within each group sort so the most likely optimization wins first try (locally-served alias before remote, then shortest first as a tiebreaker). The resolver walks the group in order and takes the first alias that produces a backend, so an unreachable preferred alias falls back to a reachable longer one instead of dropping the model. Refactor extracts a small `resolve_one_worker_from_aliases` helper to keep `build_moa_config` under the cognitive-complexity limit. Tests (4 new) in moa_gateway.rs: * group_aliases_keeps_all_aliases_per_canonical_base * group_aliases_prefers_locally_served_alias_even_when_longer * group_aliases_falls_back_to_shortest_when_no_local * group_aliases_distinct_models_stay_in_separate_groups Validation ---------- * cargo test -p mesh-llm-host-runtime --lib moa_gateway: 13/13 pass * cargo test -p mesh-llm-host-runtime --lib: 1450/1450 pass * cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): strip dead Session continuation/running-summary machinery PR #566 review (Copilot, item #11): `Session::classify_turn` used a `self.turns` counter to decide Fresh vs Continuation, but the gateway constructs a fresh `Session` per inbound request and never invokes `record_assistant_response` / `record_turn_outcome` in production. As a result `turns` was always 1, `Continuation` never fired, and the entire deterministic running-summary feature (`accepted_facts`, `AcceptedFact`, `record_turn_outcome`, `rebuild_summary`, `running_summary` accessor) silently never executed. `pack_fast` had a `turn_count() > 1` branch that appended the summary to the worker's system prompt; that branch was dead too. This commit reflects the gateway's actual design: MoA is request-scoped, and the caller (Goose, OpenCode, an SDK) owns the multi-turn loop and sends the full history each request. Continuation context comes from `session.messages()`, not from a gateway-owned summary. Changes ------- session.rs * Drop `TurnType::Continuation` (only `Fresh` and `ToolResult` remain). The header comment documents the request-scoped lifetime. * Drop fields: `turns`, `last_was_tool_call`, `accepted_facts`, `running_summary`. * Drop methods: `record_assistant_response`, `record_turn_outcome`, `rebuild_summary`, `running_summary`, `accepted_facts`, `turn_count`, `has_unprocessed_tool_results`. * Drop struct: `AcceptedFact`. * Rewrite `ingest` to rebuild `pending_tools` from the caller-provided history each call (delta tracking only worked when Sessions were persisted across requests). Both assistant-emitted tool_calls and tool-result messages are picked up; agent harnesses unchanged. * Simplify `classify_turn`: walk history backwards, return `ToolResult` if a `role: "tool"` message appears before any `role: "assistant"`, else `Fresh`. lib.rs * Drop the `Continuation` match arm. context.rs * Drop the `turn_count() > 1` running-summary injection from `pack_fast`. Context comes from `session.messages()`. Tests ----- * tool_result_turn test rewritten to not rely on the removed `record_assistant_response`; verifies the same behaviour with caller-provided history. * No other test changes needed. Net: 132 LoC of dead machinery removed. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 83/83 pass * cargo test -p mesh-mixture-of-agents (incl. integration): all pass * cargo test -p mesh-llm-host-runtime --lib: 1450/1450 pass * cargo check -p mesh-llm: clean * cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): tighten error code, linearize strip_thinking, sanitize header names, drop dead branch PR #566 review (Copilot) \u2014 batch of MEDIUM cleanups. mesh-mixture-of-agents ---------------------- * `error_response` now takes a `code: &str` parameter and the two call sites pass distinct constants: - `MOA_ERR_ALL_WORKERS_FAILED` for the fanout/arbiter failure path. - `MOA_ERR_ALL_REDUCERS_FAILED` for the tool-result reducer path. Previously both paths emitted `code = "all_workers_failed"`, which was misleading for clients branching on `error.code`. * `strip_thinking` rewritten as a single linear pass over the input. The previous shape rebuilt the entire string on every think block (`format!` + `replace` in a loop), which is O(n*k) for long worker outputs with many `<think>` blocks. Three new tests cover the new shape: orphan close-tag handling, fifty-block linear behaviour, and UTF-8 preservation through multibyte content. mesh-llm-host-runtime --------------------- * `network::openai::transport`: header NAMES are now validated against the RFC 7230 tchar grammar via a new `is_valid_header_name` helper. Names that fail the grammar are dropped with a tracing warning rather than written verbatim. CR/LF in header values is still stripped.\n Both `send_json_ok_with_headers` and\n `send_json_with_status_and_headers` route through a shared\n `append_safe_header(\u2026)` so the validation can't be bypassed by a\n future caller. Four new tests cover the validator and the safe\n header writer.\n\n* `network::openai::moa_gateway::send_moa_as_sse` now reuses\n `append_safe_header` so the SSE adapter gets the same name validation\n as the JSON writers.\n\n* `network::openai::ingress::try_intercept_moa`: dropped the\n unreachable `if let Some(_unused_stream) = \u2026 { tracing::error!(\u2026) }`\n branch. `try_handle_moa` self-gates on the model name and the outer\n gate guarantees it matches, so the inner call always returns\n `None` here. Replaced with `let _ = \u2026.await;` and a comment\n explaining the invariant.\n\nValidation\n----------\n* cargo test -p mesh-mixture-of-agents --lib: 86/86 pass (3 new)\n* cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass (4 new)\n* cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime\n --all-targets -- -D warnings: clean\n* cargo fmt --all -- --check: clean * docs(moa): align stale comments and doc claims with current code PR #566 review (Copilot) \u2014 batch of LOW comment/doc corrections so future readers don't trust outdated narrative. * mesh-llm-host-runtime/src/network/openai/moa_gateway.rs: `try_handle_moa` doc said "returns `true` if handled" but the signature is `Option<TcpStream>`. Rewrote the doc to describe the actual contract: `Some(stream)` means "not MoA, fall through", `None` means "MoA consumed the stream and responded; do not respond again". * model-ref/src/lib.rs: comment claimed the function "emits a lowercase selector", but the implementation slices from the\n original (case-preserving) stem and lowercasing is only used for\n marker-position lookup. Reworded so the comment matches the code.\n\n* skippy-cache/src/resident/prefix.rs: regression-test comment\n referenced a "4*min_tokens floor" \u2014 that derivation was replaced by\n a hard `MIN_CTX_FOR_CELL_CAP = 8192` threshold in a follow-up\n commit. Updated to describe the floor that actually ships.\n\n* mesh-llm-host-runtime/src/inference/skippy/family_policy.rs:\n inline comment said the 16-entry cap "does not fully eliminate"\n unified-KV starvation and that the "proper fix is out of scope".\n The proper fix (token-based budget via `max_resident_tokens` in\n `ResidentCacheConfig::from_stage`) shipped in PR #566. Updated to\n describe entry-count as the coarse lever and `max_resident_tokens`\n as the complementary fine-grained budget.\n\n* docs/design/MOA_GATEWAY.md:\n - Role assignment section claimed "the largest of the big tier gets\n Strong". The impl only partitions small vs big, no\n within-tier sort by parameter count. Reworded to describe the\n actual heuristic and call out the future option.\n - Test plan said "29 unit tests" with a stale per-area breakdown.\n Updated to 86 with current coverage areas, and added a note to\n bump the number when adding tests.\n\nValidation\n----------\n* cargo test -p mesh-mixture-of-agents --lib: 86/86 pass\n* cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass\n* cargo test -p skippy-cache --lib: 13/13 pass\n* cargo test -p model-ref --lib: 10/10 pass\n* cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime\n -p skippy-cache -p model-ref --all-targets -- -D warnings: clean\n* cargo fmt --all -- --check: clean * fix(moa): drop malformed tool_calls instead of inserting empty-id placeholders PR #612 review (Copilot) \u2014 two follow-ups. 1. session.rs: malformed tool_calls in caller history ----------------------------------------------------- The earlier shape defaulted missing/empty `id` and `function.name` to `""` and still pushed a `PendingToolCall`. With adversarial or buggy caller history this had two real consequences: * Two malformed calls share `call_id == ""` and become indistinguishable, so any later `role: "tool"` whose `tool_call_id` is also missing matches the first one rather than the intended call. * The `tool_call_response` wire-shape invariant (non-empty function name) is violated downstream. Now both `assistant`-side ingestion and `tool`-side matching skip entries with missing or empty `id`/`tool_call_id` (and `assistant` ingestion also skips entries with empty `function.name`), emitting a `tracing::warn!` so the issue is visible in logs. Well-formed history is unaffected. 2. normalize.rs: doc/code mismatch on tool_arguments ---------------------------------------------------- The `extract_tool_arguments` doc claimed missing/null `arguments` collapses to `Some({})`. The code returns `None` and relies on the downstream `tool_call_response` to substitute `"{}"` when serializing the wire shape. The wire output is correct either way, but the doc misled future maintainers about the invariant. Rewrote the comment to describe the actual invariant ("`None` or an object; `None` means emit `{}` at wire time") so a future refactor doesn't reintroduce the literal-"null" bug. Regression test --------------- `malformed_tool_calls_are_dropped_not_collapsed_to_empty_id` covers four malformed shapes in one history (missing id, missing function.name, empty id, plus an orphaned tool result) and asserts only the one well-formed call survives and the orphaned result doesn't attach. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 87/87 pass (+1 new) * cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass * cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean Live end-to-end validation -------------------------- Built locally and deployed to the 2-node private mesh (M4 Qwen3-8B + Studio MiniMax). All four agent harness shapes from PR #566/#612 exercised the changed code paths: * mini-agent.py model=auto \u2014 2 turns, correct. * mini-agent.py model=mesh \u2014 2 turns, correct. * mini-agent2.py model=mesh \u2014 4 turns multi-file, correct. * Goose model=mesh \u2014 1 tool call, correct. No regressions in MoA fan-out / tool-result reducer routing. * fix(moa): route streaming failures to HTTP 502 (drop in-band SSE error) PR #612 review feedback (Nick) \u2014 two related findings on the streaming failure path. 1. Streaming MoA failures returned HTTP 200 ------------------------------------------- `write_moa_response` previously routed all streaming responses through `send_moa_as_sse`, which always emits `HTTP/1.1 200 OK` because SSE clients expect 200 before they start parsing the event stream. That meant `stream: true` MoA failures (`all_workers_failed`, `all_reducers_failed`, reducer hedge exhaustion) arrived as a 200 SSE that *happened* to carry `finish_reason: "error"` \u2014 dumb HTTP clients saw a "successful" stream and didn't realise inference had failed. The body is fully available before we decide how to write it, so we can collapse failure-shaped streaming responses to a non-streaming HTTP 502 JSON response with the structured error body. This matches the OpenAI API shape (failures on streaming endpoints come back as a single non-streaming JSON error response with the right HTTP status) and means streaming and non-streaming MoA failures are now consistent at the HTTP layer. 2. Error-only SSE chunk could break OpenAI-shape clients -------------------------------------------------------- The previous error chunk emitted on failure was `{ object: chat.completion.chunk, choices: [], error: \u2026 }`. Many OpenAI client SDKs assume every `chat.completion.chunk` has `choices[0]` and index blindly into it, so an empty `choices: []` array crashes those clients with an index error. With finding #1's routing change, failure-shaped bodies never reach `send_moa_as_sse` anymore, so the error-chunk emission is dead code. Drop it. `send_moa_as_sse` now does one clear thing: emit a single delta chunk + a `finish_reason: "stop" | "tool_calls"` stop chunk, both with a real `choices[0]`. A `debug_assert!` pins the invariant in tests. Tests ----- Four new unit tests covering the four corners of the routing decision: * `streaming_success_routes_to_sse` * `streaming_failure_routes_to_json_502_not_sse` * `non_streaming_success_routes_to_json_200` * `non_streaming_failure_routes_to_json_502` Validation ---------- * cargo test -p mesh-llm-host-runtime --lib: 1458/1458 pass (4 new) * cargo test -p mesh-mixture-of-agents --lib: 87/87 pass * cargo clippy ... --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean Live end-to-end validation on 2-node mesh (M4 + Studio MiniMax): * Non-streaming MoA happy path: HTTP 200 + structured response. * Streaming MoA happy path: HTTP 200 + `text/event-stream` + `[DONE]`. * mini-agent.py model=mesh: 2 turns, correct. * mini-agent2.py model=mesh: 4 turns multi-file, correct. * Goose model=mesh: 1 tool call, correct. The failure path is exercised by unit tests; production-triggering streaming failures would require fault injection beyond what the harness covers, but the routing logic and the SSE invariant are locked down at the code level.

Address Copilot review on PR #629: 1. `apply_enable_thinking` silently dropped the flag if a caller passed a non-object for `chat_template_kwargs` (string, array, number, null). The `entry().or_insert_with()` doesn't replace a non-object, so `as_object_mut()` then returns None and the override is lost. We now normalize a bogus shape to {} before injecting, so the flag always applies. New test `apply_enable_thinking_normalizes_non_object_kwargs` pins all four bogus-input shapes. 2. `effective_enable_thinking_for_moa` had two doc comments merged together because there was no blank line / item separating them. The merged doc said both 'returns None when caller hasn't expressed a preference' (true of the inner extractor) AND 'MoA picks for them: off' (true of the outer policy). Split into one doc per function. No behavior change beyond fix #1.

…answers (#629) * feat(moa): propagate enable_thinking to every worker and the reducer For `model: "mesh"`, the OpenAI-style reasoning knobs (`reasoning_effort: "none"`, `reasoning: { enabled: false }`, `enable_thinking: false`, `thinking_budget: 0`, `chat_template_kwargs.enable_thinking`, and the THINKING_BOOLEAN_ALIASES) were silently dropped in MoA. The fast worker's 256-token budget then got burned inside an unclosed `<think>` block on reasoning models and never reached the actual answer. Live lab on a 3-node mesh (M4 Qwen2.5-3B + studio MiniMax-M2.5 + mini Qwen3.5-9B), "reply with one short word" prompt: | Run | default thinking | reasoning_effort: none | |-----|--------------------------|------------------------| | 1 | 32s, leaked "Thinking Process: 1. Analyze..." | 1s, "Okay." | | 2 | 17s, "Yes" | 2s, "Hello" | | 3 | 32s, leaked reasoning prose | 1s, "Hello" | Worker log on the slow runs shows Qwen3.5-9B returning 1700+ char payloads at 31s \u2014 the full runaway think block. On the no-think runs all three workers finish in <1.3s with 4-5 char clean answers. Implementation: * `SamplingParams` gains `enable_thinking: Option<bool>` and a builder-style `with_thinking(...)` helper. `None` (default) is "don't override" \u2014 callers without a preference see no behavior change. * New `backend::apply_enable_thinking(body, hint)` helper centralises the wire shape: injects `chat_template_kwargs.enable_thinking` (canonical llama.cpp chat-template knob) and `reasoning_effort: "none"` when disabled. Merges into existing `chat_template_kwargs` instead of clobbering. * `HttpBackend`, `LocalModelBackend`, and `RemoteModelBackend` all call it after building their request bodies. * `GatewayConfig` gains `enable_thinking: Option<bool>` so the choice flows in one place to every worker (`SamplingParams::worker().with_thinking(...)`) AND the hedged reducer (`SamplingParams::reducer().with_thinking(...)`). * `moa_gateway.rs::try_handle_moa` extracts the override from the inbound request body via the new `extract_enable_thinking_override` helper, which mirrors every shape that `openai_frontend::common::normalize_reasoning_template_options` recognises (so MoA users get the same surface as direct callers). Tests: * mesh-mixture-of-agents lib: 98 pass (6 new for apply_enable_thinking and SamplingParams::with_thinking). * mesh-mixture-of-agents/tests/sim_enable_thinking_propagation.rs (new): 3 mock-backend integration tests pinning that chat_template_kwargs.enable_thinking reaches every worker AND that no spurious fields appear when no override is requested. * mesh-llm-host-runtime lib: 1478 pass (10 new for extract_enable_thinking_override covering every JSON shape). * clippy --all-targets -D warnings: clean. * cargo fmt --all -- --check: clean. Compat: * No mesh wire-protocol change. The new fields travel inside the existing chat-completion request JSON over the QUIC tunnel; peers on older binaries simply forward the body to llama.cpp which already understands `chat_template_kwargs`. Additive only, safe across mixed-version meshes. * No skippy ABI change. No plugin protocol change. * MoA callers without a preference get the same behavior as before this commit \u2014 `enable_thinking: None` is the default. Lab follow-up: a new `mesh_no_think` probe in the stability lab (`/tmp/lab/probe-stable.sh`) hits `/v1/chat/completions` with `reasoning_effort: "none"` so we can A/B latency and quality over hours of probe traffic. * feat(moa): opinionated no-think default — workers don't think unless asked Per Mic: MoA shouldn't have to ship behind a UI toggle. The MoA gateway now defaults to `enable_thinking = Some(false)` for every `model: "mesh"` request. Callers can still opt-in by passing any recognised reasoning knob explicitly (`reasoning_effort: "low"`, `enable_thinking: true`, etc.) — but the default is reasoning off. Rationale: * Workers are short-budget internal slots, not user-facing reasoning steps. The fast worker has a 256-token budget that doesn't fit `<think>...</think>` + answer. * The reducer doesn't want reasoning prose as candidate input. * MoA chat UX is the worst case for thinking models — every turn pays the reasoning latency penalty without giving the user the reasoning output. Implementation: * Extracted the policy into a tiny pure function, `effective_enable_thinking_for_moa(&body) -> Option<bool>`, that returns `extract_enable_thinking_override(body).or(Some(false))`. * 4 unit tests cover the contract: silent caller → no-think; explicit disable → no-think; explicit enable → thinking on (escape hatch); tool turn → still no-think by default. * The existing `extract_enable_thinking_override` tests are unchanged (they test the parser, not the gateway policy). Live verification on the 3-node lab mesh (M4 + studio + mini), release binary: * `model=mesh` with no knobs: 3/3 clean short answers ("Hello", "Okay", "Hello"), zero think leakage. * `model=mesh` + `reasoning_effort: "low"`: thinking turns back on, raw "Thinking Process:" prose appears as expected. * `model=mesh` + `tools=[read_file]`: tool_calls path unchanged, `finish_reason: tool_calls`, correct args. No-think default applies. Tests: 1486 host-runtime lib pass (4 new), all mesh-mixture-of-agents tests pass, clippy + fmt clean. * fix(moa): grace fires on diverse fast answers, not just sole answer Live public-mesh lab data showed a real failure: when 3-5 workers all return short answers in <1s but the answers don't textually agree (e.g. "Hello" / "Yes" / "Ready" / "Okay"), the arbiter's consensus rule ("\u22652 workers agree on answer") doesn't fire. MoA then waits for the slow tail worker before deciding, even though every fast worker has already produced a confident answer. Sample from the public-mesh lab (5 runs of "reply with one short word"): | Run | mesh time | what happened | |----:|----------:|---| | 1 | 45.5s | waited for slow Qwen3-8B | | 2 | 36.0s | waited for slow Qwen3-8B | | 3 | 38.2s | waited for slow Qwen3-8B | | 4 | 0.7s | 2 workers happened to textually agree | | 5 | 44.9s | waited for slow Qwen3-8B | The previous grace logic only armed when (sole answer). That covers the case where one worker has answered and the rest are still pending. It does NOT cover the case where multiple workers have answered fast but disagree. This fix relaxes grace eligibility to: "at least one Answer-kind output with confidence \u2265 0.5". When grace fires with multiple qualifying answers, pick the highest-confidence one. The previous sole-answer case is naturally preserved (max_by on a single element returns that element). Worker role and short-message variance is the dominant input here \u2014 fast and specialist workers running on different model families rarely produce textually-identical short answers, so consensus is a high bar. Grace gives us a sensible time-bounded escape hatch. Tests: * mesh-mixture-of-agents::fanout: 6 grace tests pass * New: grace_fires_with_multiple_diverse_answers \u2014 pins the real-world public-mesh case * New: grace_picks_highest_confidence_when_multiple_qualify \u2014 pins the highest-confidence picking rule * Existing 4 grace tests unchanged behavior; one was renamed internally but still asserts \u2018sole-answer grace fires\u2019 Compat: no API change. The relaxed eligibility is strictly more permissive than before, so existing callers see grace fire in cases where it didn't before (the public-mesh latency-tail case). No agentic regression \u2014 `!has_tools` still gates grace entirely. * fix(moa): also tighten chat grace default 6s -> 3s Bundled into the same PR (#624) as the eligibility relaxation. The two changes are inseparable in the chat user journey: * Eligibility relaxation (this PR's main change): grace fires on diverse fast answers, not just sole answers. Cures the 40s tail on the public mesh. * Tighter default (this commit): chat latency floor moves from ~6s to ~3s now that grace is the dominant chat path. Lab data on the public mesh: median mesh_chat went from ~6s to ~2s after this value, no quality regression on factual, arithmetic, or short-creative prompts. Without the eligibility relaxation, dropping the timer doesn't help much (grace rarely fires anyway). Without the tighter default, the relaxation just trades 40s for 6s instead of 40s for 2s. Reviewers should agree on both as one user-facing improvement. * fix(moa): normalize non-object chat_template_kwargs + clarify doc Address Copilot review on PR #629: 1. `apply_enable_thinking` silently dropped the flag if a caller passed a non-object for `chat_template_kwargs` (string, array, number, null). The `entry().or_insert_with()` doesn't replace a non-object, so `as_object_mut()` then returns None and the override is lost. We now normalize a bogus shape to {} before injecting, so the flag always applies. New test `apply_enable_thinking_normalizes_non_object_kwargs` pins all four bogus-input shapes. 2. `effective_enable_thinking_for_moa` had two doc comments merged together because there was no blank line / item separating them. The merged doc said both 'returns None when caller hasn't expressed a preference' (true of the inner extractor) AND 'MoA picks for them: off' (true of the outer policy). Split into one doc per function. No behavior change beyond fix #1.

macos menu app

5332ea5

michaelneale force-pushed the main branch from 42fbadf to 899740f Compare March 14, 2026 16:05

michaelneale pushed a commit that referenced this pull request Mar 23, 2026

Docs/release prereqs (#1)

23c1731

* docs: update prereqs for RELEASE * docs: remove unnecessary --console option from RELEASE

michaelneale closed this Mar 28, 2026

michaelneale mentioned this pull request Apr 3, 2026

Explore CommitLLM as an optional verified-inference backend #147

Open

Copilot AI mentioned this pull request Apr 9, 2026

docker: multi-stage images for most architectures, CI, and fly rewrite #220

Merged

ndizazzo mentioned this pull request Apr 12, 2026

feature: make mesh-llm app load aware to stop/start models depending on local usage #256

Open

michaelneale mentioned this pull request Apr 13, 2026

Harden Nostr mesh discovery #268

Closed

michaelneale mentioned this pull request Apr 13, 2026

Harden Nostr discovery: validate tokens, candidate fallback #269

Merged

ventz mentioned this pull request Apr 16, 2026

A30 / driver 535: v0.60.3 CUDA bundle fails at matmul despite sm_80 cubins being present #304

Closed

Copilot AI mentioned this pull request May 21, 2026

chat: real data by default, MoA for 'Auto', sole-answer grace for slow meshes #615

Merged

michaelneale mentioned this pull request May 22, 2026

feat(moa): no-think default on workers + grace fires on diverse fast answers #629

Merged

michaelneale mentioned this pull request May 24, 2026

fix(kv-eviction): proactive prefix cache eviction before decode #661

Merged

michaelneale mentioned this pull request Jun 2, 2026

Fix models not appearing: stop /api/models from re-downloading the catalog every call #776

Merged

michaelneale mentioned this pull request Jul 2, 2026

Fix daily layer-package split jobs failing on first artifact #945

Merged

This was referenced Jul 27, 2026

Add Inkling Q2 staged runtime and certify text split serving #1017

Closed

Ingress control plane (/v1/models, health) head-of-line blocks behind in-flight inference #1126

Closed

ndizazzo mentioned this pull request Aug 18, 2026

fix(skippy): stop a stalled SSE consumer pinning a generation worker #1367

Merged

michaelneale mentioned this pull request Aug 18, 2026

Stream tool calls as they are generated instead of after the turn ends #1371

Merged

i386 mentioned this pull request Aug 24, 2026

Design: from-scratch iteration-level scheduler for concurrent staged serving #1416

Open

11 tasks

michaelneale mentioned this pull request Aug 24, 2026

Rescue terminal Buzz replies in MoA #1417

Open

Navigation Menu

Uh oh!

Uh oh!

macos menu app - #1

macos menu app#1
i386 wants to merge 1 commit into
Mesh-LLM:mainfrom
i386:macos

i386 commented Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

i386 commented Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants