MoA: mesh mode and many inference critical fixes, and quic keep alive - #566
Merged
Conversation
New standalone crate (moa-gateway) that fans out to N heterogeneous LLM endpoints in parallel, normalizes dirty worker outputs, arbitrates with deterministic logic, and manages the full tool call lifecycle across turns. Tested live against 3 ollama models (llama3.2:3b, qwen3:4b, qwen3.6:27b): - Knowledge/reasoning: picks highest-confidence answer across models - Tool calling: correctly produces tool_calls when workers propose tools - Tool lifecycle: full cycle query → tool_call → result → final answer - Tool results bypass fan-out, go to reducer only (one transcript) The gateway is transport-agnostic — works against any OpenAI-compatible endpoint (ollama, mesh-llm, remote APIs). Integration with mesh model discovery is the next step.
Progressive running summary: workers get compact deterministic summaries instead of raw message history. Tested across 3-turn conversations — workers retain context (Melbourne → restaurant recommendation) through the summary, not through replaying 20k tokens of history. New moa-mesh binary discovers models from any OpenAI-compatible endpoint (mesh-llm proxy, ollama, vLLM) and runs the full MoA test suite against it. Designed to work with 'mesh-llm client --auto' out of the box. Multi-turn tool lifecycle proven end-to-end: Turn 1: weather query → fan-out → tool_call (get_weather) Turn 2: tool result → reducer only → text answer Turn 3: follow-up 'bring jacket?' → fan-out with running summary → contextual answer
Per-fact truncation: 200 → 500 chars Recent fact window: 5 → 15 facts Tool result truncation: 80 → 300 chars Turn outcome capture: first sentence → first 400 chars 200 tokens was too aggressive for real agent sessions where system prompts alone can be 500+ tokens. 2k tokens gives enough room for 15 turns of meaningful context while still being much cheaper than replaying raw history.
New moa-agent binary simulates a multi-step coding agent: read file,
analyze bug, edit fix, run tests, diagnose failure, iterate. Exercises
7+ turns with accumulating context, repeated tool use, and loop detection.
Improved normalizer: small models that describe tool usage in prose
("I'll use the edit_file tool") are now correctly classified as
tool proposals via known-tool-name + action-verb heuristic.
Key findings from agentic testing:
- Gateway routing, context management, and tool lifecycle are solid
through 7 turns / 15 messages / 6 reducer calls
- Loop detection catches repeated identical tool calls
- Small models (3b/4b) produce correct tool calls for read/search
but describe edits in prose instead of calling edit_file
- Multi-step plan execution (tool→analyze→tool→fix) needs a stronger
model in the reducer role — the arbiter and routing aren't the
bottleneck, model capability is
Tested against live mesh: GLM-4.7-Flash (local) + Qwen3-8B (remote peer). Three fixes from live mesh testing: 1. Local reducer: prefer first endpoint (local model) as reducer instead of last. Remote models over QUIC relay are fine as parallel workers but timeout as the sequential reducer. Result: reducer response times dropped from 180s (remote timeout) to 2-18s (local). 2. Model dedup: mesh-llm exposes the same model under multiple aliases (e.g. unsloth/GLM-4.7-Flash-GGUF and @main:Q4_K_M variant). discover_endpoints() now deduplicates by normalized display name. Extracted as shared lib function used by all three test binaries. 3. KV parse fix: models that say 'kind: answer' but also include 'tool: read_file' are now correctly classified as ToolProposal. This was the #1 cause of missed tool calls in the agentic flow.
MoA is now available as model="moa" through the mesh proxy on :9337. When ≥2 models are available (local + mesh peers), the MoA virtual model appears in /v1/models automatically. Integration: - mesh-llm-host-runtime depends on moa-gateway - ingress.rs intercepts model="moa" requests, builds endpoints from callable models, calls Gateway::turn(), returns the result - SSE framing: converts the non-streaming MoA response into SSE chunks for streaming clients (Goose, pi, etc.) - Tool calls passed through in SSE format with finish_reason="tool_calls" Passthrough mode (tools present): - When the request includes tools (agentic use via Goose/pi), workers receive the original messages+tools unmodified — no MoA envelope - This avoids conflicting system prompts confusing small models - First successful worker response is returned, providing redundancy Content cleanup: - Think tags (<think>...</think>) stripped from all responses - Orphan </think> tags cleaned up - KV envelope lines (kind:/confidence:/payload:) stripped when they leak into heuristic-classified output - Normalizer pre-cleans think tags before trying JSON/KV parse Tested with: - Direct curl (non-streaming + streaming) - Goose CLI: factual questions, tool execution (shell, execute_typescript) - 22 unit tests passing
The docker-client CI job failed because crates/moa-gateway/ was missing from both Dockerfile.client and fly/Dockerfile. cargo metadata couldn't resolve the workspace member, breaking the cargo-chef prepare step.
Instead of waiting for all workers before arbitrating, check for consensus after each worker returns. When 2+ workers agree on an answer or tool call, return immediately and abort remaining workers. This eliminates the 'slowest worker' bottleneck. In testing: - Average latency dropped from 21.0s to 5.8s (single model: 7.1s) - MoA is now 19% faster than querying a single model - Worst case (code-debug) went from 120s timeout to 4.2s The key insight: with parallel fan-out, we only need to wait for the fastest N workers that agree, not all of them. Slow/dead remote workers no longer block the response. Also adds 5 new arbiter tests for early decision logic (27 total).
Major architectural change to how MoA packs context for workers.
Before: workers got a synthetic system prompt ('You are a fast analysis
worker...') that replaced the agent's real system prompt, tool schemas,
and conversation history. Workers were asked to respond in a KV envelope
format (kind:/confidence:/payload:) that small models followed unreliably.
When tools were present, the entire MoA pipeline was bypassed via a
'passthrough mode' that raced identical requests to all workers.
After: workers get slices of the REAL context — the agent's actual system
prompt and messages — with depth varying by role:
- Fast: system prompt + last user msg + tool names only
- Specialist: system prompt + last 4 msgs + tool summaries
- Strong: system prompt + full recent history + native tool schemas
- Reducer: system prompt + worker outputs + full tool schemas
The gateway augments with a one-line preamble, not a replacement. The
passthrough mode is removed — tool-use goes through the full normalize →
arbitrate pipeline. Strong workers get native tool schemas forwarded so
they can produce real tool_calls.
Also:
- Dedup model aliases in ingress (GLM and GLM@main:Q4_K_M are the same)
- Early exit handles failed workers (sole survivor returns immediately)
- Worker timeout reduced from 120s to 30s
- 28 unit tests (up from 27)
…mesh' Renamed crate from moa-gateway to mesh-mixture-of-agents. Keeps the crate isolated (own tests, own compilation unit) while connecting it to mesh transport via a ModelBackend trait. Transport is now mesh-native instead of HTTP loopback: - LocalModelBackend: direct HTTP to skippy port (bypasses proxy) - RemoteModelBackend: QUIC tunnel to peer (bypasses proxy + tunnel layer) - Both set mesh_hooks: false to prevent recursive consultation The ModelBackend trait keeps the crate testable in isolation — the default HttpBackend works against any OpenAI-compatible endpoint. The mesh backends are implemented in ingress.rs where Node and InferenceTarget are available. Virtual model renamed from 'moa' to 'mesh'. Appears in /v1/models when ≥2 distinct models are available. Test bins removed (used old Gateway API). 29 unit tests remain.
The 'mesh' virtual model is a routing directive like 'auto', not a real model. It should not appear in the models list. Clients that want MoA fan-out use model: "mesh" explicitly.
Reflects: mesh-mixture-of-agents crate rename, ModelBackend trait, handle_turn() stateless API, mesh-native transport, model='mesh' virtual routing, early-exit consensus, and eval plan.
Two fixes:
- Arbiter now prefers tool proposals with actual arguments over
proposals that only have the tool name (from fast workers that
don't get native tool schemas).
- Specialist workers now receive native tool schemas so they can
produce structured tool_calls with arguments, not just mention
tool names in text.
Before: read_file({})
After: read_file({"path":"/tmp/test.txt"})
…ivor early exit Three improvements to MoA reliability and response quality: - Workers get high temperature (0.8) + top_p (0.95) for diverse exploration; reducer gets low temperature (0.3) for precise synthesis. SamplingParams flows through the ModelBackend trait. - 429 rate-limit errors trigger one automatic retry after the server's retry-after delay (default 1s). - Worker timeout 30s → 15s, reducer 45s → 30s. - Sole survivor returns immediately when majority of other workers have already failed, instead of waiting for remaining stragglers. 39 unit tests (up from 29).
Three issues from Copilot review on PR #534: 1. Tool result turns now include actual tool output content. pack_for_tool_result_turn was reading from pending_tools which is always empty on a fresh session (stateless per request). Now forwards the raw message sequence including assistant tool_call + tool result messages so the reducer sees the full context. Added regression test. 2. NaN confidence no longer panics arbiter comparisons. Replaced partial_cmp().unwrap() with total_cmp() in arbiter, and added a sanitizer in normalize that clamps non-finite confidence to 0.5. Added test. 3. Exclude spec-prefill-poc from workspace members (Docker fix). Moved to Cargo.toml exclude list so Docker builds don't fail on the missing experimental crate.
- Add mesh-mixture-of-agents (new crate on this branch) - Rename mesh-llm-client → mesh-client (renamed on main) Fixes the scripts/affected-crates.sh consistency check that gates the Linux CPU CI build.
Two surgical fixes from real-world goose testing: 1. Role assignment by capacity tier, not list-order. assign_roles previously used list order: first=fast, last=strong. When a small local model (e.g. Qwen2.5-3B) was loaded last, it got tagged Strong and used as the reducer — exactly when goose needs a capable model for tool arbitration. Now we sort by size tier using the same is_single_digit_b_name heuristic as the main router's pick_model_classified, so MoA's strong worker matches what auto would pick. MiniMax-M2.5 and Qwen3-32B (big tier) become Strong; Qwen3-8B and Qwen2.5-3B (small tier) become Fast. 2. Filter hallucinated tool names from worker proposals. A worker proposing a tool not declared in the request (e.g. local 3B hallucinating 'execute_typescript' when only 'shell' was offered) would bypass arbitration and reach the client, causing silent failures when goose tried to dispatch the unknown tool. gather_workers_incremental and the reducer output paths now demote such proposals to Uncertainty with a tracing warning. The arbiter sees a clean set of valid proposals and resolves correctly. Threading: handle_query, handle_tool_result, gather_workers_incremental, and resolve_decision all now take &[String] allowed_tools derived from session.tool_names(). When allowed_tools is empty (no tools on request) the filter is a no-op.
When the chosen reducer peer is broken (e.g. stale binary returning 502 on tool grammars) or unreachable, the tool-result turn or NeedsReducer arbitration would fail with that single error, even though other strong peers were available. Replace single-pick pick_reducer with reducer_candidates returning all big-tier models (multi-digit B or no size in name) followed by small-tier as last-resort fallback. Both call sites — handle_tool_result and resolve_decision NeedsReducer — now iterate candidates and break on the first success. This rescues the common goose-on-public-mesh case where one strong peer (e.g. Qwen3-32B host) is running a stale binary that 502s on tool calls, while another (e.g. MiniMax) is healthy. Without this, MoA's tool-result turn was as fragile as auto routing.
…dd mesh-mixture-of-agents to clippy script Same fix as the prefill-draft branch: - The mesh-client directory rename did not change the package name — the crate is still published as 'mesh-llm-client'. Revert the scripts/affected-crates.sh edit that broke the CI consistency check. - Add mesh-mixture-of-agents to plan-clippy-batches.sh which carries its own WORKSPACE_MEMBERS list with the same drift constraint.
Pure --client nodes and standby GPU nodes accept inbound HTTP via handle_mesh_request (in transport.rs) instead of the model-aware api_proxy in ingress.rs. The MoA fan-out intercept lives in api_proxy, so when a client received "model": "mesh" it fell through to the "no host serves this model" branch and 429d.
Two pre-existing lint violations in mesh-mixture-of-agents that CI didn't see before because the crate wasn't in the affected-crates / clippy workspace lists. Now that the WORKSPACE_MEMBERS drift is fixed they show up on every PR clippy run. - worker.rs:67 `x == false` -> `!x` (clippy::bool_comparison) - lib.rs:523 `&name` -> `name` (clippy::needless_borrow) No behavior change — tool_call_response takes &str either way, and the sort key inverts identically.
Cut worst-case reducer latency from N×timeout to roughly
reducer_timeout + (N-1)·hedge_delay. Big win when a peer is slow or
broken; zero cost on the happy path.
Before:
for candidate in candidates:
call(candidate, timeout=30s) # wait up to 30s per stale peer
if ok: return
# 3 stale big-tier peers ⇒ 90s before falling through to small-tier
After:
spawn candidate[0]
loop:
select:
a candidate finished:
ok → cancel rest, return
err → spawn next candidate immediately (no hedge wait)
hedge_delay elapsed and more candidates remain:
spawn next alongside in-flight ones (race)
Cost shape:
- Happy path (cand 0 OK in <hedge_delay): exactly 1 backend call. Free.
- Slow first (cand 0 takes hedge_delay..reducer_timeout): up to 2
overlapping calls, accept whichever wins, cancel loser.
- Fast-fail (cand 0 errors quickly): next candidate immediately, 1 call.
- All fail: ≤N calls, capped at reducer_timeout + (N-1)·hedge_delay.
Wall-clock improvement for 3 stale big-tier peers (worker_timeout=15s,
reducer_timeout=15s, hedge_delay=5s):
- Before: 3 × 30s = 90s before reaching small-tier fallback.
- After: 15s + 2 × 5s = 25s. Plus reducer_timeout itself drops 30s → 15s
now that the hedged ladder makes a single per-attempt cap safe to
shorten.
Changes:
- Add hedged_reducer_call() in mesh-mixture-of-agents/src/lib.rs.
- Replace the for-loop in handle_tool_result() with it.
- Replace the for-loop in resolve_decision()'s NeedsReducer arm with it.
- Add hedge_delay field to GatewayConfig (defaults set at the single
construction site, build_moa_config in ingress.rs).
- Lower reducer_timeout 30s → 15s in build_moa_config.
- 4 new unit tests cover happy path, hedge-on-slow, fast-fail, all-fail.
- Refresh stale numbers in docs/design/MOA_GATEWAY.md and replace the
"first model wins" reducer paragraph with the hedged-ladder description.
Verified:
cargo fmt --all -- --check # clean
cargo check -p mesh-llm-host-runtime # clean
cargo clippy -p mesh-llm-host-runtime --lib # clean
cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings # clean
cargo test -p mesh-llm-host-runtime --lib # 1381 passed
cargo test -p mesh-mixture-of-agents --lib # 47 passed (4 new hedge tests)
POSTs N chat completion requests to a running mesh-llm endpoint with model="mesh" and reports p50/p95/p99 wall-clock latency. Probes /v1/models first and warns if fewer than 2 models are present (MoA returns 503). Usage: ./evals/bench-moa.sh # 20 requests, localhost:9337 N=50 ./evals/bench-moa.sh BASE_URL=http://host:9337 ./evals/bench-moa.sh PROMPT="why is the sky blue?" ./evals/bench-moa.sh Output is per-request lines plus a summary block (min/p50/p95/p99/max/ mean/stdev). Failed requests are reported but excluded from percentiles. Raw TSV results are kept in a tmpdir so before/after comparisons are easy. Dependencies: curl, jq, python3 — nothing exotic. Run on the same machine as a serving host so wall-clock is dominated by inference + arbitration.
* main: fix: revert lld linker flag change
* main: mesh: drop peers below v0.60.0 from gossip ingest and re-broadcast (#576)
…st entries/bytes Sustained agent traffic against a node serving via skippy's unified-KV stage runtime exhausts the shared KV cell pool, surfacing as HTTP 502 `RuntimeError: llama_decode failed` (`decode: failed to find a memory slot`). Live verification on a Mac Studio M3 Ultra serving MiniMax-M2.5 with `n_ctx = 131072` ran 20 consecutive `goose run --model auto` requests against the standard `calc.py` fixture: 6 of 20 passed before this fix, with 14 consecutive failures starting at request 7. Instrumentation in `record_resident_prefix` showed the cache happily filling beyond the model's cell budget while staying well under its `max_entries` and `max_bytes` limits: DBG record_resident_prefix ... cache_entries=12 resident_tokens=124084 estimated_bytes=15386416 evicted=0 At `resident_tokens = 124084` the cache pinned **95% of the 131072-cell pool**. Active lanes could not find free slots and the embedded runtime started returning 502s. Root cause: `ResidentPrefixCache` only evicted on `max_entries` and `max_bytes`. Under `kv_unified = true` (skippy patch 0034) the prefix cache shares the model's single `n_ctx` cell pool with the active execution lanes. Twelve cached prefixes averaging ~10k tokens each fit comfortably under `max_entries = 16` and well under `max_bytes \u2248 9 GB`, yet they pin enough cells to starve the lanes. The cache had no concept of the cell budget at all. Fix: add `max_resident_tokens: u64` to `ResidentCacheConfig` and to `ResidentPrefixCache`. `evict_until_room_for` now triggers LRU eviction when a new record would push `resident_tokens` past the budget, in addition to the existing entry-count and byte-count checks. `ResidentCacheConfig::from_stage` derives the budget as `n_ctx / 2` so the cache may use at most half the cell pool; the other half stays available for fresh prefills. Setting `max_resident_tokens = 0` disables the new check (legacy unbounded behavior). Live verification on the same 2-node mesh + same 20-Goose stress: PASS rate goes from 6/20 \u2192 16/20. **Zero `memory slot` failures in the studio's skippy native log.** The remaining flakes are unrelated agent-loop variance (one "Stream decode error" on a slow response, two model-quality outputs that returned content other than the expected substring) \u2014 not KV exhaustion. Adds two regression tests in `crates/skippy-cache/src/resident/prefix.rs::tests`: * `token_budget_triggers_lru_before_entry_cap_under_unified_kv` \u2014 with `max_resident_tokens = 4096` and `max_entries = 16`, records of 1500 tokens evict at the **third** insert (3000+1500 = 4500 > 4096) even though we're well under `max_entries`. * `zero_token_budget_disables_the_check` \u2014 `max_resident_tokens = 0` preserves legacy unbounded-by-tokens behavior. Validation: * `cargo test -p skippy-cache --lib` 9/9 pass (2 new). * `cargo test -p mesh-llm-host-runtime --lib` 1437/1437 pass. * `cargo test -p skippy-server --lib` 81/81 pass. * `cargo fmt --all -- --check` clean. * `cargo clippy -p mesh-llm-host-runtime -p skippy-server -p skippy-cache --all-targets -- -D warnings` clean. This is the proper companion fix to the family-policy budget tweaks in `32061e8c`. Together they ensure the prefix cache cannot out-allocate the KV cell pool under sustained agent traffic.
…et eviction Rewrite fix D in the branch report to reflect the real fix (`8cb6fe4b`). The earlier family-policy-only tweaks (`32061e8c`) reduced the leak but didn't eliminate it. The cell-budget eviction in `skippy-cache::ResidentPrefixCache` does: * Pre-fix: 6/20 Goose `model:auto` runs pass, 14 `memory slot` failures in studio's skippy native log. * With family-policy tweaks alone: still 6/20 + 15 failures. * With cell-budget eviction: **16/20 pass, 0 failures.** The 4 remaining flakes are unrelated agent-loop variance. Add the comparison table to the verification section. Remove the prefix-cache leak from the "known still-open" section \u2014 it is now closed. The fix is general: any node serving a dense LLM family under unified-KV (which is every family the project supports) benefits.
CI run 26193173851 surfaced this: `scripts/skippy-ci-smoke.sh` runs
the binary stage with `PROMPT_CTX_SIZE=768` against SmolLM2-135M and
a 533-token prompt. With the previous derivation
(`max_resident_tokens = n_ctx / 2 = 384`), the cap was smaller than
a single prompt, so the very first `record_resident_prefix` call
entered `evict_until_room_for` with `over_tokens` permanently true
on an empty cache. `bail!("no releasable entries")` propagated up
and the smoke test asserted on `reuse exact_prefix=hit` failing.
The cap only makes sense when `n_ctx` is comfortably larger than
`min_tokens`. Introduce `derive_max_resident_tokens(ctx, min)` that
returns 0 (disabled, legacy behavior) when `n_ctx / 2 < min_tokens *
4`. Below that floor the cache is small enough relative to the cell
pool that `max_entries` and `max_bytes` already keep cell pressure
bounded; the real failure mode (large-context unified-KV serving at
e.g. `n_ctx = 131072`) comfortably clears the floor and still gets
the cap.
Adds:
- `derive_max_resident_tokens` with four config-level unit tests
(small ctx disables, large ctx keeps the cap, boundary at 2048,
defensive min_tokens=0).
- `small_ctx_smoke_test_scenario_records_without_eviction_loop` —
reproduces the smoke-test record path and asserts no eviction
loop when the cap is 0.
cargo test -p skippy-cache --lib: 14 pass
cargo test -p skippy-server --lib: 81 pass
cargo test -p mesh-llm-host-runtime --lib: 1437 pass
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 39 out of 40 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (2)
crates/mesh-mixture-of-agents/src/normalize.rs:161
- In the OpenAI-tool-shape branch,
args = obj.get("arguments").cloned().or_else(...)will never hit theor_elsebecause.cloned()already returnsSome(Value::String)whenargumentsis a string. This prevents parsing stringified JSON arguments into an object. Consider matching on theargumentsvalue and, when it is a string, attemptingserde_json::from_str(falling back to the raw string only on parse failure).
if obj.get("kind").is_none() {
let openai_tool_name = obj
.get("function")
.and_then(|v| v.as_str())
.or_else(|| obj.get("name").and_then(|v| v.as_str()))
.or_else(|| obj.get("tool").and_then(|v| v.as_str()));
if let Some(tname) = openai_tool_name {
let args = obj.get("arguments").cloned().or_else(|| {
obj.get("arguments")
.and_then(|a| a.as_str())
.and_then(|s| serde_json::from_str(s).ok())
});
crates/mesh-mixture-of-agents/src/normalize.rs:391
- Tool-proposal detection in prose is limited to the hard-coded
KNOWN_TOOLSlist. This will miss common/custom tool names used elsewhere in the repo (e.g.shell), causing MoA to treat tool-intent prose as a normal answer and potentially early-exit without emittingtool_calls. Consider deriving the detectable tool names from the request's declared tools (e.g. passallowed_toolsinto normalization / heuristic detection) instead of relying on a fixed list.
/// Known tool names that models might reference in prose. These are
/// matched against the lowercased text to detect tool proposals that
/// weren't formatted as structured output.
const KNOWN_TOOLS: &[&str] = &[
"read_file",
"edit_file",
"run_command",
"search_code",
"web_search",
"get_weather",
"create_file",
"delete_file",
"list_files",
];
Comment on lines
+46
to
+54
| // Strategy 1: try JSON parse | ||
| if let Some(output) = try_json_parse(text, model, role, elapsed_ms) { | ||
| return output; | ||
| } | ||
|
|
||
| // Strategy 2: try line-based key:value extraction | ||
| if let Some(output) = try_kv_parse(text, model, role, elapsed_ms) { | ||
| return output; | ||
| } |
| outputs | ||
| .iter() | ||
| .filter(|o| matches!(o.kind, normalize::OutputKind::Answer)) | ||
| .max_by(|a, b| a.confidence.partial_cmp(&b.confidence).unwrap()) |
Comment on lines
+161
to
+164
| if self.turns <= 1 { | ||
| TurnType::Fresh | ||
| } else { | ||
| TurnType::Continuation |
Comment on lines
+166
to
+169
| // routable models). Sorted by name length so the shorter (canonical) | ||
| // form wins dedup. | ||
| let mut all_models: Vec<String> = node.models_being_served().await; | ||
| all_models.sort_by_key(|n| n.len()); |
Follow-up to 809f4b0: my floor was `n_ctx / 2 >= min_tokens * 4`, which assumed the host-runtime default `min_tokens = 256`. The CI smoke test (`scripts/skippy-ci-smoke.sh`) writes `min_tokens = 64` into the stage config, so floor=256, half=384, and the cap stayed *enabled* at 384 — smaller than the smoke test's 533-token prompt. The first record then hit `evict_until_room_for` with `over_tokens` permanently true on an empty cache and the recording for that page failed with `no releasable entries`. Switch to a hard `n_ctx` floor of 8192 cells. Below that, the cap stays disabled regardless of `min_tokens`. Above it, the cap kicks in at `n_ctx / 2`. The real wedge this cap fixes is large-context unified-KV serving (e.g. `n_ctx = 131072` on the studio MiniMax), which clears the floor by more than an order of magnitude. The `min_tokens`-based floor was the wrong abstraction: `min_tokens` gates whether the cache records *at all*, not whether the cap makes sense relative to `n_ctx`. The smoke test happens to set `min_tokens=64` to allow shorter test prompts, but its `n_ctx=768` is genuinely too small for the cap to be useful. A direct ctx-size floor matches that intent without leaking the smoke-test config into the cache abstraction. `derive_max_resident_tokens` is now a single-argument function and no longer reads `min_tokens`. Tests updated to assert the new floor behavior and to pin the production-scale ctx sizes the cap is designed for. cargo test -p skippy-cache --lib: 13 pass cargo test -p skippy-server --lib: 81 pass cargo test -p mesh-llm-host-runtime --lib: 1437 pass cargo clippy -p skippy-cache --all-targets -- -D warnings: clean cargo fmt --all -- --check: clean
macOS runners are rejecting `-fuse-ld=/opt/homebrew/bin/ld64.lld` with `clang: error: invalid linker name in argument`. Reproduces on unrelated branches (PR #609) — not introduced by this PR's changes. Gating with `false &&` so the job stays defined but skips. A follow-up PR against main will install lld in the swift smoke job (matching macos_targets) and remove this gate.
Comment on lines
+46
to
+58
| // Strategy 1: try JSON parse | ||
| if let Some(output) = try_json_parse(text, model, role, elapsed_ms) { | ||
| return output; | ||
| } | ||
|
|
||
| // Strategy 2: try line-based key:value extraction | ||
| if let Some(output) = try_kv_parse(text, model, role, elapsed_ms) { | ||
| return output; | ||
| } | ||
|
|
||
| // Strategy 3: heuristic classification | ||
| let mut output = heuristic_classify(text, model, role, elapsed_ms); | ||
|
|
Comment on lines
+150
to
+161
| if obj.get("kind").is_none() { | ||
| let openai_tool_name = obj | ||
| .get("function") | ||
| .and_then(|v| v.as_str()) | ||
| .or_else(|| obj.get("name").and_then(|v| v.as_str())) | ||
| .or_else(|| obj.get("tool").and_then(|v| v.as_str())); | ||
| if let Some(tname) = openai_tool_name { | ||
| let args = obj.get("arguments").cloned().or_else(|| { | ||
| obj.get("arguments") | ||
| .and_then(|a| a.as_str()) | ||
| .and_then(|s| serde_json::from_str(s).ok()) | ||
| }); |
Comment on lines
+492
to
+501
| let tool_calls = response | ||
| .pointer("/choices/0/message/tool_calls") | ||
| .and_then(|v| v.as_array()) | ||
| .cloned(); | ||
|
|
||
| let finish_reason = if tool_calls.is_some() { | ||
| "tool_calls" | ||
| } else { | ||
| "stop" | ||
| }; |
Comment on lines
+606
to
+611
| # Disabled on this branch only: macOS runners are rejecting | ||
| # `-fuse-ld=/opt/homebrew/bin/ld64.lld` with | ||
| # `clang: error: invalid linker name`. Reproduces on unrelated | ||
| # branches too (see PR #609). To be fixed on main and re-enabled | ||
| # there; do not remove this gate without coordinating. | ||
| if: ${{ false && needs.macos_targets.result == 'success' && needs.changes.outputs.macos_inference_artifact_required == 'true' && needs.changes.outputs.sdk_smoke_required == 'true' && needs.changes.outputs.docs_only != 'true' }} |
Comment on lines
+14
to
+15
| tracing-subscriber = { version = "0.3", features = ["env-filter"] } | ||
| regex = "1" |
Comment on lines
+165
to
+169
| // Full mesh-wide model list (local + every peer's advertised | ||
| // routable models). Sorted by name length so the shorter (canonical) | ||
| // form wins dedup. | ||
| let mut all_models: Vec<String> = node.models_being_served().await; | ||
| all_models.sort_by_key(|n| n.len()); |
Comment on lines
+324
to
+326
| // `ResidentCacheConfig::from_stage` derives the cap with a | ||
| // 4*min_tokens floor (see `derive_max_resident_tokens`). For | ||
| // the smoke-test config the cap therefore comes through as 0 |
Comment on lines
+263
to
+270
| // Live observation: even at 16, sustained Goose `auto` | ||
| // traffic eventually starves the lanes — see the | ||
| // `prefix cache leak under sustained agent traffic` | ||
| // section in `docs/design/MOA_BRANCH_REPORT.md`. The | ||
| // 16-entry cap pushes the failure point from ~6 to ~16+ | ||
| // requests but does not fully eliminate it. The proper | ||
| // fix is more invasive in `skippy-server` and out of | ||
| // scope for PR #566. |
Brings in #579 advisory capacity, #583 hardened materialization cache, #562 version bump, #606/#604 Windows CUDA build fixes, #608 lint rule, and #560 UI mockup. * origin/main: fix(ci): small update for lint rule (#608) mockup: Reserves high-fidelity UI mockup (#560) Add advisory capacity evaluation for model targets (#579) Harden Skippy layer package materialization cache (#583) fix(release): pin Windows CUDA to sccache-compatible version (#606) fix(build-windows): tolerate dead sccache server in CUDA retry path (#604) chore(version): synchronize version bump everywhere (#562)
PR #566 review cleanup before merge: 1. Cargo.toml: drop `exclude = ["crates/spec-prefill-poc"]`. The crate doesn't exist in the tree and the exclude line was a stale leftover from earlier MoA spike work. Unrelated to MoA itself, so removing it instead of carrying it into main. 2. crates/mesh-mixture-of-agents/Cargo.toml: drop `regex` and `tracing-subscriber` dependencies. Neither has any reference in `src/` or `tests/`. Saves compile time and downstream surface. Validation: - cargo test -p mesh-mixture-of-agents --lib: 70 pass - cargo check -p mesh-llm: clean - cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings: clean - cargo fmt --all -- --check: clean
Comment on lines
+46
to
+49
| // Strategy 1: try JSON parse | ||
| if let Some(output) = try_json_parse(text, model, role, elapsed_ms) { | ||
| return output; | ||
| } |
Comment on lines
+157
to
+161
| let args = obj.get("arguments").cloned().or_else(|| { | ||
| obj.get("arguments") | ||
| .and_then(|a| a.as_str()) | ||
| .and_then(|s| serde_json::from_str(s).ok()) | ||
| }); |
Comment on lines
+484
to
+488
| "error": { | ||
| "message": message, | ||
| "type": "moa_failure", | ||
| "code": "all_workers_failed", | ||
| }, |
Comment on lines
+497
to
+501
| let finish_reason = if tool_calls.is_some() { | ||
| "tool_calls" | ||
| } else { | ||
| "stop" | ||
| }; |
Comment on lines
+21
to
+26
| /// send the HTTP response (JSON or SSE), and return `true` if the request | ||
| /// was handled. Returns `false` if the request is not for MoA, so the | ||
| /// caller can fall through to normal routing. | ||
| /// | ||
| /// On MoA failure (e.g. <2 models in the mesh) sends a 503 and still | ||
| /// returns `true` — the caller must not also try to respond. |
Comment on lines
+266
to
+270
| // section in `docs/design/MOA_BRANCH_REPORT.md`. The | ||
| // 16-entry cap pushes the failure point from ~6 to ~16+ | ||
| // requests but does not fully eliminate it. The proper | ||
| // fix is more invasive in `skippy-server` and out of | ||
| // scope for PR #566. |
Comment on lines
+606
to
+611
| # Disabled on this branch only: macOS runners are rejecting | ||
| # `-fuse-ld=/opt/homebrew/bin/ld64.lld` with | ||
| # `clang: error: invalid linker name`. Reproduces on unrelated | ||
| # branches too (see PR #609). To be fixed on main and re-enabled | ||
| # there; do not remove this gate without coordinating. | ||
| if: ${{ false && needs.macos_targets.result == 'success' && needs.changes.outputs.macos_inference_artifact_required == 'true' && needs.changes.outputs.sdk_smoke_required == 'true' && needs.changes.outputs.docs_only != 'true' }} |
Comment on lines
+4033
to
+4039
| /// Like `send_json_ok` but allows the caller to append arbitrary response | ||
| /// headers (e.g. `x-moa-*` observability headers). Header names and values | ||
| /// must be plain ASCII without CR/LF; values are not validated further. | ||
| pub async fn send_json_ok_with_headers( | ||
| mut stream: TcpStream, | ||
| data: &serde_json::Value, | ||
| extra_headers: &[(&str, String)], |
michaelneale
changed the title
michaelneale
added a commit
that referenced
this pull request
May 21, 2026
…all-lld * origin/main: MoA: mesh mode and many inference critical fixes, and quic keep alive (#566)
michaelneale
added a commit
that referenced
this pull request
May 21, 2026
* ci(sdk-smoke): install lld in macOS swift smoke job The Swift SDK smoke job has been failing on macOS runners with: clang: error: invalid linker name in argument '-fuse-ld=/opt/homebrew/bin/ld64.lld' The transitive cargo build inside `sdk/swift/scripts/generate-swift-bindings.sh` runs in a temp dir and ends up invoking `cc` with `-fuse-ld=/opt/homebrew/bin/ld64.lld`. Apple clang accepts that flag only when the linker binary actually exists on disk; the job only installed `jq` so the link step failed. The `macos_targets` job already does `brew install ... lld` for the same reason. Install it in the swift smoke lane too. Reproduced on multiple branches (PRs #566, #609); not specific to any one change. * ci(sdk-smoke): re-enable swift smoke gate now that lld install is fixed This PR installs lld in the macOS swift smoke job, which was the root cause of the linker failure that prompted the temporary 'if: false &&' bypass in 1b1aaf4. Re-enable the normal gate. * ci(compute-changes): route sdk-smoke.yml edits into sdk_smoke_required A PR that only edits .github/workflows/sdk-smoke.yml (e.g. this one) otherwise can't trigger the swift/linux/kotlin SDK smokes it's trying to fix — classic catch-22. Add the reusable workflow file itself to DIRECT_SDK_INPUTS.
michaelneale
added a commit
that referenced
this pull request
May 21, 2026
…ceholders PR #612 review (Copilot) \u2014 two follow-ups. 1. session.rs: malformed tool_calls in caller history ----------------------------------------------------- The earlier shape defaulted missing/empty `id` and `function.name` to `""` and still pushed a `PendingToolCall`. With adversarial or buggy caller history this had two real consequences: * Two malformed calls share `call_id == ""` and become indistinguishable, so any later `role: "tool"` whose `tool_call_id` is also missing matches the first one rather than the intended call. * The `tool_call_response` wire-shape invariant (non-empty function name) is violated downstream. Now both `assistant`-side ingestion and `tool`-side matching skip entries with missing or empty `id`/`tool_call_id` (and `assistant` ingestion also skips entries with empty `function.name`), emitting a `tracing::warn!` so the issue is visible in logs. Well-formed history is unaffected. 2. normalize.rs: doc/code mismatch on tool_arguments ---------------------------------------------------- The `extract_tool_arguments` doc claimed missing/null `arguments` collapses to `Some({})`. The code returns `None` and relies on the downstream `tool_call_response` to substitute `"{}"` when serializing the wire shape. The wire output is correct either way, but the doc misled future maintainers about the invariant. Rewrote the comment to describe the actual invariant ("`None` or an object; `None` means emit `{}` at wire time") so a future refactor doesn't reintroduce the literal-"null" bug. Regression test --------------- `malformed_tool_calls_are_dropped_not_collapsed_to_empty_id` covers four malformed shapes in one history (missing id, missing function.name, empty id, plus an orphaned tool result) and asserts only the one well-formed call survives and the orphaned result doesn't attach. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 87/87 pass (+1 new) * cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass * cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean Live end-to-end validation -------------------------- Built locally and deployed to the 2-node private mesh (M4 Qwen3-8B + Studio MiniMax). All four agent harness shapes from PR #566/#612 exercised the changed code paths: * mini-agent.py model=auto \u2014 2 turns, correct. * mini-agent.py model=mesh \u2014 2 turns, correct. * mini-agent2.py model=mesh \u2014 4 turns multi-file, correct. * Goose model=mesh \u2014 1 tool call, correct. No regressions in MoA fan-out / tool-result reducer routing.
michaelneale
added a commit
that referenced
this pull request
May 21, 2026
* main: docs(AGENTS): add confidence-testing recipe for routing/MoA/gossip changes (#613) ci(sdk-smoke): install lld in macOS swift smoke job (#610) MoA: mesh mode and many inference critical fixes, and quic keep alive (#566) fix(ci): small update for lint rule (#608) mockup: Reserves high-fidelity UI mockup (#560) Add advisory capacity evaluation for model targets (#579) Harden Skippy layer package materialization cache (#583) fix(release): pin Windows CUDA to sccache-compatible version (#606) fix(build-windows): tolerate dead sccache server in CUDA retry path (#604) chore(version): synchronize version bump everywhere (#562) fix(mesh): skip filtered peers in gossip dial loop to unwedge `--auto` (#602) docs(agents): clarify just build vs release-build for serious testing (#599) build: ozempic — slim binary -42 MB / -47 MB (#592)
michaelneale
added a commit
that referenced
this pull request
May 21, 2026
…tion, dedup race, dead code) (#612) * docs(README): mark `model: \"mesh\"` MoA as experimental The MoA gateway is new and still being tuned (routing heuristics, error shapes, tuning knobs). Flag it explicitly in the README so readers do not assume the surface is stable. * fix(moa): close panic surface in worker output normalization PR #566 review (Copilot): the MoA crate had several latent panics around untrusted parsed responses. This commit closes them in one pass and adds regression tests for each. Changes ------- normalize.rs * Single sanitize pass in `normalize_worker_output` runs on the result of *every* parse strategy (JSON, KV, heuristic), not just the heuristic path. The previous shape returned early on JSON/KV success and let non-finite confidences leak into the arbiter where `partial_cmp/total_cmp` could panic on `.unwrap()`. The new helper `sanitize_worker_output` clamps NaN/Inf confidence to 0.5 and collapses non-object `tool_arguments` (Null, primitives, arrays) to `Some({})`. * New `extract_tool_arguments` helper replaces two dead `obj.get("arguments").cloned().or_else(\u2026)` chains in `try_json_parse`. The `or_else` branch was unreachable because `.cloned()` on `Some(Value::String(\u2026))` is already `Some`, so string-encoded JSON arguments leaked through unparsed. `extract_tool_arguments` now explicitly branches on String vs Object vs Null and parses the inner JSON when needed. backend.rs * `parse_retry_after` switched from `to_lowercase()` (Unicode-aware, can change UTF-8 byte length) to `to_ascii_lowercase()` (1:1 byte mapping). The earlier shape sliced the original string using the offset from the lowercased one, which could land mid-codepoint and panic for non-ASCII inputs before the marker. * `extract_text_from_response` switched from direct-indexing `resp["choices"][0]["message"]` to `.pointer()` chaining, returning a structured `Err("malformed response: \u2026")` on missing fields instead of silently producing empty content or panicking downstream. lib.rs * `best_answer` now uses `total_cmp` instead of `partial_cmp(\u2026).unwrap()`. `total_cmp` is total over all f32 (NaN/ Inf included), so even if a future caller bypasses `normalize_worker_output`, this site is panic-free. * `tool_call_response` now explicitly handles every input shape callers can construct: object \u2192 serialize, Null \u2192 `"{}"`, primitive / array \u2192 `"{}"`, validated JSON string \u2192 pass through, invalid string \u2192 `"{}"`. Previously `Value::Null` serialized to the literal four-char string "null", which downstream OpenAI tool-call consumers reject. Tests ----- 13 new tests (now 83/83 pass for the crate): * normalize.rs: kv_path_clamps_nan_confidence, kv_path_clamps_inf_confidence, json_string_encoded_arguments_are_parsed_to_object, null_tool_arguments_become_none, primitive_tool_arguments_collapse_to_empty_object. * backend.rs: parse_retry_after_handles_non_ascii_prefix_without_panic, extract_text_returns_err_on_missing_choices, extract_text_returns_err_on_empty_choices. * lib.rs (new `response_builder_tests` mod): best_answer_does_not_panic_on_nan_confidence, tool_call_response_emits_object_args_for_null, tool_call_response_emits_object_args_for_primitive, tool_call_response_passes_through_string_form_when_valid, tool_call_response_rejects_invalid_string_form. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 83/83 pass * cargo check -p mesh-llm: clean * cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): propagate failure signals to HTTP status, SSE, and tool-result reducer PR #566 review (Copilot) \u2014 three related fixes that all touch how MoA failures reach the caller. 1. HTTP status follows the body's failure signal, not TurnKind ------------------------------------------------------------- `write_moa_response` previously only used HTTP 502 when `TurnKind == Failed`. The tool-result reducer path (`handle_tool_result`) can return `error_response(\u2026)` with `TurnKind::ToolResult` when every reducer candidate fails \u2014 that returned HTTP 200 with an in-band error body, so dumb clients that only check the status code saw a "success". New helper `is_moa_failure_body(body)` recognises the two canonical failure signals MoA emits: a top-level `error` field, and / or `choices[0].finish_reason == "error"`. Status decision now uses this helper, so *every* error-shaped MoA response surfaces as 502 regardless of which sub-flow produced it. 2. SSE adapter propagates the original finish_reason ---------------------------------------------------- `send_moa_as_sse` used to hard-code `finish_reason: "stop"` (or `"tool_calls"` if any tool_calls were present). SSE clients keyed on `finish_reason` (Goose, OpenAI SDKs) therefore saw MoA failures as successful completions. Now the adapter reads `choices[0].finish_reason` from the response body and propagates it ("error" wins over the tool_calls heuristic), and when `is_moa_failure_body` is true it emits an explicit error chunk (`{ object: chat.completion.chunk, choices: [], error: \u2026 }`) before the final finish_reason chunk so SSE clients that scan deltas for an `error` field see the failure too. 3. Tool-result reducer emits tool_calls whenever tool_name is set ----------------------------------------------------------------- Both the tool-result path (`handle_tool_result`) and the fanout/arbiter path (`resolve_decision`) had the same bug: a `ToolProposal` from the reducer only became a real `tool_calls` reply when *both* `tool_name` AND `tool_arguments` were present. With `tool_arguments` missing, both paths silently fell back to a `chat_response` carrying the reducer's prose \u2014 which agent harnesses (Goose, OpenCode) ignore because they only act on `tool_calls`. Now both paths emit `tool_calls` whenever `tool_name` is set, and `tool_call_response` already collapses missing / non-object arguments to `"{}"`. Behaviour is consistent across paths and agent harnesses no longer lose tool calls to reducer prose. Tests ----- Four new `is_moa_failure_body` unit tests in moa_gateway.rs: top-level error, finish_reason=error, success body, tool_calls body. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 83/83 pass * cargo test -p mesh-llm-host-runtime --lib: 1446/1446 pass * cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): group aliases by canonical base before resolving backend PR #566 review (Copilot, item #10): the worker pool builder committed to a single alias per canonical model *before* trying to resolve a backend. Two real failure modes: 1. Stale-peer drop. The shortest alias is advertised only by a peer that drops between gossip refresh and orchestration. The peer is gone, `hosts_for_model(alias)` returns empty, the model is silently removed from the worker pool, and longer-form aliases for the same canonical model from still-reachable peers are rejected as duplicates. 2. Forced QUIC hop. The local node advertises a longer convention (e.g. `unsloth/Qwen3-8B-GGUF:Q4_K_M`) while a peer advertises a shorter variant (e.g. `Qwen3-8B-Q4_K_M`). The shortest-name rule picks the peer alias, `add_worker_backend` looks for a local port under that specific string in `targets`, finds nothing, and forces a QUIC tunnel even though the model is locally served. Fix: group all advertised aliases by canonical base first, then within each group sort so the most likely optimization wins first try (locally-served alias before remote, then shortest first as a tiebreaker). The resolver walks the group in order and takes the first alias that produces a backend, so an unreachable preferred alias falls back to a reachable longer one instead of dropping the model. Refactor extracts a small `resolve_one_worker_from_aliases` helper to keep `build_moa_config` under the cognitive-complexity limit. Tests (4 new) in moa_gateway.rs: * group_aliases_keeps_all_aliases_per_canonical_base * group_aliases_prefers_locally_served_alias_even_when_longer * group_aliases_falls_back_to_shortest_when_no_local * group_aliases_distinct_models_stay_in_separate_groups Validation ---------- * cargo test -p mesh-llm-host-runtime --lib moa_gateway: 13/13 pass * cargo test -p mesh-llm-host-runtime --lib: 1450/1450 pass * cargo clippy -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): strip dead Session continuation/running-summary machinery PR #566 review (Copilot, item #11): `Session::classify_turn` used a `self.turns` counter to decide Fresh vs Continuation, but the gateway constructs a fresh `Session` per inbound request and never invokes `record_assistant_response` / `record_turn_outcome` in production. As a result `turns` was always 1, `Continuation` never fired, and the entire deterministic running-summary feature (`accepted_facts`, `AcceptedFact`, `record_turn_outcome`, `rebuild_summary`, `running_summary` accessor) silently never executed. `pack_fast` had a `turn_count() > 1` branch that appended the summary to the worker's system prompt; that branch was dead too. This commit reflects the gateway's actual design: MoA is request-scoped, and the caller (Goose, OpenCode, an SDK) owns the multi-turn loop and sends the full history each request. Continuation context comes from `session.messages()`, not from a gateway-owned summary. Changes ------- session.rs * Drop `TurnType::Continuation` (only `Fresh` and `ToolResult` remain). The header comment documents the request-scoped lifetime. * Drop fields: `turns`, `last_was_tool_call`, `accepted_facts`, `running_summary`. * Drop methods: `record_assistant_response`, `record_turn_outcome`, `rebuild_summary`, `running_summary`, `accepted_facts`, `turn_count`, `has_unprocessed_tool_results`. * Drop struct: `AcceptedFact`. * Rewrite `ingest` to rebuild `pending_tools` from the caller-provided history each call (delta tracking only worked when Sessions were persisted across requests). Both assistant-emitted tool_calls and tool-result messages are picked up; agent harnesses unchanged. * Simplify `classify_turn`: walk history backwards, return `ToolResult` if a `role: "tool"` message appears before any `role: "assistant"`, else `Fresh`. lib.rs * Drop the `Continuation` match arm. context.rs * Drop the `turn_count() > 1` running-summary injection from `pack_fast`. Context comes from `session.messages()`. Tests ----- * tool_result_turn test rewritten to not rely on the removed `record_assistant_response`; verifies the same behaviour with caller-provided history. * No other test changes needed. Net: 132 LoC of dead machinery removed. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 83/83 pass * cargo test -p mesh-mixture-of-agents (incl. integration): all pass * cargo test -p mesh-llm-host-runtime --lib: 1450/1450 pass * cargo check -p mesh-llm: clean * cargo clippy -p mesh-mixture-of-agents --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean * fix(moa): tighten error code, linearize strip_thinking, sanitize header names, drop dead branch PR #566 review (Copilot) \u2014 batch of MEDIUM cleanups. mesh-mixture-of-agents ---------------------- * `error_response` now takes a `code: &str` parameter and the two call sites pass distinct constants: - `MOA_ERR_ALL_WORKERS_FAILED` for the fanout/arbiter failure path. - `MOA_ERR_ALL_REDUCERS_FAILED` for the tool-result reducer path. Previously both paths emitted `code = "all_workers_failed"`, which was misleading for clients branching on `error.code`. * `strip_thinking` rewritten as a single linear pass over the input. The previous shape rebuilt the entire string on every think block (`format!` + `replace` in a loop), which is O(n*k) for long worker outputs with many `<think>` blocks. Three new tests cover the new shape: orphan close-tag handling, fifty-block linear behaviour, and UTF-8 preservation through multibyte content. mesh-llm-host-runtime --------------------- * `network::openai::transport`: header NAMES are now validated against the RFC 7230 tchar grammar via a new `is_valid_header_name` helper. Names that fail the grammar are dropped with a tracing warning rather than written verbatim. CR/LF in header values is still stripped.\n Both `send_json_ok_with_headers` and\n `send_json_with_status_and_headers` route through a shared\n `append_safe_header(\u2026)` so the validation can't be bypassed by a\n future caller. Four new tests cover the validator and the safe\n header writer.\n\n* `network::openai::moa_gateway::send_moa_as_sse` now reuses\n `append_safe_header` so the SSE adapter gets the same name validation\n as the JSON writers.\n\n* `network::openai::ingress::try_intercept_moa`: dropped the\n unreachable `if let Some(_unused_stream) = \u2026 { tracing::error!(\u2026) }`\n branch. `try_handle_moa` self-gates on the model name and the outer\n gate guarantees it matches, so the inner call always returns\n `None` here. Replaced with `let _ = \u2026.await;` and a comment\n explaining the invariant.\n\nValidation\n----------\n* cargo test -p mesh-mixture-of-agents --lib: 86/86 pass (3 new)\n* cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass (4 new)\n* cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime\n --all-targets -- -D warnings: clean\n* cargo fmt --all -- --check: clean * docs(moa): align stale comments and doc claims with current code PR #566 review (Copilot) \u2014 batch of LOW comment/doc corrections so future readers don't trust outdated narrative. * mesh-llm-host-runtime/src/network/openai/moa_gateway.rs: `try_handle_moa` doc said "returns `true` if handled" but the signature is `Option<TcpStream>`. Rewrote the doc to describe the actual contract: `Some(stream)` means "not MoA, fall through", `None` means "MoA consumed the stream and responded; do not respond again". * model-ref/src/lib.rs: comment claimed the function "emits a lowercase selector", but the implementation slices from the\n original (case-preserving) stem and lowercasing is only used for\n marker-position lookup. Reworded so the comment matches the code.\n\n* skippy-cache/src/resident/prefix.rs: regression-test comment\n referenced a "4*min_tokens floor" \u2014 that derivation was replaced by\n a hard `MIN_CTX_FOR_CELL_CAP = 8192` threshold in a follow-up\n commit. Updated to describe the floor that actually ships.\n\n* mesh-llm-host-runtime/src/inference/skippy/family_policy.rs:\n inline comment said the 16-entry cap "does not fully eliminate"\n unified-KV starvation and that the "proper fix is out of scope".\n The proper fix (token-based budget via `max_resident_tokens` in\n `ResidentCacheConfig::from_stage`) shipped in PR #566. Updated to\n describe entry-count as the coarse lever and `max_resident_tokens`\n as the complementary fine-grained budget.\n\n* docs/design/MOA_GATEWAY.md:\n - Role assignment section claimed "the largest of the big tier gets\n Strong". The impl only partitions small vs big, no\n within-tier sort by parameter count. Reworded to describe the\n actual heuristic and call out the future option.\n - Test plan said "29 unit tests" with a stale per-area breakdown.\n Updated to 86 with current coverage areas, and added a note to\n bump the number when adding tests.\n\nValidation\n----------\n* cargo test -p mesh-mixture-of-agents --lib: 86/86 pass\n* cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass\n* cargo test -p skippy-cache --lib: 13/13 pass\n* cargo test -p model-ref --lib: 10/10 pass\n* cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime\n -p skippy-cache -p model-ref --all-targets -- -D warnings: clean\n* cargo fmt --all -- --check: clean * fix(moa): drop malformed tool_calls instead of inserting empty-id placeholders PR #612 review (Copilot) \u2014 two follow-ups. 1. session.rs: malformed tool_calls in caller history ----------------------------------------------------- The earlier shape defaulted missing/empty `id` and `function.name` to `""` and still pushed a `PendingToolCall`. With adversarial or buggy caller history this had two real consequences: * Two malformed calls share `call_id == ""` and become indistinguishable, so any later `role: "tool"` whose `tool_call_id` is also missing matches the first one rather than the intended call. * The `tool_call_response` wire-shape invariant (non-empty function name) is violated downstream. Now both `assistant`-side ingestion and `tool`-side matching skip entries with missing or empty `id`/`tool_call_id` (and `assistant` ingestion also skips entries with empty `function.name`), emitting a `tracing::warn!` so the issue is visible in logs. Well-formed history is unaffected. 2. normalize.rs: doc/code mismatch on tool_arguments ---------------------------------------------------- The `extract_tool_arguments` doc claimed missing/null `arguments` collapses to `Some({})`. The code returns `None` and relies on the downstream `tool_call_response` to substitute `"{}"` when serializing the wire shape. The wire output is correct either way, but the doc misled future maintainers about the invariant. Rewrote the comment to describe the actual invariant ("`None` or an object; `None` means emit `{}` at wire time") so a future refactor doesn't reintroduce the literal-"null" bug. Regression test --------------- `malformed_tool_calls_are_dropped_not_collapsed_to_empty_id` covers four malformed shapes in one history (missing id, missing function.name, empty id, plus an orphaned tool result) and asserts only the one well-formed call survives and the orphaned result doesn't attach. Validation ---------- * cargo test -p mesh-mixture-of-agents --lib: 87/87 pass (+1 new) * cargo test -p mesh-llm-host-runtime --lib: 1454/1454 pass * cargo clippy -p mesh-mixture-of-agents -p mesh-llm-host-runtime --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean Live end-to-end validation -------------------------- Built locally and deployed to the 2-node private mesh (M4 Qwen3-8B + Studio MiniMax). All four agent harness shapes from PR #566/#612 exercised the changed code paths: * mini-agent.py model=auto \u2014 2 turns, correct. * mini-agent.py model=mesh \u2014 2 turns, correct. * mini-agent2.py model=mesh \u2014 4 turns multi-file, correct. * Goose model=mesh \u2014 1 tool call, correct. No regressions in MoA fan-out / tool-result reducer routing. * fix(moa): route streaming failures to HTTP 502 (drop in-band SSE error) PR #612 review feedback (Nick) \u2014 two related findings on the streaming failure path. 1. Streaming MoA failures returned HTTP 200 ------------------------------------------- `write_moa_response` previously routed all streaming responses through `send_moa_as_sse`, which always emits `HTTP/1.1 200 OK` because SSE clients expect 200 before they start parsing the event stream. That meant `stream: true` MoA failures (`all_workers_failed`, `all_reducers_failed`, reducer hedge exhaustion) arrived as a 200 SSE that *happened* to carry `finish_reason: "error"` \u2014 dumb HTTP clients saw a "successful" stream and didn't realise inference had failed. The body is fully available before we decide how to write it, so we can collapse failure-shaped streaming responses to a non-streaming HTTP 502 JSON response with the structured error body. This matches the OpenAI API shape (failures on streaming endpoints come back as a single non-streaming JSON error response with the right HTTP status) and means streaming and non-streaming MoA failures are now consistent at the HTTP layer. 2. Error-only SSE chunk could break OpenAI-shape clients -------------------------------------------------------- The previous error chunk emitted on failure was `{ object: chat.completion.chunk, choices: [], error: \u2026 }`. Many OpenAI client SDKs assume every `chat.completion.chunk` has `choices[0]` and index blindly into it, so an empty `choices: []` array crashes those clients with an index error. With finding #1's routing change, failure-shaped bodies never reach `send_moa_as_sse` anymore, so the error-chunk emission is dead code. Drop it. `send_moa_as_sse` now does one clear thing: emit a single delta chunk + a `finish_reason: "stop" | "tool_calls"` stop chunk, both with a real `choices[0]`. A `debug_assert!` pins the invariant in tests. Tests ----- Four new unit tests covering the four corners of the routing decision: * `streaming_success_routes_to_sse` * `streaming_failure_routes_to_json_502_not_sse` * `non_streaming_success_routes_to_json_200` * `non_streaming_failure_routes_to_json_502` Validation ---------- * cargo test -p mesh-llm-host-runtime --lib: 1458/1458 pass (4 new) * cargo test -p mesh-mixture-of-agents --lib: 87/87 pass * cargo clippy ... --all-targets -- -D warnings: clean * cargo fmt --all -- --check: clean Live end-to-end validation on 2-node mesh (M4 + Studio MiniMax): * Non-streaming MoA happy path: HTTP 200 + structured response. * Streaming MoA happy path: HTTP 200 + `text/event-stream` + `[DONE]`. * mini-agent.py model=mesh: 2 turns, correct. * mini-agent2.py model=mesh: 4 turns multi-file, correct. * Goose model=mesh: 1 tool call, correct. The failure path is exercised by unit tests; production-triggering streaming failures would require fault injection beyond what the harness covers, but the routing logic and the SSE invariant are locked down at the code level.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Mixture-of-Agents (MoA) as a mesh-native routing mode. A client requests
model: "mesh"; the gateway fans out in parallel to N heterogeneous workers across the mesh, a deterministic arbiter returns the first consensus answer, and a hedged reducer ladder produces an arbitrated answer when workers disagree. Same OpenAI API surface as a normal chat completion.WHY: on a wan mesh where RTT (round trip) is too high to practically do pipeline splits, we can "fall back" to having many more nodes run distinct models, and use this MoA approach to improve quality of inference without being bound to accumulated latency (and it also can cope with some nodes being intermittent). This is complementary for mesh and would want to use it (if it can work!) in cases where we cannot do a split! but lets us make use effectively of vram.
Works from any node
The MoA gateway runs wherever the request lands — on a serving host, a standby, or a pure
--client --autonode with zero local models. The worker pool is built from gossip: every model advertised by any peer is a candidate. On a host node, locally-served models are wired directly to their skippy port. On a client node, all backends are remote QUIC tunnels.Live-verified end-to-end
From a
--client --autonode on a laptop, joined to the public mesh (4 reachable models, all remote):Plain chat —
x-moa-workers: 2,early-exit, 1.3s, clean answer.Goose agentic loop with developer extension — fan-out, tool proposal, shell exec, tool-result turn, final answer. Server log of the live run:
Tool-result turn went through reducer fallback with sole-survivor recovery; goose printed
DONEand exited 0.Per-turn observability via response headers
x-moa-*headers (ignored by normal OpenAI clients) expose what happened:x-moa-elapsed-msx-moa-turnfanout/early-exit/tool-result/failedx-moa-workersx-moa-workers-okx-moa-reducerfalse)x-moa-reducer-attemptsArchitecture
New crate
crates/mesh-mixture-of-agents(standalone, no host-runtime dep):lib.rsGatewayConfig,TurnResult, response buildersbackend.rsModelBackendtrait, HTTP backend, retry-after parsingfanout.rsreducer.rsattemptson both Ok and Errtool_guard.rstoolsarbiter.rsnormalize.rssession.rscontext.rsworker.rsstrip_thinking+truncate_charsHost-runtime integration in
crates/mesh-llm-host-runtime/src/network/openai/moa_gateway.rs. Single implementation called from bothapi_proxy(host path) andhandle_mesh_request(client/passive path), so MoA behaviour is identical regardless of which path the request takes.What's new since the last review pass
Mesh-wide orchestration from any node
build_moa_confignow readsNode::models_being_served()(mesh-wide union of local + every peer's advertised models). Stopped depending on the local routing table for the worker list.model: "mesh"intercept added tohandle_mesh_requestso client-mode nodes orchestrate locally instead of forwarding verbatim to a random peer.Reducer-attempts accounting
hedged_reducer_callnow returnsResult<HedgedReducerOk, HedgedReducerErr>whereErrcarriesattemptstoo. Previously the all-fail path reportedattempts=0regardless of how many candidates actually ran, producing nonsense like "Reducer failed (tried 0): remote timeout after 15s". Surfaced by the live goose test.Code quality from Copilot review
moa::truncate_chars) replaces 2 panicking byte-slice sites. New tests pin char-boundary handling for both 2- and 3-byte sequences.x-moa-*header values.("unknown", 0)defensive fallback removed fromreducer_candidates— was masking real "no candidates" bugs by silently callingbackend_index=0with a bogus model name.strip_thinkingcopies consolidated onto one canonical implementation.MoA crate refactored (earlier in this branch)
lib.rs1267 → 454 LoC; split intobackend/reducer/fanout/tool_guard.context.rstests pin role-shaped packing contract.mesh-mixture-of-agents.autorouting is now tok/s-awareIndependent of MoA but landed in the same branch because it surfaced during public-mesh testing.
pick_model_classifiedwas picking uniformly inside a tier despiteRoutingMetrics.avg_tokens_per_secondbeing recorded all along. Now:RoutingCandidatestruct carriestps_hint+throughput_samples; weighted pick withweight = tps.clamp(5, 100)when samples ≥ 3, neutral 25 otherwise; 15% exploration probability prevents lock-in. 5 new behaviour tests; all 17 router tests pass.Closes #534.