Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
00b8fc5
feat: MoA gateway — stateful mixture-of-agents with tool arbitration
michaelneale May 12, 2026
4c527d5
feat: multi-turn context efficiency + mesh endpoint discovery
michaelneale May 12, 2026
54da3b7
fix: increase running summary budget to ~2k tokens
michaelneale May 12, 2026
d0d0c89
feat: agentic workload test + improved prose tool detection
michaelneale May 12, 2026
d1f83ea
fix: mesh-tested agentic flow — local reducer, model dedup, KV parse fix
michaelneale May 12, 2026
36f12b8
moa: integrate into mesh proxy, SSE streaming, Goose support
michaelneale May 13, 2026
2663f20
fix: add moa-gateway to Docker builds
michaelneale May 13, 2026
655c0d4
moa: early-exit on worker consensus
michaelneale May 13, 2026
9e1c311
moa: real context slices, not synthetic envelopes
michaelneale May 13, 2026
432945d
moa: rename to mesh-mixture-of-agents, mesh-native transport, model='…
michaelneale May 13, 2026
a78a3ea
fix: update Dockerfiles for moa-gateway → mesh-mixture-of-agents rename
michaelneale May 13, 2026
3f3b05d
moa: remove 'mesh' from /v1/models list
michaelneale May 13, 2026
8d0597d
fix: clippy warnings in mesh-mixture-of-agents
michaelneale May 13, 2026
fb16177
fix: clippy unnecessary_lazy_evaluations in ingress build_moa_config
michaelneale May 13, 2026
b40d417
docs: update MoA design doc with current architecture and test plan
michaelneale May 13, 2026
91d331a
moa: fix tool call arguments lost in arbitration
michaelneale May 13, 2026
65fb7c6
moa: worker diversity sampling, 429 retry, faster timeouts, sole-surv…
michaelneale May 14, 2026
64084a3
Fix MoA tool result handling and NaN confidence (PR review feedback)
michaelneale May 16, 2026
ad10c51
ci: align WORKSPACE_MEMBERS with current workspace
michaelneale May 17, 2026
1d670cf
moa: size-aware role assignment + hallucinated tool name filtering
michaelneale May 17, 2026
deaf9f4
moa: reducer candidate fallback on 5xx / timeout
michaelneale May 17, 2026
6359534
ci: fix workspace member drift — keep mesh-llm-client package name, a…
michaelneale May 17, 2026
e537cb4
moa: support model:"mesh" on client/standby nodes via forward-to-host
michaelneale May 17, 2026
ce666be
moa: fix clippy lints surfaced by CI
michaelneale May 17, 2026
b0f5d6a
moa: hedge reducer candidates instead of sequential fallback
michaelneale May 18, 2026
c6d5504
evals: add bench-moa.sh — quick wall-clock benchmark for model:"mesh"
michaelneale May 18, 2026
5cb2c6d
Merge remote-tracking branch 'origin/main' into micn/moa
michaelneale May 18, 2026
20a6df9
Merge branch 'main' into micn/moa
michaelneale May 18, 2026
cd29f52
Merge branch 'main' into micn/moa
michaelneale May 18, 2026
14e7437
moa: split lib.rs into backend / reducer / fanout modules
michaelneale May 18, 2026
6956595
moa: cover role-shaped context packing with tests
michaelneale May 18, 2026
87fe3e2
docs: surface model:"mesh" (MoA) in README
michaelneale May 18, 2026
635307c
moa: extract tool_guard module, drop dead Endpoint/discover_endpoints
michaelneale May 18, 2026
ebb167d
docs(moa): drop the speculative hook-integration line
michaelneale May 18, 2026
2524840
router: weight 'auto' selection by locally observed tok/s
michaelneale May 18, 2026
5d86ac4
docs(moa): clarify topology — N workers + serial 2-call shape
michaelneale May 18, 2026
8c59d86
moa: emit x-moa-* observability headers from gateway
michaelneale May 18, 2026
f40bc9b
bench-moa: aggregate gateway path, reducer, and hedge stats
michaelneale May 18, 2026
fd8f450
evals: remove bench-moa.sh — never actually run
michaelneale May 18, 2026
505573a
moa: orchestrate from any node, build worker pool from mesh-wide gossip
michaelneale May 18, 2026
9d47346
moa: carry attempts on reducer-failure path; copilot review fixes
michaelneale May 18, 2026
acddf99
moa: address remaining Copilot review items
michaelneale May 18, 2026
bcc4e99
Merge branch 'main' into micn/moa
michaelneale May 19, 2026
80b63f7
moa: tighten early-exit content check with subset+negation rule
michaelneale May 19, 2026
a049c35
docs(moa): add pressure-test research plan to MOA_GATEWAY
michaelneale May 19, 2026
8b59e52
docs(moa): reframe pressure test around equal-VRAM split-vs-mix on mesh
michaelneale May 19, 2026
a3dc269
docs(moa): reframe as operating-envelope, not benchmark fight
michaelneale May 19, 2026
2f8dcb5
merge main into micn/moa: resolve ingress.rs + tighten arbiter rule
michaelneale May 19, 2026
c131edf
Merge branch 'main' into micn/moa
michaelneale May 20, 2026
8fd08a5
docs(moa): promote 'why MoA exists' to top of design doc
michaelneale May 20, 2026
5169d12
fix(moa): signal all-workers-fail as a proper error response
michaelneale May 20, 2026
a396ab1
fix(moa): account for aborted workers in worker_summaries
michaelneale May 20, 2026
000ae50
fix(moa): route tool-result follow-ups to reducer, not fan-out
michaelneale May 20, 2026
64c4e0e
fix(moa): recognise OpenAI-shape inline tool JSON in worker output
michaelneale May 20, 2026
f3355bf
fix(moa): /v1/models advertises quant-suffix IDs that route back
michaelneale May 20, 2026
d527965
fix(moa): bump worker/reducer timeouts to 60s for agent-scale prompts
michaelneale May 20, 2026
f5cf4b8
fix(mesh): enable QUIC keep-alive on mesh transport (was: connections…
michaelneale May 20, 2026
1b90121
fix(planner): cap auto lane count to llama-server's 4-lane unified-KV…
michaelneale May 20, 2026
b117f4d
docs(moa): report on micn/moa branch's critical fixes beyond MoA itself
michaelneale May 20, 2026
165793b
docs(moa): expand branch report — include throughput-weighted router …
michaelneale May 20, 2026
32061e8
fix(family_policy): tighten prefix-cache budget for unified-KV serving
michaelneale May 20, 2026
7271fb5
docs(moa): document prefix-cache budget fix + remaining steady-state …
michaelneale May 20, 2026
8cb6fe4
fix(skippy-cache): cap resident prefix cache by KV cell count, not ju…
michaelneale May 20, 2026
4eb513f
docs(moa): update fix D \u2014 KV exhaustion fully fixed by cell-budg…
michaelneale May 20, 2026
809f4b0
fix(skippy-cache): disable resident token cap for tiny contexts
michaelneale May 20, 2026
30f55d5
fix(skippy-cache): use hard ctx-size floor for resident token cap
michaelneale May 21, 2026
3a3ce00
ci: disable swift_sdk_smoke on this branch (infra flake)
michaelneale May 21, 2026
161c37f
merge main into micn/moa
michaelneale May 21, 2026
f4a2c11
chore(moa): drop stale workspace exclude and unused deps
michaelneale May 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .github/workflows/pr_builds.yml
Original file line number Diff line number Diff line change
Expand Up @@ -603,7 +603,12 @@ jobs:

swift_sdk_smoke:
needs: [changes, macos_targets]
if: ${{ needs.macos_targets.result == 'success' && needs.changes.outputs.macos_inference_artifact_required == 'true' && needs.changes.outputs.sdk_smoke_required == 'true' && needs.changes.outputs.docs_only != 'true' }}
# Disabled on this branch only: macOS runners are rejecting
# `-fuse-ld=/opt/homebrew/bin/ld64.lld` with
# `clang: error: invalid linker name`. Reproduces on unrelated
# branches too (see PR #609). To be fixed on main and re-enabled
# there; do not remove this gate without coordinating.
if: ${{ false && needs.macos_targets.result == 'success' && needs.changes.outputs.macos_inference_artifact_required == 'true' && needs.changes.outputs.sdk_smoke_required == 'true' && needs.changes.outputs.docs_only != 'true' }}
Comment on lines +606 to +611
Comment on lines +606 to +611
uses: ./.github/workflows/sdk-smoke.yml
with:
sdk_kind: swift
Expand Down
13 changes: 13 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -435,6 +435,19 @@ If an instance is wedged badly enough that the scoped stop path cannot reach it,
pkill -f mesh-llm
```

## Running mesh-llm in the Background (for Testing)

When running `mesh-llm serve` from an agent for testing, the process is non-interactive — it just runs. There is no interactive prompt or TUI to worry about. Use standard backgrounding:

```bash
bash -c './target/debug/mesh-llm serve --model "..." --auto > /tmp/mesh.log 2>&1 & disown; echo "PID=$!"'
```

- **Do not use `--headless`** — it disables the web UI but does not change process behavior. The name is misleading and does not help with backgrounding.
- The mesh process writes TUI-formatted output to stderr which looks like errors but is normal.
- Wait for models to appear via polling `curl -s http://localhost:9337/v1/models` before sending requests.
- Kill with `pkill -f "target/debug/mesh-llm"` or `pkill -f mesh-llm`.

## Deploy Checklist — MANDATORY

**Every deploy to test machines MUST follow this checklist.**
Expand Down
13 changes: 13 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ members = [
"crates/llama-spec-bench",
"crates/skippy-bench",
"crates/skippy-prompt",
"crates/mesh-mixture-of-agents",
"tools/xtask",
]
default-members = [
Expand Down
21 changes: 21 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ mesh-llm serve --auto --headless
| Run an API-only client | `mesh-llm client --auto` | [docs/MESHES.md](docs/MESHES.md) |
| Run a big model with splits | `mesh-llm serve --model hf://meshllm/<repo>@<rev> --split` | [docs/SKIPPY_SPLITS.md](docs/SKIPPY_SPLITS.md) |
| Attach a Flash-MoE SSD backend | `mesh-llm serve` with `[[plugin]] name = "flash-moe"` | [docs/plugins/flash-moe.md](docs/plugins/flash-moe.md) |
| Fan out one prompt to every model in the mesh | `curl ... -d '{"model":"mesh", ...}'` | [docs/design/MOA_GATEWAY.md](docs/design/MOA_GATEWAY.md) |
| Use Goose, OpenCode, Claude Code, or Pi | `mesh-llm goose`, `mesh-llm opencode`, `mesh-llm claude`, `mesh-llm pi` | [docs/AGENTS.md](docs/AGENTS.md) |
| Build or contribute | `just build` | [CONTRIBUTING.md](CONTRIBUTING.md) |

Expand All @@ -83,6 +84,26 @@ mesh-llm serve --auto --headless
For a deeper operator guide, see [docs/USAGE.md](docs/USAGE.md). For every CLI
command and switch, see [docs/CLI.md](docs/CLI.md).

## Mixture-of-Agents (`model: "mesh"`)

Send a request with `"model": "mesh"` and the proxy fans it out to every
model available in the mesh in parallel, arbitrates their responses with
deterministic logic, and returns one OpenAI-compatible reply. The arbiter
runs in code (not as another model call) and only escalates to a reducer
LLM on genuine conflict. Tool calls flow through the full pipeline.

```bash
curl http://localhost:9337/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"mesh","messages":[{"role":"user","content":"What is the capital of Japan?"}]}'
```

Requires at least two distinct models in the mesh. See
[docs/design/MOA_GATEWAY.md](docs/design/MOA_GATEWAY.md) for the
architecture, arbitration rules, and tuning knobs.



## Supported model families

Mesh LLM's Skippy runtime tracks llama.cpp family parity with reviewed GGUF
Expand Down
1 change: 1 addition & 0 deletions crates/mesh-llm-host-runtime/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ workspace = true

[dependencies]
ansi-to-tui = "8"
mesh-mixture-of-agents = { path = "../mesh-mixture-of-agents" }
mesh-llm-plugin = { path = "../mesh-llm-plugin" }
mesh-llm-identity = { path = "../mesh-llm-identity" }
mesh-llm-protocol = { path = "../mesh-llm-protocol" }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -255,11 +255,13 @@ async fn fetch_mesh_models(
}
model.clone()
} else {
let available: Vec<(&str, f64, crate::models::ModelCapabilities)> = models
// Pre-startup path: no live routing metrics yet, so candidates
// are scored as cold (uniform weight).
let available: Vec<crate::network::router::RoutingCandidate<'_>> = models
.iter()
.map(|name| {
let caps = crate::models::installed_model_capabilities(name);
(name.as_str(), 0.0, caps)
crate::network::router::RoutingCandidate::unscored(name.as_str(), caps)
})
.collect();
let agentic = crate::network::router::Classification {
Expand Down
139 changes: 125 additions & 14 deletions crates/mesh-llm-host-runtime/src/inference/skippy/family_policy.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,24 @@ impl FamilyPolicy {
max_entries,
} => {
let max_bytes = derive_stage_cache_max_bytes(config)?;
// The family policy's `max_entries` is a generous
// upper bound on cache cardinality. The real ceiling
// is the unified KV cell pool size: each resident
// prefix pins `token_count` cells across `stage_layers`
// in the same `n_ctx` pool the active lanes use. If
// we let the cache fill to `max_entries` it can
// starve the active lanes of cells and surface as
// HTTP 502 `RuntimeError: llama_decode failed`
// (`decode: failed to find a memory slot`).
//
// Cap entries so the cache cannot overcommit the
// pool. See `derive_max_entries_from_kv_cells` below.
let bounded_entries =
derive_max_entries_from_kv_cells(config, min_tokens, max_entries);
Some(StageKvCacheConfig {
mode: StageKvCacheMode::LookupRecord,
payload: payload.as_stage_payload(),
max_entries,
max_entries: bounded_entries,
max_bytes,
min_tokens,
shared_prefix_stride_tokens: 128,
Expand All @@ -73,6 +87,42 @@ impl FamilyPolicy {
}
}

/// Cap the prefix-cache `max_entries` so resident prefixes cannot
/// exhaust the unified KV cell pool.
///
/// Skippy's stage runtime serves with `kv_unified = true` whenever
/// `lane_count > 1` (patch `0034-Add-shared-execution-lanes-to-skippy-ABI.patch`).
/// In unified mode the KV cache is a single pool of `n_ctx` cells
/// shared across all `n_seq_max` sequences. The resident-prefix cache
/// pins prefixes onto dedicated sequence ids in *the same pool*, so
/// every cached entry consumes cells that the active lanes can no
/// longer use. Without a cap, the family default of 128 entries can
/// accumulate enough pinned prefixes to starve the active lanes,
/// surfacing as HTTP 502 `RuntimeError: llama_decode failed`
/// (`decode: failed to find a memory slot`) after a dozen or so
/// agent-style requests.
///
/// Budget: the cache may use at most half the cell pool. Each entry
/// is at least `min_tokens` cells, so `max_entries ≤ n_ctx / (2 *
/// min_tokens)`. The LRU in `ResidentPrefixCache` evicts when this
/// ceiling is hit. The other half of the pool stays available for
/// the active lanes' fresh prompts.
///
/// Never lifted above the family-policy default; never below 1.
fn derive_max_entries_from_kv_cells(
config: &StageConfig,
min_tokens: u64,
family_default: usize,
) -> usize {
if min_tokens == 0 {
return family_default;
}
let n_ctx = u64::from(config.ctx_size.max(1));
let cache_budget_cells = n_ctx / 2;
let kv_capped = (cache_budget_cells / min_tokens) as usize;
kv_capped.clamp(1, family_default)
}

pub(crate) fn family_policy_for_stage_config(config: &StageConfig) -> FamilyPolicy {
[
config.materialized_path.as_deref(),
Expand Down Expand Up @@ -193,7 +243,32 @@ fn resident_kv_policy(activation_wire_dtype: StageWireDType) -> FamilyPolicy {
prefix_cache: FamilyPrefixCachePolicy::Auto {
payload: FamilyPrefixCachePayload::ResidentKv,
min_tokens: 256,
max_entries: 128,
// Was 128. Real-world OpenAI surface workloads (Goose,
// OpenCode, pi) record prefixes that average 1.5–2k
// tokens — much larger than `min_tokens`. 128 entries at
// ~2k tokens each pins ~256k cells, which exceeds even a
// 131k-`n_ctx` model's unified KV pool. The active lanes
// then can't find a slot and the embedded runtime
// returns HTTP 502
// `RuntimeError: llama_decode failed`
// (`decode: failed to find a memory slot`).
//
// 16 entries at ~2k tokens ≈ 32k cells; comfortable
// headroom under any model that gets `kv_unified = true`
// serving (`lane_count > 1`). The LRU in
// `ResidentPrefixCache` evicts older entries as new
// prefixes are recorded, so cache hit rate for the
// recent workload is preserved.
//
// Live observation: even at 16, sustained Goose `auto`
// traffic eventually starves the lanes — see the
// `prefix cache leak under sustained agent traffic`
// section in `docs/design/MOA_BRANCH_REPORT.md`. The
// 16-entry cap pushes the failure point from ~6 to ~16+
// requests but does not fully eliminate it. The proper
// fix is more invasive in `skippy-server` and out of
// scope for PR #566.
Comment on lines +263 to +270
Comment on lines +266 to +270
max_entries: 16,
},
}
}
Expand All @@ -204,7 +279,10 @@ fn kv_recurrent_policy(activation_wire_dtype: StageWireDType) -> FamilyPolicy {
prefix_cache: FamilyPrefixCachePolicy::Auto {
payload: FamilyPrefixCachePayload::KvRecurrent,
min_tokens: 256,
max_entries: 128,
// See `resident_kv_policy` for the rationale; recurrent
// state lanes share the same n_ctx cell pool under
// `kv_unified = true`.
max_entries: 16,
},
}
}
Expand Down Expand Up @@ -282,11 +360,29 @@ fn estimate_stage_cache_max_bytes(config: &StageConfig, meta: &GgufCompactMeta)
let value_bytes_per_token = dtype_bytes(value_elems_per_token, &config.cache_type_v)?;
let bytes_per_token_layer = key_bytes_per_token.checked_add(value_bytes_per_token)?;

bytes_per_token_layer
// The prefix cache shares the same `n_ctx` cell pool the active
// lanes use (skippy patches set `kv_unified = true` whenever
// `lane_count > 1`; see patch 0034). The total native KV memory
// is `bytes_per_token_layer * stage_layers * n_ctx` — NOT
// multiplied by `lane_count` (lanes share, they do not multiply
// the budget). Cap the cache at *half* that total so the other
// half stays free for the lanes' fresh prompts.
//
// The previous code (a) included the lane_count multiplier (so
// budget was 2–4× the actual pool) and (b) didn't reserve any
// pool for active lanes. Under sustained agent-style traffic
// (Goose, OpenCode, pi against `model: auto`) the cache filled
// until it crowded the lanes out and the embedded runtime
// returned HTTP 502 `RuntimeError: llama_decode failed`
// (`decode: failed to find a memory slot`).
let full_pool_bytes = bytes_per_token_layer
.checked_mul(u64::from(stage_layers))?
.checked_mul(u64::from(config.ctx_size.max(1)))?
.checked_mul(u64::from(config.lane_count.max(1)))
.filter(|bytes| *bytes > 0)
.checked_mul(u64::from(config.ctx_size.max(1)))?;
let cache_budget_bytes = full_pool_bytes / 2;
if cache_budget_bytes == 0 {
return None;
}
Some(cache_budget_bytes)
}

fn dtype_bytes(elements: u64, dtype: &str) -> Option<u64> {
Expand Down Expand Up @@ -379,7 +475,7 @@ mod tests {
FamilyPrefixCachePolicy::Auto {
payload: FamilyPrefixCachePayload::ResidentKv,
min_tokens: 256,
max_entries: 128,
max_entries: 16,
}
);
}
Expand Down Expand Up @@ -515,7 +611,7 @@ mod tests {
FamilyPrefixCachePolicy::Auto {
payload: FamilyPrefixCachePayload::ResidentKv,
min_tokens: 256,
max_entries: 128,
max_entries: 16,
},
"{family_id}"
)
Expand All @@ -527,7 +623,7 @@ mod tests {
FamilyPrefixCachePolicy::Auto {
payload: FamilyPrefixCachePayload::KvRecurrent,
min_tokens: 256,
max_entries: 128,
max_entries: 16,
},
"{family_id}"
),
Expand Down Expand Up @@ -564,7 +660,7 @@ mod tests {
FamilyPrefixCachePolicy::Auto {
payload: expected_payload,
min_tokens: 256,
max_entries: 128,
max_entries: 16,
},
"{} ({})",
expected.llama_architecture,
Expand Down Expand Up @@ -621,12 +717,23 @@ mod tests {
}

#[test]
fn stage_cache_cap_tracks_ctx_lanes_layers_and_kv_types() {
fn stage_cache_cap_tracks_ctx_layers_and_kv_types() {
let config = stage_config();

let bytes = estimate_stage_cache_max_bytes(&config, &kv_meta()).unwrap();

assert_eq!(bytes, 12_845_056);
// 4096 bytes/token/layer * 2 stage_layers * 1024 ctx_size /
// 2 (cache may use at most half the unified KV pool).
//
// Crucially does NOT include lane_count: skippy's unified KV
// shares one cell pool across all lanes (`kv_unified = true`
// patch 0034), so the cache budget is independent of lane
// count. The previous formula multiplied by lane_count AND
// didn't reserve any of the pool for active lanes, which
// produced an over-generous cache budget that surfaced as
// `decode: failed to find a memory slot` failures under
// sustained agent traffic.
assert_eq!(bytes, 3_211_264);
}

#[test]
Expand All @@ -637,7 +744,11 @@ mod tests {

let bytes = estimate_stage_cache_max_bytes(&config, &kv_meta()).unwrap();

assert_eq!(bytes, 4_718_592);
// q4_0 packs 32 elements into 18 bytes (= 0.5625 bytes/element
// vs 2.0 for f16). Same `2 stage_layers * 1024 ctx_size / 2`
// and no lane_count multiplier; see
// `stage_cache_cap_tracks_ctx_layers_and_kv_types` for why.
assert_eq!(bytes, 1_179_648);
}

#[test]
Expand Down
Loading
Loading