Add advisory capacity evaluation for model targets - #579
Merged
Conversation
IvGolovach
force-pushed
the
codex/advisory-capacity-evaluator
branch
from
May 18, 2026 05:25
3a389aa to
b76fb8b
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds advisory capacity evaluation to /api/model-targets, surfacing capacity context under derived.capacity_advice without changing ranking or routing behavior.
Changes:
- Adds a new API-local capacity evaluator for model targets.
- Moves runtime model fit headroom helpers into shared runtime capacity code.
- Updates model-target response shaping, docs, and tests for capacity advice.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/design/TESTING.md | Updates API testing expectations for capacity advice. |
| docs/design/DESIGN.md | Documents advisory capacity states and semantics. |
| crates/mesh-llm-host-runtime/src/runtime/mod.rs | Re-exports runtime required-byte helper and updates imports. |
| crates/mesh-llm-host-runtime/src/runtime/local.rs | Uses capacity helper functions from runtime capacity module. |
| crates/mesh-llm-host-runtime/src/runtime/capacity.rs | Hosts shared model fit/headroom helpers. |
| crates/mesh-llm-host-runtime/src/api/tests.rs | Adds management API tests for capacity advice output. |
| crates/mesh-llm-host-runtime/src/api/status.rs | Adds capacity advice payload/status types. |
| crates/mesh-llm-host-runtime/src/api/routes/model_targets.rs | Nests capacity advice under derived. |
| crates/mesh-llm-host-runtime/src/api/model_targets.rs | Passes node/peer capacity inputs into target payload construction. |
| crates/mesh-llm-host-runtime/src/api/model_target_capacity.rs | Implements advisory model-target capacity evaluation. |
| crates/mesh-llm-host-runtime/src/api/mod.rs | Registers the new capacity evaluator module. |
Comment on lines
+126
to
+132
| advice( | ||
| ModelTargetCapacityAdviceState::InsufficientCapacity, | ||
| "capacity_shortfall", | ||
| capacity, | ||
| AdviceDetails { | ||
| required_bytes: Some(required_bytes), | ||
| shortfall_bytes: Some(required_bytes.saturating_sub(comparable_capacity)), |
Comment on lines
+207
to
+213
| fn model_size_hint(query: &str) -> Option<ModelSizeHint> { | ||
| let entries = models::remote_catalog::catalog_entries()?; | ||
| let normalized_query = normalize_match_key(query); | ||
| for entry in entries { | ||
| let mut variants = entry.variants.iter().collect::<Vec<_>>(); | ||
| variants.sort_by(|left, right| left.0.cmp(right.0)); | ||
| for (variant_name, variant) in variants { |
IvGolovach
force-pushed
the
codex/advisory-capacity-evaluator
branch
2 times, most recently
from
May 18, 2026 15:51
35d2590 to
a8aec6f
Compare
Validation * Validation tier: Tier 3 - additive management API response plus shared runtime capacity helper reuse. * git fetch --no-tags origin main:refs/remotes/origin/main codex/advisory-capacity-evaluator:refs/remotes/origin/codex/advisory-capacity-evaluator: PASS * git diff --check: PASS, no output * git diff --cached --check: PASS, no output * cargo fmt --all -- --check: PASS * LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime model_target_capacity --lib: PASS, 2 passed * LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime capacity_advice --lib: PASS, 3 passed * LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal /opt/homebrew/bin/cargo-clippy clippy -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS * LLAMA_STAGE_BUILD_DIR=<repo>/.deps/llama-build/build-stage-abi-metal cargo check -p mesh-llm: PASS * Ledger: not applicable - not required for selected validation tier/change family. * Version: not applicable - no release/version sync required for this non-deploy PR branch. * Not run: just build - not required for selected validation tier; no UI build artifact changed. Rollback * git revert HEAD
IvGolovach
force-pushed
the
codex/advisory-capacity-evaluator
branch
from
May 20, 2026 18:19
a8aec6f to
a40cd32
Compare
michaelneale
added a commit
that referenced
this pull request
May 21, 2026
Brings in #579 advisory capacity, #583 hardened materialization cache, #562 version bump, #606/#604 Windows CUDA build fixes, #608 lint rule, and #560 UI mockup. * origin/main: fix(ci): small update for lint rule (#608) mockup: Reserves high-fidelity UI mockup (#560) Add advisory capacity evaluation for model targets (#579) Harden Skippy layer package materialization cache (#583) fix(release): pin Windows CUDA to sccache-compatible version (#606) fix(build-windows): tolerate dead sccache server in CUDA retry path (#604) chore(version): synchronize version bump everywhere (#562)
michaelneale
added a commit
that referenced
this pull request
May 21, 2026
* main: docs(AGENTS): add confidence-testing recipe for routing/MoA/gossip changes (#613) ci(sdk-smoke): install lld in macOS swift smoke job (#610) MoA: mesh mode and many inference critical fixes, and quic keep alive (#566) fix(ci): small update for lint rule (#608) mockup: Reserves high-fidelity UI mockup (#560) Add advisory capacity evaluation for model targets (#579) Harden Skippy layer package materialization cache (#583) fix(release): pin Windows CUDA to sccache-compatible version (#606) fix(build-windows): tolerate dead sccache server in CUDA retry path (#604) chore(version): synchronize version bump everywhere (#562) fix(mesh): skip filtered peers in gossip dial loop to unwedge `--auto` (#602) docs(agents): clarify just build vs release-build for serious testing (#599) build: ozempic — slim binary -42 MB / -47 MB (#592)
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds an advisory capacity signal to the ranked
/api/model-targetssurface, so operators can see whether a wanted target appears already served, single-node fit, split-capable through aggregate capacity, capacity-constrained, or currently unknown because model size or VRAM information is missing.derived.capacity_adviceto/api/model-targetswhile keeping existing rawsignalsand ranking fields unchanged.Clientrole exclusion separate from worker/host missing VRAM, so the API can explain role vs. capacity uncertainty clearly.ModelTargetsbehavior unchanged.Why
#276 calls for Phase 4 advisory capacity evaluation before automatic load/unload reconciliation. This PR makes the current model-target ranking more explainable without taking runtime action: wanted models remain visible and now carry enough capacity context for operators and later UI work to understand why a target may or may not be immediately actionable.
Architecture
The evaluator is API-local under
mesh-llm-host-runtime/src/apiand consumes existing node role, VRAM, peer, catalog, serving, interest, requested, and demand signals. Runtime fit sizing uses the sharedruntime_model_required_byteshelper so API advice matches the existing startup headroom rule.Compatibility
/api/model-targetsflat-field hiding remains unchanged; new capacity data lives underderived.Validation
git fetch --no-tags origin main:refs/remotes/origin/maingit diff --checkgit diff --cached --checkcargo fmt --all -- --checkLLAMA_STAGE_BUILD_DIR=<local stage ABI build dir> cargo test -p mesh-llm-host-runtime model_targets --lib— PASS, 6 passedLLAMA_STAGE_BUILD_DIR=<local stage ABI build dir> cargo test -p mesh-llm-host-runtime model_target_capacity --lib— PASS, 1 passedLLAMA_STAGE_BUILD_DIR=<local stage ABI build dir> cargo check -p mesh-llm-host-runtime— PASSNot run:
just build— no UI build artifact changed; targeted backend validation covers the API/runtime diff.Rollback
Revert this PR.
Known Residual Risk
split_candidateis deliberately a coarse advisory signal. Real activation still depends on runtime split/package readiness checks before any model is loaded.Related
Related: #276