Skip to content

Cover asymmetric KV cache width pricing - #594

Merged
IvGolovach merged 1 commit into
mainfrom
codex/kv-context-slot-auto-planning
May 21, 2026
Merged

Cover asymmetric KV cache width pricing#594
IvGolovach merged 1 commit into
mainfrom
codex/kv-context-slot-auto-planning

Conversation

@IvGolovach

@IvGolovach IvGolovach commented May 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

This is now a narrow regression-test follow-up after current main absorbed and simplified the runtime auto-planning work from the original branch.

  • Keeps current main runtime planning behavior, including the conservative unified-KV auto slot ceiling from the recent multi-node fixes.
  • Drops the stale RuntimePlanningProfile / public-mesh lane escalation from this branch instead of fighting the newer runtime work.
  • Adds coverage that GGUF KV-cache pricing handles asymmetric key/value widths independently.

Why

The maintainer feedback was to update against current main and re-check the shape after the multi-node runtime changes. After rebasing, the right professional scope was not to reintroduce the older runtime planning profile. The remaining useful piece is the model-artifact regression coverage for asymmetric K/V metadata, which protects the capacity math used by the planner without changing runtime behavior.

Diff scope

  • crates/model-artifact/src/gguf.rs
    • Adds a regression test for independent K/V cache width pricing.

Compatibility

  • Test-only change.
  • No runtime behavior change.
  • No mesh wire protocol, gossip, protobuf, Skippy ABI, or plugin protocol compatibility change.

Validation

Local validation:

  • git fetch --no-tags origin main:refs/remotes/origin/main codex/kv-context-slot-auto-planning:refs/remotes/origin/codex/kv-context-slot-auto-planning - PASS
  • git rebase origin/main - PASS
  • git diff --check origin/main...HEAD - PASS, no output
  • git diff --check - PASS, no output
  • git diff --cached --check - PASS, no output
  • cargo fmt --all -- --check - PASS
  • cargo test -p model-artifact gguf --lib - PASS, 16 passed
  • LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime runtime::context_planning --lib - PASS, 9 passed
  • cargo check -p model-artifact - PASS
  • LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo check -p mesh-llm - PASS
  • /opt/homebrew/bin/cargo-clippy clippy -p model-artifact --all-targets -- -D warnings - PASS

Remote validation:

  • PR Quality Checks - PASS
  • PR Builds - PASS
  • PR Docker Build - PASS

Not run:

  • Live performance smoke - not required after the final diff no longer changes runtime planning behavior. Current main runtime planning remains covered by targeted runtime::context_planning tests and remote CI.

Rollback

Revert this PR.

Related: #359

Comment thread crates/mesh-llm-host-runtime/src/runtime/context_planning.rs Outdated
@IvGolovach
IvGolovach force-pushed the codex/kv-context-slot-auto-planning branch from 4cda985 to b02caf0 Compare May 19, 2026 20:53
@IvGolovach
IvGolovach requested a review from ndizazzo May 19, 2026 21:33
@IvGolovach
IvGolovach force-pushed the codex/kv-context-slot-auto-planning branch from b02caf0 to 4fadfeb Compare May 19, 2026 23:48
@ndizazzo

Copy link
Copy Markdown
Collaborator

@IvGolovach thanks for pushing a fix!

@michaelneale

Copy link
Copy Markdown
Collaborator

@IvGolovach had to make some mega changes to this based on multi node testing I found - might want to update this to main, and check how it performs (I did consult this one but had to simplify it), am still a little confused on how to avoid things overcommitting and falling over, but glad this is looked at

Validation
* Validation tier: Tier 2R - post-review conflict repair narrowed to a model-artifact KV pricing regression after current main absorbed the runtime planner changes and kept the conservative 4-slot auto ceiling.
* git fetch --no-tags origin main:refs/remotes/origin/main codex/kv-context-slot-auto-planning:refs/remotes/origin/codex/kv-context-slot-auto-planning: PASS
* git rebase origin/main: PASS, kept current main runtime planning behavior and dropped stale RuntimePlanningProfile/PublicMesh lane escalation.
* git diff --check origin/main...HEAD: PASS, no output
* git diff --check: PASS, no output
* git diff --cached --check: PASS, no output
* cargo fmt --all -- --check: PASS
* cargo test -p model-artifact gguf --lib: PASS, 16 passed
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo test -p mesh-llm-host-runtime runtime::context_planning --lib: PASS, 9 passed
* cargo check -p model-artifact: PASS
* LLAMA_STAGE_BUILD_DIR=/Users/Funtland/Downloads/mesh-llm/.deps/llama-build/build-stage-abi-metal cargo check -p mesh-llm: PASS
* /opt/homebrew/bin/cargo-clippy clippy -p model-artifact --all-targets -- -D warnings: PASS
* Ledger: not applicable - not required for selected validation tier/change family.
* Version: not applicable - test-only regression coverage; no release/version sync required.
* Not run: live performance smoke - not required after the final diff no longer changes runtime planning behavior; current main's runtime planner remains covered by targeted context_planning tests and remote CI.

Rollback
* git revert HEAD
@IvGolovach
IvGolovach force-pushed the codex/kv-context-slot-auto-planning branch from 4fadfeb to a884363 Compare May 21, 2026 07:02
@IvGolovach IvGolovach changed the title Make auto planning account for KV cache and parallel slots Cover asymmetric KV cache width pricing May 21, 2026
@IvGolovach
IvGolovach merged commit 27d46a6 into main May 21, 2026
18 checks passed
@IvGolovach
IvGolovach deleted the codex/kv-context-slot-auto-planning branch May 21, 2026 14:46
michaelneale added a commit that referenced this pull request May 21, 2026
…able-thinking

* origin/main:
  Cover asymmetric KV cache width pricing (#594)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants