Repository navigation
[kimi-k3] HiCache: size each rank's host pool from the full host headroom (upstream #38157) - #35
Merged
Venkat2811 merged 1 commit intoOct 27, 2026
Conversation
… headroom // ranks (upstream sgl-project#38157) Layer 2 of the Kimi-K3 nightly line on upstream main 5884e4f (the lmsysorg/sglang:nightly-dev-cu13-20261027-5884e4fe image). host_memory_budget_bytes() outside a budget scope returned free // ranks_per_host(), so on an 8-GPU host a rank that asks after its peers have pinned their pools divides a smaller headroom by 8 again and is refused although every pool fits. The guard now checks a pool against the whole remaining headroom; the callers that size a pool keep this rank's share through host_memory_budget_bytes_per_rank(). Source: 894d2ed on the kimi-k3-nightly-3c560089 line (the change of fork PR #14 on the v0.5.22 line), cherry-picked unchanged; the test hunk applies at a 4-line offset. Upstream sgl-project#38157 does the same job a different way and is still open. Callers added upstream since 3c56008 were checked: the MiniMax-M3.1 NVFP4 host pool (pool_host/minimax_m3_nvfp4.py) and its HiCache guard in hybrid_pool_assembler.py check an explicitly sized pool, so they take the whole headroom like the other host pools; no new caller sizes a pool. Tests (CPU): test_mem_pool_host.py with this commit's version on 5884e4f: 2 failed, the two cases that pinned the split (the budget came back as the headroom divided by 4 and by 8 ranks), 28 passed; with this commit 30 passed. Fork PR (v0.5.22 line): #14 Upstream: sgl-project#38157 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Venkat2811
force-pushed
the
venkat/n5884e4fe-02-host-budget
branch
from
October 27, 2026 15:27
413b52d to
e4aecc2
Compare
Venkat2811
changed the base branch from
venkat/n5884e4fe-01-pd-early-send-hicache
to
kimi-k3-nightly-5884e4fe
October 27, 2026 15:27
Venkat2811
marked this pull request as ready for review
October 27, 2026 15:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 2 of the Kimi-K3 serving line on upstream main 5884e4fec6 (base branch
kimi-k3-nightly-5884e4fe, thenightly-dev-cu13-20261027-5884e4feimage). Stacked: this PR's base is the previous layer's branch.What: each rank's HiCache host pool is checked against the whole remaining host headroom instead of headroom // ranks; auto-sizing keeps the per-rank share. Commit 413b52da44 = 894d2ed8b5 (the 3c56008 line; fork PR #14 on v0.5.22) unchanged.
Why: on an 8-GPU host a rank that sizes its pool after its peers pinned theirs divided an already smaller headroom by 8 and was refused although every pool fits. Upstream #38157 (same job, other approach) is still open. Callers added upstream since 3c56008 (MiniMax-M3.1 NVFP4 host pool and its guard) are explicit-size checks and fit this rule.
Tests (CPU):
test_mem_pool_host.pywith the commit's version on the parent: 2 failed (budget split by 4 and 8 ranks), 28 passed; with the commit: 30 passed.Risk: low; unchanged from the line in service since 20 Oct. Trade-off: the guard checks one pool against the whole headroom, so concurrent first checks on 8 ranks can over-commit; size
--hicache-sizex ranks within the container limit (as the K3 recipes do).🤖 Generated with Claude Code
CI States
Latest PR Test (Base): 🚫 Run #40531230677
Latest PR Test (Extra): 🚫 Run #40531230412
Latest PR Test (AMD ROCm 10): 🚫 Run #40531230577