Repository navigation
[HiCache] Size each rank's host pool from the full host headroom, not headroom // ranks (upstream #38157) - #14
Merged
Conversation
… headroom // ranks (upstream sgl-project#38157) The kimi-k3 v0.5.22 prefill/decode line: v0.5.22 plus this one layer. It is 331f644 (the v0.5.21 kimi-k3 line's layer 5b, fork PR #13) rebased onto v0.5.22, with the two conflicts against v0.5.22's pool_host changes resolved. host_memory_budget_bytes() outside a budget scope returns free // ranks_per_host(), where free is the host headroom left when the rank asks. On an 8-GPU pod the ranks arrive at different times, so a rank that checks after its peers have pinned their pools divides a smaller headroom by 8 again and is refused although every pool fits (seen 2026-10-01 on Kimi-K3: "Requesting 24.77 GB but only have 22.74 GB free"). v0.5.22 still does this with an explicit --hicache-size (hicache_auto_size.py opens no budget scope; pool_host/base.py and pool_host/mamba.py check against free // ranks_per_host()). The guard now checks a pool against the whole remaining headroom; the two callers that size a pool (HiCache auto-sizing, the unified page-envelope pool when --hicache-size and --hicache-ratio are 0) keep this rank's share through host_memory_budget_bytes_per_rank(). The unit tests that pinned the split are updated. Not carried from the v0.5.21 line, for the prefill/decode copy (prefill at DCP 1 with HiCache and Mooncake L3, decode without HiCache): sgl-project#41692 to sgl-project#41695 (file L3 and DCP > 1 L3 only) and sgl-project#42039 (in v0.5.22 as sgl-project#42420 and sgl-project#42788). The aggregated copy (Mooncake L3 at DCP 8) stays on verda/kimi-k3-2026-10-03. Validated on HGX B300 with Kimi-K3 (19 Oct 2026, v0.5.22 image + this layer as a Python overlay): same-time A/B against the v0.5.21 line on a 1 prefill + 2 decode copy (weka median 24 twice, AgentX 48), and a 2 prefill + 2 decode copy (AgentX 56 twice, 64): level or better, no cross-request leaks, no broken answers. Upstream: sgl-project#38157 Issue: sgl-project#38156 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Venkat2811
added a commit
that referenced
this pull request
Oct 27, 2026
… headroom // ranks (upstream sgl-project#38157) Layer 2 of the Kimi-K3 nightly line on upstream main 5884e4f (the lmsysorg/sglang:nightly-dev-cu13-20261027-5884e4fe image). host_memory_budget_bytes() outside a budget scope returned free // ranks_per_host(), so on an 8-GPU host a rank that asks after its peers have pinned their pools divides a smaller headroom by 8 again and is refused although every pool fits. The guard now checks a pool against the whole remaining headroom; the callers that size a pool keep this rank's share through host_memory_budget_bytes_per_rank(). Source: 894d2ed on the kimi-k3-nightly-3c560089 line (the change of fork PR #14 on the v0.5.22 line), cherry-picked unchanged; the test hunk applies at a 4-line offset. Upstream sgl-project#38157 does the same job a different way and is still open. Callers added upstream since 3c56008 were checked: the MiniMax-M3.1 NVFP4 host pool (pool_host/minimax_m3_nvfp4.py) and its HiCache guard in hybrid_pool_assembler.py check an explicitly sized pool, so they take the whole headroom like the other host pools; no new caller sizes a pool. Tests (CPU): test_mem_pool_host.py with this commit's version on 5884e4f: 2 failed, the two cases that pinned the split (the budget came back as the headroom divided by 4 and by 8 ranks), 28 passed; with this commit 30 passed. Fork PR (v0.5.22 line): #14 Upstream: sgl-project#38157 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Venkat2811
added a commit
that referenced
this pull request
Oct 27, 2026
… headroom // ranks (upstream sgl-project#38157) Layer 2 of the Kimi-K3 nightly line on upstream main 5884e4f (the lmsysorg/sglang:nightly-dev-cu13-20261027-5884e4fe image). host_memory_budget_bytes() outside a budget scope returned free // ranks_per_host(), so on an 8-GPU host a rank that asks after its peers have pinned their pools divides a smaller headroom by 8 again and is refused although every pool fits. The guard now checks a pool against the whole remaining headroom; the callers that size a pool keep this rank's share through host_memory_budget_bytes_per_rank(). Source: 894d2ed on the kimi-k3-nightly-3c560089 line (the change of fork PR #14 on the v0.5.22 line), cherry-picked unchanged; the test hunk applies at a 4-line offset. Upstream sgl-project#38157 does the same job a different way and is still open. Callers added upstream since 3c56008 were checked: the MiniMax-M3.1 NVFP4 host pool (pool_host/minimax_m3_nvfp4.py) and its HiCache guard in hybrid_pool_assembler.py check an explicitly sized pool, so they take the whole headroom like the other host pools; no new caller sizes a pool. Tests (CPU): test_mem_pool_host.py with this commit's version on 5884e4f: 2 failed, the two cases that pinned the split (the budget came back as the headroom divided by 4 and by 8 ranks), 28 passed; with this commit 30 passed. Fork PR (v0.5.22 line): #14 Upstream: sgl-project#38157 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 1 of the
kimi-k3-v0.5.22stack (P/D readiness). d0058e64da is fork PR #13 (331f64442c) rebased onto v0.5.22, with its two conflicts against v0.5.22'spool_hostchanges resolved.host_memory_budget_bytes()stops dividing the remaining host headroom by the ranks on the host again; the two callers that size a pool (HiCache auto-sizing, the unified page-envelope pool) keep this rank's share throughhost_memory_budget_bytes_per_rank(). Without it, with an explicit--hicache-sizea rank that checks after its peers pinned their pools is refused although every pool fits (v0.5.22 still opens no budget scope there). Retire this layer when upstream sgl-project#38157 or a successor merges. sgl-project#42039 (fork PR #12) is not carried: v0.5.22 has it as sgl-project#42420 and sgl-project#42788.Upstream: sgl-project#38157 (issue sgl-project#38156)
lmsysorg/sglang:v0.5.22-cu130)test_mem_pool_host.py, against v0.5.22 / against this head (CPU)The Kimi-K3 rows ran this commit's Python diff as an overlay on the v0.5.22 image (byte-identical to
git diff v0.5.22 d0058e64da -- python/).🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #39277201405
Latest PR Test (Extra): ❌ Run #39277201168
Latest PR Test (AMD ROCm 10): ❌ Run #39277200984