[HiCache] Auto-size the host pool to fit available host memory by xiezhq-hermann · Pull Request #40135 · sgl-project/sglang

xiezhq-hermann · 2026-09-18T07:49:47Z

Motivation

With only --enable-hierarchical-cache, the default HiCache ratio (2.0, or 1.2 in buffer-only mode) is applied blindly. On hosts where several ranks pin host memory at once, or where the engine runs inside a memory cgroup smaller than the machine, the host pools either fail the "Not enough host memory available" check at startup or oversubscribe the host. Each pool also re-sampled psutil at construction time, so sibling ranks that were already pinning memory made the later pools' checks inconsistent.

Modifications

New --hicache-host-memory-fraction (default 0.8): fraction of the available host memory, after the existing 10 GiB reserve, that the host pools of all ranks on this machine may use. It applies only when neither --hicache-ratio nor --hicache-size is given; an explicit ratio or size is honored as before and resolves the fraction to None (auto-sizing off). Values outside (0, 1] are rejected.
mem_cache/hicache_auto_size.py: before any host pool is built, estimate the device bytes of every pool that gets a host mirror (full, SWA, mamba, packed and sidecar draft pools), reduce the default ratio until they fit the per-rank share of the budget, agree the fitted ratio across the world group with one MIN all-reduce, and build all pools inside a single budget snapshot.
mem_cache/host_memory.py: available_host_memory_bytes() bounds psutil availability by the visible cgroup v2 memory.max/memory.high (or v1 limit) minus current usage, so independent engines with separate container budgets size against their own allowance.
pool_host/base.py: host_memory_budget_bytes(requested_bytes=0) books the request against the snapshot when inside the scope, otherwise keeps the previous psutil-based split by ranks per host. MHA/DSA/Mamba host pools pass their request. kv_cache_builder.build_kv_cache wraps tree-cache creation and draft registration in the auto-size scope.
Tests: new test_hicache_auto_size.py and test_host_memory.py; test_mem_pool_host.py::TestHostMemoryBudget now stubs available_host_memory_bytes instead of the removed psutil import.
test_serving_chat.py (second commit, CI unblock): ServingChatTestCase now closes the current event loop and runs gc.collect() after each test. get_or_create_event_loop() creates a fresh loop per call and leaves the previous one unclosed; on the Python 3.10 CPU runners the cyclic GC finalized one of those loops in the middle of test_dpsk_v32_encoding_path's lazy import, and the ResourceWarning's import tracemalloc inside an in-flight import trips KeyError in importlib._bootstrap._ModuleLock.acquire (fixed upstream in 3.12). This PR merely shifted GC timing enough to hit it deterministically on base-a-test-cpu (5); the leak is pre-existing.

Scope note: the DeepSeek-V4 host pools in memory_pool_host.py still call host_memory_budget_bytes() without a request, so they see the snapshot but do not book against it. Kept as in the source change; can follow up.

Original commits

8e0d767b8e

Accuracy Tests

N/A (startup sizing only; no numerics change).

Benchmarking and Profiling

N/A.

Checklist

Format your code according to the Code Formatting with Pre-Commit.
Add unit tests as outlined in the Running Unit Tests.
Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
Please feel free to join our Slack channel at https://slack.sglang.io to discuss your PR.

Ran locally: pytest test/registered/unit/mem_cache/test_hicache_auto_size.py test/registered/unit/mem_cache/test_host_memory.py test/registered/unit/server_args/test_server_args.py -> 245 passed; pytest test/registered/unit/mem_cache/test_mem_pool_host.py test/registered/unit/mem_cache/test_hicache_host_register.py test/registered/unit/mem_cache/test_minimax_sparse_pool_host_unit.py -> all passed after the stub fix; python test/registered/unit/entrypoints/openai/test_serving_chat.py -> 140 OK, also with -W error::ResourceWarning (6 unclosed-loop warnings before the fix, 0 after); ruff format/check and isort clean on all touched files. No multi-node run locally; the all-reduce path is covered by the unit test with a stubbed world group.

CI States

Latest PR Test (Base): 🚫 Run #35464131213
Latest PR Test (Extra): ❌ Run #35464131040
Latest PR Test (AMD ROCm 10): 🚫 Run #35464131173

xiezhq-hermann · 2026-09-18T07:50:03Z

/tag-and-rerun-ci

xiezhq-hermann · 2026-09-18T17:44:25Z

/tag-and-rerun-ci

xiezhq-hermann · 2026-09-18T20:37:48Z

/tag-and-rerun-ci

xiezhq-hermann · 2026-09-19T01:50:03Z

CI note: none of the NVIDIA failures on this PR so far were caused by the change.

attempt 1: base-b-test-1-gpu-large (2) failed on test_inkling_unified.py::test_input_output_logprobs_match (KL 0.018 > 0.01, a known flake that doesn't use hierarchical cache); the two 2-GPU-large shards then sat queued for ~3h, so I cancelled and reran failed jobs.
attempt 3 (full rerun): base-b-test-1-gpu-large (0) failed in test_mla_int8_deepseek_v3.py setUpClass with server exit -9, caused by a HF Hub 429 Too Many Requests while resolving the NextN draft model; the same shard passed on attempts 1 and 2. Everything else red in stage B was fast-fail cascade.

The stage-A CPU failure from the first run (test_serving_chat.py KeyError in importlib on Python 3.10) was a pre-existing event-loop leak that this PR happened to expose; it is fixed in the second commit and all CPU shards have been green since.

Attempt 4 is a "rerun failed jobs" of attempt 3 with the 23 green jobs carried over.

With only --enable-hierarchical-cache, the default HiCache ratio is now reduced at startup until the host pools of all ranks on the machine fit --hicache-host-memory-fraction (default 0.8) of the available host memory after the 10 GiB reserve. An explicit --hicache-ratio or --hicache-size is honored as given and disables the fitting. Ranks per host come from world_size // nnodes, so TP8 over two 4-GPU nodes splits four ways. The device bytes of every pool that gets a host mirror (full, SWA, mamba, packed and sidecar drafts) are estimated before any pool is built, the fitted ratio is agreed with one world-wide MIN all-reduce, and every host pool then books against a single budget snapshot instead of re-sampling psutil while sibling ranks are already pinning memory. Available host memory is bounded by the visible cgroup v2 memory.max/memory.high or v1 limits, so co-located engines size against their own container allowance.

get_or_create_event_loop() creates a fresh loop on every call outside a coroutine and leaves the previous one unclosed, so the cyclic GC finalizes those loops at arbitrary points. When that happens in the middle of an import, the loop's ResourceWarning imports tracemalloc while the outer import still holds the module-lock bookkeeping, and importlib._bootstrap raises KeyError on Python < 3.12. Close the current loop and collect after each ServingChatTestCase test so finalizers run between tests instead.

merrymercy · 2026-09-19T19:51:09Z

pass all tests https://github.com/sgl-project/sglang/actions/runs/35430408005 before the conflict resolution

nightly-dev-20261002-67eab570 includes sgl-project/sglang#40135, which bounds HiCache host pools by the job cgroup's remaining memory, counting the page cache from the model-weight load. On these GB300 nodes that leaves 100-117 GB per rank, so the 135 GB host pools fail at startup. Use the newest nightly before that change (bundled Dynamo 1.5.0.dev20260917) and keep the benchmarked HiCache sizes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… not headroom // ranks (upstream sgl-project#38157) `host_memory_budget_bytes()` outside a budget scope returned `free // ranks_per_host()`, where `free` is whatever host headroom is left at the moment the rank asks. On an 8-GPU pod the ranks arrive at different times, so a rank that checks after its peers have pinned their pools sees a smaller `free`, divides it by 8 again, and is refused although every pool fits. Seen on 2026-10-01 in the Kimi-K3 L3 gates (one rank refused with "Requesting 24.77 GB but only have 22.74 GB free"); the same recipe passed on other runs because it is a race. This commit returns the whole remaining headroom (clamped at 0) for the explicit-size guard and updates the two unit tests that pinned the split. The guard now rejects a single pool that cannot fit at all; it no longer tries to catch co-located pools that jointly exceed the host, which the recipes size explicitly. HiCache auto-sizing keeps its per-rank budget: the division moves to its call site in hicache_auto_size.py, so the default-ratio path and its tests are unchanged. Upstream fix: sgl-project#38157 (issue sgl-project#38156) reads host memory once per TP group before splitting the budget. It predates sgl-project#40135's rewrite of this file and does not apply to v0.5.21, so this layer carries the one-line form instead. Drop it when sgl-project#38157 or a successor merges. Upstream: sgl-project#38157 Issue: sgl-project#38156 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… not headroom // ranks (upstream sgl-project#38157) `host_memory_budget_bytes()` outside a budget scope returned `free // ranks_per_host()`, where `free` is whatever host headroom is left at the moment the rank asks. On an 8-GPU pod the ranks arrive at different times, so a rank that checks after its peers have pinned their pools sees a smaller `free`, divides it by 8 again, and is refused although every pool fits. Seen on 2026-10-01 in the Kimi-K3 L3 gates (one rank refused with "Requesting 24.77 GB but only have 22.74 GB free"); the same recipe passed on other runs because it is a race. This commit returns the whole remaining headroom (clamped at 0) for the explicit-size guard and updates the two unit tests that pinned the split. The guard now rejects a single pool that cannot fit at all; it no longer tries to catch co-located pools that jointly exceed the host, which the recipes size explicitly. The two places that use the budget to *size* a pool rather than to check one keep this rank's share through the new host_memory_budget_bytes_per_rank(): HiCache auto-sizing (hicache_auto_size.py) and the unified page-envelope pool when both --hicache-size and --hicache-ratio are 0 (pool_host/unified.py). Their behaviour is unchanged. Upstream fix: sgl-project#38157 (issue sgl-project#38156) reads host memory once per TP group before splitting the budget. It predates sgl-project#40135's rewrite of this file and does not apply to v0.5.21, so this layer carries the one-line form instead. Drop it when sgl-project#38157 or a successor merges. Upstream: sgl-project#38157 Issue: sgl-project#38156 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…stream sgl-project#42039) Squash of the kimi-k3 integration PR #12 (layer 5a): upstream sgl-project#42039 applied verbatim (host_memory.py and its unit test), nothing else. Since sgl-project#40135 the host-pool budget is limit - memory.current, and memory.current includes the page cache charged to the cgroup, including the checkpoint the server just read. This counts that cache as reclaimable headroom so an explicitly sized HiCache pool is not refused at startup. Retire when sgl-project#42039 merges. If sgl-project#41181 lands instead (same lines, different text), drop this layer on the rebase. Upstream: sgl-project#42039 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… not headroom // ranks (upstream sgl-project#38157) `host_memory_budget_bytes()` outside a budget scope returned `free // ranks_per_host()`, where `free` is whatever host headroom is left at the moment the rank asks. On an 8-GPU pod the ranks arrive at different times, so a rank that checks after its peers have pinned their pools sees a smaller `free`, divides it by 8 again, and is refused although every pool fits. Seen on 2026-10-01 in the Kimi-K3 L3 gates (one rank refused with "Requesting 24.77 GB but only have 22.74 GB free"); the same recipe passed on other runs because it is a race. This commit returns the whole remaining headroom (clamped at 0) for the explicit-size guard and updates the two unit tests that pinned the split. The guard now rejects a single pool that cannot fit at all; it no longer tries to catch co-located pools that jointly exceed the host, which the recipes size explicitly. The two places that use the budget to *size* a pool rather than to check one keep this rank's share through the new host_memory_budget_bytes_per_rank(): HiCache auto-sizing (hicache_auto_size.py) and the unified page-envelope pool when both --hicache-size and --hicache-ratio are 0 (pool_host/unified.py). Their behaviour is unchanged. Upstream fix: sgl-project#38157 (issue sgl-project#38156) reads host memory once per TP group before splitting the budget. It predates sgl-project#40135's rewrite of this file and does not apply to v0.5.21, so this layer carries the one-line form instead. Drop it when sgl-project#38157 or a successor merges. Upstream: sgl-project#38157 Issue: sgl-project#38156 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… headroom // ranks (upstream sgl-project#38157) Squash of the kimi-k3 integration PR #13 (layer 5b). host_memory_budget_bytes() outside a budget scope returned free // ranks_per_host(), where free is whatever host headroom is left at the moment the rank asks. On an 8-GPU pod the ranks arrive at different times, so a rank that checks after its peers have pinned their pools divides a smaller headroom by 8 again and is refused although every pool fits (seen 2026-10-01 in the Kimi-K3 L3 gates: "Requesting 24.77 GB but only have 22.74 GB free"). The guard now checks a pool against the whole remaining headroom; the two unit tests that pinned the split are updated. The two callers that size a pool rather than check one, HiCache auto-sizing (hicache_auto_size.py) and the unified page-envelope pool when both --hicache-size and --hicache-ratio are 0 (pool_host/unified.py), keep this rank's share through the new host_memory_budget_bytes_per_rank(); their behaviour is unchanged. Upstream fix sgl-project#38157 (issue sgl-project#38156) predates sgl-project#40135's rewrite of this file and does not apply to v0.5.21. Retire this layer when sgl-project#38157 or a successor merges. Upstream: sgl-project#38157 Issue: sgl-project#38156 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ttings / GB300 GLM-5.2 NVFP4 SGLang AgentX：聚合 MTP 5-1-6、删除无效设置 (#3659) * perf(sglang): refresh GLM-5.2 GB300 AgentX configs * fix(sglang): size aggregate request limits by concurrency * fix(sglang): extend high-throughput scheduler watchdog * fix(sglang): leave prefill workspace headroom * fix(sglang): cap c260 prefill chunks * fix(ci): restore checkout on self-hosted sweep runners * fix(ci): use scoped token for benchmark checkout * fix(sglang): avoid HiCache direct eviction hang * perf(sglang): submit validated GLM-5.2 GB300 AgentX configs * fix(sglang): avoid c192 HiCache direct stall 将 c192 预填充配方切换到 HiCache kernel I/O 后端，避免 direct 后端驱逐停滞导致四小时超时。 * fix(agentx): follow current acceptance and draft policy 移除配方内硬编码的合成接受率与已禁止的 FP8 NextN draft 开关，改由当前 srt-slurm 连接器选择黄金接受率，并保持 checkpoint 自带 draft 精度。 * Refresh GLM-5.2 NVFP4 GB300 SGLang AgentX on nightly-dev-20261002 - Use lmsysorg/sglang:nightly-dev-20261002-67eab570 (bundled Dynamo 1.6.0.dev20261001) for the aggregate and disaggregate recipes. - Use EAGLE MTP 5-1-6 on the aggregate concurrency 1, 2 and 4 points; the connector selects the matching golden acceptance length. - Drop env vars and flags that are no-ops or SGLang defaults in this image, keep NVFP4 dispatch only on the DeepEP DEP16 decode, and disable the TP memory-imbalance check via its current name. - Keep the benchmark checkout step identical to main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Set perf-changelog PR link to #3659 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the TP memory-imbalance check override from GLM-5.2 GB300 recipes The check only fails startup when one rank in a worker has over 10% less free GPU memory than the others before model load. Disabling it only downgrades that error to a warning while KV-cache sizing still uses the minimum across ranks, so it never helps throughput. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin GLM-5.2 GB300 SGLang AgentX to nightly-dev-20260918-20518d85 nightly-dev-20261002-67eab570 includes sgl-project/sglang#40135, which bounds HiCache host pools by the job cgroup's remaining memory, counting the page cache from the model-weight load. On these GB300 nodes that leaves 100-117 GB per rank, so the 135 GB host pools fail at startup. Use the newest nightly before that change (bundled Dynamo 1.5.0.dev20260917) and keep the benchmarked HiCache sizes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin GLM-5.2 GB300 SGLang AgentX to nightly-dev-20260914-4358a161 nightly-dev-20260918-20518d85 produced corrupted GSM8K output on the 1P4D point (0.46 vs 0.97) and a prefill GPU page fault at c192, with server arguments identical to the passing 4358a161 runs. Return to the image validated on all seven points, which also predates the HiCache host-memory check that rejects the 135 GB pools. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>

xiezhq-hermann requested review from Ying1123, alphabetc1, hanming-lu, hnyls2002, huangtingwei9988, hzh0425, ispobock, merrymercy and yizhang2077 as code owners September 18, 2026 07:49

github-actions Bot added the hicache Hierarchical Caching for SGLang label Sep 18, 2026

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 18, 2026

xiezhq-hermann force-pushed the hicache-auto-size-host-pool branch from 2acf656 to 7deadd3 Compare September 18, 2026 17:44

xiezhq-hermann added the bypass-fastfail label Sep 18, 2026

xiezhq-hermann force-pushed the hicache-auto-size-host-pool branch from 97f8b3e to 42bee35 Compare September 19, 2026 02:02

ZYHowell requested review from BBuf, Edwardf0t1, Fridge003, HaiShaw and ch-wan as code owners September 19, 2026 06:27

xiezhq-hermann added 2 commits September 19, 2026 12:20

xiezhq-hermann force-pushed the hicache-auto-size-host-pool branch from 4747375 to b473fdd Compare September 19, 2026 19:22

merrymercy merged commit 7a6c652 into main Sep 19, 2026
108 of 178 checks passed

merrymercy deleted the hicache-auto-size-host-pool branch September 19, 2026 19:50

bingxche mentioned this pull request Sep 20, 2026

[CI Monitor] Daily Report - 2026-09-20 bingxche/sglang-ci-bot#201

Open

alphabetc1 mentioned this pull request Sep 20, 2026

[Simulator] Fix meta host memory budgets on constrained runners #40440

Merged

xiezhq-hermann mentioned this pull request Sep 22, 2026

[HiCache] fix: handle HiCache OOM with graceful server shutdown #20136

Closed

1 of 5 tasks

paulzhang-tm mentioned this pull request Sep 24, 2026

[HiCache] Account for reclaimable page cache in cgroup headroom #41181

Closed

4 of 5 tasks

nvjullin mentioned this pull request Oct 1, 2026

Fixed incorrect host memory counting #42039

Closed

3 of 5 tasks

nvpohanh mentioned this pull request Oct 2, 2026

GLM-5.2 NVFP4 GB300 SGLang AgentX: MTP 5-1-6 aggregate, drop no-op settings / GB300 GLM-5.2 NVFP4 SGLang AgentX：聚合 MTP 5-1-6、删除无效设置 SemiAnalysisAI/InferenceX#3659

Merged

5 of 11 tasks

This was referenced Oct 3, 2026

[HiCache] Count charged page cache as reclaimable cgroup headroom (upstream #42039) datacrunch-research/sglang#12

Merged

[HiCache] Size each rank's host pool from the full host headroom, not headroom // ranks (upstream #38157) datacrunch-research/sglang#13

Merged

metamergebot mentioned this pull request Oct 3, 2026

[HiCache] Fix cgroup page-cache accounting and the sizing fallback when cgroup discovery fails #42420

Merged

3 of 5 tasks

OrangeRedeng mentioned this pull request Oct 11, 2026

[NPU] Diffusion CI fixes #43210

Open

2 of 5 tasks

Navigation Menu

Repository navigation

[HiCache] Auto-size the host pool to fit available host memory - #40135

[HiCache] Auto-size the host pool to fit available host memory#40135
merrymercy merged 2 commits into
mainfrom
hicache-auto-size-host-pool

xiezhq-hermann commented Sep 18, 2026 •

edited by github-actions Bot

Loading

Uh oh!

xiezhq-hermann commented Sep 18, 2026

Uh oh!

xiezhq-hermann commented Sep 18, 2026

Uh oh!

xiezhq-hermann commented Sep 18, 2026

Uh oh!

xiezhq-hermann commented Sep 19, 2026

Uh oh!

Uh oh!

merrymercy commented Sep 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

xiezhq-hermann commented Sep 18, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Original commits

Accuracy Tests

Benchmarking and Profiling

Checklist

CI States

Uh oh!

xiezhq-hermann commented Sep 18, 2026

Uh oh!

xiezhq-hermann commented Sep 18, 2026

Uh oh!

xiezhq-hermann commented Sep 18, 2026

Uh oh!

xiezhq-hermann commented Sep 19, 2026

Uh oh!

Uh oh!

merrymercy commented Sep 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xiezhq-hermann commented Sep 18, 2026 •

edited by github-actions Bot

Loading