Repository navigation
Disk-backed primary tier, and correct the concurrency claim - #1
Merged
Merged
Conversation
Two changes, one of which is a correction to a claim this README already made. CORRECTION. The Limits section predicted that concurrent deep restores beyond the batch capacity would "retry and serialise rather than fail". Measurement contradicted that: N=2 and N=4 are clean, but from N=6 requests end in WAITING_FOR_REMOTE_KVS with the final batch never issued. A bisect with GLM53_OFFLOAD_STREAM_RESTORE=0 showed pristine vLLM completing the same wave, isolating the defect to patch_offload_streaming_restore.py rather than to vLLM. Anyone running the published patch concurrently needed to know this, so it now says so plainly instead of reassuring. NEW DESIGN. vLLM admits a restore only when every group's chunks are resident in the primary tier together. The streamed restore dodges that invariant; MiaAI PR#58 satisfies it instead, by backing the staging region with a sparse file on NVMe so the tier can simply be large enough. Adds patch_offload_mmap_region.py (four edits, gated on GLM53_OFFLOAD_MMAP_DIR) and documents the design, with only PR#58's region half taken -- its group-exclusion half is already covered here, and ours additionally carries the region-copies fix. The sizing rule is the part worth stealing: the tier must be LARGER than the GPU KV pool. Both are LRU over the same stream, so a smaller tier holds a strict subset of what the pool already has and can never be the thing that serves -- and it fails silently, with every store-side metric still climbing. Also records where the block ids go: 33 of the 38 ids per 3584-token segment belong to one sliding-window group that offload excludes, so estimating tier capacity from total GPU block ids is wrong by ~7.6x. Numbers stated are read off the boot log or confirmed against the created region file (54,329,344 B per row: 32 GiB -> 632 rows, 260 GB -> 4785 rows, both exact). Concurrency under the new design is still being measured and is deliberately NOT claimed here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L36XXMwx5s97Bpp2hiPLk6
The capacity comparison published a minute ago gave 100 GB as 1.42M tokens and called it a strict subset of the 1.90M pool. That rested on 5 rows per 3584-token segment, inferred from the boot log's per-group id breakdown and never confirmed end to end. Measurement narrows it but does not settle it. kv_offload_store_bytes_total against tokens actually stored gives 31.7 KB/token per rank, which rules OUT 1 row/segment -- one row is 54,329,344 B, i.e. 15.2 KB/token, and you cannot store more than the row holds -- but leaves 3 through 5 open. At 3 rows/segment 100 GB gives 2.36M and clears the pool; at 5 it gives 1.42M and does not. So 100 GB was borderline, not wrong, and the earlier text overstated it. The structural rule survives unchanged and is what actually matters: size the tier to clear the GPU pool under the least favourable reading, and the ambiguity stops being load-bearing. 260 GB clears it under every reading. Also warns against the naive sizing anyone would try first: summing KV-group page sizes predicts 2.7 KB/token here, 12x below what is measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L36XXMwx5s97Bpp2hiPLk6
Fills in the result the previous commit promised and deliberately withheld. The streamed restore stalls from N=6. Under the disk-backed tier the same ramp is clean to N=8: 20 sessions of ~60k tokens, each evicted by pushing 2.00M tokens of filler through the pool, 20 of 20 restored with 5/5 needle recall, zero stalls and zero tier allocation retries at any rung. Records the reading error the raw output invites. Per-session TTFT decays from 4.0x at N=2 to 0.9x at N=8, which looks like the benefit evaporating; it is an artifact of comparing concurrent restores against a sequential cold prefill. Against the same tokens prefilled at the measured 902 tok/s the gain is flat at 7.0-8.4x across every rung. Concurrency costs per-request latency, not throughput. Two counters also read as failures and are not: disk_hits is structurally 0 because the probe counts tiers excluding the primary and only tier="0:primary" now exists -- the tier IS the disk -- and the 89.6% restored fraction is the Mamba alignment tail, not a truncation. Tightens the capacity bound too: the tier held 3.20M tokens with the oldest session still restoring in full, so rows_per_segment <= 5.36. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L36XXMwx5s97Bpp2hiPLk6
The repo documented two designs at once -- the FS secondary tier with a peer cascade and streamed restore, and the disk-backed primary tier that replaced it -- with diagrams, capacity figures and sizing advice from the retired one still presented as current. Two pieces of guidance flatly contradicted each other: the old design says the tier is a staging buffer to be sized for restores in flight, the new one says it is a cache that must exceed the GPU pool. README is now rewritten around the deployed design only. Every number in it is measured on that design: the N=2..8 ramp (clean to N=8, 20/20 restores at 5/5 needle recall, 7.0-8.4x throughput against a measured 902 tok/s prefill), a single restore at 63.2s -> 15.9s, 53,760 of ~60,000 tokens restored with the remainder being the Mamba alignment tail, 31.7 KB/token per rank stored, and a tier that absorbed 3.20M tokens for 88 G of actual disk. The headline changed with it. The old one -- 250k restored in 8.3s vs 250.6s -- was measured with the streamed restore and cannot describe a design that no longer ships, and no 250k figure has been measured on the disk-backed tier, so none is claimed. The headline is now the concurrency result, which was. Deletes the retired implementation rather than leaving it unexplained: patch_offload_streaming_restore.py, vllm_bounded_fs_tier.py, peer_kv_agent.py, their four test suites, the streamed-restore probes and the GB10 allocator probes that no longer back any published number. Git history keeps them. UPSTREAM_ISSUE.md drops the cross-node secondary-tier defect, which only existed in the retired design, and renumbers. The other three still apply -- including the un-unlinked staging region, which moving to disk relocates rather than fixes: a stale 32 GiB region survived a restart here today. The README table says four defects, not three, for that reason. Ships tests/test_mmap_overlay.py, which had a hardcoded path to a private overlay directory and crashed instead of skipping when pristine sources were absent. It now resolves the patch from the repo, honours PRISTINE=, skips cleanly, and is verified passing against real pristine sources from a clone of just the two files it needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L36XXMwx5s97Bpp2hiPLk6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes. One is a correction to a claim this README already made publicly, which is why this is a PR rather than a push — it changes what the repo asserts, and the measurement that settles the remaining question is still running.
1. Correction: the streamed restore stalls at N ≥ 6
The Limits section predicted that surplus concurrent restores would "retry and serialise rather than fail". Measurement contradicted that. N=2 and N=4 are clean; from N=6 requests end in
WAITING_FOR_REMOTE_KVSwith the final batch never issued.A bisect with
GLM53_OFFLOAD_STREAM_RESTORE=0showed pristine vLLM completing the same wave — so the defect is inpatch_offload_streaming_restore.py, not in vLLM. One real bug was found and fixed along the way (async-lookupRETRYchurn, fixed with a memo); the stall itself is not fixed.Anyone running the published patch under concurrency needed to know this, so the README now says it plainly instead of reassuring.
2. New design: back the primary tier with NVMe
vLLM admits a restore only when every group's chunks are resident in the primary tier together. The streamed restore dodges that invariant batch by batch. MiaAI PR#58 satisfies it instead — back the staging region with a sparse file on NVMe so the tier can simply be large enough. No promotion step, no
HIT_PENDINGlivelock, no tier-size ceiling.Adds
patch_offload_mmap_region.py(four edits, gated onGLM53_OFFLOAD_MMAP_DIR). Only PR#58's region half is taken; its group-exclusion half is already covered here, and ours additionally carries the region-copies fix.The sizing rule worth stealing
The tier must be larger than the GPU KV pool. Both are LRU over the same stream of blocks, so a tier smaller than the pool holds a strict subset of what the pool already has — the GPU answers first and the tier can never serve. Under-sizing is not a smaller cache, it is no cache, and it fails silently with every store-side metric still climbing.
Where the block ids go
33 of the 38 GPU block ids per 3584-token segment belong to one sliding-window group that offload excludes. Estimating tier capacity from total GPU block ids is wrong by ~7.6×.
What is deliberately not claimed
Concurrency under the new design is still being measured and is not reported. It is expected to be structurally absent — there is no promotion step left to stall on — but expected is not measured, and this README does not carry unmeasured claims. A
[!WARNING]callout says so; the result gets added when the ramp finishes.Every number here is read off the boot log or confirmed against the created region file (54,329,344 B per row: 32 GiB → 632 rows, 260 GB → 4785 rows, both exact). The headline 250k results were measured with the streamed restore and now carry a provenance callout so they are not read as describing the current design.
🤖 Generated with Claude Code
https://claude.ai/code/session_01L36XXMwx5s97Bpp2hiPLk6