Fix dockerfile and triton cache manager - #720
Merged
Merged
Conversation
zhyncs
reviewed
Jul 25, 2024
hnyls2002
changed the title
Collaborator
Author
hnyls2002
marked this pull request as draft
July 25, 2024 07:45
Closed
hnyls2002
marked this pull request as ready for review
July 25, 2024 09:21
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…t the cutover; bound the read spike sgl-project#719 -- THE REBIND, and the finding that shapes it. A REBIND IS NOT A POINTER SWAP. The host pool is CONSTRUCTED FROM the device pool (hybrid_pool_assembler.build_kv_host_pool(kv_pool=...)), so its layer_num and buffer sizes are that phase's. The two phases have different per-rank layer counts -- a PP stage holds 7/5/4 of the 16 attention layers, the TP stack holds all 16 -- so repointing the controller at the other phase's device pool while its host pool still describes this phase's is a NEW corruption wearing the fix's clothes: matching row ids, mismatched widths, and the copy RUNS. The rebind therefore refuses unless handed a host pool whose shape matches the incoming device pool, and phase_pools_for refuses when the boot did not build one -- which is today's real state, reported with its reason instead of proceeding onto the wrong pool. Supplying that second host pool is a boot-time question (it costs host RAM, the binding constraint per DESIGN_706 C1) and is named, not mocked away. THREE READERS, ALL OR NONE. The pool identity is captured in three places (controller: mem_pool_device / _hybrid / _allocator; radix cache: hybrid_kv_cache / kvcache / host pool; scheduler: token_to_kv_pool_allocator). A rebind that moves some and not others is strictly worse than none: the readers then name different memory for the same row id and EVERY CALL STILL SUCCEEDS. So the rebind stamps a generation on each reader and a coherence check compares them afterwards -- the failure is invisible to inspection, so it is verified by counter. A stamp that fails mid-way rolls the state to a generation nothing can match, so the tier cannot re-arm onto a torn binding. THE PAYOFF, wired through sgl-project#718's predicate: "disarmed" now asks whether the ACTIVE phase is the BOUND phase, not whether TP is active. With no rebind those are the same question (the binding is always the boot phase), so the default path is byte-identical -- sgl-project#718's 12 tests pass unchanged. After a coherent rebind the device tier is usable in the phase it moved to. Call site: after the stack swap in _cutover (the mirror of sgl-project#703's writeback, which runs before anything moves because it reads the OUTGOING pools). Refusal is logged, not raised: a refused rebind is SAFE by construction because the binding does not move and sgl-project#718 keeps the tier disarmed, while a raise at the seam takes down an instance that was serving fine. sgl-project#720 -- THE READ SPIKE, bounded and charged. Every storage read took its target from host_pool.get_dummy_flat_data_page(), a fresh pinned tensor per read that the joint budget cannot see (it accounts pools declared at attach; this is neither). ReadBufferPool is a fixed ring, allocated once, declared through check_and_register_pinned_post and unregistered on close. Exhaustion falls back to a fresh allocation -- today's behaviour, counted -- rather than blocking, because stalling a prefetch worker to save memory trades a bounded spike for unbounded latency. SGLANG_HICACHE_READ_BUFFERS=0 (default) keeps the current path exactly. Tests (hermetic, CUDA_VISIBLE_DEVICES=""), 18 new, 137 in the family: * test_hicache_rebind_719.py (11) -- all three readers move together; a reader left behind is CAUGHT (planted deliberately, since without generations that state is invisible); an absent reader refuses the whole rebind; a failing stamp leaves the set unusable rather than half-moved; shape mismatch and unmeasurable shapes refused; a phase with no host pool refused WITH the reason; the disarm lifts only after a coherent rebind and re-arms on the return leg; unarmed is byte-identical. * test_read_buffer_pool_720.py (7) -- the falsifier pair (25 reads = 25 allocations today, 4 with a ring of 4), bounded overflow, a raised read still returns its buffer, the registry sees the ring and stops seeing it on close, off by default. Regression, same env, base c3e9487 vs this commit: unit/mem_cache 940 failed / 779 passed -> 940 failed / 894 passed (+115) unit/server_args 1 failed / 626 passed -> 1 failed / 639 passed (+13) unit/managers 4 failed / 2145 passed -> 4 failed / 2145 passed scheduler/test_phase_flip_runtime.py: 67 passed (the file the call site edits) phase_flip_runtime.py diff: 23 insertions, 0 deletions. Live validation of both joins F4-r4's window list; the flip+hicache boot they need is exactly the boot sgl-project#719 makes safe -- and sgl-project#719 cannot arm until that boot also builds the second phase's host pool. CAN-FAIL PROOF (mutation applied, suite re-run, reverted): Q1 coherence check always passes -> test_a_reader_left_behind_is_caught, test_a_failing_reader_leaves_the_set_unusable_not_half_moved Q2 shape check removed (the pointer-swap trap) -> test_shape_mismatch_is_refused, and ONLY that one Q3 partial rebind allowed (absent readers ignored) -> test_an_absent_reader_refuses_the_whole_rebind, and ONLY that one Q4 predicate ignores the binding (reverts to the raw TP-active test) -> 6 failures spanning BOTH suites, incl. test_disarm_lifts_only_after_a_coherent_rebind, test_the_return_leg_rebinds_back, and sgl-project#718's own test_disarmed_is_false_without_the_flip -- i.e. the tie between the two features is load-bearing in both directions Q5 ring never reuses (release drops every buffer) -> test_green_the_ring_allocates_once_and_reuses, test_concurrent_borrows_beyond_the_ring_fall_back, test_a_raised_read_still_returns_its_buffer Q6 ring not declared to the pinned registry -> test_the_ring_is_declared_to_the_pinned_registry, and ONLY that one Restored tree re-verified green after every mutation (30 passed).
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…the first Flip+HiCache boot Everything mechanical is landed (canonical format 04c6736, GDN blob a38f39f, sharded backend 19f4c68, cutover rebind ec117fa, ReadBufferPool, sgl-project#718 disarm). This is the boot that composes them, and its one open cost, priced. THE OPEN COST, stated first and found by inspection: sgl-project#719's rebind refuses unless the incoming phase owns a shape-matched host pool, and that pool HAS NO BUILDER -- phase_pools_for reads scheduler.phase_flip_host_pools (hicache_phase_binding.py:287) and nothing in the tree writes it. So the rebind refuses at every cutover today, logged and never raised, with sgl-project#718 keeping the device tier disarmed in the phase that did not build the binding. That is a safe state, and the design recommends booting IN it. HOST-RAM BUDGET, from the deployed [32,16,16] cut. A host row is that rank's OWN layers x 2048 B, so PP rows are 16,384 / 8,192 / 8,192 B while a TP row is 32,768 B on every rank -- multipliers 2.00x / 4.00x / 4.00x, i.e. 16/own_layers. Against the MEASURED pinned load (C1 at 591add2: 25.87 GB usable, 20.50 GB of flip weight images, 5.37 GB remainder): rows fitting in 5.37 GB rank0 rank1/2 PP pool only (today) 327,759 655,518 BOTH pools 109,253 131,104 So the second pool cuts the host tier to a third on rank0 and a fifth on ranks 1-2 at fixed budget; at the 9.01 GB the C1 boot actually requested it needs ~27 GB on rank0's ratio against 5.37 available -- it does not fit, and not marginally. WHICH INSTRUMENT BINDS, because reasoning from free -g gives the wrong answer: the PINNED budget refuses first (it already refused 29.51 GB at C1), while the sgl-project#721 available floor of 24.3 G has ~12.7 G of slack today (available 37 G). A second host pool is refused by the pinned check long before it threatens the OOM floor. MARKED ABSENT rather than estimated: whether the 9.01 GB host-pool figure is one rank's or an aggregate, and the per-rank split of the three weight images. The refusal message does not say and no ledger entry resolves it, so every number is given per rank AND against the shared 5.37 GB remainder, so the conclusion does not depend on the ambiguity. RECOMMENDATION: the first boot does NOT add the second pool. It does not fit, and the cross-phase path does not need it -- sgl-project#706 made the DISK tier geometry-neutral, sgl-project#703 pushes warm prefixes there before the cutover, and both phases resolve the same content key (100 GB disk = 3,051,758 canonical-page tokens against a 5.37 GB host staging tier). The safe states are the DEFAULT states, so nothing has to be remembered. Also in the document: the exact flag set and env (including why --phase-flip-rebind-hicache is deliberately NOT set), the boot-before-hold sequence with the two log lines whose ABSENCE is the stop condition, the sgl-project#630 PP=3 x disk-HiCache warmup wedge with its root fix (9da9dfd) and what to watch anyway, acceptance (byte-identical cross-phase continuation with A-vs-A first -- the sgl-project#718 shape is a WRONG ANSWER, not a miss; hits counted from log lines because cache_hit_rate reads 0.0 with real hits; ReadBufferPool overflow counter zero; the sgl-project#720 falsifier pair), and the ranked risk list. Design only. No boot, no GPU. The boot goes on the window list. Symlinked beside the other DESIGN_706 docs in evidence-665-f1/.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…; retract my "resize is untested" claim NOTHING BUILT, because it is already built. attach (PUT), detach (DELETE), status (GET), resize (POST /resize) and clear all exist on /hicache/storage-backend (http_server.py:1395-1510), each behind @auth_level(AuthLevel.ADMIN_OPTIONAL) PLUS an explicit admin_api_key check -- the sgl-project#510 regime with the belt-and-braces the most sensitive routes should get -- reaching attach/detach/resize_storage_backend on the tree cache via scheduler handlers at scheduler.py:7625/7681/7728. The semantics the brief specified are the semantics implemented: attach/detach refuse a non-idle scheduler BY NAME ("Reject attach: scheduler is not idle. #queue-req=... #running-req=..."), and resize-down evicts inline and "returns once usage is back under the new cap", with in-flight (reserved but uncommitted) writes never evicted -- so it does not truncate live pages, and the write interlock is _pending_writes rather than a race with the backup queue. resize deliberately does NOT require idleness, which is a narrower and better interlock than attach/detach's whole-scheduler gate. RETRACTION. I wrote a hermetic pin file on the finding that "resize has no coverage at all", having grepped for resize INSIDE the E2E attach/detach test. Wrong: test/registered/unit/mem_cache/test_hicache_runtime_resize_545.py exists with 21 tests covering every property I pinned and several I did not -- grow-evicts-nothing, shrink-until-under-cap, LRU victim order, enable-at-runtime-adopts-existing-files, in-flight-write-not-evicted, lifting-the-cap-disables-eviction, non-owner-MLA-rank-inert, the request validation layer, and both cache classes. My file was 100% duplicate and is DELETED; shipping it would have created a second authority for the same properties, which is what I refused in sgl-project#536. Third instance of the same error (after sgl-project#726 and sgl-project#677): concluding absence from the file I happened to open instead of grepping for the thing itself. "No coverage" requires a search FOR the coverage. My harness also passed max_size_bytes/min_free_bytes as extra_config keys when the real names are max_size/min_free_space, so every evictor it built came up UNCONFIGURED and most pins still passed -- a harness passing for the wrong reason. TWO MORE BRIEF PREMISES THAT DO NOT EXIST. MixedLayoutError: zero hits across python/. The cited commit 19f4c68 describes it in its message but no such class is in the worktree; the nearest relative guards PD draft-KV, not HiCache attach. What attach actually uses is a same-backend check (hiradix_cache.py:544-568) refusing a DIFFERENT backend by name. ReadBufferPool / sgl-project#720: also absent, no class and no reference. I did not design around either guess. PRIOR ART the brief did not mention: docs/dev/NOTE_544_hicache_runtime_ preserve_thinking.md is a desk-complete investigation of this same ticket, and docs/advanced_features/hicache_storage_runtime_attach_detach.md is the user-facing documentation of the shipped feature. THE ONE REAL GAP, and it lands on this rig's model family: UnifiedRadixCache.attach_storage_backend and detach_storage_backend (unified_radix_cache.py:2769,:2783) are HARD STUBS that always fail -- "does not support runtime HiCache storage attach yet". resize works there; attach and detach do not. UnifiedRadixCache is what registry.py:191 constructs on the path that appends the MAMBA component for is_hybrid_ssm, so for the hybrid-GDN family the shipped story is resize yes, attach/detach no -- the ticket's headline capability is exactly the half that is stubbed. I did NOT verify which cache class our specific boot instantiates; that needs the boot config, not the source. Also NOT ESTABLISHED: no test raises or mocks a real OSError(ENOSPC); disk-full is covered only through the min_free_space watermark refusal. RECOMMENDATION: implement UnifiedRadixCache attach/detach (a real implementation task -- the stubs exist because that cache's controller lifecycle differs from HiRadixCache's), then a real ENOSPC injection test. Live-window acceptance filed in the note, including that attach and detach are expected to fail on a hybrid-GDN model with the named refusal -- that is the acceptance for the gap, not a second bug report. Existing suite re-verified: 21 passed. codespell clean. Desk only.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…by hand --hicache-host-role staging (bfabd2d) refuses --hicache-ratio and requires an explicit --hicache-size. That refusal is only worth having if the number it demands comes from somewhere; a hand-picked --hicache-size is exactly the pool pin the planner's sole authority over the memory budget (sgl-project#584/sgl-project#785) exists to prevent. planner/hicache_staging.py derives it. Two consumers, and sizing for one of them silently breaks the other: * WRITE STAGING, a bandwidth-delay product: a page stays pinned from the moment it is handed to the storage backend until the write acks, so the resident set is drain rate x that residency, widened by a burst margin (write-through arrives in bursts, the drain is steady). * READ LANDING, which scales with CONCURRENCY, not bandwidth: a prefetch takes its destination slot from mem_pool_host.alloc() BEFORE the storage read is issued (hiradix_cache.py's _prefetch path). Sizing from the write side alone yields a tier that is correct and quietly serialises prefetch -- invisible to every capacity metric, visible only as latency. The tier takes the larger, rounded UP: rounding down would emit a tier smaller than the derivation justifying it. NO RIG FIT. Drain rate and latency are PARAMETERS, not constants. The planner probes rather than carrying a measured vector from one box to another, so nothing in this module encodes what this rig happens to do (~0.5 GB/s on its ZFS-backed file tier); that figure appears only as a test input. WHAT THIS DELIBERATELY REFUSES TO DO. If write-through produces bytes faster than the backend drains them, NO finite size holds -- residency is set by the drain, so a faster producer grows the in-flight set without bound. sustainable() answers that question so a caller can refuse instead of emitting a number that cannot be correct. The remedy there is backpressure at the producer, a runtime mechanism and not a size. Note sgl-project#720's ReadBufferPool made the OPPOSITE choice on purpose for reads -- on exhaustion it allocates a fresh uncounted buffer rather than blocking, trading "a bounded spike for an unbounded latency" -- which is right for a prefetch worker and wrong for a write path under a RAM budget, where re-inflating the pinned footprint is the failure the budget exists to prevent. That ring is the next posten, not this one. fits_pinned_host_budget() asks the sgl-project#729 authority and registers the post like every other pinned producer, so a tier that does not fit is refused where it is derived rather than discovered at allocation. Also: docs/dev/NOTE_810_dangling_design_706_c1.md records that DESIGN_706_BOOT cites a DESIGN_706_constraints document (C1/C1a) that exists nowhere in the tree. sgl-project#810's premise is quoted verbatim at the citing site and does not depend on it, but the supporting argument is unrecoverable. Recorded rather than reconstructed: rebuilding it from four citation fragments would read as primary evidence while being inference, and the next reader would cite it as the original. The tasked "add to DESIGN_706-C1" cannot be executed until the target exists. Tests: test/registered/unit/planner/test_hicache_staging_810.py, 19 new cases, hermetic (CUDA_VISIBLE_DEVICES=""). FIVE mutants, all dying on the intended tests: read consumer dropped (write-only sizing) -> 1 failed rounds down instead of up -> 1 failed zero-drain guard removed -> 2 failed burst-margin floor removed -> 1 failed sustainability predicate inverted -> 2 failed Source green again at 19 passed; 28 passed together with the role gate. Suite: test/registered/unit/planner/ is 2889 passed / 1 failed after this change. The single failure is test_rejected_evidence_pins.py's sgl-project#797 evidence assertion, confirmed PRE-EXISTING by re-running it on a clean checkout of dd6967c with these files removed -- same test, same assertion. No pre-existing test changed status. ruff/black/codespell clean. Not wired to a probe yet, and named so it is not assumed: nothing measures the drain rate automatically, so a staging profile still needs the number supplied. Measuring it needs load against a running server, which belongs to another strand's boot window right now.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…ount the refusals A staging host tier is small on purpose, and nothing at runtime bounded what write-through put into it. The tier itself did, by running out -- which is not a bound but two silent failures. The read consumer starves: a prefetch takes its landing slot from mem_pool_host.alloc() BEFORE issuing the storage read, so a tier full of undrained write-through pages makes it evict, retry, truncate and abandon; a correct miss, invisible to every capacity metric and visible only as latency. And the write refusal itself is silent: write_backup learns about exhaustion by reading a None back out of an allocation that already happened, then -- with no rank-uniform floor published -- takes evict_host(), a RANK-LOCAL tree edit, which is the sgl-project#645 divergence. NOT sgl-project#720's ReadBufferPool. Its acquire() (read_buffer_pool.py:99-105) answers exhaustion by allocating a fresh UNCOUNTED pinned buffer, deliberately: "stalling the prefetch worker to save memory would trade a bounded spike for an unbounded latency". Right for a prefetch worker, and precisely wrong for a write path under a RAM budget -- the footprint that overflow re-inflates is the one --hicache-host-role staging exists to cap. StagingWriteRing never allocates; its only outcomes are admitted and refused-and-counted. THE RESIDENCY IS TWO-PHASE, and only one phase is refusable. write_backup allocates the host slots; write_backup_storage calls node.protect_host() and the protection drops only at the backup ack (ongoing_backup.pop -> release_host). The bandwidth-delay product that planner/hicache_staging sizes the tier from is the SECOND phase, so a ring spanning only the device->host copy would claim a bound it does not have. Phase one is admit(), taken before any allocation, where a refusal is actionable. Phase two is occupy(), which cannot be refused -- the page is already resident, so refusing would free nothing and merely hide the drain queue from the next admission. Phase two is keyed by the STORAGE OPERATION, not the node, and that is not cosmetic: in UnifiedRadixCache one write-through ack fans out into several storage backups after a node split, each with its own id, and lock_node is by then no longer among publish_nodes. A node-keyed charge would be stranded there -- and a leak shrinks the ring permanently, a strictly worse failure than the overshoot it protects against. So the node-keyed charge is retired at exactly one site (_finish_write_through_ack) before the hand-off. CAPACITY WITHOUT A NEW NUMBER. The read consumer is already bounded at runtime by cache_controller.prefetch_capacity_limit = int(0.5 * mem_pool_host.size). Both consumers share one tier, so the write bound is that number's complement. The module introduces no constant of its own and does not become a second sizing authority beside the planner (sgl-project#584/sgl-project#785). Built after _symmetrize_prefetch_capacity(), so the bound derives from the group-agreed number: a rank-dependent admission bound on this path is the sgl-project#645 defect. No hysteresis, deliberately: a refused backup is not lost work (the node stays in the tree and the next insert offers it again), so there is no thrash cost for a watermark to damp, and a resume fraction would be an invented number. Wired into both cache classes that have a construction site -- HiRadixCache (registry.py:115) and UnifiedRadixCache (registry.py:191). HiMambaRadixCache has none, stated at registry.py:108-109. Default unchanged: the ring is built only under --hicache-host-role staging. Under retention -- the default -- the attribute is None and every call site is one `is None` test. The diff is purely additive. Tests: test/registered/unit/mem_cache/test_staging_write_ring_810.py, 33 cases, hermetic (CUDA_VISIBLE_DEVICES=""). Both real write_backup implementations are driven UNBOUND over a fixture, in both directions: a ring with room admits exactly as before, a full ring refuses, and a refusal is shown never to reach evict_host and never to call cache_controller.write. The drain edges are driven through the real write_backup_storage and the real _drain_storage_control_queues_impl. Mutants, 15, all confirmed dying: hiradix admit gate inert -> 4 failed hiradix abort on failed write removed -> 1 failed hiradix abort on floor refusal removed -> 1 failed hiradix drain-phase occupy removed -> 2 failed hiradix release at D2H ack removed -> 1 failed hiradix release at backup ack removed -> 1 failed hiradix release on forced detach removed -> 1 failed unified admit gate inert -> 2 failed unified abort on failed write removed -> 1 failed unified drain-phase occupy removed -> 1 failed unified release at backup ack removed -> 1 failed unified release at D2H ack removed -> 1 failed role gate removed (retention gets a ring) -> 2 failed capacity is the whole tier, not the complement -> 2 failed exhaustion allocates instead of refusing -> 8 failed Source green again at 33 passed. "unified release at D2H ack removed" SURVIVED the first battery at 32/32 green: that call edge was untested. The test it now dies on (UnifiedDrainPhaseTest.test_the_two_phases_do_not_double_count_one_page) was added for it. Without the mutant round a leak edge would have shipped unchecked. ruff/black/codespell clean on the new files; the modified files carry 372 ruff findings before and after this change, i.e. none in these hunks.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 22, 2026
…the order does not matter sgl-project#809 asks for a hybrid residency of the phase-flip weight images: a small pinned share (a "hot head, in wave order"), the rest file-backed as today, and a reload with prefetch overlap on the sgl-project#125 double-buffer pattern. The prior-art gate falsified two of those three, and the third rests on a comparison nobody has run. Recording that before writing code, because the cheapest outcome here is the one where nothing is rebuilt. THE OVERLAP IS ALREADY SHIPPED. `_staged_file_refill` (sgl-project#802, landed 2026-08-22, weights_arena.py:522) reads the file-backed image with bounded preadv into sgl-project#720's pinned ring and overlaps the next chunk's read with the previous chunk's H2D DMA -- depth streams, depth events, an inflight flag, and a synchronize before a buffer is refilled. The depth is SGLANG_PHASE_FLIP_REFILL_DEPTH, default 2: literally a double buffer. Rebuilding it would be the sgl-project#720-verbatim mistake in a new place. THERE IS NO HOT HEAD. `arena_refill` copies the whole payload and only then checksums it; nothing reads the arena before that checksum passes. There is no partial publication and no per-wave consumption, so WHICH bytes are pinned cannot matter -- only HOW MANY. sgl-project#254's wave order exists because a MoE forward consumes experts wave by wave; a flip consumes the arena once, whole. A "hot head selection" would be a knob with no effect. TWO OF THE THREE CITED NUMBERS ARE NOT IN THIS TREE. Grepped as rates over python/sglang/srt/ and docs/dev/: zero hits for 2850, 4263, 1763 and 1844 as a MiB/s, MB/s or GB/s figure, and the near-misses are named in the note so the next reader does not "find" them by accident (1763 is a byte budget in an affordability check and an NVML corridor floor; 1844 is a seam-staging free memory delta). The pinned figure IS real: _PINNED_REF_LO_GBPS / _HI_GBPS = 4.93 / 8.88, phase_flip_boot.py:502-509. What the numbers that DO exist say: pinned 4.93/7.08/8.88 GB/s per rank (sgl-project#690) against staged O_DIRECT 2651/2602/ 3751 MiB/s per rank in the real flip on metal (sgl-project#802), i.e. roughly 1.8-2.6x -- but from two separate campaigns, never against each other, and the note says so rather than presenting it as an A/B. COMPRESSION IS DISCARDED, WITH THE ARITHMETIC, not deferred. This tree's own ANALYSE_306 already ends "do not build sgl-project#306 as a codec", and sgl-project#456's sparse write records a 0 byte win on /spinning because ZFS folds the same holes -- which is where the flip images live. Independently, at ratio 1.145 the bytes saved are 12.7%, so a serial win needs decompression at 7.9x the read rate (64 GiB/s synthetic, 20-29 GiB/s at metal rates) and even a fully overlapped one needs 9.29 GiB/s per rank, 10.07 GiB/s aggregate across three concurrent ranks. The measured decompression ceiling in ANALYSE_306 is 4.3-4.8 GB/s. Both bars are out of reach by 2-13x. THE DANGER DIRECTION, AND THE ONE POSITION THIS DOES NOT REVERSE. The file-backed arm exists so ~68.7 GiB of unreclaimable host RAM cannot OOM-kill a swapless boot, and it refuses a missing or tmpfs image dir rather than "silently allocating a pinned image the host ledger would then double-count as reclaimable". The tree has ALREADY decided that image posts are registered but not checked -- "a new refusal path here could break a boot that works today", weights_arena.py:918-924. That decision stands. It does not bind a new, opt-in, default-off pinned share, for its own stated reason: refusing a share no current boot requests cannot break a boot that works today. So only the new share may be checked, and a hybrid image must register exactly its pinned head and nothing else, because the file-backed bytes are deliberately unregistered (the registry sums NON-reclaimable bytes). THE HAZARD ANY IMPLEMENTATION MUST HANDLE, recorded because it is silent: _staged_file_refill takes O_DIRECT only when `at % 4096 == 0`. If reads resume at an unaligned pinned-head boundary, EVERY chunk misses that test, the whole refill falls back to the buffered fd -- 8304 -> 2595 MiB/s, a 3.2x regression with no error and no log line -- and the pinned head costs more than it saves. The head must be floored to _DIRECT_ALIGN. NOT BUILT HERE, deliberately: the share's size is the whole feature, and the comparison that would set it (pinned vs staged-O_DIRECT on ONE binary, same load, same bytes) has never been run. A size derived from the two-campaign mixture above would be the rig-fit the planner rules forbid, and shipping the actuator with the size left as an unmeasured parameter would put a knob in the tree nobody can set -- while a helper with no caller is exactly the defect removed from planner/hicache_staging.py in this same branch. The A/B is named as a window item, with the instrumentation that already exists for it (_timed_arena_refill / refill_report). Analysis only: no code changes, no test changes, no behaviour change. codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…all three falsifiers First slice of the user's flip-image design: RAM holds ONE layout image plus a small overshoot, and at the flip the incoming layout streams RAM -> VRAM while the outgoing one streams VRAM -> RAM into the pages just freed. PCIe is full duplex, so the copy-back rides the idle return direction. THE COPY-BACK IS NOT WRITE-BACK. The weights are immutable and nothing is saved; it is residency PLACEMENT for the next flip, which is what a single-layout RAM budget requires. Written into the module docstring because a later reader who mistakes it for a write-back will optimise it away and break the following flip. WHY THIS AND NOT THE PARTIAL PIN: W26 proved the dual pin impossible here -- both pin arms OOM-killed in the LAUNCH phase, before any flip. One layout plus eps (~30 GiB vs ~68.7 GiB) fits AND takes the disk off the steady-state critical path, which is what reaches the physics floor; a partial pin leaves a disk share behind, and W26 measured the leg 99.8-100 % storage-bound. THIS SLICE IS THE ARITHMETIC ONLY, deliberately. The overshoot sizing and the interleaved schedule are pure functions over byte counts, so every invariant the scheme rests on is falsifiable WITHOUT a GPU -- the same split sgl-project#852's estimator and sgl-project#856(a)'s bound phrase use, and for the same reason. OVERSHOOT = size asymmetry + in-flight window, sized from the LARGER direction. The asymmetry is W26's measured one (PP0 15925.8/16362.7, PP1 8573.8/8961.3, PP2 8573.8/9481.6 MiB); a single fixed reservation has to cover whichever direction the next flip takes, so a mean is the OOM. The in-flight term is separate and pinned: an implementation returning only the asymmetry gives 0 for equal layouts and stalls immediately. ALL THREE NAMED FALSIFIERS ARE ASSERTED: * no actual overlap -- `rotation_totals` counts co-scheduled steps; a real rotation must have them and must have them as the DOMINANT shape (>90 % of steps), not as an accident of the tails. Its can-fail partner: a one-sided rotation must report zero overlap. * RAM leak across cycles -- three full A->B->A cycles must return host occupancy exactly to its start. Three, because W27-retry's leak fired on the THIRD cycle, not the first. * checksum -- verified against the real source: the image is `payload = image[:layout.total_bytes]` plus an int64 trailer, checked with `uint8_checksum(dst)` over the ARENA. That last part is what makes a D2H reproducible: bytes returned from VRAM verify exactly as bytes read from disk do, so only the 8-byte trailer is new. A FINDING THE TESTS PRODUCED, and it is why the budget test first passed vacuously: THE RAM BUDGET BINDS IN ONLY ONE DIRECTION. Pressure exists solely when the OUTGOING layout is LARGER than the incoming one, because only then does the copy-back need more RAM than the H2D frees -- PP0 copying back its 16362.7 MiB tp image while the smaller 15925.8 MiB pp image streams in leaves 436.9 MiB with nowhere to go. The opposite direction schedules cleanly at zero overshoot. Both halves are now asserted so the asymmetry is recorded rather than rediscovered. Under-sizing STALLS LOUDLY rather than proceeding: a scheduler that kept going would be holding both layouts, which is precisely the state that OOM-killed W26's pin arms. GATE (foreground, family-batched): managers core (45 PP files excluded) 3554 passed, 336 subtests, 0 failed model_executor 777 passed / 15 failed (+14 new) The 15 are the pre-existing sgl-project#815 family. ZERO new failures. NOT YET BUILT, and not claimed: the device-side execution (streams, the pinned ring registered once per sgl-project#720/sgl-project#729, the planner-priced host post per sgl-project#721/sgl-project#770), and the separately-instrumented priming flip. This slice is the plan those will execute.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 24, 2026
…the aliasing defect the arithmetic could not see Slice 1 fixed the arithmetic. This is the executor that runs it, and building it surfaced a defect in the plan that no byte-count model can express. THE ARENA IS ONE BUFFER AND THE ROTATION IS AN IN-PLACE PERMUTATION. `allocate_arena` returns a single contiguous device tensor sized max(pp, tp) and `arena_refill` overwrites `arena[: layout.total_bytes]` in place (weights_arena.py:1184,1192). Under a single-layout RAM budget the host image is likewise ONE buffer. So at every chunk offset k the two directions are CIRCULARLY dependent: the H2D wants to write arena[k], which the D2H has not read yet, and the D2H wants to write image[k], which the H2D reads. Slice 1's plan emits h2d_offset == d2h_offset while both are active, so a literal execution of it aliases on every step. Serialising removes the duplex the scheme exists for; running concurrently corrupts the image -- and it corrupts it in the direction THIS flip's checksum cannot catch, because the damage lands in the image the NEXT flip streams in. THE RING IS THEREFORE LOAD-BEARING, not an optimisation, and this is the part a later reader is most likely to undo. Per chunk: save image[k] into a ring slot, D2H arena[k] -> image[k], then H2D ring slot -> arena[k] gated on that D2H. Chunk k+1's D2H is enqueued before chunk k's H2D is waited on, so the lanes genuinely run together with no aliasing anywhere in the pipeline. The one host-to-host memcpy per chunk is the intrinsic cost of an in-place rotation, not an accident of this implementation. AND THE TWO SLICES THEN AGREE BY CONSTRUCTION rather than by coincidence: one max-sized host buffer (one image PLUS the size asymmetry) plus depth*chunk of ring IS `rotation_overshoot_bytes`. The same number reached twice, from two directions. OVERLAP IS MEASURED ON THE EXECUTOR, NOT THE PLAN. A step counts as overlapped when, at the instant its D2H is enqueued, an earlier H2D has not been waited on. Slice 1 could only count co-scheduled steps, which a serialised implementation would also produce. Its can-fail partner is pinned: a ring of depth 1 must report exactly zero overlap and must still be byte-correct. PRIOR ART REUSED, NOT REBUILT. The ring is sgl-project#720's ReadBufferPool, which charges the pinned-host registry BEFORE allocating (sgl-project#729), exactly as `weights_arena._refill_staging_pool` already composes it; registered once per process and reused by every flip. The readout is sgl-project#856(a)'s RefillLegTiming / refill_bound_phrase -- no new telemetry. The launcher PRICES the ring with the pure `joint_pinned_host_error` and does NOT register it: a planner-side registration of bytes the launcher never pins is the helper commit 272d0d9 deleted, and reintroducing it was the obvious wrong move here. THE DEFERRED WAVE-LOOP SUCCESSOR, FOLDED IN. W27-retry measured 16 empty waves for ~314 ms. Skipping the loop was deferred because `finalize_wave` is what marks the destination pool resident, and a bare skip leaves `backing_is_resident` answering no. The replacement is the whole-pool swap ALREADY on the same object (`WavedBackingSwap.__call__`): release source, reclaim, restore destination. It is not merely equivalent, it is strictly cheaper -- waving exists to bound the transient of holding a source layer live while its destination is written, and with no bytes crossing there is nothing to bracket, while `__call__` releases before it restores so its peak is max(src, dst). Gated on a new `PhaseFlipTransition.moves_nothing` that checks ALL THREE legs: a predicate reading only the peer exchange would skip a plan that still has a LOCAL move and drop KV silently, which is the one outcome worse than the 314 ms. R1's directional budget law is carried forward unregressed: pressure exists only when the outgoing layout is the larger, and both halves stay pinned. THE PRIMING FLIP IS INSTRUMENTED APART (P4), with its own stats and its own timing record, so a steady-state mean can never absorb it. WHAT IS DESK-PROVEN vs WHAT NEEDS THE WINDOW. Every test here executes the REAL executor over REAL byte patterns on CPU tensors: byte-exactness in both directions, the checksum, the absence of drift across three A->B->A cycles, and the ring returning to full are EXECUTED, not modelled. Only the CUDA lane mapping needs metal. A mutant that bypasses the ring save kills 13 of the tests, so the suite detects the corruption it is written against. GATES (foreground, family-batched, PYTHONPATH pinned to this worktree): model_executor 15 failed / 803 passed (baseline 15 / 780) -> 0 new managers batch A (240 f) 2640 passed, 1 failed + 1 collect error, both identical at HEAD managers batch B (63 PP files, one process each) 684 passed, 0 failed mem_cache (chunked) 31 failed / 2108 passed, byte-identical to HEAD; one chunk segfaults at HEAD too (pre-existing) server_args 4 failed / 793 passed, the same 4 at HEAD new suites 48 passed (rotation executor 23, plan 14, wave 11) ZERO new failures. ruff F/E7/E9 clean on new files; the two large touched files are 358 -> 358, unchanged. NOT CLAIMED: nothing here has run on a GPU. The boot-side change that allocates ONE max-sized host image instead of two, and the wiring of the executor into PhaseFlipStacks.refill, are the remaining steps before the proof window.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 15, 2026
…h groups (256 page buffers) Boot xsn129 (2eb120d, write_back on both groups): the round trip held (P leg 1 -> publish sweep -> D leg 2 cached 4314/4316 and 11806/11808; extent raised to the anchor: kv=4031 extent=4095, OFF-EXTENT 0; pin-skips 0, anchor drops 0), the flips ran 1.8-2.0 s sleep / 2.0-2.5 s wake -- and then D's leg-2 store reads for the two parallel 11k requests came back short: 'sgl-project#1157 PREFETCH REAPED requested_pages=11126 hit_pages=11126 completed=5585 elapsed=12.87s' -- 100-430 pages per second against a raw file-read rate of 15k files/s (measured on the same store, ZFS ARC-warm). The per-page cost is not the disk and not the filesystem: with SGLANG_HICACHE_READ_BUFFERS=0 (default, the launcher never set it) every `_read_page` borrows `host_pool.get_dummy_flat_data_page()`, a FRESH `torch.zeros(..., pin_memory=True)` -- cudaHostAlloc + cudaFreeHost per page, under the driver lock the decode loop shares. The sgl-project#720 ring (read_buffer_pool.py) allocates once and reuses; it was simply off. build_env now sets SGLANG_HICACHE_READ_BUFFERS=256 unless the operator set it. Tests: rotation/flip-instrument/refill (read_buffer_pool users) 31 passed. Metal: xsn130.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 15, 2026
…ce probe, ring buffers and one indexed host write per batch for KV and draft pages, preadv
Boots xsn129-132: the D group read a store prefix at 250-430 pages/s per
rank (11k pages -> 13 s, past the 13.5 s reap budget: completed=2680 of
11806 on xsn132), so every leg-2 request parked, the box idled, and the
W1b abort had to end the drain. py-spy on D TP0 during the in-flip read
of xsn132 (profile in the job dir, 2663 samples):
prefetch issuer thread: 691/710 samples in batch_exists_v2, 533 of them
in os.path.exists -- the probe of one stem was exists(sharded) +
exists(flat) + exists(sharded) + getsize, and an 11k prefix probes
33k stems (KV, mamba, draft). Measured on the store: 11.5 us a stem
as written, 2.7 us as one os.stat.
prefetch IO thread: ~half in set_from_flat_data_page (2*layer_num
strided slivers per 32 KiB page), the rest split between a fresh
pinned get_dummy_flat_data_page() per page (the sgl-project#720 ring served only
the extra-pool route, never the KV/draft pages), read_extents' pread
alloc+copy, _existing_path's second exists, and the evictor touch.
The disk itself: 6-7 us a warm 32 KiB page (readinto/preadv measured).
Changes, all on the same read path:
hicache_storage._stat_stem: ONE os.stat answers presence, path and
size; the legacy flat layout is consulted only when the top directory
holds flat .bin files (decided once per backend). _stem_exists,
_existing_path and _stem_readable are its three views.
cache_controller: _generic_page_get and _draft_page_get_generic borrow
their targets from the backend's read ring (_borrow_read_pages, module
level so the harness doubles that bind curated methods keep working);
a draft miss zeroes its borrowed buffer (sgl-project#993 stays: the zero page is
the miss); the served prefix of a batch is written with ONE indexed
copy (_set_host_pages -> set_from_flat_data_pages).
pool_host/mha.py: set_from_flat_data_pages for layer_first/page_first
(one index assignment over the token axis); base.py default loop;
memory_pool_host's bound-tier wrapper keeps the per-index stray check.
canonical_page_store.read_extents: preadv straight into the target.
Tests: test_read_ring_prefix_routes_1402 (6: ring borrow/return, dirty
buffer -> zero page, no-ring fallback, raised read returns buffers,
batched write == per-page loop for both layouts x page_size 1/2, served
prefix written once and stops at the miss) + sharding_558, canonical
1233/706, mem_pool_host, stray_718, draftkv, reissue_939, 1157, 869b,
905, 937, 966, 1063, 861, 0828, 1324, 1401: 221 passed, 7 failed -- all
seven red at the tip without this change (3 mem_pool_host + 1 in 905 +
2 in 1063 + the 0828 text scan that trips on a 2026-09-07 comment).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Fix #548
Modification
Add a custom cache manager.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.