Bump version to 0.1.24 by Ying1123 · Pull Request #718 · sgl-project/sglang

Ying1123 · 2024-07-24T22:54:55Z

No description provided.

fix 0608

… it is not bound to REACHABILITY VERDICT FIRST: LATENT, not live -- and it stops being latent on exactly the boot sgl-project#703/sgl-project#706 are aiming at. `HiCacheController` exists only when a hierarchical cache is built, and today's serving flagset (evidence-665-f1/argv_*.txt) carries --enable-phase-flip with its whole family and NO hicache flags, so no controller is constructed and none of this I/O is reachable on the live line. What makes it reachable is the combination flip + hierarchical cache, which nothing refuses any more: the sgl-project#630 blocker was deliberately removed from both the boot-time and the runtime guard lists so that a prefix cache could ride the flip. That is the intended configuration of the next boot. FAILURE SHAPE: silent corruption, BOTH directions -- not a dead cache. `cache_controller.__init__` captures the device pool once (`token_to_kv_pool_allocator.get_kvcache()`), the scheduler's allocator is likewise assigned once, and the cutover reassigns neither. Both transfers name the captured object directly: * WRITE (`backup_from_device_all_layer(self.mem_pool_device, ...)`, :866) during the TP phase reads the PP pool's rows -- the stale cutover snapshot at best, unbacked pages under the flip's VA-backed pools at worst -- and those bytes are keyed by TOKEN CONTENT and, under sgl-project#706, persisted to disk. A page claiming tokens it does not hold is a wrong ANSWER later, in either phase, and it survives a reboot. * LOAD (`load_to_device_per_layer(self.mem_pool_device, ...)`, :994) fills rows in the PP pool that the model does not read, while the radix tree reports the prefix resident -- so the scheduler skips recomputing it and attention reads rows nobody filled. Also a wrong answer, also silent. THE CUT (minimal, per the fix-direction call): refuse device-tier I/O while the flip routes to its TP stack. A refused write is a prefix not cached; a refused load is a prefetch that does not land. Misses are the correct failure here and they are cheap; both alternatives are wrong output. The host and storage tiers are untouched, and the PP phase resumes normal staging after the next flip -- which is also why sgl-project#703's flip-time writeback (running at the seam, in the bound phase) composes with this rather than being blocked by it. NO FLAG, deliberately: `phase_flip_tp_routing_active()` is False whenever the flip's secondary groups were never built, so every deployment without --enable-phase-flip is byte-identical by construction and cannot enter the guarded state. An off switch would exist only to restore the corrupting behaviour. The predicate encodes "the binding belongs to the boot phase" in ONE place, so the real fix (rebinding the controller's pool at the cutover, which touches a once-assigned binding) has a single site to update. Tests (hermetic, CUDA_VISIBLE_DEVICES=""), 11 new: * test_hicache_phase_guard_718.py -- the guard is invisible when the flip is not routing (proved by an EXPLODING pool that the real prologue reaches); both directions are refused when it is; the refusal happens BEFORE any pool is touched (the executable form of "it does not copy against the wrong pool"); each direction warns once and names its own hazard; an unreadable phase module fails OPEN, since disarming every deployment on an import error would be a bigger outage than the hazard. * TestReachability -- the premise, pinned so it cannot rot: hierarchical cache is off by default, and flip + hierarchical cache is ACCEPTED. If a refusal is ever added back, that test says the guard's premise changed. Regression, same env, base c3e9487 vs this commit: unit/mem_cache 940 failed / 779 passed -> 940 failed / 873 passed (+94) unit/managers 4 failed / 2145 passed -> 4 failed / 2145 passed cache_controller.py diff: 14 insertions, 0 deletions. NOT done here, and named: rebinding the controller to the active phase's pool is the real fix and would make the device tier usable in BOTH phases instead of one. It touches the once-assigned binding that three other subsystems read, so it wants its own slice and its own flag. CAN-FAIL PROOF (mutation applied, suite re-run, reverted): P1 guard removed from write() -> test_write_is_refused_while_the_flip_routes_to_tp, test_the_refusal_happens_before_any_pool_is_touched P2 guard removed from load() -> test_load_is_refused_while_the_flip_routes_to_tp, +1 P3 predicate never disarms -> 5 failures, i.e. every claim the guard makes P4 warns on every operation (the once-set bypassed) -> NOTHING FAILED at first. Real test gap: the once-test counted the FIRST call's output and then made a second call whose silence it never asserted, so a warn-every-time regression would have passed. Closed with assertNoLogs on the repeat call, for both directions; the mutation now fails test_each_direction_warns_once. P5 fails CLOSED on an unreadable phase module -> NOTHING FAILED at first, and for a worse reason: the test named the import-failure path but never exercised it -- it called the real predicate, which returns False here because no flip groups exist. So it was pinning a different fact under a misleading name. Split in two: the honest one (no flip groups -> not disarming) keeps its assertion under an accurate name, and a new test replaces the phase module with one that cannot supply the predicate, which is what an import failure looks like from inside the guard. The mutation now fails test_an_unreadable_phase_module_fails_OPEN. Restored tree re-verified green after every mutation (12 passed).

… nothing to register Spec note answered for the sgl-project#706 store, with the reasoning recorded rather than the conclusion alone. DECISION: the canonical store allocates NO host memory of its own, so it registers nothing with the process-wide pinned budget (pinned_host_budget.check_and_register_pinned_post; today's consumers are the kvso host pool, the HiCache host pool and the phase-flip weight images). Both directions take CALLER-OWNED buffers -- write_extents reads the payload it is handed, read_extents fills the target it is handed -- and the file I/O is os.pwrite / os.pread against those. There is no steady-state buffer of the store's to account for, so "register at attach/resize" does not apply here rather than having been overlooked. Pinned by CONTRACT, which is the part that can rot: TestNoStoreOwnedBuffers asserts the read fills the caller's tensor and the write does not relocate it. A future change that gave the store its own staging buffer breaks those two tests, and that is precisely the change that would have to register. FLAGGED FOR WHOEVER OWNS sgl-project#550, and NOT fixed here because it is not ours: the file backend takes its read target from host_pool.get_dummy_flat_data_page(), which allocates a fresh PINNED tensor PER READ on both pool families (pin_memory=self.pin_memory -- pool_host/mha.py:471, memory_pool_host.py:563). It predates this work and the joint budget cannot see it. It scales with concurrent reads rather than with tier size, so it is a per-operation spike rather than a missing pool -- but on a prefetch-heavy path it is real pinned memory nobody counted, and the budget refuses over-commitment on numbers that do not include it. Recorded in evidence-665-f1/PLAN_PERF_PIPELINE_2026-08-16.md. Tests: 32 in the store suite (2 new), 119 across the sgl-project#706/sgl-project#703/sgl-project#718 family.

…t the cutover; bound the read spike sgl-project#719 -- THE REBIND, and the finding that shapes it. A REBIND IS NOT A POINTER SWAP. The host pool is CONSTRUCTED FROM the device pool (hybrid_pool_assembler.build_kv_host_pool(kv_pool=...)), so its layer_num and buffer sizes are that phase's. The two phases have different per-rank layer counts -- a PP stage holds 7/5/4 of the 16 attention layers, the TP stack holds all 16 -- so repointing the controller at the other phase's device pool while its host pool still describes this phase's is a NEW corruption wearing the fix's clothes: matching row ids, mismatched widths, and the copy RUNS. The rebind therefore refuses unless handed a host pool whose shape matches the incoming device pool, and phase_pools_for refuses when the boot did not build one -- which is today's real state, reported with its reason instead of proceeding onto the wrong pool. Supplying that second host pool is a boot-time question (it costs host RAM, the binding constraint per DESIGN_706 C1) and is named, not mocked away. THREE READERS, ALL OR NONE. The pool identity is captured in three places (controller: mem_pool_device / _hybrid / _allocator; radix cache: hybrid_kv_cache / kvcache / host pool; scheduler: token_to_kv_pool_allocator). A rebind that moves some and not others is strictly worse than none: the readers then name different memory for the same row id and EVERY CALL STILL SUCCEEDS. So the rebind stamps a generation on each reader and a coherence check compares them afterwards -- the failure is invisible to inspection, so it is verified by counter. A stamp that fails mid-way rolls the state to a generation nothing can match, so the tier cannot re-arm onto a torn binding. THE PAYOFF, wired through sgl-project#718's predicate: "disarmed" now asks whether the ACTIVE phase is the BOUND phase, not whether TP is active. With no rebind those are the same question (the binding is always the boot phase), so the default path is byte-identical -- sgl-project#718's 12 tests pass unchanged. After a coherent rebind the device tier is usable in the phase it moved to. Call site: after the stack swap in _cutover (the mirror of sgl-project#703's writeback, which runs before anything moves because it reads the OUTGOING pools). Refusal is logged, not raised: a refused rebind is SAFE by construction because the binding does not move and sgl-project#718 keeps the tier disarmed, while a raise at the seam takes down an instance that was serving fine. sgl-project#720 -- THE READ SPIKE, bounded and charged. Every storage read took its target from host_pool.get_dummy_flat_data_page(), a fresh pinned tensor per read that the joint budget cannot see (it accounts pools declared at attach; this is neither). ReadBufferPool is a fixed ring, allocated once, declared through check_and_register_pinned_post and unregistered on close. Exhaustion falls back to a fresh allocation -- today's behaviour, counted -- rather than blocking, because stalling a prefetch worker to save memory trades a bounded spike for unbounded latency. SGLANG_HICACHE_READ_BUFFERS=0 (default) keeps the current path exactly. Tests (hermetic, CUDA_VISIBLE_DEVICES=""), 18 new, 137 in the family: * test_hicache_rebind_719.py (11) -- all three readers move together; a reader left behind is CAUGHT (planted deliberately, since without generations that state is invisible); an absent reader refuses the whole rebind; a failing stamp leaves the set unusable rather than half-moved; shape mismatch and unmeasurable shapes refused; a phase with no host pool refused WITH the reason; the disarm lifts only after a coherent rebind and re-arms on the return leg; unarmed is byte-identical. * test_read_buffer_pool_720.py (7) -- the falsifier pair (25 reads = 25 allocations today, 4 with a ring of 4), bounded overflow, a raised read still returns its buffer, the registry sees the ring and stops seeing it on close, off by default. Regression, same env, base c3e9487 vs this commit: unit/mem_cache 940 failed / 779 passed -> 940 failed / 894 passed (+115) unit/server_args 1 failed / 626 passed -> 1 failed / 639 passed (+13) unit/managers 4 failed / 2145 passed -> 4 failed / 2145 passed scheduler/test_phase_flip_runtime.py: 67 passed (the file the call site edits) phase_flip_runtime.py diff: 23 insertions, 0 deletions. Live validation of both joins F4-r4's window list; the flip+hicache boot they need is exactly the boot sgl-project#719 makes safe -- and sgl-project#719 cannot arm until that boot also builds the second phase's host pool. CAN-FAIL PROOF (mutation applied, suite re-run, reverted): Q1 coherence check always passes -> test_a_reader_left_behind_is_caught, test_a_failing_reader_leaves_the_set_unusable_not_half_moved Q2 shape check removed (the pointer-swap trap) -> test_shape_mismatch_is_refused, and ONLY that one Q3 partial rebind allowed (absent readers ignored) -> test_an_absent_reader_refuses_the_whole_rebind, and ONLY that one Q4 predicate ignores the binding (reverts to the raw TP-active test) -> 6 failures spanning BOTH suites, incl. test_disarm_lifts_only_after_a_coherent_rebind, test_the_return_leg_rebinds_back, and sgl-project#718's own test_disarmed_is_false_without_the_flip -- i.e. the tie between the two features is load-bearing in both directions Q5 ring never reuses (release drops every buffer) -> test_green_the_ring_allocates_once_and_reuses, test_concurrent_borrows_beyond_the_ring_fall_back, test_a_raised_read_still_returns_its_buffer Q6 ring not declared to the pinned registry -> test_the_ring_is_declared_to_the_pinned_registry, and ONLY that one Restored tree re-verified green after every mutation (30 passed).

…the first Flip+HiCache boot Everything mechanical is landed (canonical format 04c6736, GDN blob a38f39f, sharded backend 19f4c68, cutover rebind ec117fa, ReadBufferPool, sgl-project#718 disarm). This is the boot that composes them, and its one open cost, priced. THE OPEN COST, stated first and found by inspection: sgl-project#719's rebind refuses unless the incoming phase owns a shape-matched host pool, and that pool HAS NO BUILDER -- phase_pools_for reads scheduler.phase_flip_host_pools (hicache_phase_binding.py:287) and nothing in the tree writes it. So the rebind refuses at every cutover today, logged and never raised, with sgl-project#718 keeping the device tier disarmed in the phase that did not build the binding. That is a safe state, and the design recommends booting IN it. HOST-RAM BUDGET, from the deployed [32,16,16] cut. A host row is that rank's OWN layers x 2048 B, so PP rows are 16,384 / 8,192 / 8,192 B while a TP row is 32,768 B on every rank -- multipliers 2.00x / 4.00x / 4.00x, i.e. 16/own_layers. Against the MEASURED pinned load (C1 at 591add2: 25.87 GB usable, 20.50 GB of flip weight images, 5.37 GB remainder): rows fitting in 5.37 GB rank0 rank1/2 PP pool only (today) 327,759 655,518 BOTH pools 109,253 131,104 So the second pool cuts the host tier to a third on rank0 and a fifth on ranks 1-2 at fixed budget; at the 9.01 GB the C1 boot actually requested it needs ~27 GB on rank0's ratio against 5.37 available -- it does not fit, and not marginally. WHICH INSTRUMENT BINDS, because reasoning from free -g gives the wrong answer: the PINNED budget refuses first (it already refused 29.51 GB at C1), while the sgl-project#721 available floor of 24.3 G has ~12.7 G of slack today (available 37 G). A second host pool is refused by the pinned check long before it threatens the OOM floor. MARKED ABSENT rather than estimated: whether the 9.01 GB host-pool figure is one rank's or an aggregate, and the per-rank split of the three weight images. The refusal message does not say and no ledger entry resolves it, so every number is given per rank AND against the shared 5.37 GB remainder, so the conclusion does not depend on the ambiguity. RECOMMENDATION: the first boot does NOT add the second pool. It does not fit, and the cross-phase path does not need it -- sgl-project#706 made the DISK tier geometry-neutral, sgl-project#703 pushes warm prefixes there before the cutover, and both phases resolve the same content key (100 GB disk = 3,051,758 canonical-page tokens against a 5.37 GB host staging tier). The safe states are the DEFAULT states, so nothing has to be remembered. Also in the document: the exact flag set and env (including why --phase-flip-rebind-hicache is deliberately NOT set), the boot-before-hold sequence with the two log lines whose ABSENCE is the stop condition, the sgl-project#630 PP=3 x disk-HiCache warmup wedge with its root fix (9da9dfd) and what to watch anyway, acceptance (byte-identical cross-phase continuation with A-vs-A first -- the sgl-project#718 shape is a WRONG ANSWER, not a miss; hits counted from log lines because cache_hit_rate reads 0.0 with real hits; ReadBufferPool overflow counter zero; the sgl-project#720 falsifier pair), and the ranked risk list. Design only. No boot, no GPU. The boot goes on the window list. Symlinked beside the other DESIGN_706 docs in evidence-665-f1/.

…t is a build Ordered step (1) says read the refusal's reason before touching it. Read, with blame: the answer is a third option, neither of the two the brief anticipated. NOT a sgl-project#718-class safety guard. git blame puts it in 05933d0 (2026-07-13), the flag's OWN introducing commit, listed among ordinary validations ("hierarchical/unified radix tree rejected") with no measured failure behind it. NOT merely never-composed either. The unified mamba component contains no reference to mamba_checkpoint_interval, is_on_interval or mamba_track_interval, while MambaRadixCache consults the grid in six places. So composing the flags today would corrupt nothing and would leave --mamba-checkpoint-interval SILENTLY IGNORED while its help promises deterministic absolute-multiple positions. That is the sgl-project#742 class exactly -- and lifting the guard would replace an honest refusal with a dishonest acceptance, which is worse than leaving it. The stated reason is also aimed at the wrong class: it blames HiMambaRadixCache, which per NOTE_745 has no construction site (registry.py:107-112 routes hybrid-SSM + hicache to the unified tree). Second time that aiming error has surfaced in this area. Scoped instead of forced: the lift is a mirror of the grid at five seams of the unified component (match validator/finalize, commit_insert, prepare_for_caching, split redistribute, evict/drive_eviction), each named with the mamba_radix_cache.py line it mirrors. The eviction seam is the subtle one and it gets EASIER under the unified tree. With the interval set, MambaRadixCache spares the deepest anchors per path because "losing the deepest one silently moves the resume point of identical requests and re-introduces run-to-run drift" (mamba_radix_cache.py:1092-1097) -- a determinism contract that exists because a spilled anchor is a DEAD anchor on a device-only pool. With a host tier an evicted anchor is still a valid match and loads back (mamba_component.py:71-74, :139-144), so the protection window can be weaker here, not stronger. That is exactly the directive's goal: 8k deterministic anchors that survive on disk. Recommended shape recorded for when it is built: cadence 8192 (16 x chunked_prefill_size), anchors host-tier-eligible with no separate pin class, red-first with a hermetic construction smoke plus an anchor-reaches-host-tier test, and the refusal deleted only in the same commit that makes the grid effective. Method precedent: sgl-project#547 -> sgl-project#550, reading a describing-not-reasoning refusal against the tree to tell impossibility from unbuilt. Docs only; nothing boot-verified. Post-boot branch, not for the closing train.

THE CHECK WAS IN THE WRONG PLACE, NOT MISSING. write() already refuses at ENQUEUE via the sgl-project#718 device-tier disarm. The copy happens later, and the flip rebinds in between, so a write-back queued before a cutover reaches backup_from_device_all_layer carrying a pointer table into the pool it was built from. Both crash specimens died three seconds AFTER a pp_to_tp cutover completed: 14:08:14 -> 14:08:17 (epoch 27) and 07:12:09 -> 07:12:12 (epoch 3), seven hours apart, same direction, same lag. WHY THE SHAPE GUARD COULD NEVER CATCH IT, and why its silence was misread as innocence -- by me, until the operator pushed back. Under layer_first the host layout EQUALS the device layout, so a stale binding is shape-IDENTICAL to the live one and check_shapes passes by construction. That is exactly what sgl-project#760 recorded: KV-TRANSFER-GUARD armed on all three ranks, zero transfers refused, SIGSEGV anyway. Matching shapes plus a crash puts the fault below the Python seam; a generation stamp is what tells "same shape" from "same pool". MEASURED, ONE VARIABLE. HiCache host+disk under sustained 4-way load with the flip REMOVED (plain TP3, which keeps speculation on -- PP+spec requires the flip, server_args.py:18385) survives five minutes with 0 segfaults and 0 admission wedges, and passes the REP gate outright (1 distinct/12, salted 0/6). The fault needs the cutover. STAMPED BY CONSTRUCTION, VERIFIED AT CONSUME. The stamp lives in CacheOperation.__init__ rather than at one enqueue site: an op built by any other path would otherwise be unstamped, and an unstamped op must be refused, which silently dropped legitimate write-backs and broke the staged-dispatch tests until I moved it. A stale op is dropped loudly and counted; its prefix simply misses later, the same cheap failure the sgl-project#718 disarm already accepts. Also folded in, both consequences of earlier sgl-project#767 work rather than new choices: the anchor-protection test that encoded "no interval means no anchors" is updated to the corrected premise it was measured against, and the SECOND-PASS EVICTION line drops from warning to info -- the second pass is documented as legitimate, and anchor eviction was falsified as the drift cause, so it is accounting rather than an alarm. 4 new tests, red-first (a pre-rebind stamp must be refused). mem_cache and managers suites: 4047 passed, 43 failed -- all 43 the pre-existing sgl-project#772 class (PhasePolicyConfig lacks idle_locked_settle_s after the sgl-project#713 revert), one fewer than the 44 baseline because the 747 premise test is now correct. ruff clean on both touched files (0 at HEAD, 0 now).

…nsumed THE ENQUEUE ANSWER WAS RIGHT; NOTHING ASKED AGAIN. write() calls device_tier_disarmed("write") and correctly gets False: the copy is queued while the model computes in PP, which IS the phase these pools are bound to, so queueing it is legitimate. start_writing() then consumes it later, and the cutover lands in between -- both crash specimens died three seconds AFTER a pp_to_tp cutover completed (14:08:14 -> 14:08:17 epoch 27; 07:12:09 -> 07:12:12 epoch 3, seven hours apart, same direction, same lag). WHY THE GENERATION STAMP COULD NOT COVER IT ALONE, which the previous commit assumed it would: with --phase-flip-rebind-hicache off, binding_state() never advances, so every stamp matches by construction and the check is dead code -- measured as 0 write-back refusals on a boot that still took 2 SIGSEGVs. The phase predicate is the one that already knows the answer; it only had to be asked a second time, at the moment the device indices are actually read. Refusing costs a cache MISS later, the same cheap failure the sgl-project#718 disarm already accepts at enqueue. Counted and named so the cost stays answerable. NOT VERIFIED ON METAL. This edit was written but never booted -- the task moves to a successor strand. HANDOVER_760.md carries the repro, the anchors and the ranked hypotheses. ruff: 1 pre-existing F541 in this file at HEAD and after, none added.

…at outlive their phase TWO HOLES, ONE WINDOW. The sgl-project#718 guard read the parallel_state routing global, which is toggled INSIDE the cutover -- one step among many -- so for the whole seam (waves moving KV rows, movers releasing the outgoing backing, the cutover rebuilding topology) the guard named a phase while pool bytes were in motion. And even a perfect predicate cannot recall a copy that is already riding the controller's private CUDA streams: write() and start_writing() run in the same Python instant (start_writing is called synchronously from write, its only caller), so the previous commit's consume-time re-check re-asks the question at the same moment it was first asked. The torn window is the STREAM's asynchrony, not the queue's: a device->host copy enqueued legitimately in PP outlives its Python call by seconds under load, and the seam releases the pool under it. Both crash specimens died exactly there -- 3 s after a pp_to_tp cutover, inside backup_from_device_all_layer, below the Python seam. THE FIX, in the two halves the window has: 1. AUTHORITY. PhaseFlipRuntime registers itself (weakly) as the phase guard's authority. Its _phase field is what the PHASE-FLIP DONE line reports -- truthful by the 3 s crash correlation -- and it alone knows the seam's extent: hicache_seam_active is raised at the no-return point (after the unanimous-abandon verdict, before the first wave) and cleared after the cutover installs the new phase, with a finally in the caller as insurance. During the seam the guard refuses device-tier I/O for EVERY binding. Outside it, the authority's phase wins over the routing global, and a disagreement logs the sgl-project#754-shape instrument line that settles whether that global was ever stale here. No authority registered (no flip runtime built) falls back to the routing global: non-flipping deployments stay byte-identical. 2. QUIESCE. At the same no-return point the runtime drains the controller's write_stream and load_stream while every pointer they hold still names live memory. Finishing those copies is correct (they become durable cache entries) and bounded (PCIe transfer of the backlog; this thread is the only device-tier producer, so nothing refills behind the drain; rank-local, so it cannot wedge the group). Ordered after the sgl-project#703 flip-writeback hook, whose staging copies are the largest legitimate producer of exactly such in-flight work. Tests: test_flip_seam_guard_760.py, 10 tests, red-first (all 10 fail on the parent commit: seam disarm and authority-wins are impossible there, quiesce/_quiesce_hicache do not exist). With the fix: 14/14 green including the sgl-project#760 stamp tests. Full mem_cache+managers sweep: 4057 passed, 43 failed -- the 43 are the pre-existing sgl-project#772 class (PhasePolicyConfig lacks idle_locked_settle_s), zero regressions, +10 passed vs the parent's 4047. ruff clean on the new code (the 13 E402 in cache_controller.py pre-exist on HEAD); codespell clean. Metal gate next: the 2-SIGSEGV repro arm must hold >=2 cutovers under 4-way load with 0 segfaults and logged seam refusals.

…-- the hole the crash went through THE METAL FALSIFIED THE PREVIOUS COMMIT'S SUFFICIENCY IN 40 SECONDS, and the faulthandler stack it produced is the whole finding. seamfix1 (52df0ff, repro arm, 4-way load) registered the authority on all three ranks, quiesced at two seams, completed pp_to_tp epoch 3 -- and segfaulted seconds later in the TP phase, through a stack the guards never see: unified_radix_cache.cache_finished_req -> insert -> _inc_hit_count -> write_backup -> hybrid_cache/hybrid_cache_controller.write -> start_writing -> backup_from_device_all_layer -> transfer_kv_direct This deployment's tree cache is UnifiedRadixCache driving HybridCacheController -- which INHERITS from HiCacheController (so the seam quiesce worked, it is inherited) but OVERRIDES write() and load() without the device_tier_disarmed checks the base methods carry. Every sgl-project#718/sgl-project#760 metal reading of 'zero disarm hits' on this stack was therefore vacuous: the guarded methods never executed; TP-phase inserts enqueued copies against the PP-bound pools unchecked, and one of them walked released backing. The overrides now ask the guard first, before any pool is touched, with the base contract: refuse -> return None -> the caller books a miss (write_backup returns 0, load-back returns False -- both verified None-tolerant). Tests: two red-first additions to test_flip_seam_guard_760.py pin both overrides (an alloc-must-not-run pool proves the guard runs FIRST); both fail on the parent commit, 16/16 green with the fix. Full mem_cache+managers sweep: 4059 passed, 43 failed -- the pre-existing sgl-project#772 class only, zero regressions. ruff and codespell clean. Metal gate re-run next on this commit: >=2 cutovers under 4-way load, 0 SIGSEGV, and the first genuinely non-zero disarm/refusal counters on this stack.

…t inventory USER DECISION 2026-08-24, binding, verbatim: "das kv soll niemals vom layer flip her stammen, einfach aus dem hicache laden fertig." Recorded BEFORE the build so the build cannot quietly become something else, and so the ledger that follows reads as validation rather than as a vote. THE TREE ALREADY AGREED, which is the part worth landing on its own. hicache_flip_writeback.py:21-23 states the premise in its own words: "a prefix's only way across the flip is the geometry-free STORE (sgl-project#706): the disk tier, whose keys carry content alone and whose pages are cut at read time for whichever geometry asks." And it names exactly why a mover exists today: "device rows survive the flip, because the live row set (radix tree values UNION parked requests' rows) is relocated between the two phase pools BY ROW ID." That relocation IS the wave mover. THE BLOCKER THAT SHAPES THE BUILD, found before writing any code. phase_flip_resident_carry.py:64-76 -- a carried Req keeps its req_pool_idx across the swap "by construction", and the reason it stays valid is that "the bytes behind those ids are what the KV and GDN movers relocate". So PHASE-FLIP-CARRY is NOT a KV mover and NOT a retirement candidate, but its correctness today DEPENDS on the mover. Retire the mover alone and a resident request's req_pool_idx points at unwritten memory. The fence and the retirement must therefore land together, and the cutover must leave the new phase's device tier in a state where a lookup MISSES rather than returning stale rows. That is the correctness core, and it is where the red-first tests must bite hardest. WHY THE PRIZE IS FUNDING, NOT LATENCY -- stated plainly because the opposite is the natural assumption. The whole KV+GDN movement is 901 ms of an 11.6 s seam (W25 epoch 11, 116502 live slots). Removing it leaves ~10.5 s. What it DOES remove is wave_peak = incoming + max(outgoing, local) + one_layer_window + backing_slack (phase_flip_runtime.py:7314) -- every term a KV quantity -- which is the 2339.11 MiB tp_to_pp staging reserve behind W25's 33 refused arms, 25 of them on the staging rate limit, and 17 FLIP ABANDONED. RETIREMENT INVENTORY, scoped to the flip path, REPLACE/RETIRE/KEEP-WITH-REASON with file:line, in the note. Rules held to: "hardened against corruption" does not count as reconciled; no bulk deletion; shared machinery with a named other consumer is KEPT (kv_reshard's sgl-project#297 domain, gdn_flip_preconditions -- whose "no other consumer found" is recorded WITH the narrow search set that failed, not as a licence to delete); anything merely deletable-later is a separate section and marked un-re-verified. EXISTING MACHINERY TO EXTEND, NOT REBUILD: hicache_flip_writeback.py already IS the fence (stage + bounded ack drain, deadline 2.0 s); hicache_demotion.py already covers evict-before-persist WITH counters and is merely off by default; sgl-project#719 rebind + sgl-project#718 disarm already own the read-path switch; mamba_ckpt_utils' anchor grid is already what anchor-resume needs. NO FALLBACKS, per standing doctrine: every gap is fixed inside the HiCache route. A flip enabled without hierarchical cache becomes a validate-early launch refusal on the sgl-project#806 precedent (c0a6347, ServerArgs.__post_init__ after materialize_declarations) -- NOT a silent mover revival. That refusal is deliberately NOT landed yet: it only becomes true once the flip carries no KV, and shipping it early would reject launches that work correctly today. Forced build order, recorded rather than discovered later. VALIDATION METRIC CHANGED, and one half of it does not exist yet: cutover- blocking time (fence + weights refill) can reuse the existing seam_census.mark("flip_writeback") and the DONE stats dict, but there is NO instrument for post-cutover warm-up cost as served-request latency -- searched and named as a build item rather than assumed present. OPEN GAPS ARE LISTED AS GAPS, with the search sets that failed: sgl-project#735's "sgl-project#706 rows on the full plan" could not be located (both greps resolve to a different topic -- non-contiguous PP placement), fp8 kv_cache_dtype against the canonical page format is unverified, and the worst-case un-hashed resident tail at the quiescent-flip instant is reasoned from the quiescence predicate rather than measured. No code changes. Documentation only.

…roject#718 rebind can arm MECHANISMUS VORHANDEN, AKTUATOR FEHLT. The whole sgl-project#718 rebind chain already existed and was already wired: `rebind_for_cutover` is called at the cutover, the sgl-project#719 generation stamp and `coherence_check` are built, and `phase_pools_for` knows exactly what it wants. It wanted `scheduler.phase_flip_host_pools[phase]` -- and across the entire tree that name appeared ONLY in its own docstring and its own refusal message. Nothing ever wrote it, so the rebind could never arm. W32 measured the consequence end to end: no host pool -> RebindRefused -> the rebind never arms -> bound_phase() stays "pp" -> device_tier_disarmed("load") is True for the whole TP phase -> HiCacheController.load() returns None -> ZERO tokens reach the device. The one transport prefill logged `#cached-token: 0` on what should have been a perfect disk hit, beside 6 `sgl-project#718 hicache-phase-guard` warnings. `build_phase_flip_host_pools` runs in `init_model_worker`, right after `build_phase_flip_tp_stack` -- the first point where BOTH device pools exist, which is required because a host pool is allocated FROM its device pool (DESIGN_706 C1) and cannot be derived after the fact. FLAG-GATED: without --phase-flip-rebind-hicache it returns {} and allocates nothing, so every other boot is byte-identical. A STAGING PIN, NOT A MIRROR (sgl-project#810). `pp` maps to the tier the boot already built; the rebind needs a HANDLE per phase, not a second pp pool. Only `tp` is new and it is sized to the WORK, never to the pool: chunked_prefill_size x max_running_requests x PHASE_FLIP_STAGING_CHUNKS tokens times the pool's own measured per-token cell. `ratio=0` is passed explicitly, because a ratio is the mirror-shaped answer sgl-project#810 forbids -- it would duplicate retention the pp tier already provides and charge the pinned host budget for capacity nothing reads. Pinned by a test: doubling the device pool changes the pin not at all; doubling in-flight work doubles it. HOST-LEDGER POST (sgl-project#721): the GB taken, the derived GiB, the token count it came from and host free after, logged AT the allocation so the ledger carries the number actually taken rather than an intention. The POST shrinks if it does not fit; the FLOOR never does. REFUSAL CONVERTED, NOT DELETED (sgl-project#847). `phase_pools_for` still raises for a genuinely absent or mis-shaped pool, pinned against the REAL guard rather than a restatement: no TP device pool -> no `tp` entry -> the cutover refuses, exactly as in W32. A constructor that throws is caught, reported loudly, and leaves the phase unbound rather than taking the boot down. TESTS (13): default boot untouched; both phases registered; the pin comes from the TP device pool; staging-not-mirror in both directions; and the can-fail set -- no host tier, no TP pool, the real guard still raising, and a throwing constructor. Clean SERIAL gate (one suite at a time): 8 failed / 4071 passed / 2 skipped -- the same pre-existing 8, +13 matching the tests added here. METHOD CORRECTION: my earlier per-file "ruff delta vs HEAD" checks compared a copy under /tmp, where ruff resolves a DIFFERENT config, so those readings were invalid. Re-done in place: the single F401 in phase_flip_boot.py is pre-existing and this change adds zero.

…, not before it W33 arm 1: the flag parsed (`phase_flip_rebind_hicache=True`), the writer ran, and it logged its OWN refusal on every rank -- sgl-project#847 PHASE-FLIP REBIND: --phase-flip-rebind-hicache is set but this boot has no HiCache host tier, so there is nothing to build a phase-matched pin from. -- followed by 6 `sgl-project#718 hicache-phase-guard` warnings, i.e. the W32 read- through miss reproduced with its fix installed and unreachable. Third time in this strand that a correct mechanism was placed where it cannot run (W31 arm 1 below the drain gate, W32 the policy's second copy, this). THE INPUTS ARE THREE, NOT TWO. I placed the writer inside `init_model_worker` because a host pool is allocated FROM its device pool and both device pools are ready there. But it also needs the HOST tier, and that hangs off `self.tree_cache`, which `__init__` assigns AFTER `init_model_worker()` has returned. So the writer ran before one of its inputs existed. This constructor already carried that lesson: the sgl-project#677 note sitting four lines below records the identical mistake made once before, when a call was put "beside init_admission_limiter" and ran before `req_to_token_pool` existed. Both now sit after their inputs. The ordering is pinned in both directions: the call must appear in `__init__`, and must NOT appear in `init_model_worker`. 13 tests green.

W33 arm 2: the flag parsed, no RebindRefused was raised, and the rebind still never armed -- because the writer refused itself again, for a second reason. It read `getattr(tree, "token_to_kv_pool_host", None)`. That attribute belongs to `HiRadixCache`. The tree this box runs is `UnifiedRadixCache` -- `tree_type=UnifiedRadixCache` in every census line of every window -- which does not have it at all and reaches the host tier through `cache_controller.mem_pool_host`. Zero occurrences of the attribute in that class. So the read returned None on the live tree, and the boot logged 6 `sgl-project#718 hicache-phase-guard` warnings: the W32 read-through miss, reproduced with its fix installed and unreachable. THIS IS THE W29 DEFECT, WRITTEN BY THE AGENT THAT ROOTED W29. `drop_prefix_tree_returning_rows` read `full_evictable_size_` -- an attribute three tree types have and `UnifiedRadixCache` does not -- and `getattr(..., 0)` turned the absence into a value that silently disabled the eviction. Same tree class, same silent default, same family, one strand later. `host_tier_of(tree)` is now a NAMED accessor that knows both routes and is the writer's only way to that pool. `None` means genuinely no host tier -- a real state the caller reports loudly -- never "the tree keeps it somewhere I did not look". TESTS (5 new, 18 in the file) * both routes, and absent-is-absent; * DRIFT-DETECTOR against the REAL `UnifiedRadixCache` source: the direct attribute must stay absent, so if that ever changes the test says so rather than the accessor quietly depending on a route only some trees have; * the writer must use the named accessor and must not re-introduce the bare getattr. Clean SERIAL gate: 8 failed / 4076 passed / 2 skipped -- the same pre-existing 8, +5 matching the tests added here. THE PATTERN THIS MAKES FOUR OF, recorded in W33-RESULT.md: every recent defect in this strand has been a correct mechanism placed where it cannot run, and the desk tests passed each time because they exercised the mechanism and never its REACHABILITY from the live boot. Asserting against real classes rather than doubles is the shape that closes it.

…nd let a group state its layers W34 arm 1 got further than any prior arm: the accessor found the host tier, the TP device pool was found, and the ALLOCATION failed -- could not allocate the phase-matched staging pin (0.000 GiB -> 1 GB): HostPoolGroup.__init__() got an unexpected keyword argument 'allocator_type' Two defects, both mine. 1. I built the pin by cloning `type(pp_host)` with the MHA/MLA pool signature. The live host tier is a `HostPoolGroup` COMPOSITE whose constructor takes `entries: list[PoolEntry]`. A type cloned without its contract is a guess. It is now assembled from the assembler's own named primitives -- `build_kv_host_pool`, `build_pool_entry`, `HostPoolGroup` -- which ARE the contract, and reusing them keeps the one-mover rule (that assembler has five call sites; this must not become a sixth hand-rolled one). The size override rides a COPY of server_args so no other reader is disturbed, with ratio 0 because a ratio is the mirror-shaped answer sgl-project#810 forbids here. 2. "0.000 GiB" -- neither cell probe answered on the live pool, so the derived size collapsed to zero and only the max(1, ...) floor kept it allocatable. A pin sized from nothing is not a derivation. A NAMED fallback now reads the pp tier's `size_per_token` (both phases hold the same token rows), and it is a fallback, not the primary reading. AND ONE THAT IS NOT MINE, found on the way and older than this work: `HostPoolGroup` delegates `dtype`, `start_layer`, `end_layer`, `kv_buffer`, `size_per_token` and `allocator` to its anchor -- but not `layer_num`. `hicache_phase_binding.check_shapes` compares `device_pool.layer_num` against `host_pool.layer_num` and refuses when either is None. So on every boot whose host tier is a composite -- this fork's live shape -- the shape check could only ever read None and refuse, host pool present or not. The sgl-project#718 rebind was unarmable on this tree for that reason alone. `layer_num` now delegates to the anchor like its six neighbours: a group is exactly as comparable as its anchor, which is the property check_shapes needs. 18 tests. The two assembly tests patch the three NAMED primitives, which is itself the assertion that the writer uses them: a writer that went back to cloning would ignore the patches and fail.

…moves W35 armed the sgl-project#718 rebind for the first time on this tree, then died under load on all three ranks: AssertionError: Double-free detected: slots not currently allocated: [0, 1, 2, ...] check_hicache_events -> drain_storage_control_queues -> _drain_release -> HostPoolGroup.free `cc.host_mem_release_queue` holds bare index tensors naming slots allocated from the OUTGOING pool. `rebind` re-points `mem_pool_host`. The next ordinary scheduler round drains those entries against a pool that never handed the ids out. The assertion is correct and caught a real corruption loudly. THE CRITERION, DECIDED BY READING. Dropping stale entries is right only if the outgoing pool dies with them. It does not: `_stamp` only re-points readers and tears nothing down, nothing in this module destroys a pool, and `phase_flip_host_pools` holds BOTH phases for process life because the flip ALTERNATES -- the outgoing pool is the next cutover's incoming pool. It survives with live allocations, so a dropped release is a host-slot leak that recurs once per cycle. Route, do not drop. SETTLE BEFORE THE SWAP, rather than route at drain. Routing later would need a per-entry generation stamp plus a generation->pool map -- a second bookkeeping scheme beside the sgl-project#719 generation, which is exactly the second-copy defect that cost W32. Settling makes the invariant true by construction: the binding changes in exactly one place, so at that instant every queued entry belongs to the binding still installed. The sgl-project#719 generation stays the single coherence primitive and gains its second CONSUMER instead of a rival. LOUD IN BOTH WRONG DIRECTIONS. A failing settle raises RebindRefused (safe by construction: the binding does not move, so sgl-project#718 keeps the device tier disarmed -- the pre-feature state). A non-empty auxiliary `extra_host_mem_release_queues` REFUSES and NAMES the queue rather than being skipped: those entries route through per-component allocators this step does not resolve, and guessing routing on a free path turns a loud crash into a silent corruption. There is no drop path at all. TESTS (12): the specimen settled against the outgoing pool; end-to-end with a disjoint incoming id space; RED-FIRST -- without settling, swap-then-drain reproduces the metal double-free; both loud directions incl. the aux queue naming itself; ORDER pinned (settle precedes rebind); one-authority pinned (no rival stamp/map names); and REAL-CLASS asserts on the actual drain line, the actual raiser in pool_host/base.py, and `_stamp` re-pointing `mem_pool_host` -- which is why settling must come first. Clean SERIAL gate: 8 failed / 4092 passed / 2 skipped -- same pre-existing 8.

…s queued Class 4's consumer half -- the last silent, DURABLE failure in the sweep. `backup_queue` is consumed by an always-running background thread that does not pause across the flip. After a rebind, `_page_backup` reads `mem_pool_host.get_data_page(...)` -- the INCOMING pool -- and writes those bytes to a CONTENT-ADDRESSED store under a hash computed from the tokens the operation was opened with. The hash does not match the payload, every later reader trusts it, and the corruption OUTLIVES THE PROCESS. Unlike the W35 double-free, which was loud, nothing catches this one. REFUSAL, NOT ROUTING, and the asymmetry against class 1 is the point. A stale RELEASE is routed to the pool its generation names, because that pool still owns those slots. A stale BACKUP cannot be: its host slots may belong to a pool that has since been repurposed, so there is no pool whose bytes are the right bytes. Declining is the only safe verb. A declined backup is a correct NON-PERSIST -- the prefix misses later and is recomputed, the same cheap failure the sgl-project#718 disarm and the sgl-project#760 write refusal already accept -- and it is acked either way, because an unacked operation stalls the queue. `operation_is_stale` is the sibling of `consume_gate` and lives beside it: one authority, two shapes (a queued batch at a consume point; a single operation on a background thread). A third copy of the rule is what cost W32. THREAD BOUNDARY: both generations are read EXACTLY ONCE, at the decision point, pinned by a test. The consumer runs on a background thread while the cutover mutates the current generation on another; a second read mid-persist could straddle a rebind and answer two different questions about one operation. TESTS (11) ASSERT ON STORE CONTENT, not on the counter -- a fix that counts and still writes is no fix. Includes the can-fail modelling the pre-fix path (remove the gate and it persists again), the read-once pin, the ack-anyway pin, the unstamped-op compatibility case, and a pin that the hybrid subclass does NOT override this loop -- the standing warning after the `append_host_mem_release` override shadow. NOT CLOSED, and filed rather than rushed: the PREFETCH consumer. Its loop runs `_all_reduce_prefetch_groups`, a COLLECTIVE, so a per-operation refusal placed before it risks splitting the group across ranks. That needs a rank-uniform formulation (most likely routing through the existing revoke path, which is already uniform), and guessing it on a collective is how a silent corruption becomes a hang. It is also the non-durable half: a stale prefetch loads into host memory, it does not persist to the store. Full SERIAL gate: 8 failed / 4114 passed / 2 skipped -- same pre-existing 8.

…re the handle sits THE ORDERING TRAP, caught at the desk before a boot paid for it. `rebind_for_cutover` runs AFTER the active stack swap (`phase_flip_runtime.py`: `scheduler.draft_worker = want_draft` at :2717, the rebind at :3026), so on the pp->tp leg the flip's drafter IS reachable through `scheduler.draft_worker`. The first cut derived "is this a flip instance" from "did I have to fall back to the stacks" -- which answers NO on exactly the leg that needs the phase term most, arming the draft half for BOTH phases and letting a PP backup persist rows no drafter ever wrote under a content-addressed key. That is the failure the phase term exists to prevent, reintroduced by the way the term was derived. Ownership now reads the stacks' EXISTENCE; the handle is looked up wherever it currently lives, and the parked algorithm is consulted when the scheduler's own pair is still the nulled boot-phase one. Also: a controller with no draft surface at all no longer breaks the TARGET rebind. A refused rebind leaves sgl-project#718's disarm standing over the whole device tier, which is strictly worse than an unarmed draft half -- the same tolerance `readers_of` already shows a missing controller. Tests (hermetic, CUDA_VISIBLE_DEVICES=""): * 42 new passed (was 41; +test_owner_phase_survives_the_active_stack_swap). * Can-fail M7: deriving owner_phase the wrong way turns 5 tests red. * Scoped gate: 679 passed, 13 skipped, 69 subtests. * ruff: zero delta on every touched file.

…e is the default path The tier probe fix (b) reads was two-valued, and the missing third value is a regression on every deployment this ticket does not touch. Without a HiCache host tier a cached prefix can only be a DEVICE radix hit. Those rows were never freed and reallocated, so the original request's `_draft_extend_for_prefill` wrote their draft half and they are warm BY CONSTRUCTION. Reading "no controller" as "the draft half is off" therefore marked EVERY prefix-cache hit on EVERY non-HiCache speculating deployment draft-cold -- a throughput regression on the default path, introduced by a fix for a flip-only defect, and exactly the shape the sgl-project#718 guard's own no-flag argument is written to avoid. The probe is now three-valued: armed / a host tier exists with its draft half off / no host tier at all, the last of which is warm. Tests (hermetic, CUDA_VISIBLE_DEVICES=""): * 44 new passed (+2: the pin and its can-fail twin, same shape with a controller present and its gate closed, which IS cold). * Can-fail M8: collapsing the third value turns 1 test red. * Scoped gate: 681 passed, 13 skipped, 69 subtests.

…r guard, and the consumers split W38-B died six seconds into the TP phase, three ranks at once: hybrid_cache_controller.py:583 start_loading hybrid_cache_controller.py:729 move_hybrid_indices cache_controller.py:1217 device_indices = device_indices.cpu() AttributeError: 'NoneType' object has no attribute 'cpu' ROOT, and it is structural rather than a missing None-check. At 17:25:53 the controller was rebound to the 'tp' pools (generation 3). The TP host tier is built at phase_flip_boot.py:2019-2032 with EXACTLY ONE entry, PoolName.KV -- on a model whose boot-time tier is `pools=KV + MAMBA`. Six seconds later a mamba state living only on the host was matched (MAMBA-HOST-RESUME, the first in any boot on this branch), load_back built the mamba PoolTransfer with device_indices unset BY CONTRACT, and `_resolve_pool_transfers_allocation` -- whose contract is "auto-alloc where they are None" -- found no entry for MAMBA and `continue`d, returning the transfer unresolved inside the same list as the resolved ones. `check_shapes` admits that rebind because it compares ONE SCALAR, the anchor's layer count. The invariant that was actually violated is structural: the incoming tier must describe every pool the outgoing one did. Nothing asked. THE CLASS, and it is why this is a sweep and not a one-line fix: a narrower binding installed under a scalar guard splits its consumers in two. * THE ONES THAT CRASH -- move_indices dereferences the unresolved index set (this specimen); its write-side twin dies one line lower at cache_controller.py:1218 on host_indices.sort() and simply had not fired yet. * THE ONES THAT SKIP SILENTLY -- HostPoolGroup.load_to_device_per_layer (:1758) and backup_from_device_all_layer (:1789) skipped a transfer whose pool they did not know. The KV moves, the recurrent state does not, and the radix tree goes on reporting the prefix RESIDENT: a wrong ANSWER. This is the site that would have HIDDEN the defect had only the crash been patched, and it is LIVE rather than latent on the kernel/page_first/write-back-JIT path, which hands op.pool_transfers to the executor RAW (hybrid_cache_controller.py:468-475), bypassing move_hybrid_indices -- so on that configuration no crash could ever have exposed it. * THE ONES THAT LEAK -- append_host_mem_release (:389) and the drain in unified_radix_cache (:3035) resolved a release through the CURRENTLY bound entry_map, so after a rebind an extra pool's host slots were neither queued nor freed. One leak per load-back, for the whole phase. REFUSE, NEVER SKIP, at every one of them. Skipping moves the KV while the state stays behind and the tree calls the prefix resident; refusing costs a recompute, which is merely slow. That trade is already the tree's own, at mamba_component.py:986-991 (slot starvation -> re-prefill the segment) and throughout hicache_phase_guard (a refused prefetch is a miss now). No mechanism is invented here: the refusal path (rollback_allocated + return None, with callers freeing what they took at :439 and :554), RebindRefused, and the `name in entry_map` idiom (hybrid_pool_assembler.py:779) all already existed. WHAT CHANGED hicache_phase_binding.check_pool_coverage the rebind now compares the POOL SET, not one scalar; called from rebind() beside check_shapes. A refused rebind leaves the sgl-project#718 device tier disarmed, which the code already documents as the correct state: every read-through misses, and a miss is recomputed. _resolve_pool_transfers_allocation refuses instead of returning a half-resolved list, at the FIRST pool it cannot resolve (not after allocating and rolling back the later ones), with a rate-limited refusal. ... and carries a POST-CONDITION: a returned list contains no None index set. That is the check that turns a boot-second-20 crash into a unit test, and it stands on its own -- a DERIVED transfer copies its source's index sets and no per-branch check inspects it, so the post-condition is the only thing between that None and move_indices. HostPoolGroup._entry_for_transfer the executor raises, naming the pool and the tier, instead of skipping. Per-LAYER skips (layer_mapper -> None) stay: those are legitimate and are covered by a control test. entry_for_extra_release both release paths resolve through the entry captured beside the queue when neither the bound tier nor the registry knows the pool, and say so (rate-limited) rather than dropping slots. RESIDUAL, handled rather than noted: match_prefix re-derives the host hit every tick, so a refused load-back REPEATS every tick for as long as the state sits on the host. The request still makes progress (it re-prefills); what must not happen is one line per tick, which is the 449 MB/20 min flood class. All three new emitters are rate-limited first-3-then-every-200th, the cadence the stale-release routing in the same file already uses, and the rate limit is under test. NOT BUILT, FILED: docs/dev/NOTE_847_tp_host_tier_pool_set.md -- the TP host view built with the full pool set, which is what would keep the device tier ARMED across a cutover. It is not two lines: _make_layer_mapper bounds ids by transfer_layer_num, the TP view reports 16 layers against the boot-time 32/18/14, and copying the boot-time MAMBA entry across yields a mapper that returns None for every mamba layer -- the same silent skip, moved one level down and harder to see. The note carries the acceptance it needs. A NEGATIVE FINDING, recorded because it was the reason this was investigated: sgl-project#767 ("cache hit = degeneration, cache miss = correct") is NOT this. The mechanism predicts that symptom exactly, but it can only bite after a cutover has narrowed the tier, and all three sgl-project#767 boots (boot_735_acc767 / _full767 / _standing767, 2026-08-19/20) ran with `phase_flip_rebind_hicache=False` and contain ZERO hicache-rebind lines against 57/189/105 completed flips -- the flag is printed in their server_args, so the emitter existed and the absence is a measurement, not a missing log line. The tier was never narrowed, entry_map always held MAMBA, and the skip was unreachable. sgl-project#767 was separately root-caused and fixed in ecef447 / b6cdbed / e80aa43 (mamba checkpoints written against the bound instead of the computing pool). Adjacent family, different producer. TESTS test_bound_pool_set_coverage_847 17 passed. Hermetic, mocks only. Mutation-proven, six mutations, each reverting one change: M1 refusal -> continue, post-condition off 6 failed M1b refusal -> continue, post-condition left in 1 failed (the early refusal is not redundant with its backstop: it must refuse BEFORE allocating and rolling back later pools) M2 post-condition removed only 1 failed M3 check_pool_coverage not called from rebind 1 failed (wiring; a check nobody calls is inert) M4 executor raise -> silent skip 2 failed M5 release path back to bound-tier-only lookup 1 failed Restored: 17 passed. mem_cache unit suite: 1757 passed, 1658 skipped, 361 subtests, 2 failed. managers unit battery: 4326 passed, 18 skipped, 356 subtests, 15 failed. All 17 failures are PRE-EXISTING and MEASURED as such, not assumed: the four changed files were reverted to the branch tip (25e7849) and the same test files re-run, reproducing the identical 15 + 2 by name and by reason. All are environmental for a hermetic (CUDA_VISIBLE_DEVICES="") run -- 13x "RuntimeError: No CUDA GPUs are available" and 4x the retry wrapper swallowing it -- in test_acceptance_emitters_758 (2), test_arena_high_water_631 (7), test_phase_flip_rotation_wiring_809 (4), test_restore_never_rebuild_677 (4). Zero new failures. black/isort/ruff/codespell clean on the changed files. The formatters' unrelated churn was reverted by hand: the installed black is older than the one the tree was formatted with, and isort moved a deliberately-late `# noqa: E402` import in memory_pool_host.py. NOT PROVEN HERE: that a mamba load-back after a cutover restores state on metal. This change makes that case refuse loudly instead of crashing or answering wrongly; making it WORK is the filed posten.

…o the guard can arm sgl-project#718/sgl-project#847 built every part of the rebind except one: the 'tp' staging pin was assembled with a SINGLE entry, PoolName.KV. On a hybrid model the live tier carries KV *and* MAMBA, so `check_pool_coverage` computed `missing={MAMBA}` and refused -- correctly, on every cutover. A refused rebind leaves the sgl-project#718 device tier DISARMED, `load()` returns None, every read-through misses, and every prefix a cutover retracted is recomputed in full. Measured on the W40 sgl-project#857 acceptance boot (boot_w40_857strict_0825_2342.log): 60 `sgl-project#719 HiCache rebind refused`, ZERO arms, and `#cached-token: 0` on all 243 prefill batch lines -- one bucket, no exceptions. THE PRECONDITION WAS ALREADY WRITTEN DOWN, in the guard's own docstring: "A phase host tier has to be built with the FULL POOL SET before this rebind can arm; until then the sgl-project#718 disarm is the correct state and a read-through miss is the correct cost." This is that precondition, met. Not a new finding -- sgl-project#856 answers whether the retraction is necessary (it is: carry moves Req scheduling metadata, never KV bytes, and sgl-project#856 deliberately retired the movers), and sgl-project#718/sgl-project#847 already named this remedy. REFUSAL CONVERSION, NOT GUARD DELETION. `check_pool_coverage` is untouched. It must stop firing because its precondition is MET, never because it was removed, and the test asserts BOTH directions -- a full pool set arms, a narrowed one still refuses. Neither assertion alone can tell a fix from a disarm. DERIVED FROM THE BOUND TIER, not from the model config: the set that must be covered is whatever the READER names, which is the same quantity the guard compares. Reading the config would be a second opinion about one fact. MIRRORS `build_hybrid_mamba_stack` rather than re-deriving it -- same primitives, same layer mappings, same transfer_layer_num rule. The controller is deliberately NOT reused: this pin needs a host VIEW, and a second HybridCacheController would be a second writer against one device pool. Both entries are rebuilt in the hybrid case. The KV-only pin used an identity map over range(layers), which is right while KV is the only entry and wrong the moment a second pool shares the transfer index space -- the two maps collide at index 0. SIZING IS PER-SLOT, NOT PER-GB, and it is the one place the mamba half must not copy the KV half. MambaPoolHost reads host_size in GB only when > 0, else `device_pool.size * ratio`. The KV pin's GB figure is derived from a token count; mamba is allocated per request slot. Ratio 1.0 with host_size 0 mirrors the device pool, which is what a phase-matched staging pin means. The mamba half is a NAMED HOST-LEDGER POST, priced from what was allocated rather than from the intention. An unpriced pinned pool is what the ledger exists to prevent. A pool set this builder cannot mirror (SWA, indexer, DeepSeek) is logged as an ERROR at the cause and still refused by the guard -- named, not swallowed. THE CHECK (sgl-project#871, third scope item): `advance_fence_blind_streak` + FENCE_BLIND_STREAK=4, aggregating the EXISTING `persisted_nothing` instrument. One empty fence is legitimate; every empty fence means the canonical store can never populate, and that shows up only as latency. Gated on `released`, mirroring sgl-project#719's busy gate at the stale-gate streak: a fence over an empty tree is correct to persist nothing, so counting quiet cutovers would build a crying-wolf alarm out of the instrument written to replace one. NO SECOND COUNTER -- a parallel "recomputed prefix tokens" counter would measure what this and #cached-token already measure between them. Extracted as a pure function so it is falsifiable without booting. A guard whose logic can only be exercised by booting is a guard that ships unexercised, which is the failure mode this ticket is about. CLASS (unchanged, carried from the sweep): a store whose only writer is a lifecycle event another mechanism systematically preempts. Retention is finish-only; the cutover retracts before finish; the store can never populate, so every recovery path reading it can never fire. TEST RESULTS test_phase_tier_full_pool_set_871.py (new): 15 passed. FALSIFIED IN THREE DIRECTIONS BEFORE THE GREEN WAS CLAIMED, count gate held on each: guard removed (disarm) -> 3 failed, and they are exactly the refusal tests. 3 extracted == 3 in summary. guard fires unconditionally -> 6 failed, exactly the arming tests. 6 == 6. streak loses its `released` gate -> 1 failed, the idle test. 1 == 1. restored -> 15 passed. Targeted regression set (7 files: sgl-project#847 writer, sgl-project#783 fence x3, sgl-project#856 empty wave, counters, and the new file): 97 passed, 0 failed, 2 subtests passed. Count gate 0 extracted. Hermetic, CUDA_VISIBLE_DEVICES verified EMPTY at the PROCESS (/proc/<pid>/environ), not at the command. AN EXISTING TEST CAUGHT A REAL BUG OF MINE and the first run of that set was 3 failed: the new HOST-LEDGER line read `tp_host.entry_map` directly, but that writer is driven in tests by stand-ins where HostPoolGroup itself is patched. Fixed with the getattr discipline this module states at its other probes -- an instrument may never be the thing that breaks a boot. Lint: ruff 0 before and after on phase_flip_runtime.py and the new test. phase_flip_boot.py reads 1 before AND after -- a pre-existing F401 on `pack_into_arena`, present in HEAD, not mine. codespell clean. Flag-off path verified byte-identical: without --phase-flip-rebind-hicache the writer still returns {} and allocates nothing. NOT CLAIMED, AND DELIBERATELY NOT GUESSED: no metal run. Whether the kv+mamba pin can actually be ALLOCATED on this box, what its real HOST-LEDGER post comes to against the 16G floor, and whether #cached-token becomes non-zero are decidable only on hardware. They belong to a boot window. The sgl-project#857 acceptance instance was left running and untouched throughout -- it is the standing proof and the operator declared it taboo.

…oute blind over the anchor W38 acceptance boot died on all three ranks, 2026-08-26 12:54:28Z: IndexError: index 76997 is out of bounds for dimension 0 with size 30518 pool_host/base.py:344 assert self.slot_used[indices_cpu].all() memory_pool_host.py:1724 return self.anchor_entry.host_pool.free(indices) unified_radix_cache.py:2853 mem_pool_host.free(host_indices[:unclaimed_to]) HostPoolGroup.anchor_entry is fixed at construction, so the anchor does not move inside one group -- the GROUP is rebuilt onto a narrower tier at a phase rebind. In-flight prefetch state does not move with it: check_prefetch_progress holds host_indices minted against the previous, wider tier and frees them after the rebind, against a pool whose slot_used is shorter. THE CLASS, not the instance. 322f331 (sgl-project#718/sgl-project#847) fixed the TRANSFER path via _entry_for_transfer and did not sweep one level up. That resolver cannot cover these: it resolves by transfer.name through entry_map, and free / alloc / get_page_buffer_meta / get_data_page / set_from_flat_data_page receive BARE INDICES with no name (it has exactly two callers, :1803 load and :1834 backup). The index axis needs a range guard, not the resolver. Its own docstring asks for this: the raise exists "so that a future producer that bypasses the resolver is LOUD rather than wrong" -- free() is exactly such a producer. Sibling axis of _host_binding_is_stale (sgl-project#760), which guards the POINTER axis (device_buffers captured at construction) and left the INDEX axis unguarded. Same time signature: sgl-project#760's specimens died three seconds after a pp_to_tp cutover completed; this one dies on the free path instead of the write path. free() DROPS a stray, the other four are LOUD. A dropped free is a no-op on a tier torn down wholesale -- nothing leaks. An accessor that returns or writes a page cannot drop: that shortens a result or skips a store and the caller proceeds on data nobody wrote, the wrong-answer-with-no-crash outcome this codebase ranks worse than a crash. StrayHostIndexError subclasses IndexError so existing handlers keep working. THIS IS A BACKSTOP, NOT THE CURE, and the ticket should not be closed on it. The cause-side cut is one of two, both open: refuse at the rebind when the narrower tier cannot cover what is in flight (sgl-project#871, the unmet sgl-project#718/sgl-project#847 precondition), or make ongoing_prefetch a cutover participant with a completeness check (sgl-project#859, cutover_participants.py, present at the pin). What this commit guarantees is only that the failure is named and survivable instead of killing three schedulers. Unlike the binding generation stamp -- dead code whenever the binding does not advance -- this guard compares against the live pool's actual size and cannot be inert. Tests: test/registered/unit/mem_cache/test_stray_host_index_718_class.py, 13 passed. Can-fail proven by two mutations: disabling the stray filter reds 2, making the loud refusal a no-op reds 5; restore returns 13 green.

…: the five index-taking accessors route blind over the anchor) into the flip train

…t#718 added The merge added ONE test module (test_stray_host_index_718_class.py) and changed ONE source file (memory_pool_host.py). `--verify` named exactly that: 1 module unclassified, all 180 existing sha256 unchanged. MEASURED, NOT ASSUMED, and cheaply because the change is narrow: * a FRESH serial reference on this tip -- 181 modules, one process, one order: 2 failed / 2683 passed / 927 skipped, 164.0s. 2683 = the previous 2670 plus exactly the 13 the new module contributes, and the 2 failures are the same two test_acceptance_emitters_758.py::RefillTiming names, compared as a LIST and not as a count. * the 180 solo logs from the Train-2 campaign carried forward, plus ONE new solo probe for the added module. Carrying the solo logs forward is safe in the direction that matters, and the builder is what makes it safe rather than a judgement of mine: a proof is admitted only when `solo failure set == serial failure set`, and the SERIAL half was just re-measured on this tree. Had memory_pool_host.py's guard moved any module's failure set, that module's carried solo log would no longer match the new reference and the builder would DEMOTE it to the serial lane with the reason recorded. None was demoted, which is itself the evidence that the guard perturbs nothing: before PARALLEL 167 RANKS 13 UNCLASSIFIED 1 after PARALLEL 168 RANKS 13 SERIAL 0 EXCLUDED 0 -> VERIFY OK GATES on this tip, hermetic (CUDA_VISIBLE_DEVICES=""), all three lanes, the count probe agreeing with the summary on BOTH axes in every lane: mem_cache 2F/1959P/65S wide | 724P/862S narrow | serial empty scheduler 1F/384P wide | 193P narrow | serial empty managers 3733P/7S wide | 274P/11S narrow | 606P serial | 0 failing The three failures are the same three names as the tip before this merge -- the diff of the two name lists is empty, so sgl-project#718 adds no failure and masks none. 1959 = 1946 + 13, the whole delta being the new module.

…gl-project#924 added, re-prove the sgl-project#718 stub `--verify` named three things, and they are not the same kind: TWO new modules (test_hicache_owner_ctx_cutover_923.py, test_mamba_double_free_924.py) plus ONE whose bytes moved and whose proof therefore expired (test_hicache_phase_guard_718.py -- its controller stub gained the new prologue methods). All three got a solo probe; nothing else in the table moved. MEASURED on this tip, hermetic (CUDA_VISIBLE_DEVICES=""): * fresh serial reference, 187 modules, one process, one order: 2 failed / 2798 passed / 927 skipped, 169.5s. The 2 failures are the same two test_acceptance_emitters_758.py::RefillTiming names, compared as a LIST. * the 184 solo logs carried forward, plus three new probes. Demotions: ZERO, counted by diffing the table's rows before and after. before PARALLEL 172 RANKS 13 UNCLASSIFIED 2 (+1 stale) after PARALLEL 174 RANKS 13 SERIAL 0 EXCLUDED 0 -> VERIFY OK THE DELTA RECONCILES EXACTLY, measured per module rather than inferred: 2798 - 2761 = 37 = 20 (sgl-project#923) + 17 (sgl-project#924). The sgl-project#718 stub contributes 0: 12 passed on this tip and 12 on 38bd92f, so the stub fix changed what the double models without changing how many tests it carries. AND THE STUB FIX IS A CO-REQUISITE, NOT COSMETIC -- proven rather than asserted. The 2d stub, checked out into this tree against the merged controller, fails: `1 failed, 11 passed` (TestPhaseGuard::test_write_runs_normally_when_the_flip_is_not_routing). So the stub had to move with sgl-project#923's prologue, and shipping the controller change without it would have gone red at the desk. Tree restored to HEAD afterwards. gate_partition.tsv (managers) unchanged and not rebuilt -- no managers test module moved bytes, and the 35-module gap's reasoning stands from Train-2c. gate_partition_scheduler.tsv untouched for the same reason. NAMED, NOT FIXED: kv_reshard.py carries 6 UP037 that the pre-commit hook would select. They are PRE-EXISTING -- the same 6 on 38bd92f, at lines this merge does not touch -- so they are not this train's regression and are recorded here rather than swept into a merge commit as unrelated churn.

… while the peers spin BOOT-PROVEN (boot 22, boot_855_1033b_0840f82601_0831_131955, 13:24-13:26). The first TP forward after a pp_to_tp cutover touches a shape specialisation never loaded in this process. Rank 0's crash-time stack (log line 53248 ff.): _dcp_write_scatter (flashinfer_backend.py:2574) run (triton/runtime/jit.py:743) -> _init_handles (compiler.py:466) loadBinary -> cuModuleLoadData (libcuda.so) + 11 libcuda frames NOT the compiler -- cuModuleLoadData. The module is built; LOADING it into the CUDA context blocks, because the load needs the device and the device is saturated by the peers' barlink BAR1 spin kernels, which are waiting in all_gather for this very rank. The cycle closes and the spin deadline fires: Bar1CollectiveAborted (ranks 1/3 and 2/3, group flip_dcp:0) -> SIGQUIT. WHY THE EXISTING MECHANISM MISSED IT, measured: 352 build-window lines in that boot, ZERO in 13:24-13:26, all 352 carrying 'full cuda-graph capture warmup'. cold_build_window had exactly three production callers (barlink BAR1 build, sampling warmup #603b, capture warmup) and none is on the path a cutover re-dispatches into. sgl-project#640 on a path sgl-project#615 never saw. The fix makes that path a caller rather than building a second mechanism beside it. A STALE COMMENT NEARLY REFUTED THIS FIX, and the correction ships with it. sampler_warmup.py said wrapping a lazy build in cold_build_window "does NOT work ... the window is PROCESS-LOCAL". True when written (8bddb93, 2026-08-06); falsified ONE DAY LATER by sgl-project#615 (38ec4fb, 2026-08-07), which hooked publication into cold_build_window so "every existing call site therefore becomes group-visible without moving". Never revised, both ancestors of this pin, verified with git log -S rather than assumed. The paragraph is annotated in place rather than deleted, because it is still right about ITS module (warm-at-boot + barrier REMOVES the race; a window only EXTENDS it). WHY A WINDOW OVER THE REAL FORWARDS, NOT AN ENUMERATED WARM SET. The warm set is not knowable by inspection -- it varies with direction, spec-decode, the sgl-project#887 one-chunk grant and any backend swap -- and missing one member reproduces the wedge exactly, at the next first-loader. Whatever loads, loads under the window. Residual stated in the code: this EXTENDS deadlines (900 s cap) rather than removing the race; boot 22's stall was ~150 s, so the cap is not binding. FIRST-LOADER CENSUS, as a standing table: devtools/CENSUS_1033c_first_loaders.md -- every first-loader site x triggering state change x covering window x collective proximity. Rows 1-6 verified at file:line (row 5 on metal); rows 7 (runtime recapture / drafter switch) and 8 (rung change sgl-project#704, resume-restore sgl-project#89) are written UNVERIFIED, so the table is a lower bound on the covered set and never a proof that nothing else is exposed. It also names the inverse column (armed windows that may cover paths that no longer first-load -- the 352 are a count of WARMUPS, not of BUILDS, and the window carries no modules-loaded counter, so live and dead coverage are indistinguishable from the log) and answers the upstream-minimal question: there are not four window mechanisms but two plus a front door, with one site (barlink_device.py:865) bypassing the front door. Named, not rebuilt. #1033d, same boot, INDEPENDENT DEFECT, minimal hardening only: the prefetch IO aux thread caught only Empty, so a page whose geometry did not match the incoming binding (mha.py:556 reshape, '[2,16,1,4,256]'=32768 against a 16384 page) ended the thread three seconds before the cutover -- killing no process, setting no exit code, appearing in no health probe, and leaving storage prefetch dead for the rest of the boot. It now fails the OPERATION (host slots released, so the requester gets a refusal instead of an unreachable completion), logs loudly with a counter, and stays alive. The broad except is defensible only because the alternative is silent thread death with no supervisor above the loop; the underlying two-geometry host pool across a flip (sgl-project#718/sgl-project#719/sgl-project#875 family, and the fork's own 'sgl-project#939 RE-HOME VIA RE-READ ... source page 16384 elems vs destination page 32768 elems' line names the same mismatch two lines earlier while handling it correctly) is NOT fixed here and is its own posten. DESK CHECKS, matched to each edit's failure class: * devtools/check_1033c_cutover_window.py -- 19 cases, hermetic, driving the real run_batch wrapper: RED-FIRST arm reproducing the boot-22 condition (counter unarmed -> no window -> nothing published), first forward covered AND published, window closes (the sgl-project#431 open-without-close shape), budget finite and steady state byte-identical, window closed and budget spent even when the forward raises, and an AST check that the arming sits under no `if` (the naive grep form of that assertion failed on a COMMENT -- fixed to ask the AST). * AST check on the aux thread: broad handler present, logs at ERROR, releases the failed operation's host slots, and continues. * Method-split check: wrapper delegates, window only on the armed path, lazy import off the fast path, inner body intact. ruff F401/F821/UP037 clean. DESK-PROVEN. Metal proof is boot 23.

The flip aliased both stacks' request-index space at boot and never rebound the scheduler at the cutover: `tp_req_pool.req_to_token` was pointed at the PP pool's tensor (and the mamba index map with it), so every row id the TP phase used had been minted by the PP allocator, and a row freed in one phase was still named by the other. This is CUT 1 of the WEG1 family spec -- the request axis (R), which is independent of the KV-token axis (K) and buys correctness, not tokens. Five edit sites, one cut: C1.1 phase_flip_boot.py -- the two alias ASSIGNMENTS are deleted. The three shape/space checks around them STAY and the `5a.` premise comment is rewritten to the reason they are load-bearing AFTER the deletion: three consumers cache `req_to_token`'s SHAPE once, at construction (hisparse_coordinator.py:109, overlap_utils.py:296 and :299), and now see the other phase's pool after a rebind. C1.2 phase_flip_runtime.py -- `rebind_req_pool_for_cutover` runs in `_cutover`, immediately BEFORE the sgl-project#719 HiCache rebind and OUTSIDE its try/except, on EVERY cutover. Deliberately NOT gated on --phase-flip-rebind-hicache: sgl-project#719's "a refused rebind is SAFE" holds because a stale HiCache binding has a disarmed state (sgl-project#718); a request pool has none, so a refusal here RAISES. C1.3 memory_pool.py -- `ReqToTokenPool` gains `binding_tag` (minted at construction, re-minted by `clear()`); `Req` gains `req_pool_binding`, stamped beside `req_pool_idx` in `alloc`; the existing `reusing` branch refuses a row minted under another binding. Both pools hold the same row count, so a carried id lands IN RANGE on someone else's row -- silent, not a device assert. `SessionSlot` carries the binding with the parked row. C1.4 phase_req_pool_binding.py -- the sgl-project#919 census on the REQUEST axis: the outgoing pool is counted before the rebind, the line is emitted on every cutover whatever it found (indicator law), and a non-zero escapee count raises with the rids and rows named. C1.5 kv_session_offload.py -- the only tree-wide cacher of the pool OBJECT now reads `self.scheduler.req_to_token_pool` through a property, so its ~20 `req_to_token[...]` writes cannot land on the outgoing phase's tensor. Evidence red-first on the parent 8fe7b60: 17 failed / 4 passed. The 4 green are exactly the must-not-change and already-true ones (the three shape checks survive; reuse within one binding stays allowed; the two default-path pins). green after: 21 passed. mutants, each verified to have actually applied before running: M1 DANGER gate the rebind on --phase-flip-rebind-hicache -> 2 failed M2 drop the binding_tag clause from the guard -> 2 failed M3 census counts only rows a request still names -> 1 failed M4 restore the req_to_token alias in phase_flip_boot -> 1 failed M5 skip clear() of the incoming pool -> 1 failed M6 re-cache the pool object in kv_session_offload -> 1 failed (a first M6 attempt cached under a DIFFERENT attribute name and survived -- an equivalent mutant, dead code nothing reads, not a test gap; the faithful form is the one counted.) ruff: per-file counts identical to HEAD on all seven modified files (1/0/8/43/0/0/0); the new module and the new test file are clean. C1.5 follow-on: four test modules set the pool ON THE MANAGER (`mgr.req_to_token_pool = ...`), which the read-at-use property makes read-only. They now set it where it lives -- on the scheduler stand-in. Found by the desk gate (test_host_finish_stream_659, +3 failures vs the HEAD baseline); the other three sites are OUTSIDE the gated directory and were found by a tree-wide sweep of the same assignment shape. Two files matching the same grep are deliberately NOT changed: test_kv_spill_destination_unit builds a bare SimpleNamespace (no class, no property, the attribute is the pool), and test_specv2_kvcache_offloading sets it on DecodeKVCacheOffloadManager -- a different class.

bump version to 0.1.24g

a9c7d04

Ying1123 merged commit 459abad into main Jul 24, 2024

Ying1123 deleted the bump-version branch July 24, 2024 22:55

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Bump version to 0.1.24 (sgl-project#718)

5e94b0c

cherryblo added a commit to cherryblo/sglang-project that referenced this pull request Jul 2, 2026

Merge pull request sgl-project#718 from hhhh1252023/patch-7

153cb55

fix 0608

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026

Merge fix/718-stray-host-index-w38 (sgl-project#718 class, index axis…

7f60db3

…: the five index-taking accessors route blind over the anchor) into the flip train

Navigation Menu

Uh oh!

Bump version to 0.1.24 - #718

Bump version to 0.1.24#718
Ying1123 merged 1 commit into
mainfrom
bump-version

Ying1123 commented Jul 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Ying1123 commented Jul 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant