Add troubleshooting doc by Ying1123 · Pull Request #856 · sgl-project/sglang

Ying1123 · 2024-07-31T23:48:04Z

No description provided.

The user asked why a tp_to_pp flip takes 11.6 s, and whether 16 waves over 116502 live slots could not just be moved in one parallel shot. The capture answers the first half and refutes the premise of the second. THE 116502 LIVE SLOTS AND THE 16 WAVES ARE NOT THE COST. W25's own seam census names one segment: seam-census] timing tp_to_pp rank 0: 10466.8 ms across 448 segment(s), worst 'refill_highwater->weights_refill' 9516.2 ms (91% of the walk) kv_write->gdn_state 174.3 | cutover->done 152.4 | flip_writeback->hicache_quiesce 74.8 | kv_pack->kv_local_read 57.4 Three consecutive tp_to_pp flips: 9516.2 / 9496.5 / 12108.2 ms in that one segment, against totals of 10466.8 / 10568.0 / 13181.2. Meanwhile the ENTIRE KV movement at the largest observed reshard (epoch 11, 116502 live slots, 995.31 MiB sent / 825.03 MiB received) is read 19.9 + exchange 486.0 + write 394.8 = 901 ms, and the GDN blob is 11.69-18.70 MiB. Across the window live slots grew 947x (123 -> 116502) while the flip grew 1.26x (5077.9 -> 6416.0 ms). The ranks are already parallel: totals agree within ~40 ms while movers legs differ by ~600 ms, and the peers absorb PP0's overrun as CUTOVER WAIT (1313.3 / 1220.8 ms vs PP0's 225.3). A rendezvous, exactly as sgl-project#690 recorded. SO THE SEAM IS A WEIGHTS-REFILL RATE PROBLEM, and the rate is direction- asymmetric for reasons nothing in the tree explains. Same rank, within 2.7% of the same bytes: pp_to_tp 15925.8 MiB 3214-3915 MiB/s 4.07-4.96 s tp_to_pp 16362.7 MiB 1351-1723 MiB/s 9.50-12.11 s FOUR CANDIDATE EXPLANATIONS WERE CHECKED AND ALL FOUR FAIL, and they are recorded so nobody re-derives them: * not a path fallback -- SGLANG_PHASE_FLIP_REFILL_STAGED defaults True (environ.py:348), unset in the boot, and arena_refill dispatches on the image being file-backed, not on direction. Both take _staged_file_refill. * not a missing fd -- both sgl-project#802 warning paths ("could not open a read fd", "O_DIRECT unavailable") appear ZERO times in the 3.45 MB capture, against 9 "flip host image FILE-BACKED" registrations. * not the O_DIRECT alignment cliff (sgl-project#809's 8304 -> 2595 MiB/s) -- the loop only issues aligned offsets: chunks are 32 MiB multiples of _DIRECT_ALIGN=4096 and `want` is rounded down to it; only the trailing checksum tail is buffered, by design. * not the pre-sgl-project#802 fault path -- on sgl-project#802's own discriminator the fault path makes rank rates CONVERGE (821/775 MiB/s on links differing 1.80x). W25 DIVERGES with the link in both directions (1.59x and 1.40x). THE INSTRUMENT IS WHAT IS MISSING, NOT THE MECHANISM. The leg reports ONE aggregate MiB/s. The read (os.preadv) and the H2D (copy_ on a stream, depth 2) are pipelined, so that aggregate is min(read_rate, h2d_rate) with no way to see which bound it hit. That is the sgl-project#851 class -- one number with several meanings -- sitting inside the term that is 91% of the seam, which is why two independent readers could not attribute the gap from code. WHAT SHIPPED. `RefillLegTiming` accumulates, on the existing path: read_s wall time inside preadv -- storage/ARC bound h2d_wait_s wall time blocked on a prior DMA -- PCIe/link bound drain_s the pipeline tail, counted apart so it can never be mistaken for either bound They are near-exclusive by construction: the ring only waits on a buffer whose copy has not landed, so a read-bound leg never blocks there and a link-bound leg blocks almost every turn. `refill_bound_phrase` is a PURE function over that record -- no GPU -- so both directions are falsifiable off metal, the same split sgl-project#852 used for the allocator-cache estimator and for the same reason: a rule that can only be exercised on metal is one this corpus has repeatedly shipped inert. THE CAN-FAIL DIRECTION IS THE WHOLE RISK and is pinned. A phrase that always named a bound would satisfy every "it says something" assertion while being exactly as useless as the aggregate it replaces. So "unattributed" (not instrumented, or no time accounted) and "MIXED" (neither half dominates) are first-class outcomes, asserted directly, and a 101-point sweep across the whole read-share range proves the three verdicts neither overlap nor leave a hole. NO BEHAVIOUR CHANGE: the timing record is optional and defaults to None, so every caller that does not pass one runs byte-identically. ONE PINNING TEST HAD TO MOVE, and it is worth naming because widening it blindly was the tempting wrong answer. `test_arena_high_water_631.py`'s `fake_arena_refill` stub replaces `arena_refill` to prove the arena is committed before it is copied into; it did not accept the new kwarg and all six of its refill tests failed with TypeError -- the sgl-project#624 stub-drift shape. The stub now accepts `timing=None` (which is the whole adaptation, since the record is an instrument with no semantics the test pins), AND it captures what it was handed, with a new `test_the_refill_leg_is_instrumented` asserting the caller really does pass a `RefillLegTiming`. Without that, the signature widening would have silently tolerated the instrument being unwired again -- a stub that accepts anything proves nothing. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_refill_bound_856.py 9 passed, 101 subtests test/registered/unit/model_executor/ 15 failed / 763 passed, against a VERIFIED HEAD baseline of 15 failed / 754 passed -- identical failure set, pre-existing, unrelated to this path (bf16 vendor fallback, coresidence budget, expert-offload kv regain). Checked by reverting this change and re-running, not asserted. ruff format clean; the one F401 in phase_flip_boot.py is pre-existing at HEAD (verified against git show HEAD:...). Full ledger, with every figure and every failed search set: docs/dev/NOTE_856_seam_cost_ledger.md

W25 went TP-sticky on 16-20k prompts and the user caught it by eye. The policy was right and the PRICE was wrong, and the price was wrong in a way this module had already written down and then not applied to itself. phase_policy.py:82 C = round-trip flip cost, seconds break_even_tokens "never repays the {flip_cost_s}s round trip" But `observe_flip_leg` fed ONE LEG per sample -- its own docstring even computes the round trip it is not feeding, "tp_to_pp 11490 + pp_to_tp 5681 = 17171 ms" -- and BOTH DIRECTIONS went into ONE EMA. THE TWO LEGS ARE NOT THE SAME QUANTITY. On the binding rank W25 measured tp_to_pp 10466-13181 ms against pp_to_tp 5078-6545 ms. An EMA fed both alternately converges to neither. REPRODUCED EXACTLY, which is what makes this measured rather than argued. Replaying PP0's eleven PHASE-FLIP DONE totals through one estimator at ALPHA=0.3 gives 5.0779 6.6944 6.2494 7.5450 7.2426 9.0241 8.2740 9.2457 8.4356 9.3990 8.5041 -- and the boot's own decision lines printed N=15853/18110/18464/18614 at exactly the samples pricing to 7.2426/8.2740/ 8.4356/8.5041. To the token, four times. The blend settled at 8.50 s: BELOW every tp_to_pp leg and ABOVE every pp_to_tp leg. The true round trip was 11.6 + 6.4 = 18.06 s. The bar also oscillated ~2000 tok with flip-direction parity (8.50 after a pp_to_tp, 9.40 after a tp_to_pp) -- an artifact of the blend that says nothing about cost. sgl-project#819's own closing sentence is the rule it broke one level up: "a component and its container are different quantities and an EMA fed both alternately converges to neither." Two directions are different quantities too. THE CORRECTION RAISES THE BAR (C 8.50 -> 18.06 s, N 18614 -> ~39500), so it makes TP-stickiness on 16-20k prompts MORE correct, not less. That is stated plainly rather than softened: the remedy for a bar that is too high is a cheaper seam, not a permanently under-priced one. dN/dC = 2188.8 tok/s says what each second of seam is worth once it is. WHAT SHIPPED. `RoundTripFlipCost` holds one `FlipCostEstimator` PER LEG and sums them. The leg estimator is REUSED, not rebuilt, so every property sgl-project#677 pinned on it holds per leg -- including that it tracks DOWN as readily as up, which is what makes a future seam fix actually lower the bar instead of latching high. The seed is split in half, so an uncalibrated instance values exactly the round-trip seed and the pre-sgl-project#856 path is unchanged. An undirected reading is treated as a whole round trip and split evenly (so callers that really measured one stay honest); a direction this class does not know is REFUSED rather than filed under a guess. PROVENANCE GAINED A THIRD WORD. `flip_cost_measured()` is a boolean over a quantity with three states and printed the middle one as "measured". `flip_cost_provenance()` returns seed / half-measured (<leg> only) / measured. Same class of fix as sgl-project#853(i) on the exposure gate and sgl-project#854 on the economy detector, one layer further in. AND THE CONSUMER IS RECONCILED, which is the half that is usually missed. The sgl-project#838 economy detector refuses to question a bar priced off the seed because "an assumption is not the policy's own claim". A HALF-measured round trip is still half assumption, so it is refused on the same ground: `flip_cost_fully_measured()` requires BOTH legs. The blast radius is one-directional -- the detector can only DECLINE more often, never alarm more often. DELIBERATE TEST CHANGES, not loosened ones: * `test_the_leg_total_becomes_the_price` now asserts the leg's own value AND the round trip (11.4901 + 3.2/2). Asserting 11.4901 for C would be asserting that one leg is the whole round trip -- the defect itself. * `_measure` floors at 2x MIN_ESTIMATE_S: each leg carries its own band, so a round trip cannot be cheaper than two leg-minimums. * `test_a_seeded_price_is_not_evidence` now also asserts "half-measured". TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_round_trip_price_856.py 19 passed test_flip_threshold_repricing_819.py unregressed test_flip_cost_calibration_677.py unregressed test_flip_threshold_honesty_777.py unregressed test_flip_cost_clamp_directions_677.py unregressed test_layout_conformance_838.py unregressed test_economy_detector_liveness_854.py unregressed ruff check + format clean on the changed source ALSO FOUND, NOT FIXED HERE (recorded in NOTE_856_seam_cost_ledger.md): `observe_flip_leg` is called only from the flip-COMPLETION branch, so a boot whose flips are all refused or abandoned prices off the seed for the whole session -- and sgl-project#777's staleness WARNING is gated on the same event, so such a boot gets neither a reprice nor a warning. W25 did not manifest it (33 completed cutovers). Same silent-zero shape, one more instance.

…t inventory USER DECISION 2026-08-24, binding, verbatim: "das kv soll niemals vom layer flip her stammen, einfach aus dem hicache laden fertig." Recorded BEFORE the build so the build cannot quietly become something else, and so the ledger that follows reads as validation rather than as a vote. THE TREE ALREADY AGREED, which is the part worth landing on its own. hicache_flip_writeback.py:21-23 states the premise in its own words: "a prefix's only way across the flip is the geometry-free STORE (sgl-project#706): the disk tier, whose keys carry content alone and whose pages are cut at read time for whichever geometry asks." And it names exactly why a mover exists today: "device rows survive the flip, because the live row set (radix tree values UNION parked requests' rows) is relocated between the two phase pools BY ROW ID." That relocation IS the wave mover. THE BLOCKER THAT SHAPES THE BUILD, found before writing any code. phase_flip_resident_carry.py:64-76 -- a carried Req keeps its req_pool_idx across the swap "by construction", and the reason it stays valid is that "the bytes behind those ids are what the KV and GDN movers relocate". So PHASE-FLIP-CARRY is NOT a KV mover and NOT a retirement candidate, but its correctness today DEPENDS on the mover. Retire the mover alone and a resident request's req_pool_idx points at unwritten memory. The fence and the retirement must therefore land together, and the cutover must leave the new phase's device tier in a state where a lookup MISSES rather than returning stale rows. That is the correctness core, and it is where the red-first tests must bite hardest. WHY THE PRIZE IS FUNDING, NOT LATENCY -- stated plainly because the opposite is the natural assumption. The whole KV+GDN movement is 901 ms of an 11.6 s seam (W25 epoch 11, 116502 live slots). Removing it leaves ~10.5 s. What it DOES remove is wave_peak = incoming + max(outgoing, local) + one_layer_window + backing_slack (phase_flip_runtime.py:7314) -- every term a KV quantity -- which is the 2339.11 MiB tp_to_pp staging reserve behind W25's 33 refused arms, 25 of them on the staging rate limit, and 17 FLIP ABANDONED. RETIREMENT INVENTORY, scoped to the flip path, REPLACE/RETIRE/KEEP-WITH-REASON with file:line, in the note. Rules held to: "hardened against corruption" does not count as reconciled; no bulk deletion; shared machinery with a named other consumer is KEPT (kv_reshard's sgl-project#297 domain, gdn_flip_preconditions -- whose "no other consumer found" is recorded WITH the narrow search set that failed, not as a licence to delete); anything merely deletable-later is a separate section and marked un-re-verified. EXISTING MACHINERY TO EXTEND, NOT REBUILD: hicache_flip_writeback.py already IS the fence (stage + bounded ack drain, deadline 2.0 s); hicache_demotion.py already covers evict-before-persist WITH counters and is merely off by default; sgl-project#719 rebind + sgl-project#718 disarm already own the read-path switch; mamba_ckpt_utils' anchor grid is already what anchor-resume needs. NO FALLBACKS, per standing doctrine: every gap is fixed inside the HiCache route. A flip enabled without hierarchical cache becomes a validate-early launch refusal on the sgl-project#806 precedent (c0a6347, ServerArgs.__post_init__ after materialize_declarations) -- NOT a silent mover revival. That refusal is deliberately NOT landed yet: it only becomes true once the flip carries no KV, and shipping it early would reject launches that work correctly today. Forced build order, recorded rather than discovered later. VALIDATION METRIC CHANGED, and one half of it does not exist yet: cutover- blocking time (fence + weights refill) can reuse the existing seam_census.mark("flip_writeback") and the DONE stats dict, but there is NO instrument for post-cutover warm-up cost as served-request latency -- searched and named as a build item rather than assumed present. OPEN GAPS ARE LISTED AS GAPS, with the search sets that failed: sgl-project#735's "sgl-project#706 rows on the full plan" could not be located (both greps resolve to a different topic -- non-contiguous PP placement), fp8 kv_cache_dtype against the canonical page format is unverified, and the worst-case un-hashed resident tail at the quiescent-flip instant is reasoned from the quiescence predicate rather than measured. No code changes. Documentation only.

…le, not by new machinery Follow-up to 3b0c031. The open item was the correctness core: after a cutover that moves no KV, the new phase's device pool holds no valid KV while the radix tree still maps prefixes to row ids, so a lookup must MISS and fall through to the host tier. Which function makes it miss was unanswered. THE OBVIOUS ACTION IS ALREADY KNOWN TO CRASH, and finding that out from the tree rather than from a boot is the point of asking first. phase_flip_runtime.py:4590-4620 records sgl-project#825 trying exactly this on 2026-08-23, on all three ranks at once: cache_finished_req -> dec_lock_ref -> full_component.py:239 `if cur.id in skip_lock_node_ids` AttributeError: 'NoneType' object has no attribute 'id' with the cause in its own words: "PARKED IS NOT UNREFERENCED. The cutover carries RESIDENT requests across, and each holds a `last_node` with a lock ref. `reset()` rebuilds the root, orphaning those nodes." sgl-project#825 withdrew the ACTION and kept detection only, noting the real fix "needs to be built against the lock refs ... and that is a design, not a flag flip." RESOLVED, AND BY THE USER'S OWN RULE. The stale `req_pool_idx` (the resident-carry blocker) and the orphaned `last_node` (sgl-project#825's crash) are the same fact wearing two hats: A RESIDENT REQUEST CARRIED ACROSS THE CUTOVER. The no-carry rule removes it, and both blockers go with it. The mechanism exists and needs no new code: `retract_all` (schedule_batch.py:1812) walks `release_req` (:1783) over every request, and `release_req` does exactly the two releases required -- `release_kv_cache(req, tree_cache, is_insert=False)` and `req.reset_for_retract()` -- taking precisely the objects the flip already holds. SEAM ORDER: fence (maybe_flip_writeback + demotion on) -> retract all -> tree reset (now safe, sgl-project#825's precondition gone) -> cutover + weights refill -> re-admit, served by HiCache read-through. That is the user's sequence verbatim. THE CARRY ONLY EVER EXISTED TO AVOID A RE-PREFILL. With read-through that re-prefill is a cache hit, which is why "no carry" is correct rather than a simplification -- and it is exactly the "honest warm-up cost as served-request latency" named as the validation metric. That cost is REAL and must be measured, not assumed small: it is what this design pays to delete the mover, the staging reserve and a crash class. TWO CAVEATS RECORDED AS UNVERIFIED rather than glossed: that `release_kv_cache` releases the lock ref along the whole parent chain (read from release_req's body, not from release_kv_cache's own), and that retraction at the seam is compatible with the flip's quiescence predicate. Both must be confirmed before this is built. No code changes. Documentation only.

… and retraction feeds the policy a lie Follow-up to 4b6ee1e, which recorded two unverified caveats rather than glossing them. Both are now answered, and only one of them was clean. CAVEAT 1 -- DISCHARGED, AND IT PROMOTES THE ORDER TO A LAW. `release_kv_cache` (mem_cache/common.py:1749) routes to `tree_cache.cache_finished_req(req, is_insert=False)`, and the live `dec_lock_ref` (mem_cache/hi_mamba_radix_cache.py:1610) is while node != self.root_node: ... node = node.parent so the lock ref is released along the whole parent chain -- PROVIDED the walk still terminates at the live root. That is exactly the loop sgl-project#825 crashed in once `reset()` had rebuilt the root and orphaned the nodes. RETRACT STRICTLY BEFORE RESET. Reversing the two reproduces the 2026-08-23 three-rank crash. A red-first test must pin the ORDER, not merely the outcome. Two bypass paths are recorded for the build to handle, both already named in that function: `req_pool_idx is None` under MambaRadixCache, and `kv_spill_state == "host"` routing to the kv-session-offload release. CAVEAT 2 -- NOT A CLEAN PASS. A NEW DEFECT THIS DESIGN WOULD CREATE. Retracted requests return to the waiting queue, so their full context reappears as `pending_prefill_tokens` -- the exact quantity the flip policy compares against N. But N is priced from X and P, the UNCACHED prefill throughputs. Retracted tokens are CACHED by construction: the fence just persisted them and read-through serves them. Their real cost is a small fraction of what the bar prices them at. So every cutover would hand the policy a large pending figure whose true cost is far smaller, in BOTH directions. That is a thrash pathway CREATED by this design, not inherited -- today's carry keeps those tokens out of the pending count entirely. Recorded as unresolved, with three candidate directions and none of them yet evidenced. The honest one is to price the pending figure by cache residency, which is the same class as sgl-project#856(b): a decision is only as good as the quantity it compares. Explicitly NOT claimed as mitigation: sgl-project#856(b) raises N (8.50 -> 18.06 s, 18614 -> ~39500), which makes this harder to trigger. That is a mitigating accident and must not be cited as a fix. No code changes. Documentation only.

…ecedented failure Follow-up to f0ed2e4, which predicted that retraction at the seam would feed the flip policy an inflated pending-prefill figure. Verified rather than left as a prediction, because a design risk nobody checks is just a worry. CONFIRMED. `Scheduler._pending_prefill_tokens` (scheduler.py:10508) computes pending = sum(len(req.origin_input_ids) for req in queued) the FULL prompt, not the uncached extend, and there is no prefix-residency term anywhere in the function. A retracted request therefore contributes its entire context regardless of how much of it the fence just persisted and read-through would serve. The bar it is compared against (N, from X and P) is priced on UNCACHED prefill throughput. The tokens are counted once, at a price that is wrong. AND THE SHAPE IS PRECEDENTED, in a comment inside that same function. sgl-project#731, measured 2026-08-17: a cutover left one request both resident and queued, so one prompt was counted twice -- "51,369 -> 102,307 tokens across one cutover, within rounding of exactly 2x. The inflated backlog drove the flip policy past its threshold -- six cutovers, nothing served." So "an inflated pending figure across a cutover drives the policy into thrash" is a MEASURED failure of this exact code path, from a different cause. That moves caveat 2 out of the speculative column entirely. sgl-project#731's fix does not cover this route. It made the carry consume the queue entry, and it deliberately refused a blanket per-rid dedup on the stated grounds that hiding a genuine double-booking would make the class silent the way this one had been. Retraction re-creates the shape without any double-counting at all -- which is exactly why the existing fix cannot catch it and why this needs pricing by cache residency rather than another dedup. Same class as sgl-project#856(b), one quantity further out: a decision is only as good as the quantity it compares, and here the quantity is honest while its PRICE is not. Still explicitly NOT claimed as mitigation: sgl-project#856(b) raising N (8.50 -> 18.06 s, 18614 -> ~39500) makes this harder to trigger. Mitigating accident, not a fix. No code changes. Documentation only.

…g but the census Half of the new validation metric. Once the flip carries no KV, cutover-blocking time is FENCE + WEIGHTS REFILL and nothing else. The refill has been in the stats dict since sgl-project#690 and now names its own bound (sgl-project#856 a). The fence had no entry at all. Its cost was visible only as a census SEGMENT -- the delta between the `flip_writeback` and `hicache_quiesce` marks, 74.8 ms on W25's binding rank -- which is a subtraction between two marks that nothing reading `last_stats` can perform. Meanwhile `maybe_flip_writeback` was already RETURNING a `FlipWritebackReport` carrying `elapsed_s`, and the seam used it as a bare truthiness test and dropped the number. So this adds no measurement. It stops discarding one. `writeback_fence_ms` goes in the stats dict and deliberately NOT into the DONE line's parenthesised list, by the same rule `drain_ms` follows and cites: ANALYSE_830 section 10's reproduction regex pins that list ending at "cutover N ms)", and silently breaking a documented grep is the failure sgl-project#830 existed to repair. None, NEVER 0.0, when no fence ran. The fence is skipped outright without a canonical store, and a defaulted zero would report such a flip as fully fenced while nothing had been persisted -- the sgl-project#606 defaulted-measurement shape, in the one place where believing it means losing KV. "This cost nothing" and "this did not happen" are exactly the two readings a seam census must keep apart. THE CAN-FAIL TEST KILLED MY FIRST IMPLEMENTATION, which is the reason it was written. `_writeback_fence_ms` first caught only (AttributeError, TypeError, ValueError); `test_an_instrument_never_raises_into_the_seam` feeds it a report whose `elapsed_s` property raises RuntimeError, and that escaped -- straight into a cutover with requests already parked. The catch is now broad, with the narrowing recorded as the thing that was tried and refuted, because this runs where the module's standing rule is that an instrument may cost a missing line and never a flip. A genuinely free fence still reads 0.0, not None, so the broad catch did not swallow the distinction the file exists to protect. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_writeback_fence_ms_856.py 6 passed, 4 subtests managers -k "flip or seam or phase or writeback or economy or layout or band" 1234 passed, 169 subtests, exit 0 ruff check + format clean Nothing else pins the stats dict's shape: a tree-wide grep for `last_stats` / `drain_ms` / `seam_waves` across test/registered/unit/ returns only this new file.

…'s other half is refused RECONCILIATION, per the standing one-job-one-mover rule: a canonical authority exists, so every bespoke copy of the same payload must be replaced, retired, or justified. `managers/tree_congruence.py` owns the (x, -x) MIN-pair ballot as `digest_pair` / `agreement`. `AllocationSteering.decide` hand-rolled it TWICE in one function -- once for the absorbing-rank proposal, once for the free-list checksum: payload[n] = proposal ; payload[n + 1] = -proposal ... if reduced[n] != -reduced[n + 1]: Three copies of "did the ranks agree?" is three places a defect in that question has to be found. It now calls the authority. The arithmetic is identical -- this is a reconciliation, not a behaviour change -- and the import direction cannot cycle: `tree_congruence` holds no steering state and imports nothing from here. THE TEST ASSERTS IDENTITY, NOT EQUIVALENT BEHAVIOUR, deliberately. A test that only compared answers would pass against a FOURTH private copy that happens to agree today, which is precisely the state being retired. Its can-fail partner checks the shared primitive still discriminates both ways, because pointing at a function that always returns True would prove nothing. F7's OTHER HALF IS REFUSED, WITH THE REASON. `agree_mamba_slots` (gdn_flip_mover.py:669) is the other named copy, and it is NOT reconciled here because it is inside the machinery the no-KV design retires: its only production caller is `_slots()` at :927, inside `build_gdn_flip_mover`, and `GdnFlipMover.move()` is a RETIRE entry in the inventory (3b0c031). Refactoring a component scheduled for removal is the same waste the one-job-one-mover rule exists to prevent, pointed the other way. Verified rather than assumed: a tree-wide grep for `agree_mamba_slots` across python/ and test/ returns that one call site plus its own test file. If the retirement is ever abandoned, this becomes live work again and the inventory is where that is recorded. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_corridor_steering_657.py 26 passed (24 before, +2) ruff check + format clean PRE-EXISTING, VERIFIED NOT MINE: mem_ledger/test_r1_private_constant_gate_584 .py::test_no_unpinned_demand_scale_constant_exists fails identically with this change reverted (1 failed / 6 passed both ways). Checked by reverting and re-running, not asserted.

… its two callers ANALYSE_851 site 4: `spendable_bytes` reads the allocator cache raw. The finding is sharper than "underated" -- the SAME figure is correct for one caller and wrong for the other, and only one of the two has an argument. `_allocator_cache_bytes` (reserved - allocated) documents overstating as the safe direction, and it IS safe where it sizes `want`: "overstating the cache understates `want`, and an understated want costs a late arm". But `spendable_bytes` ADDS it to a budget -- return free + cache - delta -- so there an overstatement WIDENS the prefill chunk this actuator grants, and the one thing that same function calls unsurvivable ("an allocation larger than what can be served") is exactly what it then permits. Same number, opposite error. AND THE OVERSTATEMENT NOW HAS A NAME AND A NUMBER (sgl-project#852 R3, 9aa5b6e). Free blocks inside a CUDA-graph PRIVATE pool are counted by `reserved - allocated`, because those are device-global `.all` counters, but an ordinary forward allocation cannot take them -- they are reachable only while capturing into that pool. W25 measured that term at a stable 88 MiB. This is the same reconciliation the one-job-one-mover rule asks for: a canonical way to price trapped cache now exists, so every bespoke reader of the same payload is checked against it. THE SHARED FIGURE IS DELIBERATELY NOT CHANGED. Subtracting inside `_allocator_cache_bytes` would have fixed the budget caller by breaking the ladder caller's documented argument. Only the budget caller subtracts, and a test pins that separation by reading both function sources. TESTABLE OFF METAL, which the first version was not. That version put the rule behind `torch.cuda.is_available()`, which under the hermetic CUDA_VISIBLE_DEVICES="" regime is always False -- so the subtraction could never be exercised, the "shipped inert" failure this corpus keeps recording. The arithmetic is now the pure `takeable_cache_bytes`, and the probe is injectable, so BOTH outcomes are asserted: inert without a device, and actually subtracting with a probe. A test proving only the first would not distinguish "inert under CVD=''" from "inert everywhere". Its own can-fail tests killed two bugs while being written: `int(1.5e400)` raised an uncaught OverflowError, and the negative-trapped case could have INFLATED a budget above the real cache -- the direction that grants an unservable chunk. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_takeable_cache_856.py 13 passed, 3 subtests test_corridor_steering_657.py 26 passed (unregressed) ruff check + format clean ATTRIBUTION, HONESTLY. A `-k "admission or corridor or steering"` sweep showed 1 failure (test_pp_admission_wraparound_never_blocks) that a reverted-tree run did not. I could NOT get a clean A/B: that `-k` selection pulls in a family of multiprocess PP-admission tests that hang on `Process.join` under load (caught live with py-spy in test_pp_admission_send_handle_dropped_796), and two attempts wedged there. So the claim rests on a direct proof instead of a suite diff: under CVD="" `_takeable_cache_bytes` returns EXACTLY `_allocator_cache_bytes()` (325058560 == 310 MiB, demonstrated), and `torch.cuda.is_available()` is False there -- the change is byte-identical in every hermetic test and cannot have caused that failure. The wraparound test also passes standalone both with and without the change. NOT VALIDATED ON METAL. The subtraction only engages with a device, so its real effect is unproven until a window. It is strictly conservative -- it can only narrow a cut, never widen one -- which is why it is safe to land ahead of that.

… cannot argue for a flip PREREQUISITE for the seam rebuild, and its own defect. The phase-flip seam will RETRACT resident requests rather than carry them -- the carry is what made sgl-project#825's tree reset crash, and read-through makes it unnecessary. But retraction puts each full prompt back in the waiting queue, and `_pending_prefill_tokens` sums `len(req.origin_input_ids)`. That figure is compared against N = C / (1/X - 1/P), where X and P are UNCACHED prefill throughputs. Retracted tokens are not uncached: their KV was computed and the fence persisted it, so re-prefilling them is a CACHE READ -- and a cache read costs the same in TP as in PP. Equal cost on both sides of an inequality cancels, so those tokens cannot make PP cheaper than TP and have no business in the comparison. Pricing them as cold prefill would hand the policy a huge backlog after every cutover, in BOTH directions. sgl-project#731 measured that outcome from a different cause: "51,369 -> 102,307 tokens across one cutover ... six cutovers, nothing served." Its fix (the carry consumes the queue entry) cannot catch this route, because nothing is double-counted here -- the tokens are counted ONCE, at the wrong price. On W25's numbers the shipped sum prices eight retracted 20k prompts at 160,000 against a live bar of 18,614: 8.6x over, from a cutover alone. THE OPTION THAT WAS CHOSEN, AND THE TWO THAT WERE NOT. * CHOSEN -- stamp the residency at retraction. `reset_for_retract` records the fill boundary (`extend_range.end`, the same notion of "computed" the pending counter already uses for a chunked remainder; a request with output has finished its prefill by construction) BEFORE clearing it. Exact, and free. * REJECTED -- residency lookup at the counter. `prefix_indices`, `num_matched_prefix_tokens` and `extend_range` are ALL cleared by `reset_for_retract`, so after the fact the information does not exist anywhere and only a fresh `match_prefix` walk could recover it -- per queued request, per policy round, under the tree lock, duplicating the matching admission already does. * REJECTED -- a dwell-window exclusion. Coarser in the wrong direction: it also excludes tokens that genuinely need computing, so a request retracted mid-chunked-prefill would have its real remainder deleted from the backlog. The stamp distinguishes them (6000 of 20000 computed -> 14000 still count). DEFAULT PATH BYTE-IDENTICAL. A request that was never retracted carries no stamp and is counted in full, so every pre-sgl-project#856 caller gets the same number. A request retracted for a reason OTHER than the flip seam -- priority preemption, sgl-project#731's own path -- has no fence behind it and also carries no stamp, which the can-fail test pins: zeroing anything merely flagged retracted would delete real backlog and pass every other assertion here. ERROR DIRECTION IS THE SAFE ONE. If the cache is evicted under the credit, the stamp overstates residency and this UNDER-reports pending, making the policy LESS eager to flip -- not more. It cannot cause the thrash it exists to prevent. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_uncached_pending_856.py 12 passed, 3 subtests incl. the shipped-sum reproduction of the sgl-project#731 shape, and a source-order assertion that the stamp precedes the clear (afterwards it is unrecoverable) pending-counter consumers (713/677/689/701/819/854/phase_policy) 156 passed retraction consumers (retract_decode_fcfs, prefill_adder, 798, 791b) 28 passed ruff check + format clean

The correctness core of the no-KV flip, encoded before the retirement that needs it. Once the flip carries no KV, the new phase's device pool holds no valid rows while the radix tree still maps prefixes to row ids, so the tree must be dropped for a lookup to MISS and fall through to the host tier. That action is already known to be fatal in the wrong order. sgl-project#825 tried it and took the instance down on all three ranks (2026-08-23): cache_finished_req -> dec_lock_ref -> full_component.py:239 `if cur.id in skip_lock_node_ids` AttributeError: 'NoneType' object has no attribute 'id' "PARKED IS NOT UNREFERENCED ... each holds a `last_node` with a lock ref. `reset()` rebuilds the root, orphaning those nodes, so the parent walk in `dec_lock_ref` no longer terminates at the live root and runs off the top into None." THE NO-CARRY RULE REMOVES THE PRECONDITION rather than working around it. Retraction releases each resident request's rows AND its tree lock ref (release_req -> release_kv_cache -> cache_finished_req -> dec_lock_ref, whose loop is `while node != self.root_node: node = node.parent`). Run it FIRST and every walk terminates at a live root; run it after reset() and it walks off exactly as sgl-project#825 did. So the ordering is not a preference, and it is now a named function instead of a comment two callers must remember. REFUSES RATHER THAN REPAIRS when either step is missing, and the asymmetry is deliberate: skipping the retraction leaves locked nodes for the reset to orphan (the crash, loud); skipping the reset leaves the tree naming rows that hold no KV (a WRONG ANSWER, silent) -- which is worse, so neither is allowed to be optional. THE TEST PINS THE ORDER, NOT THE OUTCOME, and that is the whole point. A test asserting only "the tree ended up empty" would pass against the fatal ordering. So the FATAL ordering is reproduced hermetically against a faithful model of the real walk -- nodes with a parent, a reset() that installs a NEW root object, and a release that walks parents until it reaches the CURRENT root, which is the entire mechanism of the crash -- and `test_reset_before_retract_reproduces_the_825_crash` asserts it still raises AttributeError('NoneType'). If that ever stops raising, the model has drifted from the crash it represents and every other assertion in the file is worthless. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_seam_order_856.py 8 passed ruff format clean Correction to the previous commit's message (3ff19e7), recorded rather than force-pushed: it said "ruff check + format clean" for scheduler.py and schedule_batch.py. Format is clean and that change added ZERO ruff errors -- verified, 144 at HEAD~1 and 144 after, ~90 of them the E402s scheduler.py's own comment already documents -- but `ruff check` on those two files is not zero and never was. The claim was imprecise. Coordination: the refill call sites (weights_arena.py, phase_flip_boot.py) are untouched here and stay that way -- W26's refill-root diagnostic owns them on fix/856-refill-root, building on the sgl-project#856(a) instrument from 7500c85.

…uild the plan empty THE RETIREMENT, performed by making the mover's INPUT empty rather than by deleting a wave loop whose extent bookkeeping (finalize_wave, span release, id-space retirement) still has to run. SEAM ORDER, now wired: fence (sgl-project#703 writeback) -> HiCache quiesce -> RETRACT ALL + DROP THE TREE -> plan rebuilt EMPTY -> weights refill -> cutover. Every downstream figure (total_slots, send/recv rows, staged bytes) derives from `tr`, so an empty `tr` is a flip that provably moves nothing. The retraction runs AFTER the fence, so every prefix worth keeping is already in the canonical store, and BEFORE the plan is rebuilt, so there is nothing left to move. `offload_kv=False`: the fence already paid for those bytes and copying them again at the one instant the instance is blocked would pay twice. REFUSES INSTEAD OF DEGRADING. No scheduler, or a tree with no `reset()` (a ChunkCache), raises SeamOrderError. Entering the next phase with a tree naming rows that hold no KV is a WRONG ANSWER, and the seam's rule is that a wrong answer is worse than a loud failure. There is deliberately no fallback to the mover. TWO QUESTIONS, TWO NAMES -- and the churn of getting this wrong is what taught it. `_staging_bytes` answers "what would this MOVE need?", and its formula is still exactly right for that question: it stays, unchanged, with every one of its pins intact (test_phase_flip_staging_reserve_631, test_seam_arena_tail_additive_656, both validated against measured corridor events). It is simply no longer the question the gate asks. `_seam_reserve_ bytes` answers "what does the SEAM reserve?" -- arena tail + draft restore + cold-stack restore, with `wave_peak` retired -- and the gate asks that. Collapsing the two under one name is the defect this build keeps removing; my first attempt did exactly that and broke 16 tests that were right. THE MOVE'S PRICE IS STILL COMPUTED AND RECORDED (`_retired_wave_peak_bytes`), because the difference between the two IS the funding claim the proof window checks, and a term that vanishes silently cannot be shown to have been retired. WHAT THIS IS EXPECTED TO REMOVE, in W25's numbers: a 2339.11 MiB tp_to_pp staging ask on PP0, behind 33 refused arms (25 on the staging rate limit) and 17 FLIP ABANDONED. TESTS REWRITTEN TO THE NEW CONTRACT -- a flip that still moves KV now FAILS: `TestStagingFormulaMatchesReality` -> `TestTheFlipCarriesNoKv`. The retired contract was "staging must cover the bytes the mover holds", asserted three ways. The new one asserts the seam's transient storage is EXACTLY ZERO in both directions (the probe measured 3.8-27.7 MiB on this fixture before), that every rank drops its prefix tree exactly once, and that the seam reserve is identical for a full plan and an empty one. The fixture still builds a real non-empty live set, which is what makes "it moved nothing" a result rather than a tautology. `test_the_unwaved_seam_cannot_afford_the_wedging_request` -> `test_the_wedging_request_is_now_affordable` -- the funding win as one assertion, on the same spendable budget that used to refuse it. `test_waving_divides_the_peak_by_about_the_wave_count` -> `test_waving_no_longer_changes_the_price_at_all` -- the wave split existed to divide a transient that no longer occurs. HONESTY NOTE: `TestMoverLiveSetIsBounded` now passes VACUOUSLY (peak 0 is below any bound) and can no longer fail. It is subsumed by the strictly stronger `test_the_seam_moves_no_kv_at_all` and is left in place only because it still guards the mover component if it is ever invoked directly. It should not be cited as evidence of anything. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): managers -k "phase_flip or seam or flip_cost or band or economy or layout_conf or uncached or takeable" 759 passed, 29 subtests, 0 failed ruff format clean; phase_flip_runtime.py ruff check clean Refill call sites (weights_arena.py, phase_flip_boot.py) untouched -- W26's refill-root diagnostic owns them on fix/856-refill-root.

…ure what warm-up costs TWO PIECES THE DESIGN OWES, both consequences of carrying no KV. 1. VALIDATE-EARLY REFUSAL (sgl-project#806 pattern, ServerArgs.__post_init__). The flip retracts every resident request and DROPS the prefix tree, so the next phase starts with an empty device tier and restores prefixes by read-through. With --enable-hierarchical-cache off there is nothing to read through: the sgl-project#703 flip-time writeback has nowhere to persist, the retracted prefixes are gone, and every conversation re-prefills from scratch on every flip. That is a correctness-shaped cost, not a tuning one, and it is INVISIBLE at runtime -- the flip completes, the requests complete, and only the token bill says anything happened. A silent, expensive, correct-looking failure is what a launch gate is for. Placed beside sgl-project#806's own check and after `materialize_declarations`, for the reason sgl-project#806 documents: the hierarchical cache is switched by handlers above, so an earlier check would pass a launch this one refuses. sgl-project#806 refuses the flip that cannot ENUMERATE what it must move; this refuses the flip that cannot RESTORE what it deliberately drops. NO FALLBACK IS OFFERED AND THE MESSAGE SAYS SO -- a test asserts the phrase. Reviving the mover here would reintroduce the seam this ticket retired, the staging reserve behind W25's 33 refused arms, and the resident carry that crashed three ranks in sgl-project#825. Both exits are named instead, because which one is right depends on what the operator wanted. 2. THE WARM-UP LEDGER (managers/warmup_latency.py), the metric the user named: not "rows carried" but served-request latency by rounds since cutover. PRIOR ART CHECKED AND IT IS A NEAR MISS, not a hit: `regime_classifier.PhaseDwellGate.rounds_since_flip` has the rounds-since-flip concept but is a GATE deciding whether a flip may happen, and carries no latency. Also searched: phase_flip_runtime for warm/latency/post-cutover, the seam census (times the seam, not what follows it), the sgl-project#605 flight recorder (the seam's own peaks). Nothing measures this. Geometric bands (<=1, <=4, <=16, <=64, steady) because the CLAIM is that the cost concentrates in the first rounds and decays; a mean cannot be right or wrong in a way that shows that. Every band reports as a ratio against THIS instance's own steady state, so the figure needs no remembered number from another boot to be readable. The can-fail directions are the file's point. `None` survives wherever nothing was compared -- a ratio against an absent control reads as "no warm-up cost" while meaning "nothing was compared", the sgl-project#606 shape in the one number this ticket is judged on. "Has not flipped" never folds into "has flipped and settled". An empty ledger still says it has nothing to say. HALF-WIRED, AND SAID PLAINLY. `note_cutover` is wired at the cutover. The request feed is NOT: request latency is assembled in `tokenizer_manager`, a DIFFERENT PROCESS, so feeding it is a cross-process change rather than a line. Recorded as an open integration point instead of guessed at -- a wrongly-wired instrument reports a number, which is worse than reporting none, and this is the number the proof window judges the design on. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_phase_flip_needs_hicache_856.py in server_args/ -- 776 passed overall test_warmup_latency_856.py 13 passed, 5 subtests managers -k "phase_flip or seam or warmup or uncached" 564 passed, 28 subtests, 0 failed ruff format clean; ruff check adds ZERO errors -- verified per file from the repo root against HEAD: server_args 358 -> 358, phase_flip_runtime 0, warmup_latency 0, corridor_admission 0. RUFF MEASUREMENT CORRECTION, since a previous commit reasoned from it: ruff must be run FROM THE REPO ROOT or it does not find pyproject.toml and reports a different rule set entirely (phase_flip_runtime.py read as "110 errors" from a stray cwd and "All checks passed" from the root). The earlier 144-vs-144 comparison was taken with the same cwd on both sides, so its conclusion -- zero added -- stands; the absolute number was just scheduler.py's 102 plus schedule_batch.py's 42.

… built The note was the spec; it now also records the build state, so a later reader cannot mistake a design paragraph for a shipped mechanism. Three lists, and the middle one is the point. BUILT AND HERMETICALLY PINNED: pending priced by residency; the seam order as a law with the FATAL order reproduced against a faithful model of dec_lock_ref's walk; the retirement (residents retracted, tree dropped, plan rebuilt empty); `_seam_reserve_bytes` with wave_peak retired while `_staging_bytes` keeps its meaning and its measured pins; the launch refusal; the old-contract tests rewritten so a flip that still moves KV fails. BUILT BUT ONLY HALF-WIRED, said plainly rather than left to be discovered: the warm-up ledger has its cutover side wired and its REQUEST side not, because request latency is assembled in tokenizer_manager -- a different process. For W27 the warm-up cost must therefore come from the client side, and if it is not collected it is UNMEASURED, never "no warm-up cost observed". That sentence is in the note because this is the number the design is judged on and it is the easiest one to accidentally report as a pass. NOT BUILT: deletion of the now-inert wave-mover code; hicache_demotion still off by default; phase_flip_rebind_hicache still False (the tree drop is what makes lookups miss, which is what correctness needs -- arming the rebind is the separate sgl-project#847 step). KNOWN VACUOUS, recorded so it is never cited: TestMoverLiveSetIsBounded now passes trivially (peak 0 is below any bound) and can no longer fail. It is subsumed by test_the_seam_moves_no_kv_at_all. Window ticket W27 appended to /spinning/gpu-arb/WINDOW-QUEUE.md with seven grep-able criteria, its preflight line left PENDING until the desk suite reports. It states explicitly that this ticket alone projects ~10.5 s of seam-blocking time and must NOT be scored as a seam-time fix -- the refill is 91% of the walk and belongs to W26 on fix/856-refill-root; the two multiply and should be read together. Documentation only.

…/851-consolidated W26 resolves the open root sgl-project#856(a) left named rather than guessed. My ledger recorded the 2.5x direction gap as UNATTRIBUTED with the four candidates I had ruled out and the search sets that failed; W26 measured it with the instrument that commit shipped (RefillLegTiming / refill_bound_phrase) and the answer is none of the four -- it is the storage read, and the gap is per-image POOL SERVICE RATE, not a code path at all. 39 paired samples, 11 flip epochs, both directions, all three ranks, EVERY sample STORAGE-BOUND: disk read 99.8-100% of the leg, H2D wait 3-7 ms, arena commit ~0 ms, with the census segment and the refill timer agreeing to the ms. Two things this settles for my line: * READ PARALLELISM IS FALSIFIED (1.15x, flat from two readers). No async or overlap work belongs in this seam: a leg waiting on unread bytes cannot be overlapped away. That closes a lever I would otherwise have costed. * The only remaining lever is sgl-project#809 section 8's PARTIAL pinned share. Whole-image pinning is not merely expensive, it is impossible here -- both W26 pin arms were OOM-killed during the launch phase, before any flip, which is itself the sizing constraint. So my standing line -- that sgl-project#856's retirement projects ~10.5 s and must NOT be scored as a seam-time fix -- now has a named successor rather than an open question. W27 carries that constraint explicitly. Doc-only merge; no code changes on either side of it.

…NCOMPLETE The proof boot did what a proof boot is for: it falsified a claim this note was making. Pin 3111539, boot_w27_0824_1510.log, operator-granted window, died on all three ranks 15:14:45Z. Full result: /spinning/gpu-arb/W27-RESULT.md WHAT PASSED, and it is the one that mattered most. At 15:14:44 all three ranks logged `RESIDENTS RELEASED ... 1 request(s) retracted and the prefix tree dropped, in that order`. Zero `NoneType' object has no attribute 'id'`, zero `dec_lock_ref` -- with the tree dropped while a request was LIVE. That is exactly the sgl-project#825 three-rank crash the no-carry rule was designed to remove, and it did not occur. The order law is now vindicated on metal and not only in its hermetic reproduction. WHAT FAILED, one second later: resident_mamba_slots (gdn_flip_mover.py:620) KvReshardError: PHASE-FLIP-GDN live request da65cfe4... has no mamba slot -- refusing to flip past unmoved linear state I retired the KV mover by rebuilding the transfer plan EMPTY after retraction. I did NOT retire `GdnFlipMover.move()` -- which this note's own retirement inventory lists as a RETIRE entry (3b0c031). So it still runs, still enumerates live requests, and finds one whose mamba slot the retraction just freed. The guard is CORRECT: refusing to flip past unmoved linear state is right, and a loud abort beats the alternative. A SECOND FINDING RIDES ON IT, and it is the more general one: `retract_all` releases rows, mamba slots and the tree lock ref, but the scheduler's batch structures still reference the `Req`. Every seam-side consumer of "live requests" therefore sees a live request with freed resources. The GDN mover is the first to hit it; the sweep for others is a build item. NO RETRY WAS ATTEMPTED, on evidence rather than caution. The obvious in-window fix -- drop the GDN mover from the flip path -- trades a LOUD CRASH for SILENT LINEAR-STATE LOSS, because the GDN anchor resume (sgl-project#745/sgl-project#747) is listed NOT BUILT in this very note. Removing a guard's caller without building what replaces it is the class of change this strand exists to refuse. THE NOTE IS CORRECTED RATHER THAN APPENDED TO. Its "BUILT AND HERMETICALLY PINNED" list claimed "the retirement"; that entry now reads "the KV retirement" and carries the falsification inline. A build-state list that survives its own disproof is worse than no list. FIVE CRITERIA HAVE NO DATA and are recorded as no-data, not as passes: no `PHASE-FLIP DONE` line was reached, so zero abandons and zero staging refusals are absence of evidence, NOT the funding claim C2 makes. Window hygiene: heartbeat stopped and its exit VERIFIED BEFORE holder release; holder -> holder.released-w27; cards 0/0/0 before and after; router PID 142 untouched (it is itself an sglang process, which is why a bare `pgrep -f sglang` matches it). `choom -n 1000` verified on the launcher AND all three rank PIDs before the weights load -- no OOM, so W26's failure mode did not recur.

…rse, and the GDN mover then retires by construction W27 killed the boot in `resident_mamba_slots` and the guard was RIGHT. This is the root behind it, and it is not the GDN mover. THE ROOT. `retract_all` frees a request's KV rows, its mamba slot and its tree lock ref -- and the scheduler's batch structures keep REFERENCING the `Req`. `_live_reqs` is the one authority for "who is resident" and reads exactly four places: every `running_mbs` slot, `running_batch`, `last_batch`, and the out-of-batch `chunked_req`. Retraction touched none of them, so every seam consumer after it was handed a live request whose resources were gone. The GDN mover was simply the first to look. FREEING A RESOURCE AND RETIRING THE REFERENCE TO IT ARE DIFFERENT JOBS. Doing only the first leaves a live object that every reader has to special-case, and the next reader added reintroduces the bug. Same shape as sgl-project#731's fix, where the carry had to CONSUME the queue entry rather than leave one request counted in two places. `consume_retracted_from_live_universe` retires the reference out of all four, using `filter_batch(keep_indices=...)` and NOT a raw `.reqs` edit -- a batch carries per-request tensors beside the list and a list edit desynchronises them. Pinned by a test that asserts filter_batch was the mechanism, because the raw edit is the tempting shortcut and it fails silently, later. FIXED AT THE AUTHORITY, AND THE SWEEP SAYS THAT IS ENOUGH. Every seam reader of the live set goes through `_live_reqs`: resident_mamba_slots (gdn_flip_mover.py:617), the KV enumeration (:835), the sgl-project#822 census (:1431), :5457, the output trace (phase_flip_output_trace.py:266), and the release itself (:8129). Grepping the seam modules for direct running_mbs / running_batch / last_batch / chunked_req reads returns comments and docstrings ONLY -- no live code bypasses the authority. THE GDN MOVER NOW RETIRES BY CONSTRUCTION, WHICH IS THE POINT. W27's no-retry refused dropping `GdnFlipMover.move()` because doing that with live linear state trades a loud crash for SILENT linear-state loss. Downstream of this fix the trade is gone: with the live universe consumed and the tree dropped, both halves of `flip_mamba_slots` -- resident slots UNION tree checkpoints -- are empty, so the mover moves nothing. No deletion; the same way the KV mover was retired by emptying its input. THE ORDER IS THE SAFETY PROPERTY, and it is pinned against the REAL guard, not a stand-in: `resident_mamba_slots` no longer refuses AFTER the consume, STILL refuses without it, and a genuinely resident request still yields its slot. So the guard is SATISFIED, never weakened -- if that middle test ever stops raising, the silent-loss trade has been made after all. W27's CONFIRMED RESULTS, kept out of the FAIL headline's shadow: * C6 PASS ON METAL: the sgl-project#825 three-rank crash did NOT occur with the prefix tree dropped under a live request -- 0 `NoneType ... has no attribute 'id'`, 0 `dec_lock_ref`. The retract-before-reset order is vindicated on hardware, not only in its hermetic reproduction. * `choom -n 1000` held on the launcher AND all three rank PIDs before the weights load; no OOM, so W26's failure mode did not recur. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_retracted_leaves_live_universe_856.py 15 passed incl. the W27 specimen reproduced (a freed request still enumerated), all four live-universe routes, the filter_batch mechanism, seam-safety (no targets / no match / a refusing filter_batch / a bare scheduler), and the three real-guard derivation tests. ruff check + format clean on phase_flip_runtime.py Full window record: /spinning/gpu-arb/W27-RESULT.md

…nd what only looks seam-only Ein-Job-ein-Mover reconciliation duty, discharged as a LIST and not as a cut. Nothing is deleted. Three changes moved the ground under the seam's funding machinery -- wave_peak retired from the ask, no KV moved, and W28 rotating the image in place -- so each piece is stated with file:line and what it still funds, for the user to decide the cut. RETIRES (payload gone): the `wave_peak` term itself; the flip's share of the staging rate limit; the GDN full-state exchange -- which retires BY CONSTRUCTION rather than by deletion, since after retract+consume+tree-drop both halves of `flip_mamba_slots` are empty. KEEPS, and this is the half that matters, because two of them look seam-only and are not: * `CorridorGuard.ensure_headroom` has THREE non-seam callers -- prefill admission, the regime dial, the VRAM dial. Retiring the seam's use does not retire the ladder. * THE ARMING FLOOR IS A PLANNER QUANTITY WITH EIGHT CONSUMERS (layout_ladder, rung_pool, seam_holdback, chunked_admission, pp_cut, prefill_frontier, phase_window, boot_instruments), only ONE of which is the seam. This is the piece most likely to be mistaken for seam machinery and cut by accident. * `_staging_bytes` keeps answering "what would a MOVE need", still pinned against measured corridor events; it is simply no longer the question the gate asks. * kv-slack's status is UNCHANGED by this ticket -- it was never funding the flip, which is precisely the sgl-project#813 complaint. UNDECIDABLE FROM THE DESK, and said so instead of guessed: whether the staging rate limiter still earns its place (W25's refusals were dominated by the KV ask; whether abandons from other causes still need pacing is a measurement the retry window's C2 counters answer), and whether `backing_slack` reaches zero in practice. The rule the list follows: a piece retires only when its payload is gone AND no other consumer names it. "Nothing calls it at the seam any more" is not retirement while eight planner modules call it. Where a piece is inert rather than dead it stays inert WITH its telemetry, because a term that vanishes silently cannot be shown to have been retired. Documentation only. (One stray `ruff format docs` reformatted six unrelated files under docs/ and was reverted before this commit; the tree carries only the new note.)

… tree drop leaks 152 rows The root fix HOLDS. 9 flips, both directions, and the two things that killed the first attempt did not happen once: has no mamba slot 0 (W27: killed all three ranks) NoneType ... attribute id 0 dec_lock_ref 0 `resident_mamba_slots` never fired. The live-universe consume removed it and the GDN mover retired BY CONSTRUCTION, exactly as derived -- the derivation is now confirmed on hardware and not only against the real guard in a unit test. C1 IS PROVEN ON METAL, which is the headline this ticket exists for. Every `PHASE-FLIP DONE` line, all ranks, both directions: 0 live slots, sent 0 cells / 0.00 MiB, received 0 cells / 0.00 MiB against W25's 116502 live slots and 995.31 MiB sent on PP0. TWO NEW COSTS, MEASURED RATHER THAN ESTIMATED, and both are mine: `hicache_quiesce->resident_release` is 753.6 ms (the retract+consume+drop), and the wave loop still walks 16 EMPTY waves for ~314 ms -- the price of retiring the mover by emptying its input instead of deleting it. That number is what turns "should we delete the dead loop" into a decidable question. THE NEW DEFECT, NAMED: ValueError: pool memory leak detected! [full] total=472864, available=126802, evictable=22, protected=0, ... withheld=345888 126802 + 22 + 345888 = 472712 against 472864 -> 152 rows belong to nobody, and it accumulates per cycle (fired on the third retract+drop). The retraction frees the RESIDENT REQUESTS' rows; the prefix tree additionally holds CACHED PREFIX rows, and `tree_cache.reset()` rebuilds the root and ORPHANS them instead of returning them to the allocator. Before sgl-project#856 the tree was never reset at the seam -- sgl-project#825 withdrew exactly that action -- so this is NEW and it is mine. Attribution checked rather than assumed: W25 ran 33 cutovers on this rig with no such error, and the broken load driver cannot be the cause because its requests failed to connect and allocated no rows. NO RETRY. The fix means freeing the tree's own rows at the drop, which means choosing the right API (`flush_cache` is a wipe, `reset` is not) and PROVING it returns rows to the allocator. Guessing a tree API at a live seam is how a loud leak becomes a silent double-free. Red-first at the desk, against this leak arithmetic; it gates the next window. OPERATOR HONESTY: my `load_w27r.sh` was broken -- curl exit 7, `code:"000"`, spinning to round 640 in 45 s. So the "responses" it logged are FAILURES, C2 and C7 were never exercised under real load, and the zero abandons above are NOT the funding claim. The staging-limiter question I left open in the funding inventory therefore stays open. The 9 flips came from the boot's own warm-up traffic. Window hygiene: heartbeat stopped and its exit VERIFIED BEFORE holder release; holder -> holder.released-w27r; cards 0/0/0; router PID 142 untouched; `choom -n 1000` held on the launcher before the weights load, no OOM.

…ip deferred with its invariant named W27-retry leaked 152 rows per retract+drop cycle and died on the third. This is the root, derived from the tree code rather than from an API guess. `MambaRadixCache.reset` (mamba_radix_cache.py:555) installs a NEW `TreeNode()` as root and zeroes `full_evictable_size_` / `full_protected_size_`. It frees no device row -- the old tree is simply dereferenced, and the rows its nodes held are orphaned. It is a BOOKKEEPING reset: right for a teardown where the pool is reset too, wrong for a seam that keeps serving. That is why the detector saw `evictable=22` (the NEW tree) while 152 rows belonged to nobody, visible only as a total mismatch. The call that actually returns rows is `evict` -> `evict_full`, whose leaf path frees through `token_to_kv_pool_allocator.free`. So the drop is EVICT-THEN-RESET (`drop_prefix_tree_returning_rows`). THE EVICTION IS LEGITIMATE ONLY BECAUSE OF THE FENCE. sgl-project#703 has already persisted these prefixes to the canonical store and the new layout re-reads them; without that, this would be data loss. That is why the seam order is fence -> retract -> drop and not any permutation of it, and it is written into the function so a later reader cannot reorder it innocently. BOTH DANGER DIRECTIONS ARE PINNED, because only one of them is loud: * orphaning -- the metal defect, modelled with the accumulate-per-cycle shape it actually had (it fired on the THIRD cycle, not the first); * DOUBLE-RETURN -- silent where the leak was loud. The test allocator raises on a second free, so an evict-and-also-free implementation fails here rather than corrupting the pool quietly. A deliberate contract change: `build_cutover_release` no longer hands back the bare `tree_cache.reset`, and `test_seam_order_856` was updated to assert that it does NOT -- the bare reset is the tempting one-liner and it is exactly what leaked. THE EMPTY WAVE LOOP IS DEFERRED, WITH THE INVARIANT NAMED. W27-retry measured 16 empty waves costing ~314 ms plus 753.6 ms for retract+consume+drop. Skipping the loop looks like free seam time and is not safe yet: `finalize_wave` calls `dst.restore_backing(layers)`, which marks the destination layers RESIDENT independently of whether any KV moved. Skipping would leave the pool answering NO to `backing_is_resident`. The cheap successor is one `restore_backing(all layers)` outside the loop; it changes backing semantics and needs its own test, so it is a follow-up and not a line here. THE LOAD DRIVER IS FIXED, and its bug was a measurement trap worth recording. W27-retry's driver reported real token counts on requests that never connected: curl failed (dead-port smoke confirms code='000', rc=7, NO file written) and the parser then read `/tmp/w26_resp_<tag>.json` -- a LEFTOVER from the W26 window, same tag scheme, same prefix -- and reported W26's numbers as this run's. Two fixes: a unique prefix plus `rm -f` before the curl, so a failure can only ever yield a MISSING file and never someone else's numbers; and a health check that ABORTS after 3 consecutive failures. Mock-smoked against the dead server: exit 3 at round 4, instead of the 640 rounds in 45 s the broken one spun. GATE (foreground, family-batched, per the sgl-project#749 lesson; no background waiters): managers core (45 PP files excluded) 3554 passed, 18 skipped, 336 subtests, EXIT=0 PP family, one file per process 45/45, 0 failures mem_cache 1710 passed, 361 subtests, 0 failed server_args 776 passed, 191 subtests, 0 failed model_executor 763 passed / 15 failed The 15 are the pre-existing sgl-project#815 family, verified earlier against HEAD (identical 15/754 set). ZERO new failures.

…all three falsifiers First slice of the user's flip-image design: RAM holds ONE layout image plus a small overshoot, and at the flip the incoming layout streams RAM -> VRAM while the outgoing one streams VRAM -> RAM into the pages just freed. PCIe is full duplex, so the copy-back rides the idle return direction. THE COPY-BACK IS NOT WRITE-BACK. The weights are immutable and nothing is saved; it is residency PLACEMENT for the next flip, which is what a single-layout RAM budget requires. Written into the module docstring because a later reader who mistakes it for a write-back will optimise it away and break the following flip. WHY THIS AND NOT THE PARTIAL PIN: W26 proved the dual pin impossible here -- both pin arms OOM-killed in the LAUNCH phase, before any flip. One layout plus eps (~30 GiB vs ~68.7 GiB) fits AND takes the disk off the steady-state critical path, which is what reaches the physics floor; a partial pin leaves a disk share behind, and W26 measured the leg 99.8-100 % storage-bound. THIS SLICE IS THE ARITHMETIC ONLY, deliberately. The overshoot sizing and the interleaved schedule are pure functions over byte counts, so every invariant the scheme rests on is falsifiable WITHOUT a GPU -- the same split sgl-project#852's estimator and sgl-project#856(a)'s bound phrase use, and for the same reason. OVERSHOOT = size asymmetry + in-flight window, sized from the LARGER direction. The asymmetry is W26's measured one (PP0 15925.8/16362.7, PP1 8573.8/8961.3, PP2 8573.8/9481.6 MiB); a single fixed reservation has to cover whichever direction the next flip takes, so a mean is the OOM. The in-flight term is separate and pinned: an implementation returning only the asymmetry gives 0 for equal layouts and stalls immediately. ALL THREE NAMED FALSIFIERS ARE ASSERTED: * no actual overlap -- `rotation_totals` counts co-scheduled steps; a real rotation must have them and must have them as the DOMINANT shape (>90 % of steps), not as an accident of the tails. Its can-fail partner: a one-sided rotation must report zero overlap. * RAM leak across cycles -- three full A->B->A cycles must return host occupancy exactly to its start. Three, because W27-retry's leak fired on the THIRD cycle, not the first. * checksum -- verified against the real source: the image is `payload = image[:layout.total_bytes]` plus an int64 trailer, checked with `uint8_checksum(dst)` over the ARENA. That last part is what makes a D2H reproducible: bytes returned from VRAM verify exactly as bytes read from disk do, so only the 8-byte trailer is new. A FINDING THE TESTS PRODUCED, and it is why the budget test first passed vacuously: THE RAM BUDGET BINDS IN ONLY ONE DIRECTION. Pressure exists solely when the OUTGOING layout is LARGER than the incoming one, because only then does the copy-back need more RAM than the H2D frees -- PP0 copying back its 16362.7 MiB tp image while the smaller 15925.8 MiB pp image streams in leaves 436.9 MiB with nowhere to go. The opposite direction schedules cleanly at zero overshoot. Both halves are now asserted so the asymmetry is recorded rather than rediscovered. Under-sizing STALLS LOUDLY rather than proceeding: a scheduler that kept going would be holding both layouts, which is precisely the state that OOM-killed W26's pin arms. GATE (foreground, family-batched): managers core (45 PP files excluded) 3554 passed, 336 subtests, 0 failed model_executor 777 passed / 15 failed (+14 new) The 15 are the pre-existing sgl-project#815 family. ZERO new failures. NOT YET BUILT, and not claimed: the device-side execution (streams, the pinned ring registered once per sgl-project#720/sgl-project#729, the planner-priced host post per sgl-project#721/sgl-project#770), and the separately-instrumented priming flip. This slice is the plan those will execute.

…the aliasing defect the arithmetic could not see Slice 1 fixed the arithmetic. This is the executor that runs it, and building it surfaced a defect in the plan that no byte-count model can express. THE ARENA IS ONE BUFFER AND THE ROTATION IS AN IN-PLACE PERMUTATION. `allocate_arena` returns a single contiguous device tensor sized max(pp, tp) and `arena_refill` overwrites `arena[: layout.total_bytes]` in place (weights_arena.py:1184,1192). Under a single-layout RAM budget the host image is likewise ONE buffer. So at every chunk offset k the two directions are CIRCULARLY dependent: the H2D wants to write arena[k], which the D2H has not read yet, and the D2H wants to write image[k], which the H2D reads. Slice 1's plan emits h2d_offset == d2h_offset while both are active, so a literal execution of it aliases on every step. Serialising removes the duplex the scheme exists for; running concurrently corrupts the image -- and it corrupts it in the direction THIS flip's checksum cannot catch, because the damage lands in the image the NEXT flip streams in. THE RING IS THEREFORE LOAD-BEARING, not an optimisation, and this is the part a later reader is most likely to undo. Per chunk: save image[k] into a ring slot, D2H arena[k] -> image[k], then H2D ring slot -> arena[k] gated on that D2H. Chunk k+1's D2H is enqueued before chunk k's H2D is waited on, so the lanes genuinely run together with no aliasing anywhere in the pipeline. The one host-to-host memcpy per chunk is the intrinsic cost of an in-place rotation, not an accident of this implementation. AND THE TWO SLICES THEN AGREE BY CONSTRUCTION rather than by coincidence: one max-sized host buffer (one image PLUS the size asymmetry) plus depth*chunk of ring IS `rotation_overshoot_bytes`. The same number reached twice, from two directions. OVERLAP IS MEASURED ON THE EXECUTOR, NOT THE PLAN. A step counts as overlapped when, at the instant its D2H is enqueued, an earlier H2D has not been waited on. Slice 1 could only count co-scheduled steps, which a serialised implementation would also produce. Its can-fail partner is pinned: a ring of depth 1 must report exactly zero overlap and must still be byte-correct. PRIOR ART REUSED, NOT REBUILT. The ring is sgl-project#720's ReadBufferPool, which charges the pinned-host registry BEFORE allocating (sgl-project#729), exactly as `weights_arena._refill_staging_pool` already composes it; registered once per process and reused by every flip. The readout is sgl-project#856(a)'s RefillLegTiming / refill_bound_phrase -- no new telemetry. The launcher PRICES the ring with the pure `joint_pinned_host_error` and does NOT register it: a planner-side registration of bytes the launcher never pins is the helper commit 272d0d9 deleted, and reintroducing it was the obvious wrong move here. THE DEFERRED WAVE-LOOP SUCCESSOR, FOLDED IN. W27-retry measured 16 empty waves for ~314 ms. Skipping the loop was deferred because `finalize_wave` is what marks the destination pool resident, and a bare skip leaves `backing_is_resident` answering no. The replacement is the whole-pool swap ALREADY on the same object (`WavedBackingSwap.__call__`): release source, reclaim, restore destination. It is not merely equivalent, it is strictly cheaper -- waving exists to bound the transient of holding a source layer live while its destination is written, and with no bytes crossing there is nothing to bracket, while `__call__` releases before it restores so its peak is max(src, dst). Gated on a new `PhaseFlipTransition.moves_nothing` that checks ALL THREE legs: a predicate reading only the peer exchange would skip a plan that still has a LOCAL move and drop KV silently, which is the one outcome worse than the 314 ms. R1's directional budget law is carried forward unregressed: pressure exists only when the outgoing layout is the larger, and both halves stay pinned. THE PRIMING FLIP IS INSTRUMENTED APART (P4), with its own stats and its own timing record, so a steady-state mean can never absorb it. WHAT IS DESK-PROVEN vs WHAT NEEDS THE WINDOW. Every test here executes the REAL executor over REAL byte patterns on CPU tensors: byte-exactness in both directions, the checksum, the absence of drift across three A->B->A cycles, and the ring returning to full are EXECUTED, not modelled. Only the CUDA lane mapping needs metal. A mutant that bypasses the ring save kills 13 of the tests, so the suite detects the corruption it is written against. GATES (foreground, family-batched, PYTHONPATH pinned to this worktree): model_executor 15 failed / 803 passed (baseline 15 / 780) -> 0 new managers batch A (240 f) 2640 passed, 1 failed + 1 collect error, both identical at HEAD managers batch B (63 PP files, one process each) 684 passed, 0 failed mem_cache (chunked) 31 failed / 2108 passed, byte-identical to HEAD; one chunk segfaults at HEAD too (pre-existing) server_args 4 failed / 793 passed, the same 4 at HEAD new suites 48 passed (rotation executor 23, plan 14, wave 11) ZERO new failures. ruff F/E7/E9 clean on new files; the two large touched files are 358 -> 358, unchanged. NOT CLAIMED: nothing here has run on a GPU. The boot-side change that allocates ONE max-sized host image instead of two, and the wiring of the executor into PhaseFlipStacks.refill, are the remaining steps before the proof window.

…chable, not merely outranked USER DIRECTIVE (2026-08-24): all pending prefill is collected and processed in the PP layout, then all decode in TP, then prefill again -- and NEVER any work in the wrong layout, "egal wie lang der flip dauert". Flip duration is recorded from here on, not traded against. THE BAND HAD TO STOP BEING CONSULTED, and that is the whole content of this slice. `drain_mode` already existed and its PP half was already strict (the DRAINED rule flips at `pending <= pp_exit_tokens`, default 0). The TP half was not: the break-even band sits ABOVE the drain exit in `_decide_from_load` and can `return _no(...)`, so a correct drain rule underneath is NOT sufficient -- the band holds TP while prefill waits, and prefill then runs in the decode layout, which is the one thing this mode forbids. `drain_mode_strict` gates the band off. One condition, not a new engine. DELIBERATELY NOT a third `--phase-flip-policy` value: that enum selects the manual-vs-auto ENGINE, not the arming rule, and it is a different axis from the `phase_purity` MODE_STRICT enum. Conflating any two of the three would make the next reader's mental model wrong in a way the tests would not catch. STRICT IMPLIES DRAIN, refused rather than half-applied: the exit strict relies on lives in the drain block, so strict-without-drain would gate the band away and then fall through to the very economics it exists to remove -- configured in appearance, neither mode in behaviour. THE ECONOMIC MODE IS UNTOUCHED for every other workload, and the tests say so in both directions: same load, same numbers, the band still holds TP without strict and cannot be the reason for anything with it. The N~=28,050 repricing from the previous slice stays DOCUMENTATION -- in this mode it is not the trigger, so it is not tuned further. TWO TEST TRAPS THIS FILE FELL INTO AND NOW NAMES, because both would have made it pass while proving nothing: * a config whose band cannot be ENTERED (threshold == N) passes whatever the code does. `TestTheBandIsOpenAtAll` pins threshold 3000 against N 1000. * comparing strict against economic-WITH-drain shows no difference, because the existing drain rule already flips a drained bundle on any backlog. The sub-N test compares against PURE economics (drain off). * a state with `bundle_at_phase_entry=0` arms the "decode phase ran EMPTY" rescue in EVERY mode, hiding the difference behind a rule neither owns. RECIPE (boot_w29.sh), from this session's own scorer finding: the W28 specimen had 153 prefill batches EXECUTING IN TP LAYOUT. That followed from `--phase-flip-purity prefill_in_tp`, so the recipe now boots `--phase-flip-purity strict --phase-policy-drain-mode-strict`. w29_score.py reads conformity from the scheduler's own `phase=pp/tp` batch markers -- independent of the sgl-project#838 detector on purpose, so the detector stays falsifiable rather than self-certifying -- and now also SCORES purity stand-downs: the `_relaxed` escape valve stays (correctness over starvation) but every stand-down is a departure from the target mode and must not pass silently. GATE (foreground, family-batched): phase policy suites 172 passed managers (chunked) 3193 passed; 1 failure + 1 collect error, both identical at HEAD server_args cli-metadata / ratchet / migration 18 passed, 19 subtests strict-batch suite 9 passed ruff F/E7/E9 clean; format clean.

…ry tree answers W29 died on all three ranks with a one-row pool deficit: pool memory leak detected! [full] total=469733, available=107041, evictable=1, protected=0, session_held=0, uncached=0, withheld=362690 The seam's own sgl-project#832 census had already named the orphan by id -- `unaccounted=1 [1]`, flat from the first flip that crossed a non-empty prefix tree onward, on every later census in both directions. ROOT. `drop_prefix_tree_returning_rows` is the W27-retry fix for the 152-row orphan: evict-then-reset, so rows go back through the allocator before `reset()` rebuilds the root. It decided how much to evict with `getattr(tree, "full_evictable_size_", 0)`. That trailing-underscore attribute belongs to exactly three caches (MambaRadixCache, SWARadixCache, HiMambaRadixCache). The live tree is UnifiedRadixCache, which keeps the same quantity in `component_evictable_size_` and has no such attribute -- so the read returned ZERO, the eviction was skipped outright, and `reset()` orphaned the tree's rows exactly as before. The W27-retry fix was inert on this tree from the day it shipped. The deficit is the SIZE OF THE TREE, not a constant. It read as a constant unit only because both ranks held exactly one cached row (a 1-token health check) at every drop. W27-retry's 152 and W29's 1 are one defect at two occupancies. FIX, one writer and one clock. `tree_evictable_full_rows` reads the BasePrefixCache contract method `full_evictable_size()`, which every cache implements and which, on the three attribute-keeping caches, returns that very attribute -- a strict superset of the old read, never less. A tree that cannot answer yields None, not 0, and the caller says so: zero is a licence to skip the eviction, and skipping it is the defect. The evict is then RE-READ, because it can stop short on a locked or un-backed node (sgl-project#841), and residue is reported before reset orphans it. The seam now prints how many rows the drop returned. It already computed the number and its own docstring said it "has to be visible"; the caller discarded it, so a drop returning zero rows logged identically to one returning all of them, and the only reader left was a census one pass too late to name an owner. Nothing in the invariant checker is softened, widened or special-cased. The abort was correct. WHY THE SUITE WAS GREEN THROUGH A DEAD BOOT: this file's `_Tree` double carried the private attribute and not the public method -- backwards from every shipped cache. The double is corrected, a UnifiedRadixCache- shaped double is added, and a drift-detector now asserts against the REAL classes that each answers `full_evictable_size`, and that neither reader's body touches the private attribute again. TESTS test_tree_drop_returns_rows_856.py + test_seam_order_856.py + test_phase_flip_mover_streaming_631.py: 43 passed. CAN-FAIL (a): TestTheCheckerStaysStrict asserts _check_pool_invariant still flags W29's 1-row deficit AND W27-retry's 152-row orphan, and passes a balanced pool. CAN-FAIL (b): reverting only phase_flip_runtime.py turns 6 tests red, including all four TestTheLiveTreeShape cases. Verified, restored. managers unit suite (4026 collected): HEAD e8bce97 9 failed / 4004 passed / 2 skipped with this fix 8 failed / 4016 passed / 2 skipped The 8 are a strict subset of HEAD's 9; no new failures. Pre-existing: test_phase_flip_staging_reserve_631 (x4), test_rank_prefill_log (x1), and, under full-suite ordering only, test_hisparse_unit (x2) and test_draft_cuda_graph_removal (x1). ruff check + format clean on both files. Specimen and full record: /spinning/gpu-arb/W29-RESULT.md, section W29-A2.

…retracted for W30 ran the strict-batch acceptance and found two blockers behind the pool leak it had come to prove fixed. Neither is in the pool accounting, which was clean across 150 flips. FIX A -- `_live_reqs` enumerates `last_mbs`. Arm 2 died on all three ranks in 21 s: ResidentCarryError: PHASE-FLIP-CARRY 1 request(s) are reachable only through last_mbs/last_batch at the cutover: ['56fddcc3c0ef...'] Two functions disagreed about what "resident" means. `_live_reqs`, the authority the RETRACTION uses, read running_mbs / running_batch / last_batch / chunked_req. `orphan_resident_reqs`, the cutover guard, checks last_mbs AND last_batch. So `last_mbs` was a route the retraction could not see and the guard did check: a request freshly prefilled under event_loop_pp sits in last_mbs[slot], was never retracted, and the guard correctly refused. Fixed at the authority, exactly as the W27 fix was -- it simply stopped one route short. TWO sites needed it: enumerating without CLEARING would retract the request and leave the guard's reference intact. The guard is untouched and stays as strict as it was; its docstring rule holds ("a bug to raise, not a carry to widen"). FIX B -- the seam re-admission is FLIP TRANSPORT, and only that. Arm 1: 150 flips in 17 minutes, 129 prefill batches, ZERO decode batches, 28 of 28 client requests timing out at 600 s. The arm auditor called it 12 times: "armed pp_to_tp (... 1 req decoding ...), the cutover COMMITTED into the target layout, and it still built no batch in 8 rounds". Every link is a shipped decision. The policy arms pp_to_tp BECAUSE a request is ready to decode; the sgl-project#856 seam then retracts that very request (no-carry); re-admitting it in TP needs a read-through prefill; strict purity forbids prefill in TP absolutely (`Prefill batch phase=tp` count for the whole arm: 0); so TP builds nothing and the policy flips back. The arm's justification is destroyed by the arm's own execution. The re-admission recomputes NOTHING -- the tokens were prefilled in the PP window and their KV is in the canonical store from the sgl-project#703 fence. It is a cache restore: seam mechanics, the same category as the KV the flip moves. So it is exempt BY NAME rather than by a purity stand-down, which would let ordinary prefill into TP and be counted a violation by both the sgl-project#838 detector and w29_score.py -- making the acceptance unpassable by our own instruments and dishonest about the user's rule. THE MARKER IS SEAM-SPECIFIC ON PURPOSE. `is_retracted`/`retracted_stain` already existed and were readable at the gate, but `reset_for_retract` is reached from four paths: decode-OOM preemption, the PD prefill path and the PP void path set the identical booleans. Keying on them would exempt every preempted request's re-prefill, which is real work. So the cutover stamps its own `Req.seam_readmit_epoch`, spent on the one re-admission it licenses. The builder additionally keeps an exempt batch to stamped requests only, so a new arrival cannot ride along inside it. Rank-uniform: the stamp comes from a group-unanimous cutover and is read off the replicated waiting_queue. THE LIVELOCK SAFETY NET. The purity valve arms on flips that are guarded, abandoned or refused. W30's flips all COMMITTED, so the valve was blind to it and stood down 0 times through a ten-minute livelock. The signal it needed was already being written to a log with no consumer: ARM-VERDICT-WRONG is now booked as a streak and read as a fourth cause. Its rank-uniformity is the weakest of the four and says so -- it is bounded away, ~32 batchless rounds deep, past anything transient. w29_score.py now counts it as a HAZARD, so if this net fires the run fails loudly: it is a net under the acceptance, never a way to satisfy it. Verified both ways -- 13 hazards on the W30 arm-1 specimen it previously scored as 0, still 0 on the W29 specimen. FIX C -- an arm may not destroy its own justification. `PhasePolicyConfig.seam_readmit_available`, static boot config. The DRAINED branch refuses with a named reason when the target cannot re-admit what the cutover retracts. NOT gated on `inp.target_can_admit`, the tempting one-liner: at arm time the residents are not yet retracted and so not yet stamped, so that term is False for the very arm that would make it true, and gating on it deadlocks the other way. TESTS 15 new (test_seam_transport_exemption_w30.py) + 4 new (test_retracted_leaves_live_universe_856.py), red-first. CAN-FAIL, the dangerous direction: a genuine new request, an OOM-preempted request, and an empty queue must ALL still be blocked in TP; the round flag must not latch; exactly one site in python/sglang/ may write the stamp. CAN-FAIL, the guard direction: `assert_no_orphan_resident_reqs` must STILL raise without the consume -- if it stops, the guard was widened rather than satisfied. DRIFT-DETECTOR: every residency route the guard checks must appear in both the authority and the consume path. managers unit suite: 8 failed / 4035 passed, against 8 / 4016 before this slice and 9 / 4004 at e8bce97. Same 8 pre-existing failures throughout; +19 passes = the 19 tests added here. Targeted policy/purity/flip family: 283 passed. ruff clean, delta 0. Window record and specimens: /spinning/gpu-arb/W30-RESULT.md

… after it W31 arm 1 reproduced the W30 livelock with the fix for it in the tree and unreachable. The seam retracted 87 requests across 39 pp_to_tp flips and logged `SEAM TRANSPORT ADMITTED` 0 times and `Prefill batch phase=tp` 0 times. The exemption sat after `prefill_allowed_in_tp`, which is BELOW the sgl-project#677 drain-mode suppression. The recipe runs --phase-policy-drain-mode, so `prefill_suppressed_in_tp` returned True and `prefill_blocked_here` returned before the exemption was ever evaluated. This function already carries a note about the identical shape: "What broke was ORDER -- suppression was checked FIRST and returned True, so the valve never ran." THE ORDER IS SUBSTANTIVE, NOT COSMETIC. Drain mode forbids TP prefill because "a TP window entered to finish a bundle must not admit the work it was entered to escape". A request the cutover ITSELF retracted a moment earlier is not that work -- it IS the bundle the window was entered to finish. Suppressing it does not defend the drain contract, it makes the contract unsatisfiable: the bundle can never complete. Moved, not copied: one call site, pinned by a test. TESTS (4 new, 19 in the file) * a stamped request IS admitted under drain mode; * CAN-FAIL: ordinary prefill is STILL suppressed under drain mode, so sgl-project#677's contract is qualified rather than dissolved; * ORDER pinned in the source -- the thing arm 1 got wrong and which no behavioural test on a passing path can see; * single-sited. Proof the ordering is load-bearing: `prefill_suppressed_in_tp` returns True for this exact config, so below the gate the request was blocked. Targeted gates incl. the sgl-project#677 drain suites: 228 passed. Specimen: /spinning/evidence-665-f1/SPECIMEN_w31_a1_exemption_below_drain_gate.log

The missing half of sgl-project#856. `retract_all` returned the list it retracted, `_release_residents_for_cutover` returned it upward, and its caller discarded it. No seam path ever called `_add_request_to_queue`. The seam's own log line promised on every flip of every boot that "the new layout re-admits them and serves the prefix by read-through"; nothing performed the re-admitting. W31 arm 2 measured the cost: 28 distinct rids, each admitted EXACTLY once ever (three ADMIT lines apiece, one per rank), 14 requests prefilled once, 78 requests retracted across 42 cutovers, and ZERO completions -- every client waited out its 600 s timeout. W30 and W31 were both read as a flip "livelock"; they were the flip ping-ponging over an instance whose work it had already dropped on the floor. `Scheduler.readmit_seam_residents`, called from the release site. ORDERING: consume FIRST (inside `_retract_and_consume`), requeue SECOND. A request that is simultaneously live-referenced and queued is double-billed by every consumer that sums the two -- the sgl-project#731 shape. QUEUE POSITION: FRONT, as a block, in original arrival order. These are the oldest work AND the flip's own justification -- the tp-ward arm fires because they are ready to decode. Appending them behind arrivals that landed during the ~6 s flip lets a busy instance starve the bundle it just flipped for. Order is restored from `kv_arrival_seq`, which `_add_request_to_queue` already preserves across a retracted re-queue. ABORT PATH: correct by construction, because the requeue happens AT the release site. There is no window in which the list exists and is owned by nobody -- if the cutover raises after this point the flip abandons, the layout is unchanged, and the requests are already on the SOURCE layout's queue, which is where an abandoned flip should leave them. Deferring this to the end of the cutover would recreate the defect for exactly the abort case. sgl-project#703 FENCE: asserted, not assumed. `_writeback_fence_ms` returns None for "NO FENCE RAN" -- a real state, since the fence is skipped without a canonical store -- and that must never read as "fenced, cost 0 ms". The seam names it when it re-admits unfenced: not a wrong answer, but the silent cliff this no-carry design exists to avoid. ONE MOVER, NOT TWO: it calls `_add_request_to_queue(req, is_retracted=True)` and then moves that block to the front, so it inherits the priority validation, the queued-limit abort, the retract timestamp and -- load-bearing here -- `_prefetch_kvcache`, which is what makes the promised read-through hit. RETRACTED MUST EQUAL READMITTED: the seam compares and logs `RE-ADMISSION MISMATCH` on any difference, so the next boot's first check is arithmetic rather than inference. TESTS (14, red-first; the stand-in binds the REAL method) * all N return, exactly once, queued as retracted; * FRONT of newer work; arrival order restored even though the seam enumerates residents by slot, which is not arrival order; * a request the queue legitimately refuses is not conjured into the block; a client that gave up is not re-admitted; the count reports what LANDED, not what was tried; * the stamp SURVIVES the round trip, checked through the real `seam_readmit_candidates` -- stripping it would put the requests back where strict purity still refuses them, i.e. the W30 livelock again; * rank-uniformity: two ranks given the same requests in different slot-enumeration orders rebuild the identical queue; * abort path and consume-before-requeue pinned by source order. CAN-FAIL, proven by probe: re-applying the original defect (discarding the return value) turns 3 tests red; restoring makes all 14 green. Full managers suite 8 failed / 4054 passed -- the same pre-existing 8; the strand reads 9/4004 -> 8/4016 -> 8/4035 -> 8/4054. ruff clean. Window record: /spinning/gpu-arb/W31-RESULT.md

…le for a no-carry seam W32 measured the policy arming straight back out of the layout the seam had just flipped into, 23 times: "arming tp_to_pp: pending prefill 1 tok > 0 (purity: prefill cannot run in tp)". The purity gate knew about the seam-transport exemption; the POLICY kept its own copy of the same rule, so re-admitted residents read as pending PP work and the exemption got to run ONCE in 144 pp_to_tp flips. That is the second time a correct mechanism has been overridden by a second site enforcing the same payload (W31 arm 1 was the same shape one level down). So this makes one function canonical and has both callers derive from it. MANDATORY INVENTORY, per Ein-Job-ein-Mover. Sites enforcing "may prefill run in TP": phase_purity.prefill_blocked_here (gate); the phase policy (fixed here); w29_score.py (fixed at W32); scheduler._phase_admits("prefill_in_tp") -- still a copy, named in the record, not yet unified. NOT this payload: scheduler.py's boot-time `prefill_runs_in_tp` collapse (static, from the mode) and model_runner_kv_cache_mixin's `survivable` (KV sizing). `seam_transport_pending_tokens` is the one authority; it and `seam_transport_exempt` both derive from `seam_readmit_candidates`. Transport tokens are subtracted at the POLICY INPUT BOUNDARY, not at each trigger, so the drain exit, the break-even band and the tp-ward arm all see the same corrected quantity rather than three more copies of the judgement. DECODE-EMPTY: the rule fired on every no-carry TP entry (26x in W32) because under sgl-project#856 the bundle is absent at entry BY CONSTRUCTION and arrives one round later via the re-admission. It enforced a carrying-seam invariant that no longer exists. New input `seam_transport_tokens`: while a re-admission is in flight the policy WAITS for the bundle instead of flipping away from it. Empty stays a loud defect when no re-admission is outstanding. TESTS * both answers must come from the same candidate function; * DIVERGENCE PROOF: monkeypatching the shared function must change BOTH callers together -- if they ever stop sharing a source of truth this fails, which is exactly what W32 measured; * the subtraction is pinned at the boundary, not per-trigger; * CAN-FAIL: unstamped pending is still PP work, or the policy would stop returning to PP at all. Targeted policy/purity/flip family: 279 passed, then 187 on the policy-focused subset after the decode-empty change. ruff clean.

…hat are not it INVENTORY RULE DISCHARGED. `_phase_admits("prefill_in_tp")` is the hypothetical `target_can_admit` term -- "could the TP layout admit prefill", asked while another layout is active -- so it cannot call `prefill_blocked_here`, which early-returns on the ACTIVE phase. It therefore carried the purity judgement itself and knew nothing about the seam-transport exemption. It now DERIVES the exemption from the one authority, `seam_readmit_candidates`, exactly as `seam_transport_exempt` and `seam_transport_pending_tokens` do: a fourth caller of the same function, not a fourth copy of the judgement. This mattered concretely: a `target_can_admit=False` computed without the exemption is precisely how the arm auditor concluded the verdict was wrong while the mechanism to make it right was already installed. The same shape has now cost two windows -- W31 arm 1 (exemption below the drain gate) and W32 (the policy's own copy) -- and this site was the next one waiting. The two sites that are NOT this payload are pinned with the reason, so the next Ein-Job-ein-Mover sweep does not re-litigate them: * scheduler.py:656 -- BOOT-TIME config from the purity MODE, collapsing the policy's break-even N. Deliberately not a caller: the seam-transport exemption is per-round state and must never be baked into static config. * model_runner_kv_cache_mixin.py -- a SIZING question ("must a TP-phase prefill be survivable in this pool"), not the per-round question. If anything the exemption makes it MORE permissive, never less. Targeted: 131 passed. ruff clean on both files.

…t; handshake liveness #861f -- THE ROOT FIX. W37-E deadlocked on the #861e formulation: seven retracted-with-output requests counted as decode work, so the demand term stayed silent for requests that were not decoding at all. They sat in the waiting queue needing a prefill pass TP may not run. GPU 0 % for 198 s, flips frozen at 9. A retracted-unfinished request IS PREFILL WORK waiting for the pp layout. Counting it as decode work was the category error that let both sides veto while neither served. decode_work_bs() now reports genuinely resident decoding ONLY; the d4 anti-chop protection moves to bundle_is_mid_flight(), gated on COMPLETED DECODE STEPS (MIN_DECODE_STEPS_PER_PHASE = 8) -- not wall seconds (d4 produced one token per flip cycle while every seconds guard was happy) and not "requests that exist somewhere" (which is what deadlocked W37-E). RED/GREEN proven behaviourally against 02bd706 in a TEMP WORKTREE, because `git stash` is banned here (shared stack across worktrees): 02bd706: decode_work_bs=7, demand=0 -> NO EXIT fix: decode_work_bs=0, mid_flight=False -> EXIT EXISTS THE EXISTENCE STAMP. `cached_prompt_tokens_at_retract` answers an ECONOMICS question and credits the ENTIRE prompt to any request with >=1 output token, so every backlog counter read 0 for the seven wedged requests. `Req.needs_prefill_pass` is stamped at reset_for_retract and answers the different question; _admissible_prefill_tokens counts the full prompt for a stamped request instead of subtracting a credit against a prefix tree the seam already dropped. #861g -- SERVABILITY INVARIANT. Two deadlocks in one night from independently correct vetoes (sgl-project#858: strict purity x sgl-project#856 no-carry, 150 flips and ZERO decode batches; W37-E as above). Two is a class, and each cost a GPU window. managers/servability_matrix.py proves per (request state x phase) that at least one path to service survives with ALL gates evaluated together. It INDEPENDENTLY rediscovers the W37-E deadlock and blames the same three terms; after the fix the deadlocked set is empty, exactly one cell closed and none opened. Carries the sgl-project#858 pair as a second red fixture. #861h -- HANDSHAKE LIVENESS. `_await_handshake` had a deadline and no liveness check, so a dead child cost the full timeout: py-spy showed the test at 0 % CPU while its child was already <defunct>. Now 250 ms slices with is_alive() checked every slice, immediate raise carrying the child's exit code, deadline as backstop. Measured: the same test now raises in ~15 s. SELF-CAUGHT: the first cut used `.poll()`; the object is a multiprocessing.Process, so it would have AttributeError'd on the very failure path it was written for. Corrected to match execute_script's idiom -- one liveness idiom per file. AND THE DEATH ITSELF, diagnosed hermetically: the child raises "No accelerator (CUDA, XPU, HPU, NPU, MUSA, MPS) or platform plugin is available" from ServerArgs.__post_init__ -> get_device() under CVD="". The whole scripted_runtime suite requires cards. Per the CUDA test discipline it now DECLARES that in its own conftest and skips (42 skipped) rather than reporting failures that say nothing about the tree. The skip consults the same get_device() the child calls, so condition and failure cannot drift apart. Two of my own #861e pins encoded the falsified design and are corrected with the reason recorded. The Cut-2 ast check is now FUNCTION-AWARE: a coherent accessor is exactly the place that may read the raw field.

…not by handover THE DEFECT. The sgl-project#856 cutover retracts every resident and keeps nothing, so the state a phase transition must preserve simply does not survive it: * PHASE-FLIP-GDN moved 0 slot(s) on ALL 228 flips of the W37 series. The GDN mover reads `flip_mamba_slots` = resident mamba slots UNION tree checkpoints, and BOTH are destroyed at phase_flip_runtime.py:10829, one seam step before the mover runs at :11139. * The sgl-project#703 fence walked an empty tree: eligible=0 on 78 of 85 cutovers, acked=0 on every fence of the boot. * #cached-token: 0 on 403 of 403 prefill batch lines (W37-H arm A, the control on this tree). WHY NOT AN INSERT. Persisting at that instant by inserting into the tree was tried and measured: W37-H arm B died 33 s after ready, all three ranks, `pool memory leak detected! total=468981, available=108565, evictable=1, withheld=360437` -- 108565+1+360437 = 469003, i.e. 22 rows claimed twice. `readmit_seam_residents` brings the population straight back ("W31: RE-ADMIT THEM") and it resumes on rows the tree had taken. Ownership transfer is structurally wrong for a population that returns; `cache_finished_req` is the finish CODE PATH, not a finished request. THE FIX, IN TWO HALVES THAT ONLY WORK TOGETHER. Half 1 -- copy at the cutover retraction. `release_req(..., copy_state=False)` calls `offload_kv_cache` BEFORE `release_kv_cache`, while the rows still hold live bytes. Row ownership is untouched, so the arm B double claim cannot recur (`is_insert` stays False). Threaded through `retract_all` and set True at EXACTLY ONE line, `build_cutover_release._retract`. That single-site property is load-bearing: `retract_all`'s only other caller is the decode-pressure path, whose rate is load-dependent, and the host budget (0.585 GiB/cutover, ~57 MiB/s aggregate) is priced at the flip cadence. "Ungated by design" means no flag -- the condition is structural. Half 2 -- restore where the rows come back. NOT in `readmit_seam_residents`, which only re-queues: by then `reset_for_retract()` has cleared `req_pool_idx`. `restore_seam_state` runs in `prepare_for_extend`'s per-req loop, after `alloc_for_extend` (rows exist) and before `is_retracted` is cleared -- the same shape as the proven disagg path, `_pre_alloc` then `load_kv_cache`. The guard is the PRESENCE OF THE COPY, not `is_retracted`: the flag is also true for pressure retractions that never copied, so keying on it would request a restore that does not exist. The copy is consumed, so a second pass cannot re-apply stale bytes. Enabling change: the mamba half of the copy did not exist on this rig's pool. `Req.offload_kv_cache` promised "copies over both the kv cache and mamba state" and passed `mamba_indices`, which `UnifiedSWAKVPool` accepts and never reads -- correctly, since a KV pool with no mamba pool has nothing to copy. Only `HybridLinearKVPool` honours it, because it owns one. So the capability is now DECLARED (`supports_mamba_cpu_copy`, default False on `KVCache`, True on `HybridLinearKVPool`, forwarded by the allocator to its `_kvcache`) and `Req` dispatches on the declaration, covering exactly the pools that have no owner to delegate to. Exactly one mover per payload, enforced rather than promised. The form reuses the house idiom (`BasePrefixCache.supports_mamba` / `supports_swa`); the allocator tree had none of its own. The declaration must take the SAME hop as the payload: allocators implement `get_cpu_copy` as `self._kvcache.get_cpu_copy(...)`, so a declaration that did not forward would let `Req` read the base False while the pool underneath copies mamba -- copying it twice. The getattr default is kept there because `UnifiedKVPool` and `DeepSeekV4UnifiedKVPool` declare no base class and genuinely cannot answer; it is removed at the `Req` callsite, where every allocator inherits the method and a missing attribute is a real defect. TESTS test_seam_state_copy_783 4 copy fires at the cutover and nowhere else; pressure retraction unchanged test_seam_state_restore_783 6 restore round-trips, is consumed, and is wired after alloc / before the clear test_seam_ownership_ledger_783 8 drives the PRODUCTION invariant (SchedulerInvariantChecker._check_pool_ invariant) over a claim ledger; gates the COMBINATION, and its can-fail shows half 1 alone balanced while the pair leaks -- the arm B shape, at the desk test_unified_mamba_cpu_copy_783 8 the declaration is pinned, including a pool that declares True and lies Each file carries at least one assertion that passes without the change, so a broken fixture cannot masquerade as a finding (it caught three of mine). Serial managers battery, cards visible (this gate is not hermetic): 4358 collected = 4348 passed + 8 failed + 2 skipped, 356 subtests, 721.20s Failure set identical BY NAME to the standing 8: staging_reserve_631 x4, rank_prefill_log x1, hisparse_unit x2, draft_cuda_graph_removal x1 Zero unexpected names, zero standing failures missing. Collected == ran, so the run is complete -- including test_pp_retracted_pass_void_797, which wedged in an earlier run while arm A was still winding down. black/isort applied, ruff gate (F401,F821,UP037) clean, codespell clean. NOT PROVEN HERE: that the prefix actually survives a flip on metal. That is the boot, and its acceptance lines are #cached-token > 0 after a flip against 403/403 zeros, and PHASE-FLIP-GDN moved > 0 against 228 zeros.

…o rules, one deleted premise `build_flip_quiescence_fn` carried two rules whose justifications both died with sgl-project#856 (2026-08-24, "the flip carries no KV"), and they pulled in opposite directions -- one let a flip through that must now be refused, the other refused one for a reason that no longer exists. Fixing either alone would leave the predicate half-governed by a deleted mechanism. SITE 1 -- the between-chunks allowance LET THROUGH what must now be refused. Written 2026-08-09 (sgl-project#631 defect O), justified verbatim in its own docstring as "exactly the state the carry moves". sgl-project#856 deleted the carry: the seam retracts residents and drops the tree, so that state is freed, not moved. The predicate was never revisited. A 6019-token prompt needs two chunks, the flip commits between them, and the re-admission restarts at prefix_lens=0 -- measured on W38-B ("prefill still chunked (allocated=4096, needs=6045)") and visible in W37-H arm A as 51 flips, 132 pp prefills, 57 tp prefills, ZERO decode rounds, zero completions. The block is added under STRICT ONLY, and that condition is the design rather than a hedge. Blocking every incomplete chunk re-creates defect O, where a flip armed FOR a prefill could not land until that prefill had finished (the 32768-token prefill that ran in the slow layout and paid two cutovers for nothing, 2026-08-09 04:23:35-54Z). Under strict batching the flip is armed for the DECODE after the drain, so waiting IS drain-and-flip. Landed in `chunk_blocks_quiescence`, NOT at the `ready_fn` call site: the helper's own docstring says its two callers must never disagree, and they drifted apart once already (2026-08-09 20:31:38Z). Both callers now pass the term -- `ready_fn` via `purity_of` behind a try/except (a purity read may never break a flip), the park site via the already-cached `self._phase_purity` (no new import, and no second purity read on the hot path). SITE 2 -- the orphan gate REFUSED for a mechanism that no longer exists. It blocked on requests "not yet merged into the resident set THE CARRY HARVESTS" (sgl-project#631 defect L). There is no harvest, and de4f541 gave `_live_reqs` the identical population (running_mbs, last_mbs, running_batch, last_batch). REMOVED, not narrowed: narrowing a gate whose entire purpose has been absorbed elsewhere leaves a third stale premise behind. RESIDUAL, WRITTEN DOWN RATHER THAN LEFT IMPLICIT (own posten). Post-sgl-project#856 there is no carry in EITHER mode, so a mid-chunk flip discards the prefill in non-strict too. We decline to block there because an unconditional block re-creates defect O. THE CORRECT NON-STRICT ANSWER IS UNKNOWN AND IS FILED, NOT SOLVED. `if strict` must not be read as evidence that the non-strict path was analysed and found sound -- it was not analysed. Three claims outlived their justifications on the night this was written; this one says what it rests on. TESTS test_quiescence_no_carry_858.py, 8 passed. Each mutation verified to kill exactly its own test and nothing else: drop the strict block -> test_strict_blocks_an_incomplete_... FAILS make the block unconditional -> test_non_strict_still_allows_... FAILS restore the orphan gate -> test_..._does_not_consult_the_carry_ FAILS Two assertions pass BEFORE and after (a completed prefill still permits the flip; non-strict unchanged) -- without them a block-everything predicate reads green, which is how three red-first files were nearly mis-reported this night. `test_live_reqs_still_covers_that_population` guards the removal's own premise: nothing else asserts that enumeration, so it is the belt rather than belt-and-braces, and it reddens if `_live_reqs` ever stops reading last_mbs/last_batch. SUITES -- all four arms hermetic (CUDA_VISIBLE_DEVICES=""), verified by `nvidia-smi --query-compute-apps` reading 0 during the runs, and run SEPARATELY per sgl-project#749 (the managers/ + distributed/ combination is order-dependent on ~50 tests and cannot gate anything). Pre-existing reds measured against the branch tip 322f331, not inherited. managers/ change 15 failed / 4334 passed / 18 skipped / 356 subtests base 15 failed / 4326 passed / 18 skipped / 356 subtests identical failure sets BY NAME, set-difference empty both ways; passed delta +8 = exactly the new tests, nothing else moved. The base arm reproduces R7's independent measurement of the same tip (4326/15) exactly. distributed/ change 95 failed / 3017 passed / 12 skipped / 1170 subtests base 95 failed / 3017 passed / 12 skipped / 1170 subtests identical BY NAME across all 95 -- 92 line-start FAILED plus 3 parametrized `FAILED(dcp_size=2/3/4)` subtest reports that a `^FAILED` pattern misses; both arms agree on all of them. No formatter was run on any file: this tree's black is older than whatever formatted it, and isort has moved a deliberately-late `# noqa: E402` import here before. The change is hand-written to the surrounding style. NOT PROVEN HERE: that sgl-project#857's acceptance follows. This removes the mechanism that made the livelock inevitable; whether a full A-B-A cycle now completes with COMPLETIONS>0 is a metal question and is not claimed.

…o the guard can arm sgl-project#718/sgl-project#847 built every part of the rebind except one: the 'tp' staging pin was assembled with a SINGLE entry, PoolName.KV. On a hybrid model the live tier carries KV *and* MAMBA, so `check_pool_coverage` computed `missing={MAMBA}` and refused -- correctly, on every cutover. A refused rebind leaves the sgl-project#718 device tier DISARMED, `load()` returns None, every read-through misses, and every prefix a cutover retracted is recomputed in full. Measured on the W40 sgl-project#857 acceptance boot (boot_w40_857strict_0825_2342.log): 60 `sgl-project#719 HiCache rebind refused`, ZERO arms, and `#cached-token: 0` on all 243 prefill batch lines -- one bucket, no exceptions. THE PRECONDITION WAS ALREADY WRITTEN DOWN, in the guard's own docstring: "A phase host tier has to be built with the FULL POOL SET before this rebind can arm; until then the sgl-project#718 disarm is the correct state and a read-through miss is the correct cost." This is that precondition, met. Not a new finding -- sgl-project#856 answers whether the retraction is necessary (it is: carry moves Req scheduling metadata, never KV bytes, and sgl-project#856 deliberately retired the movers), and sgl-project#718/sgl-project#847 already named this remedy. REFUSAL CONVERSION, NOT GUARD DELETION. `check_pool_coverage` is untouched. It must stop firing because its precondition is MET, never because it was removed, and the test asserts BOTH directions -- a full pool set arms, a narrowed one still refuses. Neither assertion alone can tell a fix from a disarm. DERIVED FROM THE BOUND TIER, not from the model config: the set that must be covered is whatever the READER names, which is the same quantity the guard compares. Reading the config would be a second opinion about one fact. MIRRORS `build_hybrid_mamba_stack` rather than re-deriving it -- same primitives, same layer mappings, same transfer_layer_num rule. The controller is deliberately NOT reused: this pin needs a host VIEW, and a second HybridCacheController would be a second writer against one device pool. Both entries are rebuilt in the hybrid case. The KV-only pin used an identity map over range(layers), which is right while KV is the only entry and wrong the moment a second pool shares the transfer index space -- the two maps collide at index 0. SIZING IS PER-SLOT, NOT PER-GB, and it is the one place the mamba half must not copy the KV half. MambaPoolHost reads host_size in GB only when > 0, else `device_pool.size * ratio`. The KV pin's GB figure is derived from a token count; mamba is allocated per request slot. Ratio 1.0 with host_size 0 mirrors the device pool, which is what a phase-matched staging pin means. The mamba half is a NAMED HOST-LEDGER POST, priced from what was allocated rather than from the intention. An unpriced pinned pool is what the ledger exists to prevent. A pool set this builder cannot mirror (SWA, indexer, DeepSeek) is logged as an ERROR at the cause and still refused by the guard -- named, not swallowed. THE CHECK (sgl-project#871, third scope item): `advance_fence_blind_streak` + FENCE_BLIND_STREAK=4, aggregating the EXISTING `persisted_nothing` instrument. One empty fence is legitimate; every empty fence means the canonical store can never populate, and that shows up only as latency. Gated on `released`, mirroring sgl-project#719's busy gate at the stale-gate streak: a fence over an empty tree is correct to persist nothing, so counting quiet cutovers would build a crying-wolf alarm out of the instrument written to replace one. NO SECOND COUNTER -- a parallel "recomputed prefix tokens" counter would measure what this and #cached-token already measure between them. Extracted as a pure function so it is falsifiable without booting. A guard whose logic can only be exercised by booting is a guard that ships unexercised, which is the failure mode this ticket is about. CLASS (unchanged, carried from the sweep): a store whose only writer is a lifecycle event another mechanism systematically preempts. Retention is finish-only; the cutover retracts before finish; the store can never populate, so every recovery path reading it can never fire. TEST RESULTS test_phase_tier_full_pool_set_871.py (new): 15 passed. FALSIFIED IN THREE DIRECTIONS BEFORE THE GREEN WAS CLAIMED, count gate held on each: guard removed (disarm) -> 3 failed, and they are exactly the refusal tests. 3 extracted == 3 in summary. guard fires unconditionally -> 6 failed, exactly the arming tests. 6 == 6. streak loses its `released` gate -> 1 failed, the idle test. 1 == 1. restored -> 15 passed. Targeted regression set (7 files: sgl-project#847 writer, sgl-project#783 fence x3, sgl-project#856 empty wave, counters, and the new file): 97 passed, 0 failed, 2 subtests passed. Count gate 0 extracted. Hermetic, CUDA_VISIBLE_DEVICES verified EMPTY at the PROCESS (/proc/<pid>/environ), not at the command. AN EXISTING TEST CAUGHT A REAL BUG OF MINE and the first run of that set was 3 failed: the new HOST-LEDGER line read `tp_host.entry_map` directly, but that writer is driven in tests by stand-ins where HostPoolGroup itself is patched. Fixed with the getattr discipline this module states at its other probes -- an instrument may never be the thing that breaks a boot. Lint: ruff 0 before and after on phase_flip_runtime.py and the new test. phase_flip_boot.py reads 1 before AND after -- a pre-existing F401 on `pack_into_arena`, present in HEAD, not mine. codespell clean. Flag-off path verified byte-identical: without --phase-flip-rebind-hicache the writer still returns {} and allocates nothing. NOT CLAIMED, AND DELIBERATELY NOT GUESSED: no metal run. Whether the kv+mamba pin can actually be ALLOCATED on this box, what its real HOST-LEDGER post comes to against the 16G floor, and whether #cached-token becomes non-zero are decidable only on hardware. They belong to a boot window. The sgl-project#857 acceptance instance was left running and untouched throughout -- it is the standing proof and the operator declared it taboo.

…ated gap and a stale double-owned reading as a leak, killing all three schedulers on idle Five on_idle firings (two boots) showed available+evictable+withheld exceeding total by exactly 22, never a deficit -- the opposite sign from sgl-project#814/sgl-project#902/sgl-project#832/sgl-project#856, and never previously routed through the sgl-project#822 ownership authority. Decomposition: 21 rows from _check_full_pool reading TokenToKVPoolAllocator.available_size() (a raw sum of free_pages+release_pages, allocator/token.py:52-54) instead of the union read_free_rows() already used by the phase-flip census and the sgl-project#822 audit (kv_row_ownership.py:743-843); 1 row from a single row simultaneously claimed by the free list and the radix tree, the sgl-project#822 authority's own EXCLUSIVITY "claimed by multiple" finding, never subtracted before. Fix, both additive, neither a tolerance/epsilon: - _check_full_pool now reads available via read_free_rows() when the allocator can enumerate; composite/watermark allocators keep ps.full_available_size unchanged. - _check_pool_invariant gained a double_owned term, subtracted, sourced from allocator.double_owned_slots (the sgl-project#822 EXCLUSIVITY count). Two defects found in review of the first cut, both fixed here: 1. The double_owned filter in phase_flip_runtime.py::_census_ownership_audit selected violations by substring match on Violation.detail ("more than one owner" in v.detail) -- prose used as control flow, against Violation's own "detail is never load-bearing" contract, the same shape as the line_gate substring defect (sgl-project#908). Fixed by giving Violation a structural kind field, set to EXCLUSIVITY_DOUBLED or EXCLUSIVITY_UNOWNED at its two EXCLUSIVITY construction sites in kv_row_ownership.py, and filtering on kind instead of detail text. 2. double_owned_slots is published only at phase-flip census (seam) events and read by on_idle at unrelated times; a reading taken before authority.retire() drops every claim it was computed from is stale past that point. Fixed by clearing double_owned_slots to None (not 0) at the exact point _retire_row_id_space calls authority.retire(). Checked analytically whether staleness alone could mask a genuine deficit: it cannot, for any non-negative reading, because double_owned is subtracted and can only push total_accounted further below total. It can mask an unrelated, coincidentally-equal-sized surplus; that residual case is disclosed, not eliminated, and bounded by the cutover-clearing fix. Tests (test/registered/unit/mem_cache/test_pool_invariant_double_owned_912.py, 14 tests, 31 subtests): reconstructs all five measured specimens, red-first against unfixed code, green after the fix; separate mutants for the original subtraction, the kind-based filter, and the retire-time clearing, each shown to fail only its own guard when reverted; a parametrized check that no non-negative stale double_owned value can turn a manufactured 100-row deficit into a false pass. Gate, hermetic (CUDA_VISIBLE_DEVICES=""), attribution proven by rerunning identical failing node-id lists with these three files restored to HEAD (git show HEAD:<path>) and diffing failure sets -- both axes' failures are identical with and without this fix, i.e. pre-existing: - mem_cache/: 2 failed, 1951 passed, 1658 skipped, 392 subtests passed (139.10s). Failures: test_acceptance_emitters_758.py RefillTiming x2, RuntimeError: No CUDA GPUs are available. - managers/: 15 failed, 4631 passed, 18 skipped, 563 subtests passed (719.09s). Failures: test_arena_high_water_631.py x7 and test_restore_never_rebuild_677.py x4 (RuntimeError: No CUDA GPUs are available), test_phase_flip_rotation_wiring_809.py x4 (Exception: retry() exceed maximum number of retries) -- all reproduced identically against HEAD-restored files. ruff check and codespell clean on every line touched by this change (both tools also flag pre-existing issues elsewhere in phase_flip_runtime.py and invariant_checker.py, outside this diff's hunks, left untouched as out of scope for sgl-project#912).

… suite relit against the sgl-project#856 contract) into the flip train

…layer ids, or refuse sgl-project#706 slice 2 (a38f39f, 2026-08-17) built the geometry-neutral {hash}.mamba blob and build_mamba_window to cut it for either phase. It has never once been built at runtime, on any model, on any boot. THE HOP. HiCacheController._canonical_mamba_window opened with cache_params = getattr(model_config, "mamba2_cache_params", None) mamba_layer_ids = list(getattr(cache_params, "layers", None) or []) if not mamba_layer_ids: return None and mamba2_cache_params is a property of the CHECKPOINT config (Qwen3NextConfig, configs/qwen3_next.py:288, layers=self.linear_layer_ids), never of sglang's ModelConfig: `grep -c mamba2_cache_params configs/model_config.py` is 0. The getattr missed on every model, the list was always empty, and the one silent branch fired every time. The correct hop is hf_text_config -- the one gdn_flip_mover.py:929 already uses for GDN geometry. MEASURED, boot 2g (boot_2f_698cd396ce_0827_0704.log). Format explicitly armed (phase_flip_canonical_kv_page=True, phase_flip_writeback=True, backend 'file', page_size 1, so the pairing guard at server_args.py:8716 held), and the log carries "sgl-project#706 canonical KV page active" THREE times -- one per rank -- against "sgl-project#706 canonical GDN blob active" ZERO times and zero refusals. Exactly the split _canonical_mamba_window's own docstring calls fatal: "a canonical KV page beside a phase-local GDN blob delivers ZERO usable prefix ... Silently running KV-only would look like the feature was on while every cross-phase lookup missed." THAT IS WHERE sgl-project#928 COMES FROM. KV crossed the flip geometry-neutrally while the recurrent anchor stayed phase-local; sgl-project#856 then drops the tree at every cutover trusting the canonical store ("Nothing is carried across"), and read-through returned a correct KV prefix beside an anchor in the writing phase's layout. The on-device fallback could not cover it either: the GDN mover runs AFTER the drop and logged "moved 0 slot(s)" on all ten flips of that boot. THE FIX IS THE SIBLING LADDER, NOT A SECOND BARE LOOKUP. resolve_attn_layer_ids was given exactly this treatment on the same day, for the same reason ("RESOLVED, not guessed" -- it could not tell NOT HYBRID from NOT POPULATED). The linear half kept a raw getattr aimed at the wrong object. canonical_page_store gains resolve_linear_layer_ids with the same four rungs: (a) ModelConfig's own list if a future one populates it; (b) the checkpoint config's own properties through hf_text_config, linear_layer_ids first and mamba2_cache_params.layers second; (c) [] ONLY on a positive dense proof; (d) otherwise a NAMED REFUSAL. The caller's `return None` is now sound rather than a hole: an empty list can only mean a proven dense model. BEHAVIOUR CHANGE THE NEXT BOOT WILL SHOW, stated rather than discovered. A hybrid whose config class exposes neither property now REFUSES the boot instead of silently running KV-only -- the conversion the caller's docstring demanded ("this must succeed or attach fails"), with the remedy in the error text. And because the window is now actually built, derive_mamba_blob_spec is reached for the first time; if this checkpoint's config lacks a GDN field it will say so loudly. So the first boot on this commit yields either "sgl-project#706 canonical GDN blob active" three times or a named refusal. Both are results. Neither is silent, and silence was the defect. TEST RESULTS (desk, CUDA_VISIBLE_DEVICES="", /spinning/htsglang-gpu/.venv): * test_canonical_gdn_blob_attaches_931.py -- 5 passed. Five arms: a Qwen3-Next-shaped hybrid resolves its 48 GDN layer ids; the wrapped source (mamba2_cache_params.layers) resolves too; a proven-dense model returns [] (a ladder that refuses everything is an outage); an unresolvable hybrid RAISES; and _canonical_mamba_window gets PAST its early return -- the attach itself, probed by making the next call raise a sentinel, because a green resolver beside a caller still holding its own getattr looks identical from outside. * MUTATION, one per claim: M1 caller reverts to the bare getattr on ModelConfig 1 failed (the attach arm) M2 resolver aims at ModelConfig instead of hf_text_config 4 failed M3 the (d) refusal degrades back to a silent skip 1 failed (the refusal arm) restored: 5 passed. * REGRESSION: test/registered/unit/mem_cache/ -- 2822 passed, 927 skipped, 471 subtests passed, 2 failed. Both are test_acceptance_emitters_758 "RuntimeError: No CUDA GPUs are available" and fail identically on the parent; the desk has no GPU. The sgl-project#706 format suites (test_canonical_mamba_blob_706, test_canonical_page_store_706, test_mamba_phase_uniform_706, test_mamba_gates_the_hit_706) are inside that run and stay green. * ruff: canonical_page_store clean; cache_controller 13 errors before and 13 after, all E402/F541 at lines 16-44 and 1414, none in the edited region. (First count of "0" was a broken extraction -- ANSI codes in the concise output defeated the per-file grep; recounted verbatim.) NOT BUILT, AND NAMED RATHER THAN IMPLIED: the end-to-end content proof -- archive a PP-phase anchor through the fence, load it back in the TP phase, compare against the reference state -- needs live pools and a storage backend and is not hermetically cheap at the desk. It is a boot-proof item, and the measurement window's coherence probe drives exactly that path.

…eeps its one ref Boot 9 died in 24s on the lock_ref>0 underflow (full_component.py:320, via cache_unfinished_req's dec at the FIRST chunk stash). R12 closed the pair on metal: ONE inc (schedule_policy.py:2021, transferred stash-to-stash by unified_radix_cache.py:1354-1355) met TWO decs -- the give-back at the void (slice line 270, lock_ref_returned=True) and the stash's own dec. Both give-back sites (sgl-project#984 park, sgl-project#986 orphan-queue) assert 're-admission takes a FRESH ref', which is FALSE for a carried chunk: the chunked continuation holds ONE admission ref across all its chunk passes (schedule_policy.py:1441/:1747/:2008, scheduler.py:9059). The sgl-project#988 guard removed the accidental compensating inc of the second waiting-queue visit, exposing the imbalance (that causal half is desk-inferred; the pair itself is measured). Fix, two lines at the ONE shared function: pp_give_back_admission_lock_ref returns False for the request currently held as self.chunked_req -- ownership discriminated by identity; the sgl-project#986 orphan route keeps giving back (its orphan has by definition LEFT the field). The assert is never weakened. Context (R12's design evaluation, register 20:xx): the user's re-entry design is ALREADY shipped as sgl-project#856/W30-FIX-B but scoped by _live_reqs to batch residents; extending it is measurement-gated on sgl-project#972/sgl-project#975 + a GDN host anchor (ABSENT today) -- registered as posten, not smuggled in here. Import smoke green; suites in the test-agent lane.

THE DEFECT, one line. Every draft page the TP phase backs up is restored correctly by `_page_transfer` -> `_draft_page_get` and then zeroed again one funnel later, because `arm_draft_cold_for_admission` scrubs a seam-re-admitted request's WHOLE cached prefix unconditionally -- the trigger asks WHERE the request came from, never WHETHER its draft rows arrived. Nothing rewrites a prefix's draft rows after admission, so such a request decodes over an all-zero draft chain for its entire life. That is the "no spec win in decode" the user named, and the 50504 draft pages in /tmp/hicache_783 are the cost already paid for it. DESK-PROVEN only (no cards in this session, no boot). Not item (d) of TASK_861. The draft key suffix does NOT move across a cutover: `config_suffix` is built once in `HiCacheFile.__init__` (hicache_storage.py: 605-637) from a `storage_config` that only `attach_storage_backend` (cache_controller.py:795) produces, and the cutover path (`hicache_phase_binding._stamp`, :267-300) re-points pool attributes and the binding generation and nothing else. Live store confirms it: every draft page carries `_0_1_3_{0,1,2}`, the BOOT geometry, and rank i therefore reads back exactly the pages rank i wrote. A canonical draft-page form buys nothing here; the blocker was the read path. TWO LINKS, both small, both default-inert without a flip: (1) `_draft_page_get_generic` writes ZEROS on a miss instead of leaving the row alone. The row is a recycled host slot holding the previous occupant's draft bytes, and leaving it made "restored" and "stale" indistinguishable downstream -- which is precisely why the scrub had to be unconditional. `get_dummy_flat_data_page()` is already `torch.zeros`, so the miss path writes the same value `scrub_draft_kv` writes, at the one site that knows which pages missed. Also removes the fp8 NaN/Inf path into the draft softmax that arbitrary recycled bytes carry. (2) The admission scrub is bounded to `not tier_armed`. With the tier armed the prefix came back THROUGH the draft read path, so its rows are real where the page existed and zero where it did not. The MARK is deliberately kept for both: splitting the scrub from the seed is sgl-project#631's own decomposition, and the seed is the cheap half (one non-drafting round whose FULL-captured hidden states start the real chain). Correctness is untouched in both directions -- the target verifies every proposed token, so this buys acceptance, not answers. INSTRUMENTS (speed mode: one mandatory line per chain link, and both are written so silence is a finding rather than success): * `sgl-project#993 draft L3 READ: N page(s) requested, H hit / M miss` -- fires on the first batch, then per doubling. A process that never reads a draft page emits nothing, and that absence is the upstream finding. * the ADMISSION line gains `%d prefix(es) KEPT (draft tier armed=%s)`. kept=0 with armed=True means the read path is not delivering and the fault is above this funnel -- a distinction the old line could not make. VERIFICATION (desk, hermetic, CUDA_VISIBLE_DEVICES=""): * import smoke on the worktree, patch confirmed applied by introspection. * test_draft_cold_admission_861 / test_draft_tier_gate_861 / test_draft_hicache_binding_861: 46 passed. * Two new pins, one per direction, and CAN-FAIL proven: reverting the `not tier_armed` bound fails the armed pin with `assert (5 == 0)` -- the five scrubbed rows are the defect itself. Cold rather than boot-observed on purpose: on metal the two behaviours differ only in an acceptance number no single boot can separate from model, traffic and flip cadence. * ruff check and ruff format: zero delta against HEAD on all three files (the 13 findings and the reformat are pre-existing, measured against `git show HEAD:` copies). SIBLING SWEEP (analysis). `arm_draft_bootstrap`, the sgl-project#631 cutover leg, scrubs too, but its input is carried DEVICE rows no read path restored, so its scrub is correct and it is de facto dead since sgl-project#856 retracts the residents anyway. The v2 draft route stays unreachable and unchanged. The class to watch: a restore path that leaves a recycled row untouched on a miss forces every downstream consumer to assume the worst about ALL rows. The component pools avoid it a different way -- `batch_exists_v2` takes the MIN and truncates the prefix rather than scrubbing behind itself.

…ONGRUENT ZUGELASSEN -- meine "nie angekommen"-Erklaerung ist FALSCH S1-Durchgang, Schritt 1. Die rid-genaue Timeline aus dem Strandungs-Log kippt meine eigene Diagnose, und zwar in die praezisere Richtung. == DIE TIMELINE (logindex, boot_855_fix1027_..._112900, voll ingested) == Nach 11:57:30 nennt das Log GENAU EINEN rid: 2fe8dab34ab1. admission n=18 11:56:14 .. 11:58:05 flip_event n=100 11:56:17 .. 12:02:21 Die 18 Admission-Zeilen sind SECHS Runden zu je drei Raengen, und JEDE ist "sgl-project#788 PP-ADMISSION verdict=ADMIT n_reqs=1 rids=2fe8dab34ab14d" auf Rang 0 UND 1 UND 2. Die Raenge waren sich EINIG. Letzte Zulassung 11:58:05; letzter Prefill 11:58:07; danach nichts mehr. Und der Flip-Apparat nennt denselben rid noch bis 12:02:21 in FLIP EXTENT PROBE auf allen drei Raengen -- der Request ist also die ganze Strandung ueber RESIDENT, nicht verschwunden. == SELBSTKORREKTUR (fuenfte, und sie stand im CODE) == Mein Commit 70d85ed schloss: "DER REQUEST IST OBERHALB DER PHASE-POLICY VERLORENGEGANGEN", und ich hatte diese Erklaerung in die sgl-project#1028-Log-Zeile geschrieben ("its request never reached this scheduler"). Das ist FALSIFIZIERT: er hat den Scheduler erreicht, wurde sechsmal von allen drei Raengen zugelassen und blieb danach resident. Die Zeile haette den naechsten Leser genau so fehlgeleitet, wie mich "decoding in tp" fehlgeleitet hat -- also derselbe Fehler, den ich am selben Tag viermal katalogisiert habe, von mir selbst begangen und in den Baum committet. Text korrigiert auf "admitted and then stopped progressing somewhere DOWNSTREAM of admission". Der Grund, dass ich es fand: die rid-Timeline, nicht erneutes Nachdenken. == FOLGE: DER VERLUSTPUNKT LIEGT NICHT AUF S0 UND NICHT AUF S1 == Intake (S0) war bereits ausgeschlossen. Admission (S1) ist hiermit ebenfalls ausgeschlossen -- sie hat kongruent ADMIT gesprochen. Der Defekt liegt ZWISCHEN Zulassung/Prefill und dem laufenden Batch, also auf der MERGE-Bahn (S2). Damit ist die Station benannt, an der weitergesucht wird. == EIN KANDIDAT, AUSDRUECKLICH NOCH NICHT ALS URSACHE BEHAUPTET == Am Strandungspunkt feuert auf allen drei Raengen "SELF-MERGE REFUSED: last_batch is running_batch (bs=1)" (scheduler.py:7427-7457). Formal ist das genau die vom Gesetz verbotene Form: ein Guard, der einen Zustand erkennt, den sein eigener Kommentar "the resident set is corrupted" nennt, und dann REFUSAL-UND-WEITER macht statt die Gruppe zu stoppen -- und der Kommentar sagt selbst "a detector that only declines to act cannot stop a doubling -- the instance still died". Nach Upstream-Minimal ist ein Defekt in einer Kompensationsschicht ein LOESCH-KANDIDATEN-Befund; die eigentliche Frage ist, warum `last_batch is running_batch` ueberhaupt gilt (Aliasing). ABER, INDIKATOR-GESETZ: der Marker feuert 166.173-mal auf diesem Boot, also auch waehrend der ~26 Minuten, in denen alles funktionierte. Er ist damit NICHT hinreichend fuer die Strandung und wird hier als KO-OKKURRENZ gefuehrt, nicht als Ursache. Wer ihn ohne diese Zahl zitiert, berichtet eine Korrelation als Wurzel. == NAECHSTER SCHRITT (nicht mehr in diesem Zug) == S2/Merge-Bahn mit derselben Disziplin: Zeilen per emittiertem Literal am Pin relokalisieren, per trapsafe zaehlen, und die Frage stellen, die die sgl-project#1031-MERGE-PATH-PROBE (scheduler.py:7462ff) bereits woertlich formuliert -- "`#running-req: 0` heisst entweder der Prefill hat NIE gemerged, oder er merged und sgl-project#856-no-carry hat ihn vor der Decode-Runde retracted; nichts Gemessenes trennt die beiden". Genau diese Trennung ist jetzt die Aufgabe, und die Probe dafuer liegt schon im Baum. BELEG-STUFE: BOOT-BEWIESEN (rid-Timeline aus dem voll ingesteten Log). Kandidat SELF-MERGE: BEOBACHTET, ausdruecklich nicht kausal belegt.

…e steht, aber der Umbau loest den Stall NICHT -- zwei Korrekturen an meiner eigenen Skizze Desk-Arbeit, kein Boot. Jeder Slot am Pin relokalisiert. Volltext: /spinning/gpu-arb/DESIGN_968_budget_verdict_to_pp0.md == DIE KLASSIFIKATION IST DREI-, NICHT ZWEIWERTIG == Die Order fragt "Divergenz-Patch vs echtes physisches MIN". Die acht Slots zerfallen tatsaechlich in drei Klassen, und die dritte ist der Grund, warum der Umbau kleiner ausfaellt als gedacht und den Stall nicht behebt: (a) DIVERGENZ-PATCH -- ein VERDIKT, das jeder Rang rechnet und ueber das man sich einigt: head_match (sgl-project#823 W9), admit_limit (:6167), Prefetch-Ballot (#791b). DIESE DREI SIND DIE LOESCHLISTE. (b) PHYSISCHE EINGABE -- eine per-Rang-Tatsache, die PP0 NICHT SELBST WISSEN KANN: local_avail (#616g), admission (sgl-project#610), host (sgl-project#639), mamba (#639b), corridor (sgl-project#794). Muessen weiter reisen, aber als EINGABE in PP0s Verdikt. (c) DIVERGENZ-DETEKTOR -- die `x, -x`-Paare, die aus einem MIN zugleich Max liefern, also Uneinigkeit ERKENNEN. Nach dem Rang-Gesetz ist die einzig legale Reaktion CRASH/STOP; heute speisen sie Kompensation. == KORREKTUR 1 AN MEINER SKIZZE: die Kadenz-Deckel fallen NICHT == §4 Punkt 5 meiner eigenen Skizze behauptete, die sgl-project#1027/sgl-project#1028-Deckel wuerden mit dem Umbau gegenstandslos. FALSCH. Die drei teuren Calls (memory_snapshot, memory_stats, mem_get_info) haengen unter Slot 5, und Slot 5 ist eine PHYSISCHE EINGABE -- PP0 kann PP1s freies VRAM nicht selbst messen. Das Verdikt zu PP0 zu verschieben nimmt den Followern die ENTSCHEIDUNG ab, nicht die MESSUNG. Der Stall liegt damit auf einer ANDEREN ACHSE als die Verdikt-Platzierung: die physische Messung darf nicht auf dem kritischen Pfad zwischen Rundenbeginn und Barriere liegen (Off-Thread-Sampling; der `corridor-trace`-Thread existiert bereits). Beides kombinierbar, aber keines folgt aus dem anderen. == KORREKTUR 2 / TRANSPORT-VERDIKT: der Ring-Lap traegt es NICHT == pp_admission_congruence.py:235 woertlich: eine Entscheidung braucht BIS ZU `pp_size - 1` Runden (hier 2) ueber den Lap. Und die sgl-project#1027-Sicherheitsanalyse UEBERTRAEGT SICH NICHT -- die Asymmetrie ist der Kern: * `trapped` (sgl-project#1027) wird ABGEZOGEN -> ein zu alter, zu grosser Wert VERENGT einen Cut. Sichere Richtung. * `corridor_width` (Slot 5) ist eine OBERGRENZE -> ein zu alter, zu grosser Wert WEITET den gewaehrten Chunk. UNSICHERE Richtung, und exakt der Fall, vor dem der sgl-project#856-F6-Kommentar an spendable_bytes:606-619 warnt. Ein Ein-bis-Zwei-Lap-altes Budget ist also fuer mindestens einen Slot in der unsicheren Richtung. Der Lap traegt das Verdikt in seiner heutigen Phase nicht. XL-Fork, geht zur Meldung statt in einen stillen Umbau. Alternativen benannt: (1) frueherer Lap-Punkt -- aendert die Rundenphasen, Risiko bei den Lockstep-Familien; (2) Piggyback auf bestehendem Kollektiv -- kein neuer Draht, aber PP0s Verdikt raeste auf demselben Reduce, den der Umbau abschaffen soll. Dritte Moeglichkeit (konservative Marge gegen die unsichere Richtung) NICHT empfohlen, aber benannt, damit sie nicht spaeter als neu auftaucht -- sie waere eine vierte Kompensationsschicht. == AUFWAND, PRAEZISIERT == Loeschliste Slots 6-8 zu PP0 M (Verdikte, kein Transportproblem) Slots 1-5 als Eingaben behalten S (Payload schrumpft, Semantik bleibt) Stall-Behebung (Messung vom Pfad) M-L (eigene Achse) Slot 5 unter PP0-Autoritaet XL-Risiko, UNGELOEST EMPFEHLUNG: Loeschliste und Mess-Achse sind unabhaengig und beide OHNE den XL-Fork machbar. Der volle PP0-Umbau von Slot 5 wartet auf die Transport-Entscheidung. BELEG-STUFE: DESK-BEWIESEN (Slots am Pin relokalisiert, Archaeologie aus den Einfuege-Kommentaren, Lap-Latenz aus pp_admission_congruence.py:235). Kein Boot, kein Code geaendert.

…en stale-UNSICHEREN Term eingefuehrt. Zurueckgenommen. Schritt A (reiner Lese-Pass, beide Quellen je Zeile) -- und die Grundlage, die kippt, ist mein eigener sgl-project#1028-Fix, der im Baum UND im laufenden Serving lag. Gemeldet und korrigiert im selben Zug. == DIE STALENESS-RICHTUNGSTABELLE (Ergebnis von Schritt A) == Treiber-Call Konsument Operation stale-groesser memory_reserved/allocated want (:447) SUBTRAHIERT under-arm SAFE memory_reserved/allocated spendable_bytes (:620) ADDIERT weitet UNSAFE mem_get_info (free_bytes) spendable_bytes (:602) ADDIERT weitet UNSAFE memory_snapshot (trapped) takeable (innen) SUBTRAHIERT verengt SAFE Die Asymmetrie steht im Code und ich hatte sie nicht gelesen: `_takeable_cache_bytes`-Docstring :720-728 sagt woertlich "the two callers want OPPOSITE errors: sizing `want` is safe when the cache is overstated, and sizing a spendable BUDGET is not." == WAS ICH FALSCH GEMACHT HABE == sgl-project#1028 cachete den GANZEN `_takeable_cache_bytes`-Wert 30 s lang und begruendete das so: "der Wert wird vom spendable Budget ABGEZOGEN (:620), ein zu grosser Altwert verengt einen Cut, er weitet ihn nie." FALSCH. Bei :620 steht `free + takeable - delta` -- takeable wird ADDIERT, und `takeable_cache_bytes`' eigener Docstring sagt es auch ("a negative budget that would then be ADDED to a free column"). Meine Aussage stimmte fuer `trapped` (innerhalb takeable subtrahiert, sgl-project#1027) und wurde falsch, sobald ich die SUMME cachete: ein 30 s alter, zu grosser `cache` WEITET die Zuteilung -- der Fall, den der sgl-project#856-F6-Kommentar :606-619 unsurvivable nennt. Ich habe das Vorzeichen meines eigenen Konsumenten aus dem Gedaechtnis zitiert statt es zu lesen; genau die Methoden-Wurzel, die gestern als Regel ins Register kam. == KORREKTUR == Zurueck auf sgl-project#1027-Umfang: nur `trapped` bleibt cadenced (beweisbar sichere Richtung). `_allocator_cache_bytes` kehrt auf den Pro-Runde-Pfad zurueck, was den #1028a-Straggler REINSTALLIERT. Das ist ein bewusster Tausch und der vom Gesetz vorgeschriebene: ein intermittenter Liveness-Stall wird ERKANNT (Deadman, er korrumpiert nichts still), ein ueber-gewaehrter Korridor ist stilles Ueber-Commit. Crash vor Korruption. OPERATIVE FOLGE, ehrlich: die Stall-Anfaelligkeit unter RM-Schwerlast ist damit wieder auf dem Stand vor sgl-project#1028. Das ist eine Verschlechterung der Liveness zum Preis einer wiederhergestellten Korrektheits-Invariante. == FOLGE FUER STUECK 3 (Mess-Achse) -- der Lese-Pass hat es vorab entschieden == Off-Thread-Sampling loest das NICHT: eine Off-Thread-Probe ist ebenfalls stale, und fuer einen ADDIERTEN Term ist Staleness die unsichere Richtung. Fuer `cache` und `free` bleiben nur drei Wege: (a) synchron billig genug machen, (b) so umbauen, dass der Term nicht pro Runde gebraucht wird, oder (c) eine GEMESSENE UNTERE SCHRANKE fuehren (Minimum ueber das Sampling-Fenster), die in der SICHEREN Richtung stale ist -- das ist keine willkuerliche Marge, sondern ein Messwert, und damit nicht die abgelehnte vierte Kompensationsschicht. Schritt B ist deshalb NICHT wie beauftragt gebaut worden: der Lese-Pass hat seine Praemisse ("off-thread nach corridor-trace-Muster") widerlegt. BELEG-STUFE: DESK-BEWIESEN (Vorzeichen an beiden Konsumenten am Pin gelesen, Docstrings zitiert). Korrektur BOOT-GEBOOTET: health 200, greedy kohaerent, nur noch die sgl-project#1027-Kadenz aktiv (3 Instrument-Zeilen), Deadman armiert.

…, and stop the two instruments that lied about it NOT AN sgl-project#1028 FIX. This was written as one and the boot refuted the attribution; it is committed as the standalone correction it actually is. The sgl-project#1028 recompute has a different root (see the retraction below). THREE CORRECTIONS, all at the flip's writeback fence: 1. THE BOUND IS ON STALLING, NOT ON DURATION. `deadline_s` was a flat wall-clock cut, which cuts a fence making steady progress at the same moment as a wedged one. Measured, boot_855_wt1016 19:22:42 and 19:22:45 -- the same four nodes fenced twice, three seconds apart: `acked=0 outstanding=4` then `acked=1 outstanding=3`, both `elapsed=2.000s/2.000s`. Acks were landing at ~1 per 3 s; the backups were slow, not stuck, and the flat bound discarded that progress. `deadline_s` is now the NO-PROGRESS bound (unchanged default 2.0 s, so a genuinely stuck backend behaves exactly as before) and a derived hard ceiling (12x) stops a slow backend holding the seam open without end. The stall clock resets only when the in-flight set SHRINKS, which is monotone by construction, so noise cannot reset it. 2. AN EXPIRED FENCE MAY NOT CONVERT INTO A FALSE PROMISE. The cutover released residents and logged "Their KV is in the canonical store from the fence" unconditionally -- measured on wt1016 one second after the fence reported `acked=1 outstanding=3`. The claim now reads the report it speaks for, and distinguishes measured-zero from unmeasured. An incomplete fence additionally joins the EXISTING unanimous abandon (`too_small`), at the point where nothing has been mutated yet, rather than becoming a check of its own: same argument the row-bounds and staging terms make in that function, and a rank-local abandon would half-flip the group. Bounded by _WRITEBACK_DEFER_LIMIT=3 so a permanently stuck backend cannot trade a recompute for a wedge -- the worse of the two -- and the acceptance is then logged out loud instead of happening silently. 3. DWELL-RELEASE SAID "the flip cohort is resident" ON TWO DIFFERENT PATHS. The cohort count reaches zero either because the cohort became resident or because a re-admission left it with a SPENT one-chunk TP grant while still mid-recompute (scheduler.py, `seam_grant_is_open` exclusion). Only the first is residency. Measured wt1016 19:23:33: "the flip cohort is resident" logged in the same second, same rank, as a 4096-token TP batch with `recomputing=True`. `_seam_cohort_pending` now also returns the count it was already skipping; no decision changes, the log stops asserting the flattering one of two causes. RETRACTION, recorded because the wrong version was reported first: I attributed the sgl-project#1028 large-prompt recompute to this fence expiring. My own boot refutes it. boot_855_1028fence ran 27/27 fences COMPLETE (outstanding=0), including `eligible=4 acked=4 outstanding=0`, and the re-admission still reported `#969B READMIT-MATCH prefix_len=0 host_hit=0 storage_hit=0 input_len=13180` and recomputed all 13180 tokens. A fully acked fence yields host_hit=0 too, so the timeout is not the cause. That inference was a correlation in a single boot read as causation. The real candidate is `prefetch_registered=False prefetch_keys=0`, present in every READMIT-MATCH of both boots -- the never-shipped sgl-project#856 read-through half. sgl-project#1028 stays OPEN; nothing here is claimed to fix it. EVIDENCE / BELEG-STUFE: BOOT-PROVEN AS RUNNING AND BEHAVIOURALLY INERT, not "fixed" at any symptom. boot_855_1028fence: all fence lines now carry `ceiling=24.000s` (execution proof the code is live), all 27 completed with `elapsed=0.000s` exactly as before, and all four new markers counted 0 -- the trigger condition (outstanding>0) never occurred, so the defer branch cannot have influenced that boot. Desk check matched to the error class of these edits (undefined name / tuple arity, which py_compile is blind to): ruff F821/F811/F841/E9 clean on the changed regions -- the only 2 hits are pre-existing duplicate imports at scheduler.py:148/171, far from the edits at 3812+/13321+ -- plus an AST scan confirming `_seam_cohort_pending` returns 3-tuples on every path and its single call site unpacks 3. Sibling census for the warn-then-continue class in phase_flip_runtime.py: 2 other sites (1529, 7933); both warn honestly and make no promise, so neither is touched. This site was a false negative of that deliberately narrow rule, which is stated rather than dressed up as a rule hit.

ROOT, corrected against the handover's framing. The handover read the store census as "15457 KV pages vs 11 mamba anchors = grid behaviour" and ordered the anchors published at every write-through. Measured on the same boot (boot_855_1028fence), that premise is a PAGE-vs-NODE denominator mix and the ordered fix is already in place: * `#969H BACKUP` = 33 lines = n=1..11 on EACH of PP0/PP1/PP2 at identical timestamps. The probe logs every call up to n<=40, so 11 lines means `write_backup` reached its component loop exactly 11 times per rank in the whole run -- and all 11 carried `mamba_value=has_value`. => mamba coverage OF THE HOST-BACKUP PATH is 11/11 = 100%, not 11/15457. * All 11 `.mamba` hashes in the store are also full-KV page hashes (intersection 11, mamba-only 0): one anchor per node at a real page boundary, shared key namespace. * The 15457 KV pages are the PAGES of those same 11 nodes: `write_backup_storage` writes `keys=node.hash_value` (every page of the node) while the mamba branch writes `keys=[node.hash_value[-1]]` (one trailing page). The real hole is one link earlier and is not mamba-specific: `_inc_hit_count` returns before any backup when `chunked=True` (upstream's "skip the hit count update for chunked requests"). Under chunked prefill NOTHING is published per chunk -- so the 11 backups are the 11 FINISHED requests, and a chunked prefill that never finishes publishes nothing at all. That is why the 13179-token prompt found its deepest anchor at 3072 and recomputed the remaining 10107 tokens: the anchors are as dense as finished requests, not as dense as chunks. FIX: allow a chunked-prefill node to reach the host tier, gated structurally on (storage tier present AND a MAMBA component present). The per-chunk node already carries a donated state on the device (`MambaComponent.prepare_for_caching_req`, is_finished=False branch); this early return was the only reason it never reached host or storage. UPSTREAM-MINIMAL: the chunked skip IS upstream (`hiradix_cache.py`), so this is a DEVIATION and carries its burden of proof. Named: (1) drain-and-flip -- a chunked prefill interrupted by a flip never reaches `cache_finished_req` (sgl-project#856 removed the carry, the flip DISCARDS it), so upstream's publish-at-finish never fires; (2) GDN hybrid -- a recurrent state is valid at exactly one token position, so the per-chunk anchor has no pure-attention analogue. Gate off = byte-identical to upstream (verified: branch truth table, 0 mismatches over all 8 chunked/write_back/force combinations). LAWS: `mamba-per-knoten-nicht-gitter` in its own words ("states per radix node/chunk like KV pages"), and it waives write volume explicitly. `kein-doppel-prefill` (sgl-project#939): loss bound becomes ONE chunk. sgl-project#968 PP0 DEBT -- WHERE THIS PATH STAYS RANK-LOCAL: the publish decision is taken at the scheduler's chunk boundary, which every rank runs for the same request at the same split, so it is unanimous BY CONSTRUCTION, not by agreement; this path holds no reduce (the `check_prefetch_progress` MIN is TP-scoped and the boot runs tp_size=1/pp_size=3, so it is structurally skipped). `raenge-nie-uneins` is met by construction and `#1028P CHUNK-PUBLISH` is how the claim gets CHECKED: identical n at identical timestamps across ranks, the evidence shape `#969H` gave for the 11. The one rank-local input reachable from here is the sgl-project#581/sgl-project#773 write-through pin budget, which fired ZERO times in that boot (trap-safe: bare 0, genuine 0) because 11 backups never approached it. Per-chunk publishing makes it reachable for the first time; `pin_skipped` rides on the same line so a nonzero count next to a divergent n is the divergence, named in advance. COST, QUANTIFIED NOT BUILT: one `.mamba` blob is 78446592 B = 74.8 MiB (measured, all 11 identical); a KV page is 32768 B. A published 4096-token chunk therefore adds 74.8 MiB of anchor on top of 128 MiB of KV (+58% L3). Host RAM is UNCHANGED -- the host mamba pool is pre-sized at boot from `hicache_ratio` (1.5) x device slots, so anchors roll through a fixed tier and land on disk. The int8-anchor idea (sgl-project#1013) would cut the 74.8 MiB and is deliberately NOT built here. ALSO: #1028B FETCH CAP instrument at the `min` in `hicache_storage.py`, the cap that decides how much of an existing KV prefix a prefetch may claim. It printed nothing: `final_pages`, `kv_pages`, `boundary=`, `hit_pages` each occur 0 times in the whole 5.87 MB log, so "anchors too sparse" and "KV prefix too short" produced the same number and were NOT separable from that boot. Now both terms print on one line. Desk checks (error-class matched): ruff F821/F811/F841/E9 clean on both changed files (new names + new attribute reads); branch truth table executed, gate-off equivalence exact. The gate is deliberately NOT memoised -- `enable_storage` is False at __init__ and only set in `init_hicache`, and a memoised early False would leave a wired-but-inert write path, the sgl-project#742/sgl-project#745 class this area has produced before. NOT YET BOOT-PROVEN. Acceptance is the next boot.

Ying1123 added 2 commits July 31, 2024 16:47

Create troubleshooting.md

a880481

Update troubleshooting.md

0c2b14f

Ying1123 merged commit 90286d8 into main Aug 1, 2024

Ying1123 deleted the Ying1123-patch-1 branch August 1, 2024 07:05

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Add troubleshooting doc (sgl-project#856)

886a03b

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026

Merge fix/905-flip-contract-suite (sgl-project#905, the flip contract…

8890a6e

… suite relit against the sgl-project#856 contract) into the flip train

Navigation Menu

Uh oh!

Add troubleshooting doc - #856

Add troubleshooting doc#856
Ying1123 merged 2 commits into
mainfrom
Ying1123-patch-1

Ying1123 commented Jul 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Ying1123 commented Jul 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant