Decouple kv by hnyls2002 · Pull Request #679 · sgl-project/sglang

hnyls2002 · 2024-07-20T21:15:07Z

As flashinfer has supported a more standard kv cache layout, we changed our kv cache to decoupled KV.

Co-authored-by: shiyang814-cpu <shiyang.814@bytedance.com>

… and die THE DEATH. 2026-08-15 23:41:01, unpinned boot 1, pool 454039, five agent lanes. available 0, evictable 0, and alloc_token_slots raised "Out of memory. Try to lower your batch size" out of get_new_batch_prefill -> prepare_for_extend -> alloc_for_extend on ALL THREE ranks at once, then "terminate called without an active exception". WHY NO GRACEFUL PATH ENGAGED -- every relief this tree owns, traced: retract_decode reachable only from update_running_batch, which runs AFTER get_new_batch_prefill in the same iteration. The prefill path cannot reach it. the sgl-project#287 ladder needs --kv-pressure-ladder to exist at all, commits transitions only every consensus_interval (8) rounds behind a bounded collective, and its actuators reshape FUTURE admission. It never touches the batch that is about to allocate. kvso try_spill wired into the decode-OOM branch and into the ladder. Not reachable from the prefill allocation. evict_from_tree_cache the one relief the alloc site does attempt -- and with nothing evictable it is a guaranteed no-op that returns no signal, so the raise was reached having tried nothing at all. AND ADMISSION HANDED IT THE WORK. PrefillAdder.add_chunked_req carried: if _rem_tokens <= 0: _rem_tokens = self.rem_chunk_tokens When the budget said the pool had NOTHING, schedule a full chunk anyway. The comment above it is right that the request must not be dropped -- it leaks if it leaves unhandled -- but "admit it anyway" was never the only way to keep it. PARKING is already a first-class state here: the hybrid-SWA branch three lines up produces it, and the scheduler documents the result at the chunked-request stash ("a parked chunk leaves extend_range.end == len(prefix_indices), so there is nothing new to cache and stashing would be a no-op"). THE FIX IS TWO LAYERS AND THEY ARE NOT INTERCHANGEABLE. PREVENTION, admission side, GROUP-UNIFORM. A chunk is scheduled only for tokens the pool can actually fund; below one page it parks and is retried when memory frees. The availability term comes from the PUBLISHED floor (uniform_avail_for_evict), not this rank's own size -- under uneven DCP the ranks differ, and a rank-local branch here splits the group across different batches, which is a hang rather than a stall. That distinction has cost this chain two boots already, so it is not being relearned a third time. Where the pool can fund something but less than the nominal chunk, the chunk now takes what exists instead of the nominal size. A NET, alloc site, RANK-LOCAL. One bounded relief, one retry, then the original error unchanged -- the shape _mem_create_reclaiming already uses for driver OOM, followed rather than reinvented. Providers are rank-local BY CONTRACT: by the time execution is here the group has committed to a batch, and a provider taking a collective would hang the first time one rank arrived and its peers did not. THE NET IS CURRENTLY EMPTY, AND IT SAYS SO. Nothing registers a provider: the rank-local relief that could pay (eviction) is already spent by then, and the ones that could genuinely free tokens are collective and belong on the admission path. So the registry is a SEAM, and rather than let it be a term that is present and inert -- the failure this tree keeps finding -- reaching the alloc site with an empty registry logs, once, that no relief existed and that admission is therefore the whole guarantee. THE 45s WINDOW IS THE AMPLIFIER, NOT THE CAUSE. SGLANG_PHASE_POLICY_PP_WINDOW_S=45 is live and admits roughly three times the concurrent prefills the 15s regime did, which is why the pool reaches zero far more often now. Nothing in this fix depends on the window length: the guard is a function of what the pool can fund at the moment of scheduling, which is exactly the quantity a longer window drives to zero. Tests: 23, against a hermetic exhaustion fixture (available 0, evictable 0 -- the crash's own state). Can-fail proven by three mutations: restoring the zero-budget override, making the admission predicate rank-local (2 fail, including the three-ranks-one-floor case), and removing the alloc-site relief (4 fail). NOTE honestly: the restored-override mutant is caught by a SOURCE pin, not an execution test -- exercising add_chunked_req itself needs a full PrefillAdder, so the decision was extracted into a pure function that the behavioural tests drive, and the wiring is pinned separately. No regressions: 989 pass across the scheduler/prefill/seam suites, with the same 5 pre-existing failures before and after (test_scheduler_chunked_req_gate and test_scheduler_pp_request_order_633, both stale test stubs whose fake Scheduler lacks phase_flip_runtime/ps -- a collision with the phase-flip work, flagged rather than forced per instruction). NOT BOOTED. Desk-only by instruction.

… and say which path sized the pool RED TESTS ON THE LINE ROT, so these are repaired rather than skipped. All five had the same shape: a hand-built stub that stopped matching the class it fakes. THREE IN test_scheduler_chunked_req_gate.py. The file already carries a comment warning that "a bare MagicMock makes EVERY flag truthy, which arms the phase-boundary actuators this gate test has nothing to do with", and pins the flags it knew about. ``enable_phase_flip`` was not among them, so the prologue entered ``_phase_flip_on_round``, which LAZY-BUILDS a PhaseFlipRuntime from a Scheduler that has none of the state a build needs. Pinned off with the rest -- the same trap the block was written for, one actuator further on. A second gap behind it: ``_make_req`` builds a Req via ``__new__``, so every field the class has gained since must be restated; ``kv_spill_state`` was missing, and its default is taken from Req.__init__ rather than invented, because a stub that guesses a default is a test that passes for the wrong reason. ONE IN test_scheduler_pp_request_order_633.py, same class: ``_pp_forward_and_process_input_requests`` now consults ``pp_phase_flip_armed()``, which reads ``server_args.enable_phase_flip``, so a bare ``SchedulerPPMixin()`` raised before the ordering under test was ever exercised. A named stub factory now pins it off once instead of twice. AND ONE STALE SOURCE PIN, which is the one worth reading carefully. ``test_every_pp_loop_calls_the_helper`` asserts that every PP loop routes received requests through the forward-first helper -- the property that stops adjacent stages deadlocking on a control request. ``event_loop_pp`` was refactored to set ``_defer_flip_round_to_pp_loop`` and delegate to ``_event_loop_pp_body``, and the helper call moved one frame down with it. The property was never lost; the pin stopped reaching the code it guards. So the pin FOLLOWS THE DELEGATION rather than being relaxed: the effective source of a loop is its own source plus the body it delegates to. Weakening the assertion would have been the easy repair and the wrong one -- a pin that no longer reaches its subject reads as protection while protecting nothing, which is exactly how this file's property could be lost silently. Can-fail re-proven: replacing the helper call with a direct ``process_input_requests`` fails both the reach pin and the no-direct-call pin, on ``event_loop_pp`` specifically. SEPARATELY, THE SIZER NOW SAYS WHICH PATH IT TOOK. sgl-project#678's acceptance turns on whether the pool was SOLVED from the at-rest free column or APPROXIMATED by the budget subtrahend, and that was not visible in a boot log -- it had to be inferred from the pool number, which is an investigation rather than a check. Both branches log one line naming themselves and their inputs, so acceptance is a grep. The approximating branch also names the record provenance, because "why is there no column" is the immediate next question. Tests: 993 pass across the scheduler/prefill/seam suites with ZERO failures -- the first clean run of this sweep in the chain. The four remaining ruff E402s in the gate test are its own deliberate post-``maybe_stub_sgl_kernel()`` imports, pre-existing and untouched. NOT BOOTED. Desk-only by instruction.

…roject#677 to compose against The sgl-project#679 close-out said the retry net's registry is empty because the reliefs that could genuinely pay are collective and belong at admission. This is that claim made concrete, written so sgl-project#677's hysteresis-drain design can consume it without re-deriving anything. WHAT IT ESTABLISHES. Four reliefs exist and only four: radix eviction, the sgl-project#287 ladder's admission_cap, kvso try_spill, and retract_decode. For each: what it frees at the admission decision point, what it costs, and how it is invoked without splitting the group. Two of them are not what one would guess: the sgl-project#287 ladder is TOO SLOW to be a rung. Its consensus boundary is every 8 rounds; a chunked-prefill burst exhausts the pool in fewer. It is the slow outer loop -- and the decode path already uses it that way, throttling before retraction to stop the freed slots being handed straight back. kvso try_spill is the BEST rung, not retraction. It frees a bounded, chosen amount (the victim's block-aligned tail overhang), is already driven from the reduced value, and costs no request's progress. Its bound is the host region supply, and exhaustion is a reachable state the decode path already documents. THE ITERATION ORDER, VERIFIED, AND IT IS FAVOURABLE. The reduce runs at scheduler.py:4777, unconditional and pre-branch by its own comment; admission at 5089; retraction at 5950 inside update_running_batch (5818). So a ladder at admission reads an ALREADY-AGREED number and needs no collective of its own, while retraction is genuinely downstream -- which is why sgl-project#679's crash had nothing to fall back on, and what rung 3 must work around. THE ORDER, mirroring the decode-OOM branch rather than inventing a second shape: evict (baseline) -> try_spill -> throttle -> retract_decode -> PARK. Parking stays the floor of the ladder, not its replacement. THE COMPOSITION CONTRACT WITH sgl-project#677, stated in three rules because both mechanisms decide admission from pool headroom: 1. The park guard is the FINAL authority, the drain gate the prior one. A phase gate cannot make memory exist; it may narrow what the park guard allows, never widen it. 2. Any headroom quantity sgl-project#677 branches on must be the GROUP-PUBLISHED floor. A rank-local reading in the drain gate reintroduces the sgl-project#603/sgl-project#583 divergence class upstream of every safeguard sgl-project#679 added -- and a park guard reading the reduced floor does not protect a gate that reads a local one. 3. Parking must be reachable from every path that reaches alloc_for_extend. AND THE LIKELIEST COMPOSITION FAILURE, called out so it can be tested on both sides before it is met on metal: hysteresis and parking can beat against each other. A parked chunk schedules ZERO tokens, so a drain that counts admission ATTEMPTS rather than SCHEDULED TOKENS will believe work was taken, hold its hysteresis, and deadlock against the park at exactly the pressure where both are needed. Section 5 lists what this note does NOT close: the ladder is unbuilt, rung 3 is a real refactor, and rung 1's host-region bound has never been measured under the 5-lane load that produced the crash. No code change. Every claim carries its file:line.

…ore parking Built to DESIGN_679_admission_relief_ladder.md. A park is not free -- it is a request that made no progress this round -- and the note costed what should be spent before accepting one. This spends it. rung 0 radix eviction already spent by the caller. Baseline, not repeated. rung 1 kvso.try_spill bounded, chosen, costs no request's progress. rung 2 throttle frees NOTHING now; stops rung 3 repeating next round. rung 3 retract_decode most tokens, loudest: the victim re-prefills. rung 4 PARK the floor, and still the final authority. RUNG 3 IS THE ONLY HARD PART, AND IT IS A REFACTOR, NOT A CALL. retract_decode is one line; what surrounds it in update_running_batch is what must not drift -- the metrics, the new_token_ratio handover, the abort dispatch, and above all for req in retracted_reqs: self._add_request_to_queue(req, is_retracted=True) A second implementation that forgot that line would LEAK every victim it retracted, which is worse than the crash this ladder exists to prevent. So the block is EXTRACTED VERBATIM into _retract_decode_and_requeue and both call sites share it. One implementation, two callers, no drift -- and a test pins that the shared actuator still contains the requeue. The precondition travels with it: rung 3 sets batch.uniform_avail_floor from the reduced value before retracting, because that bound governs the retraction loop AND the last-survivor test. sgl-project#583 is exactly the case where the entry decision was uniform and the loop bound was not, so ranks entered together and popped DIFFERENT numbers of victims. GUARD (a) -- EXHAUSTION IS AN OUTCOME, NEVER AN ERROR. try_spill returns False when no host region is free; the ladder falls through to the next rung. Same for a rung that frees less than asked, and same for a rung that RAISES: every rung is wrapped, the ladder continues, and nothing here can turn a relief bug into an instance death. That host-region bound is still unmeasured under the 5-lane load that produced the crash, which is precisely why it is treated as ordinary. GUARD (b) -- EVERY DECISION IS GROUP-UNIFORM. Each rung reads uniform_min_avail(), the value the pre-branch reduce published at the top of this iteration, so no rung takes a collective of its own and none can split the group. The shortfall is sized from that same reduced value, so every rank asks its rungs for the same tokens -- sizing it locally would be sgl-project#583 one layer up from where sgl-project#583 was found. GUARD (c) -- OFF BY DEFAULT. SGLANG_ADMISSION_RELIEF_LADDER unset returns 0 before touching anything: byte-identical to c4b88e1, the boot currently serving. Rung 3 carries a SECOND flag (SGLANG_ADMISSION_RELIEF_RETRACT) because it is the only rung that destroys progress, and it is inert unless the ladder itself is on. THE PARK REMAINS FINAL (DESIGN_679 rule 1). The ladder changes what there is to decide from; add_chunked_req still decides. It runs immediately before that call and nowhere else, and a source pin refuses can_run_list / add_chunked_req / set_extend_range inside it -- a ladder that admitted anything would be a second admission authority. Tests: 18 hermetic, on a Scheduler stub carrying only what the ladder touches. Mutation-proven on the four decisions that could invert silently: default flipped ON -> the off-by-default case fails rung 3 without uniform_avail_floor -> the sgl-project#583 precondition case fails exhausted spill treated as terminal -> the fall-through case fails a paid spill no longer short-circuiting -> the ordering case fails Each fails alone, by name. One test of my own was caught by its first run reading the wrong frame (get_new_batch_prefill delegates to _get_new_batch_prefill_raw) -- the same stale-pin class repaired in the PP loop tests, fixed the same way. No regressions: 1011 pass across the scheduler/prefill/seam suites, zero failures. The extraction adds no lint (94 before, 94 after, all pre-existing). NOT BOOTED. Desk-only; validation bundles with the sgl-project#678 acceptance.

…64 an invariant THE DEATH. 2026-08-16 00:25:07, nineteen minutes into the c4b88e1 boot under five-lane load, all three ranks at once: RuntimeError: Expected 'candidates' to be of type long (torch.int64) tree_speculative_sampling_target_only <- eagle_sample <- verify <- run_batch The iteration before: full token usage 0.97, a 257-token chunked prefill admitted, 143,984 pending. THE HYPOTHESIS WAS HALF RIGHT. Not a float tensor from a defaulted torch.empty -- int32, deliberately: eagle_sample candidates = verify_input.draft_token.reshape(..) _build_trivial_verify_input draft_token = draft_input.bonus_tokens verify bonus_tokens = torch.empty_like(accept_lens, dtype=torch.int32) bonus_tokens = torch.empty((0,), dtype=torch.int32) draft_token IS the candidates tensor, and one construction path hands it over int32. int32 is CORRECT at its source -- bonus_tokens is an *input* to build_tree_kernel_efficient on the ordinary path, never the candidates tensor. The dtype is wrong at the door it came through, not where it was made. WHY ONLY UNDER PRESSURE. The trivial path is taken when drafting is disabled at high batch size, or on a phase-flip draft bootstrap -- both load states. The ordinary path takes draft_token from build_tree_kernel_efficient, which returns int64. A quiet instance never sees it. LATENT, NOT INTRODUCED BY sgl-project#679, and this was checked before anything was written. The int32 dates to upstream sgl-project#24724 (2026-05-08). The trivial path and all three int32 constructions are byte-identical at 1d1dbf9, and no commit in this chain touches python/sglang/srt/speculative/ at all. What sgl-project#679 changed is SURVIVAL: the instance used to die at the allocator before it could keep taking the trivial path at 0.97 usage. The park removed the earlier death and exposed this one. That changes the framing, not the necessity. THE FIX IS AT THE CONSTRUCTION SITE. A .to(long) in eagle_sample would fix this crash and leave every other consumer free to be handed the wrong dtype. EagleVerifyInput.__post_init__ enforces the contract for EVERY construction path, present and future. The class already DECLARED that contract in its own create_idle_input default (torch.empty((0,), dtype=torch.long)) -- a default is not an invariant, and this makes it one. The source is deliberately NOT changed: bonus_tokens must stay int32 for its other consumers, so converting at its origin would trade this crash for a different one. CONVERT, DO NOT RAISE -- a serving instance degrades rather than dies, which is the whole subject of sgl-project#679 -- and the conversion is a no-op on every compliant path. But it is ANNOUNCED, once per offending dtype, because a fix that silences its own trigger is how the defect returns: the next path to hand over the wrong dtype would otherwise be absorbed without a word. Tests: 12, extending the exhaustion work to the verify path at ~1.0 usage. RED-FIRST against 82ba7e2: 7 fail there, including the crash reproduction itself (int32 bonus_tokens -> int64 candidates). Covered: both bonus_tokens branches, the float case the hypothesis proposed, value preservation across the conversion, None, announce-once, silence on the compliant path, and three pins that this stays a construction-site fix (post_init enforcement; the trivial path still hands over bonus_tokens; eagle_sample still derives candidates from the field -- if any of those move, the pin must move with them). No regressions: 1017 pass, zero failures. NOT BOOTED. Desk-only; READY-FOR-BOOT.

…, by design not by accident The build brief named group-uniform rung decisions as a guard to be mutation-proven. It was implemented -- every decision reads uniform_min_avail() -- but the tests caught a rank-local read only by ACCIDENT: the stub carried no allocator, so a mutation raised AttributeError, the trigger's own except swallowed it, and the ladder silently did nothing. Tests that pass because the wrong code crashes are not tests of the property. So the stub now carries a rank-local availability that DISAGREES with the reduced one -- 10x the chunk, i.e. comfortable, while the group is starved -- and two cases assert the ladder acts on the group's number: it spends rungs when the GROUP is short even though this rank looks fine, and it sizes the shortfall from the group value (sgl-project#583 one layer up: ranks that size the ask differently retract different numbers of victims). Mutation-proven properly now. Pointing the TRIGGER at this rank's pool fails five cases including both new ones by name; pointing the ladder's internal comparison at it fails the ordering case. Neither is an AttributeError any more -- each is a wrong verdict, which is what the group would actually experience. 20 tests. No production change.

… two readings that invert DESIGN_679 section 6. The ladder is off by default; this is the evidence required to change that, written so the window is judged rather than described. THE NULL RESULT IS STATED FIRST, because it is the likeliest outcome of a short window and the easiest to mis-file: if KV-ADMISSION-LADDER never appears, the pressure regime was not reached and the window proves NOTHING. Not a pass. The companion script refuses to exit 0 without engagement rather than leaving that to discipline. THE METRIC IS A RATIO. The ladder runs immediately before the park decision, so every engagement is a park that was about to happen: rungs_paid = engagements - parks Judging `parks` alone is wrong in both directions -- more parks can simply mean more pressure. A ratio near 1.0 WITH engagement is not a failure of the design; it is the HOST-REGION BOUND showing itself, the quantity section 5 says has never been measured, and it says the cheap arm cannot pay on this rig and rung 3 is required. TWO INTERACTIONS THAT INVERT IF READ CARELESSLY: the sgl-project#680 line is a BATCH-SHAPE PROBE here, not a dtype signal. It fires when the trivial verify path is taken, which happens when drafting is disabled at HIGH BATCH SIZE -- and the ladder shrinks batches. So it going quiet is NOT evidence the sgl-project#680 fix is inert; it is consistent with the ladder working. It getting LOUDER is the warning sign: rung-3 over-actuation collapsing batches into the degenerate path. the park counters cannot be read without a ladder-off control at comparable pressure. The script takes one as $2 and says so plainly when it is absent. The arm order is fixed: rungs 1-2 first (bandwidth and latency), rung 3 only in a SECOND window and with one extra requirement -- every retracted request must be observed coming back. The shared actuator requeues them and a test pins it, but a leak costs a user their request, so it is confirmed on metal rather than assumed. Reject list is explicit, and the split-batch wedge is an immediate reject rather than a regression to weigh. Instrument: /spinning/evidence-665-f1/accept_ladder_679.sh, proven on four synthetic logs (null / accept / host-region-bound / wedge+OOM). Not committable -- the evidence dirs are not git repos -- so it lives in place beside accept_678.sh. No production change.

…a tombstone leaf THE DIAGNOSIS IN 7752dc8 IS WRONG, AND CORRECTING IT CHANGES THE FIX. It said tokens behind a LOCKED chain are counted but unreachable, so the counter promises what the actuator cannot pay. The second clause is right. The first cannot happen: `inc_lock_ref` / `dec_lock_ref` walk from a node to the ROOT, and `_split_node` copies `full_lock_ref` onto the new upper half. So `full_lock_ref(parent) >= full_lock_ref(child)` holds on every edge at every moment. An unlocked node therefore has no locked descendant, its whole subtree is unlocked, and the peel always reaches it. Measured on the 01:46:10 tree itself, reconstructed from the dump all three ranks printed: of 65766 evictable tokens, 65254 sat in fully-unlocked subtrees, and the 512-token remainder is an artifact of one mis-ordered line in the interleaved three-rank output. The locked-chain term is ZERO. Subtracting it from the admission budget -- the repair this chain was pointed at -- would have moved the number the scheduler admits against by nothing, and the crash would have reproduced unchanged. That is why it is not what this commit builds. The reconstruction is not taken on trust: it is checked against two totals the process printed independently, `#full_tokens: 140683` and `full_evictable_size_=65766`, and it matches both to the token. THE GAP IS AT THE OTHER END OF THE FRONTIER. `evict_full` SELECTS with `get_leaf_lru_no_lock` -- unlocked and childless -- but `_evict_leaf_node` CONSUMES only nodes with a mamba value, and asserted when one was missing. An unlocked mamba TOMBSTONE leaf satisfies the selector and violates the consumer, and the cache produces that state itself: `_iteratively_delete_tombstone_leaf` breaks on `node.parent.full_lock_ref > 0`. A tombstone that loses its last child while a request holds it survives as a LOCKED tombstone leaf. When that request finishes nothing revisits it, so it becomes unlocked, childless, counted in `full_evictable_size_`, and first in line at the frontier. The 01:46 tree held exactly one -- node 5937, fr=0, mv=None, childless, in the full LRU list -- beside a single payable leaf, 5959. Replaying that dumped tree through the deployed code selects 5937 and dies on `AssertionError: leaf node mamba value is not None`. Only 6 of its 290 nodes held a mamba value at all; 128 of the 130 unlocked nodes were tombstones. This is a crowded state, not a freak one. THE REPAIR IS ON THE ACTUATOR, NOT THE COUNTER. Freeing an unlocked tombstone leaf is not a new capability: it is the same deletion `_iteratively_delete_tombstone_leaf` already performs one step earlier, taken now that the lock which deferred it is gone. Both routes now go through one `_free_tombstone_leaf`, so `full_evictable_size_` and the LRU list stay in step by construction rather than by two copies of the same five lines. Raising the actuator rather than lowering the counter is the stronger closure of "a counter must never promise what the actuator cannot pay": the promise becomes TRUE instead of becoming smaller, and every consumer -- admission budget, the sgl-project#679 park guard, the in-flight sgl-project#677 drain gate -- is repaired at once without any of them learning a second quantity. DELIBERATELY NOT EXTENDED PAST A LOCK. A LOCKED tombstone leaf is still refused and still uncounted; reaching behind a live reference is a different repair and stays filed. `test_a_locked_tombstone_leaf_is_still_refused` is the mutation proof that the new branch is gated on the lock and not on the tombstone alone. GROUP-UNIFORMITY needs no new channel. The branch is a pure function of replicated tree state -- the tree is a replica, `full_lock_ref` and `mamba_value` are replicated -- so every rank takes it on the same iteration. The existing `uniform_avail_floor` still decides WHETHER to evict; this only changes what the peel does once asked. No collective added. HONEST CAVEAT, ONE. Deleting a tombstone leaf cascades up through its now-childless tombstone ancestors, so a 512-token request can free far more: 54502 tokens on the replayed production tree. That overshoot is the pre-existing semantics of `_iteratively_delete_tombstone_leaf`, which is unbounded on the ordinary path too and exists to restore the "no tombstone leaves" invariant. Bounding it would leave the invariant broken, so it is left as it is and named here rather than discovered later. THE sgl-project#681 BACKSTOP STAYS, WITH ITS MECHANISM CORRECTED. The shortfall note also had a defect of its own: it read `tree_cache.evictable_size()`, which RAISES NotImplementedError on MambaRadixCache and SWARadixCache -- the very classes it was written for -- so on the crashing boot it would have reported -1. It now asks for `full_evictable_size()` first. Its text no longer asserts the falsified locked-chain mechanism and instead says what firing means now: a REGRESSION SIGNAL, a new class of node being counted that the peel cannot consume. WHAT IS STILL NOT EXPLAINED, SAID PLAINLY. The 01:46 process died with RuntimeError, not with the AssertionError the replay produces, and its own numbers say the tree was untouched by the eviction that ran immediately before (available 273 and evictable 65766 both unchanged at the raise). Every branch that could skip that eviction was checked and excluded: `is_chunk_cache` is False, `disable_radix_cache` is False, and `uniform_avail_floor` is None on this boot because tp_size=1 takes the single-rank early return. So a third mechanism remains unidentified. The receipt added in 7752dc8 is the instrument that will name it on the next boot; this commit removes one guaranteed way for the next eviction on that tree to kill the group, and does not claim to be the whole story. TESTS. `test_evictable_reachability_681`, 9 cases, CPU-only, no GPU: - RED FIRST on 7752dc8: `test_the_frontier_pays_the_tombstone_leaf_...` and `test_the_counter_never_over_promises` both fail with the production assert, `leaf node mamba value is not None`. Green after. - The ancestor-closure proof runs on MambaRadixCache and on the base RadixCache, across a lock, a deeper lock, and a split under a lock, with `test_the_detector_can_fail` hand-building the shape sgl-project#681 assumed so the three green assertions cannot be satisfied by a detector that always returns empty. - `test_the_state_the_crash_tree_was_in_is_reachable` reaches node 5937's exact signature through public transitions only. - `test_the_ordinary_path_is_untouched` pins the no-tombstone peel. Suites: 79 passed across the 681 file, both sgl-project#679 files and test_mamba_lock_ref_pairing_581.

… not hold MEASURED 2026-08-16 02:07:22, all three ranks, mid-cutover, on --max-running-requests 4: PHASE-FLIP POOL CENSUS pre-cutover pp_to_tp: ... cur_slot_reqs=5 resident_reqs=5 resident_slots=[0, 1, 2] PHASE-FLIP-CARRY carried 5 resident request(s) ... into the tp phase ResidentCarryError: running_batch claims 5 resident request(s), above max_running_requests=4 The carry had already succeeded twice; the raise came from `resident_req_identity`'s re-harvest, and it took the group down. THE FIFTH RESIDENT IS NOT A CORRUPTED SET. It is a state the scheduler creates ON PURPOSE. `_get_new_batch_prefill_raw` suspends the running-request cap for as long as a chunked prefill is in flight, and its comment names the reason it must: # Ignore the check if self.chunked_req is not None. # In PP case, chunked requests (or dllm requests) can start in one # microbatch and end in another microbatch, so the max_running_requests # per microbatch should not be strict. Instead, we should always allow # chunked requests to be added, otherwise, there will be a memory leak. So the maintained bound is `max_running_requests + 1`, and the guard was asserting `max_running_requests`. The competing explanation -- that admission simply over-admitted -- is excluded by `AdmissionLimiter`'s own contract: the ceiling "is what the pools were built for and can never be exceeded", so nothing could reach 5 through the admission limit. EXACTLY ONE, because the scheduler holds exactly one: `self.chunked_req` is a single slot, asserted empty before a new one is stashed. `cap + 2` therefore remains a corrupted resident set and still raises -- this widens defect M's ceiling by the one the scheduler documents, it does not remove it. THE ALLOWANCE IS UNCONDITIONAL, AND THAT IS THE GROUP-UNIFORMITY ARGUMENT. Gating it on `scheduler.chunked_req is not None` would be tighter and would be a hang: that flag is per-rank scheduler state, the PP ranks sit at different pipeline positions, and at one cutover instant a peer can hold it while this rank has just cleared it -- the same resident set legal on one rank and fatal on another. Deriving the ceiling from `max_running_requests` alone keeps the verdict replicated: `init_admission_limiter` documents that value as "uniform across ranks by construction: every input to `ceiling` is min-reduced before it gets here". No collective added, no second channel. THE SECOND COPY, WHICH IS WHY FIXING ONE WOULD NOT HAVE HELPED. `phase_flip_draft_bootstrap.arm_draft_bootstrap` carries an INDEPENDENT ceiling check -- correctly so, since `committed_slots` is where the one-tensor-per-request allocation actually happens and it "checks that input itself rather than trusting every present and future caller". It inherited the same too-tight bound. The crashing configuration is PP->TP with NEXTN, so that leg runs: repairing only the carry would have moved the same raise one function later. Both now import ONE `IN_FLIGHT_CHUNKED_ALLOWANCE`, and `test_the_two_ceilings_agree` pins that they cannot drift apart -- two guards asserting two different bounds is the same defect with a longer fuse. A RECEIPT, BECAUSE THE ATTRIBUTION IS AN INFERENCE. That the fifth resident was the in-flight chunked prefill is established by ELIMINATION, not by observation: no chunked request is visible in the 02:07 log, and the nearest sgl-project#679 park was at 02:04:08, three minutes earlier. The park is NOT required for this state -- with chunked_prefill_size 512 against 100k-token prompts a chunked prefill is in flight most of the time -- but that also means the attribution rests on the cap bypass being the only documented route past the limit. So the harvest now logs, once, what was actually true when the allowance was spent, including `chunked_req=SET|CLEAR`. A future excursion reporting CLEAR means the elimination has a hole and this widening is covering something else. It is a LOG, never a decision: the verdict reads only `max_running_requests`, so the line cannot make two ranks disagree whatever it prints. CORRECTED IN PASSING: the `IMPLAUSIBLE_RESIDENT_REQS` comment asserted "no batch on this server holds more requests than max_running_requests (4 on the production recipe)", which is the same falsified claim one guard down. TESTS. Red-first on e778276, both legs: - carry: `test_the_cap_plus_the_in_flight_chunk_is_carried`, `test_the_error_names_the_effective_ceiling_not_the_raw_cap`, `test_a_ceiling_of_one_still_admits_its_chunk` (additive, not proportional -- --max-running-requests 1 is a real configuration) and the group-uniformity subtests all fail with ResidentCarryError before, pass after. - draft: `test_the_chunked_prefill_excursion_arms_instead_of_raising` and `test_the_two_ceilings_agree` fail before, pass after. - OPPOSITE DIRECTION, GREEN FROM THE START so the widening cannot degenerate into removal: `test_a_sixth_resident_still_raises`, `test_a_resident_set_two_above_the_cap_still_refuses_to_arm`, `test_the_absurd_length_is_still_refused` (defect M's real shape, 5000), and the pre-existing defect-M class unchanged. The constant is pinned from BOTH sides -- 5 accepted, 6 refused -- so its value cannot drift silently. - receipt: fires above the cap with the chunked state named, silent at or below it. Suites: 81 passed across both guard files; 868 passed, 0 failed across every phase-flip test in unit/managers.

…l; the new-request gate never did sgl-project#679 taught `add_chunked_req` not to schedule a chunk the pool cannot fund, and it made that decision GROUP-UNIFORMLY: `fundable_extend_tokens` reads the published group MIN through `uniform_avail_for_evict`. Its sibling gate -- the one every FRESH request passes -- was left reading this rank's own pool: PrefillAdder.rem_total_tokens available_size() + evictable_size() - offset - dcp_avail_deficit That is the same over-promise one door further along, and it fails twice over: * A rank roomier than the binding rank admits work the GROUP cannot fund, and the batch dies where it is allocated -- `alloc_for_extend`, 2026-08-16 01:46:10, all three ranks together. * It is a rank-local BRANCH upstream of a collective. Two ranks can build different batches and enter the next extend with different token axes, which is a HANG rather than a stall -- the family sgl-project#583/sgl-project#603/#616g/sgl-project#639 was paid for. `fundable_extend_tokens`' own docstring names the first half. It was written for the chunked gate and never wired to this one; `fundable_extend_tokens` appeared nowhere in scheduler.py, which the red-first run confirms. WHAT THIS IS NOT. It is not the cause of the 01:46 traceback. That was e778276 -- an unlocked mamba tombstone leaf the eviction frontier selected and `_evict_leaf_node` could not consume, so eviction under-delivered while the counter honestly promised 65766 evictable. THE COUNTER AND THE ACTUATOR NOW AGREE; this commit is about the second consumer of that counter, which reads a rank-local copy of it and can over-promise even when it is honest. A CEILING, NOT A SUBSTITUTE. `rem_total_tokens` also subtracts reservations the floor knows nothing about (the running batch's hold, mamba gap, page overhead), so replacing the local term would talk a rank that is ITSELF short back UP. The budget is the MIN of the two. Both spend down the same `rem_total_token_offset` -- a floor consulted per request without that shared accounting is not a bound at all, since every request in the round would compare itself against the same untouched pool. `dcp_avail_deficit` is deliberately NOT subtracted from the floor term: the floor is already the group MIN, and the deficit is the other mechanism for pinning a local number to the binding rank. WHY THE MIN RESOLVES TO THE FLOOR IN PRODUCTION, and it is a test, not a claim: `floor = uniform_avail + evictable` and `local = local_avail + evictable - offset` with `uniform_avail <= local_avail` by definition of a MIN, so `floor - offset <= local` on every rank. A rank BELOW the floor is not a state the reduce can produce; the clamp's behaviour there is pinned separately because refusing to be talked up by a stale floor is the wanted behaviour. THE CEILING MUST NOT BE ABLE TO WEDGE THE INSTANCE, and this is why the wiring goes through `published_fundable_floor` rather than calling `fundable_extend_tokens` directly. That helper returns 0 for BOTH "the pool is empty" and "the pool could not be read". The chunked gate can live with the ambiguity -- 0 parks a chunk and the next round retries, a self-clearing state. As a budget ceiling a mis-read 0 admits NOTHING, for every request, on every subsequent round: a harder wedge than the crash it prevents. So the ceiling is applied only where a floor was actually PUBLISHED, which is the one state (uneven rank pools) it exists for. A published floor of genuine zero still binds -- the distinction is "was a floor published", not "is it non-zero", or the ticket's own crash state would be exempt from its own fix. DEFAULT PATH UNTOUCHED, BY CONSTRUCTION AND BY ASSERTION. With no floor published -- single rank, or pools that agree -- `fundable_extend_floor` is None and `rem_total_tokens` returns exactly what it returned before. TESTS. RED-FIRST against 7936bc4: 12 of the 18 fail there, including the 01:46 shape itself (a rank holding 100000 tokens admitting against a group that holds none) and all three wiring pins. The guard against the wedge was falsified separately by removing it -- 2 of its 4 cases go red -- because an instrument that cannot fail is not evidence. Covered: the cap; the floor spent down by admissions; group agreement across ranks with different local pools; the floor binding before the local term whenever the reduce's invariant holds; the local term still binding when it is the smaller one; inertness with no floor and with a roomy floor; the published-vs-readable distinction including a genuine zero; and three pins that this stays wired (the scheduler constructs with a floor, it comes through the guarded helper, and that helper still delegates to the group-uniform one). NO REGRESSIONS, measured against the same subset on the unpatched tree: 944 failed / 2776 passed / 725 skipped before, 944 failed / 2794 passed / 725 skipped after -- identical failures (all CUDA-gated collection, hermetic run with CUDA_VISIBLE_DEVICES=""), +18 = exactly the new cases. NOT BOOTED. Desk-only; READY-FOR-BOOT. The metal arm that would exercise it is a multi-rank boot with uneven pools under max fill -- the arm that produced the 01:46 specimen -- watching for the admission ceiling binding without a wedge.

…ad only STAGED VERIFIED INDEPENDENTLY FROM SOURCE, then against F4-r4's chain -- both arrive at the same place, which is why this ships rather than another hypothesis. THE SPECIMEN (2026-08-16 13:58:37, sgl-project#693, all three ranks byte-identically): RuntimeError: Out of memory. Try to allocate 512 tokens. Available full tokens: 167743 (full_available_size=392 + full_evictable_size=167351) and NO `EVICTION UNDER-DELIVERED` line. That absence is the evidence, not a hole in it: `_eviction_shortfall_note` returns "" only when `evicted >= asked`, so eviction reported delivering its full 512 while the pool's free count stayed at 392. THE CHAIN, every hop read: 1. `_evict_leaf_node` (mamba_radix_cache.py:915-916) calls `token_to_kv_pool_allocator.free(x.value)` and counts `len(x.value)`. Same for the tombstone route `_free_tombstone_leaf` (:1705-1706). 2. `TokenToKVPoolAllocator.free` (allocator/token.py:67-80) applies pages only `if self.is_not_in_free_group`; otherwise `self.free_group.append(idx)`. 3. `available_size` (:52-54) is `len(free_pages) + len(release_pages)` -- the staged list is in NEITHER -- and `alloc` (:56-61) compares against `len(free_pages)` alone. 4. `free_group_begin` is called from inside the event loop (batch_result_processor.py:92 and :741). An eviction landing in that window frees into the staging list. So the receipt is TRUE about the tree and FALSE about the pool: the nodes are gone and their tokens counted, and the pages sit in `allocator.free_group` waiting for `free_group_end()`. NOT A POOL-TIER MISMATCH, which was the other axis: `release_pages` IS counted by `available_size`, so a tier split would still show the tokens. And not a race -- the group opens at a fixed point in a replicated event loop, which is exactly why three independent processes printed identical counters. THE FIX, AND WHY THIS ONE OF THE THREE. (a) flush-and-close before an eviction-driven alloc -- rejected: it ends a batching window the caller still owns, silently unbatching its remaining frees and turning its own `free_group_end` into a no-op it never asked for. (b) make `num_tokens_evicted` count only non-deferred frees -- rejected as THE fix: it makes the counter honest and leaves the crash, since the caller's last word is still the sgl-project#679 hard raise. It repairs the message, not the state. (c) SHIPPED: `flush_free_group()` applies the staged frees WITHOUT closing the group. Safety is the whole question and it is answerable from the source: `free_group_end` is PURE BATCHING -- `self.free(torch.cat(self.free_group))`, one concat and one `_notify_free` (allocator/base.py:203-206). It defers nothing for correctness: no in-flight reference, no graph-replay barrier, and the same iteration applies these very pages a few lines later regardless. Flushing moves that application earlier; it makes nothing reachable that was not already about to be. The flag is restored so the caller's window and its later end-call are untouched, and the staged list is consumed so that end-call cannot double-free. Called from `alloc_token_slots` BEFORE the relief ladder, because it is not relief: the rungs below spend host bandwidth or a victim's decode progress, this spends nothing. Cold path only -- it runs after an allocation has already failed, so a healthy alloc pays nothing at all. `backup_state` is re-taken after a flush; no caller passes it on this path today, and re-taking keeps that a fact about the callers rather than a dependency of this fix. TESTS. Red-first: 5 of 8 fail, and the three that matter fail with the specimen's own `RuntimeError: Out of memory` -- sgl-project#693 reproduced hermetically. The stand-in allocator mirrors token.py's free/alloc/group semantics and binds the PRODUCTION `flush_free_group` onto itself rather than reimplementing it, so the method under test is the real one. Covered: the allocation succeeding once the staged frees are applied; the group still OPEN afterwards; the caller's `free_group_end` still safe (no double free); no group open and nothing staged both byte-identical; a genuinely empty pool still raising (fail-loud preserved); and the base class carrying the method so every subclass inherits it. NO REGRESSIONS, measured both sides on the same subset: mem_cache 940 failed / 757 passed before, 940 failed / 765 passed after -- identical failures (all CUDA-gated collection), +8 = exactly the new cases. managers 2084 passed, 0 failed. Hermetic (CUDA_VISIBLE_DEVICES=""). NO DEPLOY. WHAT THIS DOES NOT CLAIM. It removes the counted-but-unpayable state. It does not make the pool larger: a genuinely exhausted pool still raises, and the admission-side fixes (sgl-project#679 park, sgl-project#681 group-uniform ceiling) remain the layer that should prevent reaching here at all.

…e two commits to hold MERGE_NOTES_602.md rewritten to cover all 18 commits: what each fixes, its test evidence, whether it touches runtime, and whether an equivalent patch is already on the serving line (verified with `git cherry`, not by message matching -- four are: c41645c, ce60358, 658ea3a, 84b0171). DRY RUN: clean. Merged into `integration/r2` -- the live line, since the serving tree descends from its tip a73a0d8 -- in a throwaway worktree, `--no-commit --no-ff`, then aborted and the worktree dropped. Zero conflicts, zero unmerged paths, so nothing was pre-resolved because nothing needed it. Verified semantically as well as textually: on the MERGED tree, managers 2093 passed / 0 failed and planner 2574 passed / 2 failed, the two being the same pre-existing test_rejected_evidence_pins pair that is already red on the base. THE FACT THE OPERATOR NEEDS, and it is not in the commit count: `7936bc4850` is NOT an ancestor of integration/r2, so merging this branch drags in its whole base lineage -- 115 commits, of which 18 are mine and 97 are the hotfix/677 work (sgl-project#662 x20, [PhasePolicy] x18, sgl-project#677 x8, sgl-project#678 x7, sgl-project#679 x6, ...), 129 files, +22731/-762. Approving this merge is approving that lineage, most of which is not mine to vouch for. If only this work is wanted it must be cherry-picked rather than merged. NOT ATOMIC, and it splits cleanly into four groups with an order: (1) the four already on serving -- merging them only reconciles integration with what is already running; (2) the three sgl-project#624 test-only drift guards, which take managers from 4 failures to 0 and should land early so the line stays green during review; (3) desk tool + docs, all planner/pp_cut.py and markdown, imported by no serving path; (4) hold. HOLD, two commits, both runtime and neither on the serving line: * e21e87f (sgl-project#690) touches the seam hot path and changes the PHASE-FLIP DONE format. Already queued to land on deploy WITH the W=8/W=4 probe after the sgl-project#694 soak verdict; merging it into integration first puts it in front of the soak meant to measure it. * 5301b94 (sgl-project#685) touches the boot sizing path. Announce-only today and abstention-guarded, but unsoaked, and the R' decision it waits on is not made. Nothing else in the chain can move serving behaviour. Docs only; no merge performed, no deploy, scratch worktree removed.

…l; the new-request gate never did sgl-project#679 taught `add_chunked_req` not to schedule a chunk the pool cannot fund, and it made that decision GROUP-UNIFORMLY: `fundable_extend_tokens` reads the published group MIN through `uniform_avail_for_evict`. Its sibling gate -- the one every FRESH request passes -- was left reading this rank's own pool: PrefillAdder.rem_total_tokens available_size() + evictable_size() - offset - dcp_avail_deficit That is the same over-promise one door further along, and it fails twice over: * A rank roomier than the binding rank admits work the GROUP cannot fund, and the batch dies where it is allocated -- `alloc_for_extend`, 2026-08-16 01:46:10, all three ranks together. * It is a rank-local BRANCH upstream of a collective. Two ranks can build different batches and enter the next extend with different token axes, which is a HANG rather than a stall -- the family sgl-project#583/sgl-project#603/#616g/sgl-project#639 was paid for. `fundable_extend_tokens`' own docstring names the first half. It was written for the chunked gate and never wired to this one; `fundable_extend_tokens` appeared nowhere in scheduler.py, which the red-first run confirms. WHAT THIS IS NOT. It is not the cause of the 01:46 traceback. That was e778276 -- an unlocked mamba tombstone leaf the eviction frontier selected and `_evict_leaf_node` could not consume, so eviction under-delivered while the counter honestly promised 65766 evictable. THE COUNTER AND THE ACTUATOR NOW AGREE; this commit is about the second consumer of that counter, which reads a rank-local copy of it and can over-promise even when it is honest. A CEILING, NOT A SUBSTITUTE. `rem_total_tokens` also subtracts reservations the floor knows nothing about (the running batch's hold, mamba gap, page overhead), so replacing the local term would talk a rank that is ITSELF short back UP. The budget is the MIN of the two. Both spend down the same `rem_total_token_offset` -- a floor consulted per request without that shared accounting is not a bound at all, since every request in the round would compare itself against the same untouched pool. `dcp_avail_deficit` is deliberately NOT subtracted from the floor term: the floor is already the group MIN, and the deficit is the other mechanism for pinning a local number to the binding rank. WHY THE MIN RESOLVES TO THE FLOOR IN PRODUCTION, and it is a test, not a claim: `floor = uniform_avail + evictable` and `local = local_avail + evictable - offset` with `uniform_avail <= local_avail` by definition of a MIN, so `floor - offset <= local` on every rank. A rank BELOW the floor is not a state the reduce can produce; the clamp's behaviour there is pinned separately because refusing to be talked up by a stale floor is the wanted behaviour. THE CEILING MUST NOT BE ABLE TO WEDGE THE INSTANCE, and this is why the wiring goes through `published_fundable_floor` rather than calling `fundable_extend_tokens` directly. That helper returns 0 for BOTH "the pool is empty" and "the pool could not be read". The chunked gate can live with the ambiguity -- 0 parks a chunk and the next round retries, a self-clearing state. As a budget ceiling a mis-read 0 admits NOTHING, for every request, on every subsequent round: a harder wedge than the crash it prevents. So the ceiling is applied only where a floor was actually PUBLISHED, which is the one state (uneven rank pools) it exists for. A published floor of genuine zero still binds -- the distinction is "was a floor published", not "is it non-zero", or the ticket's own crash state would be exempt from its own fix. DEFAULT PATH UNTOUCHED, BY CONSTRUCTION AND BY ASSERTION. With no floor published -- single rank, or pools that agree -- `fundable_extend_floor` is None and `rem_total_tokens` returns exactly what it returned before. TESTS. RED-FIRST against 7936bc4: 12 of the 18 fail there, including the 01:46 shape itself (a rank holding 100000 tokens admitting against a group that holds none) and all three wiring pins. The guard against the wedge was falsified separately by removing it -- 2 of its 4 cases go red -- because an instrument that cannot fail is not evidence. Covered: the cap; the floor spent down by admissions; group agreement across ranks with different local pools; the floor binding before the local term whenever the reduce's invariant holds; the local term still binding when it is the smaller one; inertness with no floor and with a roomy floor; the published-vs-readable distinction including a genuine zero; and three pins that this stays wired (the scheduler constructs with a floor, it comes through the guarded helper, and that helper still delegates to the group-uniform one). NO REGRESSIONS, measured against the same subset on the unpatched tree: 944 failed / 2776 passed / 725 skipped before, 944 failed / 2794 passed / 725 skipped after -- identical failures (all CUDA-gated collection, hermetic run with CUDA_VISIBLE_DEVICES=""), +18 = exactly the new cases. NOT BOOTED. Desk-only; READY-FOR-BOOT. The metal arm that would exercise it is a multi-rank boot with uneven pools under max fill -- the arm that produced the 01:46 specimen -- watching for the admission ceiling binding without a wedge.

…ad only STAGED VERIFIED INDEPENDENTLY FROM SOURCE, then against F4-r4's chain -- both arrive at the same place, which is why this ships rather than another hypothesis. THE SPECIMEN (2026-08-16 13:58:37, sgl-project#693, all three ranks byte-identically): RuntimeError: Out of memory. Try to allocate 512 tokens. Available full tokens: 167743 (full_available_size=392 + full_evictable_size=167351) and NO `EVICTION UNDER-DELIVERED` line. That absence is the evidence, not a hole in it: `_eviction_shortfall_note` returns "" only when `evicted >= asked`, so eviction reported delivering its full 512 while the pool's free count stayed at 392. THE CHAIN, every hop read: 1. `_evict_leaf_node` (mamba_radix_cache.py:915-916) calls `token_to_kv_pool_allocator.free(x.value)` and counts `len(x.value)`. Same for the tombstone route `_free_tombstone_leaf` (:1705-1706). 2. `TokenToKVPoolAllocator.free` (allocator/token.py:67-80) applies pages only `if self.is_not_in_free_group`; otherwise `self.free_group.append(idx)`. 3. `available_size` (:52-54) is `len(free_pages) + len(release_pages)` -- the staged list is in NEITHER -- and `alloc` (:56-61) compares against `len(free_pages)` alone. 4. `free_group_begin` is called from inside the event loop (batch_result_processor.py:92 and :741). An eviction landing in that window frees into the staging list. So the receipt is TRUE about the tree and FALSE about the pool: the nodes are gone and their tokens counted, and the pages sit in `allocator.free_group` waiting for `free_group_end()`. NOT A POOL-TIER MISMATCH, which was the other axis: `release_pages` IS counted by `available_size`, so a tier split would still show the tokens. And not a race -- the group opens at a fixed point in a replicated event loop, which is exactly why three independent processes printed identical counters. THE FIX, AND WHY THIS ONE OF THE THREE. (a) flush-and-close before an eviction-driven alloc -- rejected: it ends a batching window the caller still owns, silently unbatching its remaining frees and turning its own `free_group_end` into a no-op it never asked for. (b) make `num_tokens_evicted` count only non-deferred frees -- rejected as THE fix: it makes the counter honest and leaves the crash, since the caller's last word is still the sgl-project#679 hard raise. It repairs the message, not the state. (c) SHIPPED: `flush_free_group()` applies the staged frees WITHOUT closing the group. Safety is the whole question and it is answerable from the source: `free_group_end` is PURE BATCHING -- `self.free(torch.cat(self.free_group))`, one concat and one `_notify_free` (allocator/base.py:203-206). It defers nothing for correctness: no in-flight reference, no graph-replay barrier, and the same iteration applies these very pages a few lines later regardless. Flushing moves that application earlier; it makes nothing reachable that was not already about to be. The flag is restored so the caller's window and its later end-call are untouched, and the staged list is consumed so that end-call cannot double-free. Called from `alloc_token_slots` BEFORE the relief ladder, because it is not relief: the rungs below spend host bandwidth or a victim's decode progress, this spends nothing. Cold path only -- it runs after an allocation has already failed, so a healthy alloc pays nothing at all. `backup_state` is re-taken after a flush; no caller passes it on this path today, and re-taking keeps that a fact about the callers rather than a dependency of this fix. TESTS. Red-first: 5 of 8 fail, and the three that matter fail with the specimen's own `RuntimeError: Out of memory` -- sgl-project#693 reproduced hermetically. The stand-in allocator mirrors token.py's free/alloc/group semantics and binds the PRODUCTION `flush_free_group` onto itself rather than reimplementing it, so the method under test is the real one. Covered: the allocation succeeding once the staged frees are applied; the group still OPEN afterwards; the caller's `free_group_end` still safe (no double free); no group open and nothing staged both byte-identical; a genuinely empty pool still raising (fail-loud preserved); and the base class carrying the method so every subclass inherits it. NO REGRESSIONS, measured both sides on the same subset: mem_cache 940 failed / 757 passed before, 940 failed / 765 passed after -- identical failures (all CUDA-gated collection), +8 = exactly the new cases. managers 2084 passed, 0 failed. Hermetic (CUDA_VISIBLE_DEVICES=""). NO DEPLOY. WHAT THIS DOES NOT CLAIM. It removes the counted-but-unpayable state. It does not make the pool larger: a genuinely exhausted pool still raises, and the admission-side fixes (sgl-project#679 park, sgl-project#681 group-uniform ceiling) remain the layer that should prevent reaching here at all.

…e catch away The remainder of the crash pair, and it is not a missing net -- it is a net that was cast and then ignored. sgl-project#679 built the degradation: on a failed extend allocation, consult the rank-local relief provider and RETRY before raising. alloc_token_slots does exactly that (common.py:534-536, `allocator.alloc(num_tokens)` after the provider returns). Its page_size > 1 twin, alloc_paged_token_slots_extend, called the same provider, logged that relief had SUCCEEDED, and then fell straight through to the raise WITHOUT retrying. The freed pages were never spent. That is the shape of the 01:46 specimen: a raise past a degradation that had already produced the memory needed to avoid it. So the three raise sites on the prefill admission path were all "covered" by RULE 3, but coverage on this one meant asking, not using -- which reads as covered in an audit and behaves as uncovered in production. That distinction is the whole finding. FIX: the sgl-project#679 discipline verbatim, no new policy. The allocation plus its DSV4 bundle unwrap is now a closure, so the retry is literally the same call as the first attempt -- the previous shape duplicated neither, which is precisely how the retry came to be missing. The raise is untouched, so fail-loud still has the last word, and the log line now reports whether the retry SUCCEEDED or still failed instead of implying relief worked. REACHABILITY, stated rather than assumed: this rig runs page_size 1, so the twin is not on today's hot path. But _alloc_page_size notes DCP swaps in an allocator whose page_size is server_args.page_size * dcp_size, so any dcp_size > 1 boot takes it, and uneven DCP is shipped. Live path on a supported configuration, not a hypothetical. Tests: 4, red first. The falsifier failed with the exact production symptom ("Prefill out of memory") while relief had already freed the memory. The other three pin that fail-loud survives when the retry also fails, that a provider freeing nothing triggers no pointless second attempt, and -- the asymmetry this closes -- that alloc_token_slots already behaved this way. mem_cache + managers: 2887 passed, 0 failed. ruff clean.

…refused on structure Desk-only classification. Verdict and fix SHAPE only -- no fix built; the boot wrapper is F4-r4's and the operator's ack is required. THE HYPOTHESIS DOES NOT HOLD, and it is refused structurally rather than for want of evidence. The store port cannot collide across boots: route_a_631_prod_boot.sh pins neither --nccl-port nor --dist-init-addr, so server_args.py:18787-18788 applies -- nccl_port = get_free_port() draws a port that is free at that moment, and a predecessor still holding its own store port is not a candidate. A lost race is caught anyway at :18873 by wait_port_available, which polls 30 s and names the holding process. There is also a shape argument that does not depend on this codebase: "Connection closed by peer" is a connection ESTABLISHED and then broken, whereas a stale predecessor holding a port yields EADDRINUSE at bind or a refused connect. The observed error is the wrong shape for the hypothesis. So the answer to "does wait_host_release check the store port" is no -- and it should not need to. Extending it there would encode a refuted hypothesis into the boot wrapper, where the next reader would trust it. AT LEAST TWO ROOTS, not one. The specimens split on evidence: - 18:23:43 is not NEAR an OOM event, it IS one. syslog:1789 records "A process of this unit has been killed by the OOM killer" at 18:23:43.842677. The same signature repeats at 18:39:05 against e66bde7's own measurement at 18:39:07 -- F4-r4 called that boot "killed by an external process exit", which is what an OOM kill of a peer looks like from inside a survivor. A third sits at 18:45:30. - 19:31:45 rank2 is NOT memory. The host ledger reads avail=103 headroom=97 at 19:32:24, 39 seconds later, and there is no OOM line anywhere in the 19:3x window. SENDBYTES IS A TOMBSTONE. In 3 of 3 recorded instances it is the SECOND event: HANDOFF_663:696-698 (peer dies, no traceback -- "exactly what a SIGKILL looks like"; that run "died of host RAM"), HANDOFF_658:256 (rank 0's TCPStore died, survivors then spun on sendBytes), and e66bde7's gloo frame naming a dead peer process. Any verdict that makes the socket primary is fighting the prior. THE REAL HOLE is elsewhere and is the recurring shape. wait_host_release.sh computes S -- the count of surviving schedulers -- on every iteration, PRINTS it in the clear-to-boot line, and never puts it in a condition. The gate is available-RAM and nothing else, so a predecessor whose schedulers are alive but whose allocations are already unmapped passes it. The single-instance guard does not cover the gap either: boot script :250 pgreps sglang.launch_server, the LAUNCHER, while the store and ranks live in the sglang::scheduler children -- an orphaned scheduler set whose launcher has exited passes cleanly. That is the counter-without-a-reachable-actuator family (sgl-project#679/sgl-project#681/sgl-project#684/sgl-project#715): the value that answers the question is discarded one line before the decision. FIX SHAPE (not built): gate on the counter that already exists -- require S -eq 0 alongside A -ge NEED -- which closes the predecessor window for RAM, GPU and the store socket at once, without a bespoke port probe. Two cautions handed to the owner: grep -c "sglang::schedul" reads ps comm, truncated at 15 chars, so the match is one rename from silently counting zero and a miscounting gate that reports "clear" is worse than none; and requiring zero turns a soft wait hard, which is the right direction but must fail with PIDs named rather than timing out anonymously. HONEST GAPS, recorded in section 6 rather than papered over: dmesg is permission-denied here and journalctl -k returns "No entries", so kernel OOM detail is UNAVAILABLE from this session, not absent -- which process the killer took at 18:23:43 is therefore not established, only that it fired. The serving tree runs under setsid outside systemd, so the absence of an sglang unit line is not evidence it survived. Specimen B's root and the third specimen's log remain open; a timestamp tied to a specimen is not the same as its log read, and I have not read the second. Adds "schedul" to .codespellrc: it is the literal 15-char truncation ps comm reports, not a typo.

… it had to prevent DETERMINISTIC, 2/2, AT EXACTLY 7 BATCHES. Pin dc4895e, boots boot_943bx_dc4895e1dc_0828_000240.log and _001113.log, no CUDA error involved: scheduler.py:9286 in _get_new_batch_prefill_raw assert self.chunked_req is None AssertionError immediately after `sgl-project#798 PP-ADMISSION pass voided on slot N` and `#797d own pass voided on slot N`. THE ASYMMETRY. `_event_loop_pp_body` runs two voids and calls the second "AND THE MIRROR OF IT". They were not mirrored. sgl-project#797 (this rank retracted) sets `_pp_admission_pass_voided` in `_pp_void_retracted_pass`, BEFORE `get_next_batch_to_run`, so scheduler.py's guard refuses the pass and it builds nothing -- which is what that guard's own comment demands: "The retraction voids the pass, so the pass must build nothing at all." sgl-project#798 (this rank's UPSTREAM did not launch) set the same flag AFTER the call, so its pass ran the whole of `_get_new_batch_prefill_raw` -- advancing `chunked_req`, possibly adopting a fresh `new_chunked_req`, taking lock refs, emptying the waiting queue -- and was then unwound retroactively by `_pp_void_own_batch`. THE UNWIND CANNOT BE COMPLETED, so this is a guard and not a bigger unwind. `_get_new_batch_prefill_raw` reaches `_retract_decode_and_requeue` (the sgl-project#679 relief ladder, the #888b seat yield), which sends `AbortReq` to the tokenizer over `ipc_channels` at scheduler.py:9626-9632. A message already delivered to another process is not scheduler state and no handler can put it back. Every other unrestored item is a matter of writing more restore code; this one is not. THE CONDITION WAS KNOWABLE EARLY ALL ALONG, which is what makes the guard possible. `_pp_upstream_launched_incoming` is written by `_pp_recv_admission_decision` (scheduler_pp_mixin.py:5482), whose own docstring records the placement -- "Positioned in `_event_loop_pp_body` strictly BEFORE `get_next_batch_to_run`" -- and the call site agrees (:1959 receive, :2073 plan). One shared predicate, `pp_upstream_void_pending`, is now read at both moments, so the refusal and the forward cannot drift apart. The void is still FORWARDED from where sgl-project#798 always forwarded it. CARRIED, NOT DROPPED. Once the guard empties the slot, `self.mbs[mb_id]` is None on exactly the passes the sgl-project#798 site used to find non-empty, so two instruments would have gone quiet unnoticed: * the sgl-project#801-spin livelock streak, which raises at 512. The guard now records `_pp_upstream_void_withheld_work` (this rank HELD work and was refused anyway) so the streak counts what it always counted. It is a pre-plan superset of "derived a batch" -- the safe direction: it can make a genuine streak visible sooner, never hide one, and an idle rank still clears it. * `_pp_idle_void_suppress_log`, whose contract is that it "can never outlive this pass". Its consume sat AFTER the empty-slot early return, so it would have leaked a True into the next, unrelated sgl-project#797 void and silenced a record nothing asked to silence. Read and cleared at the top now. has_chunked_req DELETED, not left to mislead again. `add_one_req` carried it as a parameter its body never read; it only ever forwarded to `add_one_req_ignore_eos`, and upstream removed that last consumer in 8cc7726 ("Super tiny remove unused argument"). Reading it as the guard against a second chunked request cost a boot window. What actually holds that invariant is budget arithmetic, and that is now recorded at the assert -- including its measured reachability: witnesses under /spinning/evidence-665-f1/witness_951/ drive the real PrefillAdder into three states where `add_chunked_req` returns the request while leaving `rem_chunk_tokens` positive, and a mid-pass replenishment of `rem_total_tokens` then lets the loop mint a second chunked request. witness_941_d2 is the negative control and does not reproduce without it. That general case is a SEPARATE posten -- it needs no PP void and its danger direction (wedging a request mid-prefill, the sgl-project#858 shape) needs its own analysis. The assert stays: it is the honest watcher. TESTS. test_pp_upstream_void_before_formation_951.py, 11 tests, red-first: 3 fail on dc4895e and pass here. The five over-fire arms (first rank, pp_size<=1, gapped wire, healthy upstream, healthy chunked continuation) are green in BOTH states, which is what makes the red mean something -- a guard that over-fires voids every pass on PP0 and serves nothing at all. Two mutants on the carried instruments, each killed by exactly its own test: consume-after-return revives the suppress leak, emptiness-only revives the streak reset. Desk gate scripts/gate_tier2_partitioned.py, frozen both sides on this tree: BEFORE 4757 passed / 2 failed / 18 skipped, AFTER 4768 passed / 2 failed / 18 skipped. Delta +11 is exactly this commit's new tests. Identical failure set both sides (test_collective_family_siblings_610.py, 2 genuine, pre-existing at the pin); count check 2 == 2. The lane shift (wide -15, narrow -30, serial +56) is the gate's own sha256 rule demoting the four touched test modules to the serial lane. ruff parity exact per file (105/105, 3/3, 21/21, 1/1) with no finding on an added line; black run only on files that were clean at the pin. NOT BOOTED. Boot half is /spinning/gpu-arb/TICKET_951_WINDOW.md.

window-958-boot died 25 s into the chunked acceptance load at scheduler.py:7010, `AttributeError: 'NoneType' object has no attribute 'end'`, one line after `sgl-project#946 PREMISE RECOMPUTE`. THE ATTRIBUTION IN THE WINDOW CLOSEOUT IS REFUTED, BY TWO INDEPENDENT INSTRUMENTS. It put the null on the sgl-project#906 seam-refusal branch (scheduler.py:8916). * LOG. The full-phrase counter `[sgl-project#906] SEAM CHUNK REFUSED rid=` is 0 in BOTH boots, and `_note_seam_chunk_refused` logs its first three occurrences unconditionally (scheduler.py:5434-5446), so the zero is a measurement and not a rate limit. Other bracketed INFO tags from the same process are present in the same file, so the sink is not the explanation. * COVERAGE, which does not depend on any logging decision. The boot rode `SGLANG_949_COVERAGE=1`; boot 1's three rank databases (evidence-665-f1/trace949_0828/.coverage.24735{12,13,14}) all report scheduler.py:8916 **not executed**, while :8911 (the truncation), :8918 (the adder), :7010 (the reader) and scheduler_pp_mixin.py:2159/:2161 (the ring actuator and the call below it) are all executed on all three ranks. A fix at that branch alone would not have touched this crash. The junction took the ELSE branch every time and re-derived, exactly as sgl-project#946 argued. THE PRODUCER IS IN THE TRACEBACK, two statements above the reader. scheduler_pp_mixin.py:2159 calls `pp_apply_dead_premise_anywhere` -- sgl-project#948's relocated actuator, armed for that boot by `SGLANG_946_ACT_AT_RING=1` in the window's own recipe -- whose terminator runs `truncate_prefix_to(0)`; :2161 then calls `get_next_batch_to_run`. Nothing re-derives in between. sgl-project#946 had justified the truncation by its NEIGHBOURHOOD ("`add_chunked_req` below derives everything from `len(req.prefix_indices)` and only THEN calls `set_extend_range`"); sgl-project#948 moved the act to a site that RUNS, for a measured reason recorded at scheduler_pp_mixin.py:2100-2109 (the old site was entered ~6 times while 9471 passes voided), and the legality argument did not travel with it. FIX AT THE WRITER, ONE PLACE. `Req.truncate_prefix_to` leaves `Range(told, told)` -- zero rows at the prefix that now exists -- instead of `None`. This satisfies `_executed_extent`'s invariant `extend_range.start == len(prefix_indices)` by construction at the only place that can break it, and closes four producers in one cut instead of one branch per boot: the ring actuator (:2159), the seam refusal (scheduler.py:8916), `add_chunked_req`'s hybrid-SWA zero-budget return (schedule_policy.py:1396, which unlike the sgl-project#679 park at :1434-1436 returns without `set_extend_range`), and the sgl-project#791 clamp sites on a `NO_TOKEN` break. None of the four can now receive a null geometry, because none is produced. ONE EDIT WAS WRITTEN AND WITHDRAWN, recorded in the code rather than dropped. Making that hybrid-SWA branch write the park geometry -- so the two park branches say the same thing -- broke `test_prefill_adder.py::test_add_chunked_req_hybrid_swa_defers_when_swa_ below_page`, which pins "returned unchanged" via `set_extend_range.assert_not_called()`. With the writer fixed the branch is no longer a producer, so the edit would have been consistency rather than a fix, and it is not free: it would overwrite the PREVIOUS chunk's range on any path reaching this branch before that chunk is stashed. In production the stash runs earlier in the same pass, so the write would be value-neutral -- but that is an argument, not a measurement, and hybrid SWA is not a configuration this fork boots. Reverted, and the divergence between the two park branches is named at the site as open. sgl-project#958's ARGUMENT IS HONOURED, NOT REVERSED. Its "NONE, NOT A RECOMPUTED RANGE" paragraph refuses `Range(told, old_end)` because keeping the old end would INVENT a pass. `Range(told, told)` invents nothing, and it is not a new state: `_park_chunked_prefill_chunk` writes `Range(start, start)`, the sgl-project#679 park writes it, and `_executed_extent` declares zero-length ranges first-class. The offer still moves -- now WITHOUT the adder: `_executed_extent` reads (0, 0), so PP0 offers told=0, the value `reconcile_pp_admission_decision` admits unconditionally. `reset_for_retract`'s `None` is deliberately untouched: two disposal sites key off that sentinel (scheduler_pp_mixin.py:6061-6075, :7222-7236) and flipping it would have silenced them. The refused-geometry exit stays reachable from that producer and is now pinned by its own test. THE COMMIT'S OWN SAFETY NET WAS DOWNSTREAM OF THE CRASH. `PPScheduleRefused` / `require_executed_geometry` fired 0 times on metal while the unguarded dereference killed the process, because it iterates `can_run_list` and a resident continuation the adder did not add is never in it. It is not made reachable here; it is made unnecessary, and the structural reason is asserted rather than argued. SIBLING, same class, fixed here so this change does not widen it: `_park_chunked_prefill_chunk` handed back the `inflight_middle_chunks` increment whenever it got past its `end is None` gate rather than only when a chunk was actually prepared. Already reachable before this change via the sgl-project#679 park's `Range(prefix, prefix)`. The predicate is now the same `end > start` the KV release beside it already used -- one expression, not two. #962a: THE SEAM PROBE COULD NOT PROVE ITS HOOK RAN. The reachability probe `cutover_participants.py` registers for `latched_batch_flags` was emitted only `if any(_stale.values())`, so "ran and found nothing" and "never ran" were byte-identical -- the sgl-project#719 shape the registry's own docstring forbids. It is now unconditional and reports `reached=`, because W37-C already showed a bare zero is not enough (it logged `checked=0` eighteen times and was still blind). sgl-project#962 ITSELF IS REFUTED, no code change warranted. `batch_is_full` does not survive the tp_to_pp cutover: the hook is unconditional in `_cutover` with no early return before the completion log; it provably ran (`cutover complete: active stack` 6 and `[sgl-project#690] CUTOVER SUB-STEPS` 6 in both boots); it cleared nothing (`#861c cleared latched batch flag(s)` 0/0); PP0 admits 8 times (boot 1) / 4 times (boot 2) after the cutover before the first latched decline; and boot 1 alternates DECLINE/ADMIT eight times in one second while having MORE latched declines (5 vs 3) and NO livelock. #962b registered, not fixed: #888b's `parked_carrier_relief` re-derivation is on the post-flip path (scheduler.py:8587) but inert, because its gate reads `_parked_decode_verdict`, whose only writer (`_note_parked_carriers`, called at scheduler.py:7675) sits behind `not running_batch.is_empty()` and is unreachable at running=0 -- the state the relief exists for. Measured 0/0 against 8 latched declines. Needs its own danger-direction analysis. TESTS. `test_truncation_geometry_961.py`, 15 tests, RED FIRST at the pin (8 failed / 7 passed before the fix). The `:7010` reader is driven through the REAL `Scheduler.get_next_batch_to_run` on an uninitialised instance carrying the five attributes that line needs, so it reproduces the production AttributeError on the production line rather than on a copy of it; `_Req` borrows the real `Req.truncate_prefix_to`. Five CANFAIL mutants pin each reader to the invariant and pass before AND after. Two further readers are driven for real (`_compute_chunked_req_next_prompt_token`, `pp_chunked_local_match`) plus the real producer (`build_pp_admission_decision`). `test_offer_delivery_958.py`: its EXIT_3 test required the refusal this fix makes unproducible; corrected to assert the moved offer, and split so EXIT 3 stays pinned against the `reset_for_retract` producer that still reaches it. `test_latched_batch_flags_861c.py`: 3 tests for the #962a receipt, including a can-fail that a blind seam is not reported as an all-clear. DESK GATE, /spinning/htsglang-gpu/.venv, CVD="". BEFORE (frozen at the pin 78d030e): serial 895 passed / 2 failed, wide 3701, narrow 202. AFTER: see NOTE below. Failure set unchanged: the two pre-existing test_collective_family_siblings_610.py failures, untouched. sgl-project#954 (test_prefetch_progress_symmetry_580.py) is outside gate scope, as before. ruff: no finding on any of the 273 changed lines (all 64 pre-existing); new test file clean and ruff-formatted. codespell: new file clean; the one hit in phase_flip_draft_bootstrap.py:558 is pre-existing. No boot was run. /spinning/gpu-arb/TICKET_961_WINDOW.md carries the boot acceptance and is drivable from that file alone.

hnyls2002 added 2 commits July 20, 2024 21:11

use decouple style

6362d46

bump flashinfer version

7fa877f

hnyls2002 merged commit 69d1918 into main Jul 20, 2024

hnyls2002 deleted the decouple-kv branch July 20, 2024 21:16

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Decouple kv (sgl-project#679)

1282952

shiyu7 pushed a commit to shiyu7/sglang that referenced this pull request Aug 5, 2026

[performance] DeepSeek-V4-Pro performance enhancement (sgl-project#679)

cec65bb

Co-authored-by: shiyang814-cpu <shiyang.814@bytedance.com>

Navigation Menu

Uh oh!

Decouple kv - #679

Decouple kv#679
hnyls2002 merged 2 commits into
mainfrom
decouple-kv

hnyls2002 commented Jul 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

hnyls2002 commented Jul 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant