Support gpt-bigcode model class by hnyls2002 · Pull Request #681 · sgl-project/sglang

hnyls2002 · 2024-07-21T01:33:55Z

No description provided.

* 代码合入后触发性能看板wrokflow * 封装成函数 * 合入upstream/251031分支 * test

… nobody read the receipt TWO CLAIMS IN THE REPORT ARE WRONG, AND CORRECTING THEM CHANGES THE FIX. 1. "The 'no relief existed' line appears ZERO times." It appears THREE times, once per rank, at 01:46:10 -- the exact moment of the crash. The net DID engage, found the registry empty, said so, and let the raise proceed, which is precisely what it was built to do. Its diagnostic worked; the guarantee it points at (admission) is what failed. 2. "Pool at 0 fundable tokens / exhaustion." Not this crash. Usage was 0.85 and the failure's own message says: Try to allocate 512 tokens. Available full tokens: 66039 (available=273 + evictable=65766) 512 needed against 65,766 evictable. This was never exhaustion. WHAT ACTUALLY HAPPENED. All three ranks reported IDENTICAL numbers, so the pools agreed, so uniform_avail_floor was None, so the eviction trigger used the local value (273 < 512) and eviction DID run. It simply did not deliver. `evict` walks the LEAF FRONTIER: it pops evictable leaves and re-pushes a parent only once all its children are gone AND it is unlocked. `evictable_size_` counts unlocked tokens ANYWHERE in the tree. Tokens sitting behind a locked chain are counted and unreachable -- so the counter promises what the actuator cannot pay. That is the same shape as the sgl-project#662-F5 watermark finding: a number that is correct as a count and wrong as a capability. AND THE RECEIPT WAS THROWN AWAY. `evict` returns num_tokens_evicted; `evict_from_tree_cache` discarded it at both call sites. An eviction that freed ZERO was therefore indistinguishable from one that freed everything asked, and the allocation three lines later raised a message quoting 66,039 available tokens. Only disaggregation/decode.py reads that receipt today -- and it uses it for exactly this purpose ("after evicting X/Y tokens"), so the precedent existed. THE FIX, in three parts: * evict_from_tree_cache RETURNS what it actually freed. Callers can no longer proceed on the strength of a count the actuator did not honour. * the allocation error NAMES the under-delivery -- asked, freed, still-reported evictable, and why the two differ. The difference between a confusing message and a diagnosis. * RULE 3 OF DESIGN_679 IS CLOSED. The audit of the prefill admission path found THREE raise sites and only one covered: alloc_req_slots request slots (and mamba states) alloc_token_slots page_size == 1 <- the covered one alloc_paged_token_slots_extend page_size > 1 All three now ask the net before raising. alloc_req_slots is asked knowing a token-shaped relief cannot pay a request-slot shortage: the ask is what tells an operator the site was covered rather than forgotten. WHAT THIS DOES NOT FIX, stated plainly. The net still has no provider, so this crash would still end in a raise -- but with a message that diagnoses itself instead of blaming a full pool. And no admission accounting can prevent it: admission budgeted 66,039 fundable tokens and it was RIGHT about the count. The real repair is one of (a) make eviction able to reach tokens behind locked chains, or (b) make the budget count only leaf-reachable evictable tokens, and both are design decisions with their own uniformity arguments, not desk edits to slip into a crash fix. Filed rather than guessed. GROUP UNIFORMITY unchanged: the trigger's rank-uniform predicate (#616g) is untouched. Only the return value and the diagnosis are added, neither of which enters a branch. Tests: 10 new (30 in the file), red-first against 3374029 -- the commit that crashed -- where 5 fail outright and 3 subtests fail on the uncovered raise sites. 1013 pass across the suites, zero failures. NOT BOOTED. Desk-only; READY-FOR-BOOT.

…a tombstone leaf THE DIAGNOSIS IN 7752dc8 IS WRONG, AND CORRECTING IT CHANGES THE FIX. It said tokens behind a LOCKED chain are counted but unreachable, so the counter promises what the actuator cannot pay. The second clause is right. The first cannot happen: `inc_lock_ref` / `dec_lock_ref` walk from a node to the ROOT, and `_split_node` copies `full_lock_ref` onto the new upper half. So `full_lock_ref(parent) >= full_lock_ref(child)` holds on every edge at every moment. An unlocked node therefore has no locked descendant, its whole subtree is unlocked, and the peel always reaches it. Measured on the 01:46:10 tree itself, reconstructed from the dump all three ranks printed: of 65766 evictable tokens, 65254 sat in fully-unlocked subtrees, and the 512-token remainder is an artifact of one mis-ordered line in the interleaved three-rank output. The locked-chain term is ZERO. Subtracting it from the admission budget -- the repair this chain was pointed at -- would have moved the number the scheduler admits against by nothing, and the crash would have reproduced unchanged. That is why it is not what this commit builds. The reconstruction is not taken on trust: it is checked against two totals the process printed independently, `#full_tokens: 140683` and `full_evictable_size_=65766`, and it matches both to the token. THE GAP IS AT THE OTHER END OF THE FRONTIER. `evict_full` SELECTS with `get_leaf_lru_no_lock` -- unlocked and childless -- but `_evict_leaf_node` CONSUMES only nodes with a mamba value, and asserted when one was missing. An unlocked mamba TOMBSTONE leaf satisfies the selector and violates the consumer, and the cache produces that state itself: `_iteratively_delete_tombstone_leaf` breaks on `node.parent.full_lock_ref > 0`. A tombstone that loses its last child while a request holds it survives as a LOCKED tombstone leaf. When that request finishes nothing revisits it, so it becomes unlocked, childless, counted in `full_evictable_size_`, and first in line at the frontier. The 01:46 tree held exactly one -- node 5937, fr=0, mv=None, childless, in the full LRU list -- beside a single payable leaf, 5959. Replaying that dumped tree through the deployed code selects 5937 and dies on `AssertionError: leaf node mamba value is not None`. Only 6 of its 290 nodes held a mamba value at all; 128 of the 130 unlocked nodes were tombstones. This is a crowded state, not a freak one. THE REPAIR IS ON THE ACTUATOR, NOT THE COUNTER. Freeing an unlocked tombstone leaf is not a new capability: it is the same deletion `_iteratively_delete_tombstone_leaf` already performs one step earlier, taken now that the lock which deferred it is gone. Both routes now go through one `_free_tombstone_leaf`, so `full_evictable_size_` and the LRU list stay in step by construction rather than by two copies of the same five lines. Raising the actuator rather than lowering the counter is the stronger closure of "a counter must never promise what the actuator cannot pay": the promise becomes TRUE instead of becoming smaller, and every consumer -- admission budget, the sgl-project#679 park guard, the in-flight sgl-project#677 drain gate -- is repaired at once without any of them learning a second quantity. DELIBERATELY NOT EXTENDED PAST A LOCK. A LOCKED tombstone leaf is still refused and still uncounted; reaching behind a live reference is a different repair and stays filed. `test_a_locked_tombstone_leaf_is_still_refused` is the mutation proof that the new branch is gated on the lock and not on the tombstone alone. GROUP-UNIFORMITY needs no new channel. The branch is a pure function of replicated tree state -- the tree is a replica, `full_lock_ref` and `mamba_value` are replicated -- so every rank takes it on the same iteration. The existing `uniform_avail_floor` still decides WHETHER to evict; this only changes what the peel does once asked. No collective added. HONEST CAVEAT, ONE. Deleting a tombstone leaf cascades up through its now-childless tombstone ancestors, so a 512-token request can free far more: 54502 tokens on the replayed production tree. That overshoot is the pre-existing semantics of `_iteratively_delete_tombstone_leaf`, which is unbounded on the ordinary path too and exists to restore the "no tombstone leaves" invariant. Bounding it would leave the invariant broken, so it is left as it is and named here rather than discovered later. THE sgl-project#681 BACKSTOP STAYS, WITH ITS MECHANISM CORRECTED. The shortfall note also had a defect of its own: it read `tree_cache.evictable_size()`, which RAISES NotImplementedError on MambaRadixCache and SWARadixCache -- the very classes it was written for -- so on the crashing boot it would have reported -1. It now asks for `full_evictable_size()` first. Its text no longer asserts the falsified locked-chain mechanism and instead says what firing means now: a REGRESSION SIGNAL, a new class of node being counted that the peel cannot consume. WHAT IS STILL NOT EXPLAINED, SAID PLAINLY. The 01:46 process died with RuntimeError, not with the AssertionError the replay produces, and its own numbers say the tree was untouched by the eviction that ran immediately before (available 273 and evictable 65766 both unchanged at the raise). Every branch that could skip that eviction was checked and excluded: `is_chunk_cache` is False, `disable_radix_cache` is False, and `uniform_avail_floor` is None on this boot because tp_size=1 takes the single-rank early return. So a third mechanism remains unidentified. The receipt added in 7752dc8 is the instrument that will name it on the next boot; this commit removes one guaranteed way for the next eviction on that tree to kill the group, and does not claim to be the whole story. TESTS. `test_evictable_reachability_681`, 9 cases, CPU-only, no GPU: - RED FIRST on 7752dc8: `test_the_frontier_pays_the_tombstone_leaf_...` and `test_the_counter_never_over_promises` both fail with the production assert, `leaf node mamba value is not None`. Green after. - The ancestor-closure proof runs on MambaRadixCache and on the base RadixCache, across a lock, a deeper lock, and a split under a lock, with `test_the_detector_can_fail` hand-building the shape sgl-project#681 assumed so the three green assertions cannot be satisfied by a detector that always returns empty. - `test_the_state_the_crash_tree_was_in_is_reachable` reaches node 5937's exact signature through public transitions only. - `test_the_ordinary_path_is_untouched` pins the no-tombstone peel. Suites: 79 passed across the 681 file, both sgl-project#679 files and test_mamba_lock_ref_pairing_581.

… what we remembered THE ROOT QUESTION IS ANSWERED, AND MY OWN CANDIDATE WAS WRONG. I had named the flip's "released 1410.0 MiB of weights-arena tail" as the reason the reservation ends up below `_rows_at_boot`. It is refuted twice over: that is the WEIGHTS arena, not the KV pool, and the KV reservation cannot move at all. memory_pool.py:2458 reserved_num_tokens=self.size # at construction kv_vmm_backing.py:979 self._reserved_num_tokens = int(...) # assigned ONCE The reservation is pinned to the pool's size at the moment the arena is built and never assigned again. `size` is NOT immutable -- the sgl-project#330 dial writes it on every step, which sgl-project#662-F4 already noted one layer up. So a grow target derived from a remembered or configured row count can sit above a ceiling that never moves, and `_check_final` refuses it identically, forever. MEASURED, AND UNCONFOUNDED: 59 times between 02:15:24 and 02:35:26 on 2026-08-16, a steady 3 per minute, once per rank per flip leg -- `recovery to 270646 rows failed: ... reserved=190596`, and the same shape on the other two ranks (180428/108912, 179466/136140). That window opens before any test-harness CUDA activity on the rig, so unlike the free-column readings from 02:29 onward it is not confounded. WHY IT IS BIGGER THAN THE LINE. Recovery is what LIFTS the backing cap. 59 refusals meant the cap never lifted, the pool stayed shrunk, and every later `free_up_to` found the backing already at its target and honestly claimed 0 MiB -- which the shrink path then reported as an exhausted ARENA. One unsatisfiable number, and the corridor guard's only rung above `allocator-cache` was dead for the whole boot while its diagnostic pointed somewhere else. That is the shape sgl-project#683 was opened on. THE REPAIR IS THE SAME CORRECTION AS sgl-project#681 AND sgl-project#682: validate against what the ACTUATOR can pay, not against the count that proposed it. sgl-project#681 was a token count against a leaf frontier, sgl-project#682 a guard ceiling against the bound the scheduler actually holds, and this is a grow target against an immutable reservation. So the clamp is deliberately NOT conditional on knowing why the remembered number went stale -- it asks the bound. CLAMP *AND* RE-DERIVE, because the clamp alone would only convert a loud failure into a quiet one: `_rows_at_boot` would still name an impossible level and every later recovery would re-clamp to the same place while believing it had further to go. Correcting it lets the existing "fully recovered" branch fire, which clears the remembered rows AND retires the exhaustion marker -- the latch that kept the rung off. RANK-LOCAL, EXPLICITLY, as the brief asks. A reservation is one card's VA span; under uneven TP the ranks hold different ones -- 190596 / 136140 / 108912 on this boot -- so there is no group quantity here to agree on. `recover` takes no collective, and this commit adds none. The module's collective, the sgl-project#656 C22 cap agreement, is on the SHRINK target and is untouched. The new accessor is also NOT `_reservation_rows` (the allocator's id space, which does feed `exposed_rows` and that agreement); the two are cross-referenced in code so a later reader cannot conflate them. SAFE DESK-SIDE, AND THE JUDGEMENT IS ASKED FOR, SO HERE IT IS. Two properties make this shippable without a GPU window: * the clamp fires ONLY where `rows > ceiling`, which is exactly the path that currently fails 100% of the time. On any boot where recovery works today the branch is inert, so there is no working behaviour for it to change. * it runs AFTER the corridor-affordability bound, so when both bite the target is the smaller of the two and the clamp can only LOWER it. Raising it would commit pages the corridor law had already refused -- the failure that drove rank 1 to 6 MiB free and OOMed inside relief. Pinned by `test_the_clamp_can_only_lower_the_target_never_raise_it`. A pool that exposes no reservation keeps its previous behaviour exactly; 0 is read as "no arena", never as a ceiling of zero, which would be a shrink wearing a grow's name. TESTS, red-first. The acceptance pin committed with the verification -- `test_recovery_is_refused_forever_because_nothing_clamps_it` -- was inverted to the post-fix expectation FIRST and failed, together with the re-derivation pin; both pass after. It keeps its name: it asserted the defect before the clamp and asserts the repair after it, which is what an acceptance pin is for. Seven cases in all, four of which exist so the fix cannot pass by being broken everywhere: the control (a reservation above the boot rows recovers normally), the affordability bound still deferring untouched, the clamp/affordability interaction, and the two backward-compatibility contracts. Sweep: 2433 passed across unit/managers + unit/mem_ledger + the 681/682 files. Four failures in that run are pre-existing and unrelated -- same four, same messages, on the untouched tree (`BudgetHarness` and `_Sched` stubs missing attributes in test_collective_family_siblings_610 and test_first_chunk_dynamic_chunking); neither file references anything this commit touches.

…es the pool forever FOURTH LATCH OF THE NIGHT, and the same cure as the other three: sgl-project#681's eviction count that could not be paid, sgl-project#682's guard ceiling the scheduler never held, sgl-project#684's `_exhausted_at_rows` process-lifetime marker. Each was a number that could only ratchet one way. WHAT IT COST. `corridor_shortfall_bytes` is added straight to the arming floor's load margin -- `(DEFAULT_MARGIN_MIB << 20) + measured` -- and the arming floor is the binding constraint on two of three ranks. On 2026-08-16 the rank-0 record carried 1004 MiB of it while every record written the day before carried 0, and the boot reading it logged NO breach of its own: it was inherited. The event it descends from is almost certainly 02:36:30 on that exact card, where a test harness belonging to this strand held 4.29 GiB and drove free to 76 MiB. A few seconds of intrusion, taxing every subsequent boot. THE OLD SEMANTICS WERE HALF RIGHT, AND THAT HALF IS KEPT. `record_corridor_shortfall` documents itself as "A MONOTONIC MAXIMUM, deliberately -- a shallower breach later does not mean the deeper one cannot recur; the pool must be sized for the worst instant that has ever been seen". Correct WITHIN an observation. Wrong ACROSS boots that never see it again, because "ever" had no end and nothing could retire a number nobody could reproduce. So: monotonic maximum while it is being OBSERVED, geometric decay across boots that observe nothing. A breach that recurs is re-observed and re-raised to its worst on the spot. One that cannot be reproduced is halved by each flip boot that measures its seam without seeing it, and written off to exactly 0 below `SHORTFALL_FORGET_BYTES` so the decay terminates instead of leaving a tail that still moves the floor. 1004 MiB is gone in seven clean boots. "OBSERVED BY THIS PROCESS" IS THE DISCRIMINATOR, and it is a pid rather than a timestamp because both writers live in the same process: the runtime's corridor audit stamps the record mid-run, and `write_seam_reserve` rewrites it at the end of the same boot's flip measurement. Same pid means this boot saw it and the value stands; a different pid means it was inherited, and a boot that completed a seam measurement without its audit firing is evidence against it. Evidence is what retires it. RANK-LOCAL. The record is per (configuration, rank) and the shortfall is one card's own measurement -- 1004 / 0 / 0 on this boot, legitimately different. No collective reads or writes it and this change adds none. TESTS, red-first: 7 cases. The three decay cases failed before and pass after; the four that pin the half worth keeping -- a breach this process observed is preserved, a deeper one still raises, a shallower one does not lower it -- passed from the start, so the fix cannot have been "delete the term". One case drives the full production scenario: a 1004 MiB one-off decaying to zero while the load margin returns to its default, and one that proves a breach observed on every boot is never decayed away.

…d the two open decisions The handoff artifact for the merge queue. Nine commits on fix/602-fill-side, what each is, and the two things that are NOT decided: * R' semantics for the cold seam. 5301b94 derives and announces the cold per-token slope but the reserve stays inactive, so cold boots still size floor-only. `SeamReserve.active` needs `id_space > 0` -- a measurement anchor a derivation does not have -- and the anchor-free `solve_pool_tokens` has no live caller, so the budget it solves against is boot-path design. F4-r4's call. * sgl-project#602 metal arm, deferred by the operator at +3.6 %. Also records the retraction in full: the earlier 31,16,17 / +227095 (+36.3 %) recommendation was an artifact of bench-priced weights and is withdrawn in favour of 29,19,16 / +17235 (+3.6 %), stage 2 keeping 16 layers. MERGE-QUEUE EVIDENCE, asked for and answered NO: cherry-picking 0274bed (sgl-project#681) into a scratch copy of this tree does NOT green the two PrefillAdmissionBudgetTest reds. They fail identically with `AttributeError: 'BudgetHarness' object has no attribute '_local_mamba_avail'`. That is a STALE TEST HARNESS, not a product defect -- `Scheduler._local_mamba_avail` is real (scheduler.py:4257, called from _update_uniform_pool_budget at :4067) and the harness stand-in never binds it. Unrelated to sgl-project#681's fundable_extend_floor. Scratch dropped. Docs only; no code touched.

…l; the new-request gate never did sgl-project#679 taught `add_chunked_req` not to schedule a chunk the pool cannot fund, and it made that decision GROUP-UNIFORMLY: `fundable_extend_tokens` reads the published group MIN through `uniform_avail_for_evict`. Its sibling gate -- the one every FRESH request passes -- was left reading this rank's own pool: PrefillAdder.rem_total_tokens available_size() + evictable_size() - offset - dcp_avail_deficit That is the same over-promise one door further along, and it fails twice over: * A rank roomier than the binding rank admits work the GROUP cannot fund, and the batch dies where it is allocated -- `alloc_for_extend`, 2026-08-16 01:46:10, all three ranks together. * It is a rank-local BRANCH upstream of a collective. Two ranks can build different batches and enter the next extend with different token axes, which is a HANG rather than a stall -- the family sgl-project#583/sgl-project#603/#616g/sgl-project#639 was paid for. `fundable_extend_tokens`' own docstring names the first half. It was written for the chunked gate and never wired to this one; `fundable_extend_tokens` appeared nowhere in scheduler.py, which the red-first run confirms. WHAT THIS IS NOT. It is not the cause of the 01:46 traceback. That was e778276 -- an unlocked mamba tombstone leaf the eviction frontier selected and `_evict_leaf_node` could not consume, so eviction under-delivered while the counter honestly promised 65766 evictable. THE COUNTER AND THE ACTUATOR NOW AGREE; this commit is about the second consumer of that counter, which reads a rank-local copy of it and can over-promise even when it is honest. A CEILING, NOT A SUBSTITUTE. `rem_total_tokens` also subtracts reservations the floor knows nothing about (the running batch's hold, mamba gap, page overhead), so replacing the local term would talk a rank that is ITSELF short back UP. The budget is the MIN of the two. Both spend down the same `rem_total_token_offset` -- a floor consulted per request without that shared accounting is not a bound at all, since every request in the round would compare itself against the same untouched pool. `dcp_avail_deficit` is deliberately NOT subtracted from the floor term: the floor is already the group MIN, and the deficit is the other mechanism for pinning a local number to the binding rank. WHY THE MIN RESOLVES TO THE FLOOR IN PRODUCTION, and it is a test, not a claim: `floor = uniform_avail + evictable` and `local = local_avail + evictable - offset` with `uniform_avail <= local_avail` by definition of a MIN, so `floor - offset <= local` on every rank. A rank BELOW the floor is not a state the reduce can produce; the clamp's behaviour there is pinned separately because refusing to be talked up by a stale floor is the wanted behaviour. THE CEILING MUST NOT BE ABLE TO WEDGE THE INSTANCE, and this is why the wiring goes through `published_fundable_floor` rather than calling `fundable_extend_tokens` directly. That helper returns 0 for BOTH "the pool is empty" and "the pool could not be read". The chunked gate can live with the ambiguity -- 0 parks a chunk and the next round retries, a self-clearing state. As a budget ceiling a mis-read 0 admits NOTHING, for every request, on every subsequent round: a harder wedge than the crash it prevents. So the ceiling is applied only where a floor was actually PUBLISHED, which is the one state (uneven rank pools) it exists for. A published floor of genuine zero still binds -- the distinction is "was a floor published", not "is it non-zero", or the ticket's own crash state would be exempt from its own fix. DEFAULT PATH UNTOUCHED, BY CONSTRUCTION AND BY ASSERTION. With no floor published -- single rank, or pools that agree -- `fundable_extend_floor` is None and `rem_total_tokens` returns exactly what it returned before. TESTS. RED-FIRST against 7936bc4: 12 of the 18 fail there, including the 01:46 shape itself (a rank holding 100000 tokens admitting against a group that holds none) and all three wiring pins. The guard against the wedge was falsified separately by removing it -- 2 of its 4 cases go red -- because an instrument that cannot fail is not evidence. Covered: the cap; the floor spent down by admissions; group agreement across ranks with different local pools; the floor binding before the local term whenever the reduce's invariant holds; the local term still binding when it is the smaller one; inertness with no floor and with a roomy floor; the published-vs-readable distinction including a genuine zero; and three pins that this stays wired (the scheduler constructs with a floor, it comes through the guarded helper, and that helper still delegates to the group-uniform one). NO REGRESSIONS, measured against the same subset on the unpatched tree: 944 failed / 2776 passed / 725 skipped before, 944 failed / 2794 passed / 725 skipped after -- identical failures (all CUDA-gated collection, hermetic run with CUDA_VISIBLE_DEVICES=""), +18 = exactly the new cases. NOT BOOTED. Desk-only; READY-FOR-BOOT. The metal arm that would exercise it is a multi-rank boot with uneven pools under max fill -- the arm that produced the 01:46 specimen -- watching for the admission ceiling binding without a wedge.

…-project#681 ceiling is bound CAUSED BY MY OWN FIX, WHICH IS THE POINT. Cherry-picking sgl-project#681 onto this branch added `fundable_extend_floor` to `PrefillAdder`, and `test_collective_family_siblings_610` builds its adder with `PrefillAdder.__new__` and hand-sets every field the predicate reads. The new one was absent, so both admission cases raised `AttributeError: 'PrefillAdder' object has no attribute 'fundable_extend_floor'` -- the same stub-drift shape as the three that came before it, this time introduced by a fix rather than by a reduce. Bound in the stub, and the stub extracted into `_budget_state_stub` so there is one place to bind rather than a literal inside a loop. GUARDED, SCOPED TO WHAT THE PATH ACTUALLY READS. The guard checks the stub against `rem_total_tokens` and NOT against the whole `budget_state` surface: the stub pins `is_hybrid_swa` / `is_all_swa` False, so the SWA members (`rem_swa_token_offset`, `rem_swa_tokens`) are never reached and demanding them would fail the guard on state this harness legitimately does not need -- the same over-reach the sibling guards are scoped to avoid. `rem_total_tokens` is the term the uneven-DCP budget is about and where the sgl-project#681 ceiling landed. CAN-FAIL, against the real drift: removing the binding makes the guard report `['fundable_extend_floor']` in its own message, and it goes red BESIDE the two admission cases rather than after them. test/registered/unit/managers: 2084 passed, 0 failed -- green again with the sgl-project#681 fix on the branch. Test file only; the production behaviour is correct and it was the stub that was incomplete.

…ad only STAGED VERIFIED INDEPENDENTLY FROM SOURCE, then against F4-r4's chain -- both arrive at the same place, which is why this ships rather than another hypothesis. THE SPECIMEN (2026-08-16 13:58:37, sgl-project#693, all three ranks byte-identically): RuntimeError: Out of memory. Try to allocate 512 tokens. Available full tokens: 167743 (full_available_size=392 + full_evictable_size=167351) and NO `EVICTION UNDER-DELIVERED` line. That absence is the evidence, not a hole in it: `_eviction_shortfall_note` returns "" only when `evicted >= asked`, so eviction reported delivering its full 512 while the pool's free count stayed at 392. THE CHAIN, every hop read: 1. `_evict_leaf_node` (mamba_radix_cache.py:915-916) calls `token_to_kv_pool_allocator.free(x.value)` and counts `len(x.value)`. Same for the tombstone route `_free_tombstone_leaf` (:1705-1706). 2. `TokenToKVPoolAllocator.free` (allocator/token.py:67-80) applies pages only `if self.is_not_in_free_group`; otherwise `self.free_group.append(idx)`. 3. `available_size` (:52-54) is `len(free_pages) + len(release_pages)` -- the staged list is in NEITHER -- and `alloc` (:56-61) compares against `len(free_pages)` alone. 4. `free_group_begin` is called from inside the event loop (batch_result_processor.py:92 and :741). An eviction landing in that window frees into the staging list. So the receipt is TRUE about the tree and FALSE about the pool: the nodes are gone and their tokens counted, and the pages sit in `allocator.free_group` waiting for `free_group_end()`. NOT A POOL-TIER MISMATCH, which was the other axis: `release_pages` IS counted by `available_size`, so a tier split would still show the tokens. And not a race -- the group opens at a fixed point in a replicated event loop, which is exactly why three independent processes printed identical counters. THE FIX, AND WHY THIS ONE OF THE THREE. (a) flush-and-close before an eviction-driven alloc -- rejected: it ends a batching window the caller still owns, silently unbatching its remaining frees and turning its own `free_group_end` into a no-op it never asked for. (b) make `num_tokens_evicted` count only non-deferred frees -- rejected as THE fix: it makes the counter honest and leaves the crash, since the caller's last word is still the sgl-project#679 hard raise. It repairs the message, not the state. (c) SHIPPED: `flush_free_group()` applies the staged frees WITHOUT closing the group. Safety is the whole question and it is answerable from the source: `free_group_end` is PURE BATCHING -- `self.free(torch.cat(self.free_group))`, one concat and one `_notify_free` (allocator/base.py:203-206). It defers nothing for correctness: no in-flight reference, no graph-replay barrier, and the same iteration applies these very pages a few lines later regardless. Flushing moves that application earlier; it makes nothing reachable that was not already about to be. The flag is restored so the caller's window and its later end-call are untouched, and the staged list is consumed so that end-call cannot double-free. Called from `alloc_token_slots` BEFORE the relief ladder, because it is not relief: the rungs below spend host bandwidth or a victim's decode progress, this spends nothing. Cold path only -- it runs after an allocation has already failed, so a healthy alloc pays nothing at all. `backup_state` is re-taken after a flush; no caller passes it on this path today, and re-taking keeps that a fact about the callers rather than a dependency of this fix. TESTS. Red-first: 5 of 8 fail, and the three that matter fail with the specimen's own `RuntimeError: Out of memory` -- sgl-project#693 reproduced hermetically. The stand-in allocator mirrors token.py's free/alloc/group semantics and binds the PRODUCTION `flush_free_group` onto itself rather than reimplementing it, so the method under test is the real one. Covered: the allocation succeeding once the staged frees are applied; the group still OPEN afterwards; the caller's `free_group_end` still safe (no double free); no group open and nothing staged both byte-identical; a genuinely empty pool still raising (fail-loud preserved); and the base class carrying the method so every subclass inherits it. NO REGRESSIONS, measured both sides on the same subset: mem_cache 940 failed / 757 passed before, 940 failed / 765 passed after -- identical failures (all CUDA-gated collection), +8 = exactly the new cases. managers 2084 passed, 0 failed. Hermetic (CUDA_VISIBLE_DEVICES=""). NO DEPLOY. WHAT THIS DOES NOT CLAIM. It removes the counted-but-unpayable state. It does not make the pool larger: a genuinely exhausted pool still raises, and the admission-side fixes (sgl-project#679 park, sgl-project#681 group-uniform ceiling) remain the layer that should prevent reaching here at all.

…l; the new-request gate never did sgl-project#679 taught `add_chunked_req` not to schedule a chunk the pool cannot fund, and it made that decision GROUP-UNIFORMLY: `fundable_extend_tokens` reads the published group MIN through `uniform_avail_for_evict`. Its sibling gate -- the one every FRESH request passes -- was left reading this rank's own pool: PrefillAdder.rem_total_tokens available_size() + evictable_size() - offset - dcp_avail_deficit That is the same over-promise one door further along, and it fails twice over: * A rank roomier than the binding rank admits work the GROUP cannot fund, and the batch dies where it is allocated -- `alloc_for_extend`, 2026-08-16 01:46:10, all three ranks together. * It is a rank-local BRANCH upstream of a collective. Two ranks can build different batches and enter the next extend with different token axes, which is a HANG rather than a stall -- the family sgl-project#583/sgl-project#603/#616g/sgl-project#639 was paid for. `fundable_extend_tokens`' own docstring names the first half. It was written for the chunked gate and never wired to this one; `fundable_extend_tokens` appeared nowhere in scheduler.py, which the red-first run confirms. WHAT THIS IS NOT. It is not the cause of the 01:46 traceback. That was e778276 -- an unlocked mamba tombstone leaf the eviction frontier selected and `_evict_leaf_node` could not consume, so eviction under-delivered while the counter honestly promised 65766 evictable. THE COUNTER AND THE ACTUATOR NOW AGREE; this commit is about the second consumer of that counter, which reads a rank-local copy of it and can over-promise even when it is honest. A CEILING, NOT A SUBSTITUTE. `rem_total_tokens` also subtracts reservations the floor knows nothing about (the running batch's hold, mamba gap, page overhead), so replacing the local term would talk a rank that is ITSELF short back UP. The budget is the MIN of the two. Both spend down the same `rem_total_token_offset` -- a floor consulted per request without that shared accounting is not a bound at all, since every request in the round would compare itself against the same untouched pool. `dcp_avail_deficit` is deliberately NOT subtracted from the floor term: the floor is already the group MIN, and the deficit is the other mechanism for pinning a local number to the binding rank. WHY THE MIN RESOLVES TO THE FLOOR IN PRODUCTION, and it is a test, not a claim: `floor = uniform_avail + evictable` and `local = local_avail + evictable - offset` with `uniform_avail <= local_avail` by definition of a MIN, so `floor - offset <= local` on every rank. A rank BELOW the floor is not a state the reduce can produce; the clamp's behaviour there is pinned separately because refusing to be talked up by a stale floor is the wanted behaviour. THE CEILING MUST NOT BE ABLE TO WEDGE THE INSTANCE, and this is why the wiring goes through `published_fundable_floor` rather than calling `fundable_extend_tokens` directly. That helper returns 0 for BOTH "the pool is empty" and "the pool could not be read". The chunked gate can live with the ambiguity -- 0 parks a chunk and the next round retries, a self-clearing state. As a budget ceiling a mis-read 0 admits NOTHING, for every request, on every subsequent round: a harder wedge than the crash it prevents. So the ceiling is applied only where a floor was actually PUBLISHED, which is the one state (uneven rank pools) it exists for. A published floor of genuine zero still binds -- the distinction is "was a floor published", not "is it non-zero", or the ticket's own crash state would be exempt from its own fix. DEFAULT PATH UNTOUCHED, BY CONSTRUCTION AND BY ASSERTION. With no floor published -- single rank, or pools that agree -- `fundable_extend_floor` is None and `rem_total_tokens` returns exactly what it returned before. TESTS. RED-FIRST against 7936bc4: 12 of the 18 fail there, including the 01:46 shape itself (a rank holding 100000 tokens admitting against a group that holds none) and all three wiring pins. The guard against the wedge was falsified separately by removing it -- 2 of its 4 cases go red -- because an instrument that cannot fail is not evidence. Covered: the cap; the floor spent down by admissions; group agreement across ranks with different local pools; the floor binding before the local term whenever the reduce's invariant holds; the local term still binding when it is the smaller one; inertness with no floor and with a roomy floor; the published-vs-readable distinction including a genuine zero; and three pins that this stays wired (the scheduler constructs with a floor, it comes through the guarded helper, and that helper still delegates to the group-uniform one). NO REGRESSIONS, measured against the same subset on the unpatched tree: 944 failed / 2776 passed / 725 skipped before, 944 failed / 2794 passed / 725 skipped after -- identical failures (all CUDA-gated collection, hermetic run with CUDA_VISIBLE_DEVICES=""), +18 = exactly the new cases. NOT BOOTED. Desk-only; READY-FOR-BOOT. The metal arm that would exercise it is a multi-rank boot with uneven pools under max fill -- the arm that produced the 01:46 specimen -- watching for the admission ceiling binding without a wedge.

…ad only STAGED VERIFIED INDEPENDENTLY FROM SOURCE, then against F4-r4's chain -- both arrive at the same place, which is why this ships rather than another hypothesis. THE SPECIMEN (2026-08-16 13:58:37, sgl-project#693, all three ranks byte-identically): RuntimeError: Out of memory. Try to allocate 512 tokens. Available full tokens: 167743 (full_available_size=392 + full_evictable_size=167351) and NO `EVICTION UNDER-DELIVERED` line. That absence is the evidence, not a hole in it: `_eviction_shortfall_note` returns "" only when `evicted >= asked`, so eviction reported delivering its full 512 while the pool's free count stayed at 392. THE CHAIN, every hop read: 1. `_evict_leaf_node` (mamba_radix_cache.py:915-916) calls `token_to_kv_pool_allocator.free(x.value)` and counts `len(x.value)`. Same for the tombstone route `_free_tombstone_leaf` (:1705-1706). 2. `TokenToKVPoolAllocator.free` (allocator/token.py:67-80) applies pages only `if self.is_not_in_free_group`; otherwise `self.free_group.append(idx)`. 3. `available_size` (:52-54) is `len(free_pages) + len(release_pages)` -- the staged list is in NEITHER -- and `alloc` (:56-61) compares against `len(free_pages)` alone. 4. `free_group_begin` is called from inside the event loop (batch_result_processor.py:92 and :741). An eviction landing in that window frees into the staging list. So the receipt is TRUE about the tree and FALSE about the pool: the nodes are gone and their tokens counted, and the pages sit in `allocator.free_group` waiting for `free_group_end()`. NOT A POOL-TIER MISMATCH, which was the other axis: `release_pages` IS counted by `available_size`, so a tier split would still show the tokens. And not a race -- the group opens at a fixed point in a replicated event loop, which is exactly why three independent processes printed identical counters. THE FIX, AND WHY THIS ONE OF THE THREE. (a) flush-and-close before an eviction-driven alloc -- rejected: it ends a batching window the caller still owns, silently unbatching its remaining frees and turning its own `free_group_end` into a no-op it never asked for. (b) make `num_tokens_evicted` count only non-deferred frees -- rejected as THE fix: it makes the counter honest and leaves the crash, since the caller's last word is still the sgl-project#679 hard raise. It repairs the message, not the state. (c) SHIPPED: `flush_free_group()` applies the staged frees WITHOUT closing the group. Safety is the whole question and it is answerable from the source: `free_group_end` is PURE BATCHING -- `self.free(torch.cat(self.free_group))`, one concat and one `_notify_free` (allocator/base.py:203-206). It defers nothing for correctness: no in-flight reference, no graph-replay barrier, and the same iteration applies these very pages a few lines later regardless. Flushing moves that application earlier; it makes nothing reachable that was not already about to be. The flag is restored so the caller's window and its later end-call are untouched, and the staged list is consumed so that end-call cannot double-free. Called from `alloc_token_slots` BEFORE the relief ladder, because it is not relief: the rungs below spend host bandwidth or a victim's decode progress, this spends nothing. Cold path only -- it runs after an allocation has already failed, so a healthy alloc pays nothing at all. `backup_state` is re-taken after a flush; no caller passes it on this path today, and re-taking keeps that a fact about the callers rather than a dependency of this fix. TESTS. Red-first: 5 of 8 fail, and the three that matter fail with the specimen's own `RuntimeError: Out of memory` -- sgl-project#693 reproduced hermetically. The stand-in allocator mirrors token.py's free/alloc/group semantics and binds the PRODUCTION `flush_free_group` onto itself rather than reimplementing it, so the method under test is the real one. Covered: the allocation succeeding once the staged frees are applied; the group still OPEN afterwards; the caller's `free_group_end` still safe (no double free); no group open and nothing staged both byte-identical; a genuinely empty pool still raising (fail-loud preserved); and the base class carrying the method so every subclass inherits it. NO REGRESSIONS, measured both sides on the same subset: mem_cache 940 failed / 757 passed before, 940 failed / 765 passed after -- identical failures (all CUDA-gated collection), +8 = exactly the new cases. managers 2084 passed, 0 failed. Hermetic (CUDA_VISIBLE_DEVICES=""). NO DEPLOY. WHAT THIS DOES NOT CLAIM. It removes the counted-but-unpayable state. It does not make the pool larger: a genuinely exhausted pool still raises, and the admission-side fixes (sgl-project#679 park, sgl-project#681 group-uniform ceiling) remain the layer that should prevent reaching here at all.

…s standing in front of 54 MINUTES OF SILENT OUTAGE, health 200 throughout. From 16:23:10, 325 consecutive lines of: BOTH BLOCKED: nothing can run in the pp layout and the target cannot admit either (0 req resident, 10495392 tok pending) Last real batch 16:23:11: full token usage 1.00, #running-req 0, mamba 0.17. The whole KV pool was radix cache with ZERO resident requests -- every row unlocked and evictable -- while 10.5M tokens queued and three GPUs sat at 0%. ONE SWALLOWED EXCEPTION. _post_evict_rows asked tree_cache.evictable_size(). MambaRadixCache does not return a number from that method; it raises NotImplementedError and says "use full_evictable_size() and mamba_evictable_size() instead". The probe caught it and used 0, so on the class this rig runs it returned `available` ALONE -- the exact error its own docstring warns about, committed three lines below the warning. At usage 1.00 that reads ~0, so every admissibility question answered no: pp could not admit, tp had nothing resident to decode, and sgl-project#688's BOTH BLOCKED branch declined. That branch returns BEFORE alloc_token_slots -- so the allocator was never reached, eviction never ran, and the unlocked cache was never freed. The receipt called it "an evict trigger" while no evict could occur, which is the same counter-vs-actuator shape as sgl-project#681/sgl-project#694 in a third place: a message naming an action nothing performs. CONFIRMED BY ABSENCE in the specimen (WEDGE-2026-08-16T1623Z.txt): zero RADIX SHAPE, zero "Out of memory", zero EVICTION UNDER-DELIVERED. The allocation path was never entered. py-spy shows the ranks spinning the event loop building nothing. THE TRAP IS ALREADY DOCUMENTED IN THIS TREE, at common.py:411-425, for these same two classes -- and I read that comment the same day while diagnosing sgl-project#694 and did not apply it here. The resolution order is now COPIED from there rather than re-derived, because two spellings of one rule is how this comes back. WHY NOW. The bug shipped with sgl-project#688's admissibility simulation and needed usage == 1.00 to bite. sgl-project#696's floor repair shrank the pool by 39,504 tokens, so full occupancy arrived sooner and the wedge began 12 minutes after that boot. sgl-project#696 EXPOSED this; it did not cause it. THE SHAPE TO LEARN: a swallowed exception that yields a PLAUSIBLE value. Zero is a legal row count, so nothing downstream could tell "the cache holds nothing" from "the cache was never asked". Every accessor is now tried in turn and only a genuine absence of all of them yields zero. Health being 200 for the entire outage is the second lesson: the endpoint answers while the scheduler builds no batch. It is not a liveness signal. TESTS: test_post_evict_rows_698.py 5 passed (red-first: pre-fix it fails "0 not greater than or equal to 150000", reproducing the wedge arithmetic). managers + mem_cache 64 failed / 1922 passed -- the same 64 baseline.

…s the ledger the host had ROOT. evict_from_tree_cache gates the eviction on uniform_avail_for_evict(...) < num_tokens. That reads tree_cache.uniform_avail_floor, published ONCE per iteration (scheduler.py:4142-4144) as the group MIN of available_size(). Allocations made later in the same iteration were never charged against it, so late in an iteration the number is stale-OPTIMISTIC: with floor >= num_tokens the eviction is SKIPPED entirely, the alloc then fails against the live pool, and the raise reports a tree full of evictable tokens nothing ever asked for. Not a new class -- the HOST sibling already had this fixed. Its own comment states the reasoning: "a stale floor over-admits ... charging admissions against the floor removes the staleness without a second collective". The DEVICE sibling never got the ledger. This adds it, mirroring sgl-project#645 exactly: uniform_admitted_since_floor, charged on the success path of alloc_token_slots, reset by the scheduler in the same call that publishes the next floor so it never outlives the number it corrects. Sufficient, by sgl-project#645's argument: live availability is at least avail_at_publish - admitted, and avail_at_publish >= floor, so a request clearing floor - admitted fits the real pool. Rank-uniform by construction: num_tokens comes from the replicated batch, so every rank charges the same amount at the same allocation and the predicate stays identical across ranks -- the #616g invariant this must not break, pinned by a test. fundable_extend_tokens reads the same predicate, so admission inherits the correction for free. OVERLAP VERDICT vs sgl-project#701(a), checked in code rather than assumed: NOT one defect, on the available evidence. * evictable and protected are DISJOINT by construction -- inc_lock_ref moves tokens out of evictable_size_ into protected_size_ (radix_cache.py:605-606) and dec_lock_ref moves them back (:622-623). So a reported evictable count never includes protected-prefix pages. * The only specimen present in any accessible log is 66039 (available=273 + full_evictable=65766, 512 requested). Its "Full LRU list evictable size: 65766" matches full_evictable_size EXACTLY. That sanity check is an independent traversal of the eviction list, and divergence is precisely what it exists to detect -- so this specimen's evictable was genuinely reachable, refuting the paper-evictable hypothesis for it. It is also the specimen sgl-project#681 already diagnosed (mamba tombstone leaf) and paid in MambaRadixCache. evict_full. * The "167k evictable" specimen is NOT in any log I can reach, so its decomposition into mamba-recoverable vs paper-only cannot be done and is not inferred. If F4-r4's 1f594e7 instrument catches a recurrence, its skipped-vs-ran line decides it directly. So the staleness defect is proven STRUCTURALLY (the host/device asymmetry) and fixed with a can-fail falsifier; it is not claimed as the cause of a specimen whose instrument output does not exist. One self-inflicted defect caught by the suite: getattr(tree_cache, "uniform_admitted_since_floor", 0) yields a Mock on an unconfigured double, and int(Mock()) is 1, not 0 -- silently shaving a token off the floor and breaking test_a_published_floor_is_returned (499 != 500). Guarded by an isinstance check, with the reason recorded. Third appearance of the sgl-project#624 stub-drift class. The host sibling carries the identical latent exposure and is left untouched here to keep the blast radius small; worth a follow-up. Tests: 7, red first, including the can-fail proof that an uncharged stale floor really does skip. managers 2113 passed / 0 failed. mem_cache is 944 failed / 772 passed both before and after this change -- a large PRE-EXISTING red suite, verified by patch round-trip on the clean tree (945/2870 baseline across managers+mem_cache without the new file). ruff clean on all three files, compared against HEAD.

…e catch away The remainder of the crash pair, and it is not a missing net -- it is a net that was cast and then ignored. sgl-project#679 built the degradation: on a failed extend allocation, consult the rank-local relief provider and RETRY before raising. alloc_token_slots does exactly that (common.py:534-536, `allocator.alloc(num_tokens)` after the provider returns). Its page_size > 1 twin, alloc_paged_token_slots_extend, called the same provider, logged that relief had SUCCEEDED, and then fell straight through to the raise WITHOUT retrying. The freed pages were never spent. That is the shape of the 01:46 specimen: a raise past a degradation that had already produced the memory needed to avoid it. So the three raise sites on the prefill admission path were all "covered" by RULE 3, but coverage on this one meant asking, not using -- which reads as covered in an audit and behaves as uncovered in production. That distinction is the whole finding. FIX: the sgl-project#679 discipline verbatim, no new policy. The allocation plus its DSV4 bundle unwrap is now a closure, so the retry is literally the same call as the first attempt -- the previous shape duplicated neither, which is precisely how the retry came to be missing. The raise is untouched, so fail-loud still has the last word, and the log line now reports whether the retry SUCCEEDED or still failed instead of implying relief worked. REACHABILITY, stated rather than assumed: this rig runs page_size 1, so the twin is not on today's hot path. But _alloc_page_size notes DCP swaps in an allocator whose page_size is server_args.page_size * dcp_size, so any dcp_size > 1 boot takes it, and uneven DCP is shipped. Live path on a supported configuration, not a hypothetical. Tests: 4, red first. The falsifier failed with the exact production symptom ("Prefill out of memory") while relief had already freed the memory. The other three pin that fail-loud survives when the retry also fails, that a provider freeing nothing triggers no pointless second attempt, and -- the asymmetry this closes -- that alloc_token_slots already behaved this way. mem_cache + managers: 2887 passed, 0 failed. ruff clean.

…e they raise ROOT, and it is not a new one. free_group_begin is called from the event loop (batch_result_processor.py:92 and :741). While that window is open, PagedTokenToKVPoolAllocator.free appends to free_group instead of extending free_pages (allocator/paged.py:293-308), so the pages sit in neither free_pages nor release_pages and available_size cannot see them -- while the tree has already counted them as evicted. That is sgl-project#681's third root, and flush_free_group (allocator/base.py:208) is its remedy. WHY IT CRASHED AGAIN ANYWAY. The remedy was wired into alloc_token_slots only. The paged twins reached their raise without ever asking whether the pages they needed were already freed and merely staged. The relief NET was carried across to the extend path under "sgl-project#681 RULE 3: every alloc path reachable from prefill admission gets the same net"; the third root was not carried with it, and the decode path had no net of any kind. So: one root, wired on one of the three paths that need it. That is the 02:18 crash -- 512 tokens refused with 147,456 counted evictable. The receipt-checking added in sgl-project#681 cannot catch it, because the eviction's receipt is HONEST: the tokens really were freed. THE LABELLED CANDIDATE IS REFUTED. The proposal was that _evict_leaf_node's allocator.free(x.value) might route rows to one sub-pool of a HybridLinearKVPool while available_size/alloc read another. It cannot produce this divergence: the accounting is entirely allocator-side over index bookkeeping (free_pages / release_pages / free_group), and available_size is computed from those two lists alone (base.py:187-188), so a free and the available_size after it read the same structure however the pool splits its tensors underneath. Pinned in TestAccountingLivesInTheAllocator, including a run with kvcache=None throughout. This says only that the hybrid pool cannot cause THIS divergence, not that it is defect-free. FIX: flush staged frees and retry, on both paged paths, before the relief ladder -- same ordering as alloc_token_slots and for the same reason, that it is not relief. It gives up nothing: it applies frees already performed and already counted. Cold path only, reached after an allocation has already failed. The raises are unchanged, so fail-loud keeps the last word. Tests (hermetic, CUDA_VISIBLE_DEVICES="", no GPU, no serving contact): test/registered/unit/mem_cache/test_paged_staged_frees_715.py 9 passed can-fail proof: the two fix-pins fail without the fix, with the exact production messages ("Prefill out of memory" / "Decode out of memory"); the other 7 hold either way test/registered/unit/mem_cache/ 797 passed, 1651 skipped, 124 subtests test/registered/unit/managers/ 2119 passed, 18 skipped, 130 subtests ruff clean on both changed files The pins drive the REAL PagedTokenToKVPoolAllocator on CPU, inheriting the whole group protocol unmodified; only alloc_extend/alloc_decode are overridden, because the production ones dispatch to CUDA kernels (sgl-project#624: the stub stays off the load-bearing path).

…refused on structure Desk-only classification. Verdict and fix SHAPE only -- no fix built; the boot wrapper is F4-r4's and the operator's ack is required. THE HYPOTHESIS DOES NOT HOLD, and it is refused structurally rather than for want of evidence. The store port cannot collide across boots: route_a_631_prod_boot.sh pins neither --nccl-port nor --dist-init-addr, so server_args.py:18787-18788 applies -- nccl_port = get_free_port() draws a port that is free at that moment, and a predecessor still holding its own store port is not a candidate. A lost race is caught anyway at :18873 by wait_port_available, which polls 30 s and names the holding process. There is also a shape argument that does not depend on this codebase: "Connection closed by peer" is a connection ESTABLISHED and then broken, whereas a stale predecessor holding a port yields EADDRINUSE at bind or a refused connect. The observed error is the wrong shape for the hypothesis. So the answer to "does wait_host_release check the store port" is no -- and it should not need to. Extending it there would encode a refuted hypothesis into the boot wrapper, where the next reader would trust it. AT LEAST TWO ROOTS, not one. The specimens split on evidence: - 18:23:43 is not NEAR an OOM event, it IS one. syslog:1789 records "A process of this unit has been killed by the OOM killer" at 18:23:43.842677. The same signature repeats at 18:39:05 against e66bde7's own measurement at 18:39:07 -- F4-r4 called that boot "killed by an external process exit", which is what an OOM kill of a peer looks like from inside a survivor. A third sits at 18:45:30. - 19:31:45 rank2 is NOT memory. The host ledger reads avail=103 headroom=97 at 19:32:24, 39 seconds later, and there is no OOM line anywhere in the 19:3x window. SENDBYTES IS A TOMBSTONE. In 3 of 3 recorded instances it is the SECOND event: HANDOFF_663:696-698 (peer dies, no traceback -- "exactly what a SIGKILL looks like"; that run "died of host RAM"), HANDOFF_658:256 (rank 0's TCPStore died, survivors then spun on sendBytes), and e66bde7's gloo frame naming a dead peer process. Any verdict that makes the socket primary is fighting the prior. THE REAL HOLE is elsewhere and is the recurring shape. wait_host_release.sh computes S -- the count of surviving schedulers -- on every iteration, PRINTS it in the clear-to-boot line, and never puts it in a condition. The gate is available-RAM and nothing else, so a predecessor whose schedulers are alive but whose allocations are already unmapped passes it. The single-instance guard does not cover the gap either: boot script :250 pgreps sglang.launch_server, the LAUNCHER, while the store and ranks live in the sglang::scheduler children -- an orphaned scheduler set whose launcher has exited passes cleanly. That is the counter-without-a-reachable-actuator family (sgl-project#679/sgl-project#681/sgl-project#684/sgl-project#715): the value that answers the question is discarded one line before the decision. FIX SHAPE (not built): gate on the counter that already exists -- require S -eq 0 alongside A -ge NEED -- which closes the predecessor window for RAM, GPU and the store socket at once, without a bespoke port probe. Two cautions handed to the owner: grep -c "sglang::schedul" reads ps comm, truncated at 15 chars, so the match is one rename from silently counting zero and a miscounting gate that reports "clear" is worse than none; and requiring zero turns a soft wait hard, which is the right direction but must fail with PIDs named rather than timing out anonymously. HONEST GAPS, recorded in section 6 rather than papered over: dmesg is permission-denied here and journalctl -k returns "No entries", so kernel OOM detail is UNAVAILABLE from this session, not absent -- which process the killer took at 18:23:43 is therefore not established, only that it fired. The serving tree runs under setsid outside systemd, so the absence of an sglang unit line is not evidence it survived. Specimen B's root and the third specimen's log remain open; a timestamp tied to a specimen is not the same as its log read, and I have not read the second. Adds "schedul" to .codespellrc: it is the literal 15-char truncation ps comm reports, not a typo.

…what the tree handed over Boot instr10 died 4m55s after health, 4 seconds after a tp_to_pp flip: RuntimeError: Out of memory. Try to allocate 512 tokens. Available full tokens: 138089 (full_available_size=189 + full_evictable_size_=137900) Eviction reported delivering >= 512 tokens and the allocator then had 189 free. The proof is an absence: _eviction_shortfall_note returns "" if and only if evicted >= asked, and the note is missing from the specimen. The one diagnostic written to explain this failure silenced itself on it. MECHANISM. --kv-backing-relief is on. KvRowCap.engage subscribes to the allocator's free listener (kv_backing_relief.py:379) and KvRowCap._apply (:490-512) moves every freed id above the cap straight back out of free_pages into _withheld -- correctly, since those rows' pages are unmapped. available_size (allocator/token.py:52-54) counts neither. FullComponent. evict_component (full_component.py:115-119) takes its count the instant it hands the free over -- `self._free_full(cd.value); freed = len(cd.value)` -- and never checks receipt. So the tree's books moved by exactly the ask while the pool's did not move at all. Measured on the boot: the cap engaged at 01:53:40 backing 137135 rows instead of 161792, withholding 24243 ids; the flip 71 s later re-seeded 160822 live slots across the whole id space, putting the peel's frontier precisely on the rows the cap confiscates. Not an admission defect: admission's 138412 evictable was real, ~114k of it below the cap and payable throughout. The peel stopped after one round because it was told it had been paid. A tighter bound computed from the same wrong receipt would have admitted the batch too. CHANGES. payable_size() is the delivery measure -- available_size() plus what an open free group still owes (so sgl-project#681's staging still reads as the delivery it is) and never withheld ids. alloc_token_slots measures `delivered` as the payable_size delta across evict_from_tree_cache and feeds that to the note; evict_from_tree_cache's own return contract is untouched, so existing callers are unaffected. New rung _evict_past_confiscation re-peels past the cap after the sgl-project#681 flush and before the relief ladder, spending only recomputable prefix, escalating on rounds that pay nothing, bounded at 8 rounds. It REFUSES under an active uniform_avail_floor: the round count is rank-local and that is the #616g divergence exactly (precedent: uniform_host_floor_active). The note now names the confiscator and drops its "THIS LINE SHOULD BE UNREACHABLE" claim, which this specimen falsifies. TESTS. New test/registered/unit/mem_cache/test_residency_cap_eviction_790.py uses a real TokenToKVPoolAllocator on CPU and a real KvRowCap as the confiscator; the tree stand-in is deliberately as careless as FullComponent, so no double supplies the guarantee whose absence is the bug. fix reverted, test present -> 3 failed / 6 passed, reproducing the specimen (same "NO relief provider is registered" warning) with the fix -> 9 passed common.py's direct dependents (679, 681 x3, 694, 616g, 631 backing relief) -> 134 passed, 7 subtests passed ruff and codespell clean on all three files.

The extend length is not `fill - prefix`. It is new_len = min(fill - prefix, _rem_tokens) schedule_policy.py:1306-1310 _rem_tokens = min(rem_chunk_tokens, rem_total_tokens) :1663 rem_total_tokens = own allocator.available_size() + own tree_cache.full_evictable_size() :948-968 The third term is RANK-LOCAL. Two ranks can therefore compute different extend lengths for the same request with an ALIGNED prefix and an ALIGNED fill -- which is exactly the observed 254-vs-301, needing neither a stale leftover nor a fill divergence. Both of those were excluded by measurement earlier in this window; this term was never examined. `sgl-project#681` caps it with a published GROUP floor -- but only when one exists. `published_fundable_floor` returns None unless `tree_cache.uniform_avail_floor` is set, and its own docstring states the assumption: "the ceiling is applied only when the scheduler actually PUBLISHED a group floor -- i.e. the ranks' pools are UNEVEN ... With no floor (single rank, or POOLS THAT AGREE) the local budget is already the group's budget" This configuration runs --rank-gpu-memory-mib 28000,17000,12000 and its derived pools differ by a factor of 35 (767780 against 21725 tokens, measured in the sgl-project#991 matrix). The assumption is violated by construction, so whether the floor is actually published decides whether the cap is group-uniform or rank-local. THIS COMMIT ASKS, IT DOES NOT ANSWER. Zero log lines for `sgl-project#681` in the dying boots is NOT evidence the mechanism is off -- it may simply not log, and that null-reading-without-coverage has caught me twice in this window already. So the value is captured where the adder is built and printed OFF-PATH in `pp_ring_note`, a site measured neutral across boots 33 and 34. The adder still receives exactly the same value; nothing about admission changes. IF IT READS None, the candidate is confirmed and the implication is larger than the fix: the inequality of the budgets would be BOTH why this configuration serves at all (the sgl-project#991 matrix: small pool + low concurrency is the only serving cell) and why it dies (unequal pools with no published floor = rank-local budgets). One property on both sides, and a published floor could resolve both at once. It would also couple to the desk line's Site #0, whose `last_chunk` would then compute on an already rank-locally capped extend_len -- two rank-local quantities in series, which would explain why that defect was desk-visible but never triggerable. Evidence: desk. py_compile; smoke asserting the value is captured, that the adder still receives it unchanged, that NO log call was added on the admission path, and that the census carries the new line. Belegstufe: DESK-BEWIESEN. The boot decides the candidate.

support gpt-bigcode

1b4c3a9

hnyls2002 merged commit caaad53 into main Jul 21, 2024

hnyls2002 deleted the gpt-big-code branch July 21, 2024 01:34

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Support gpt-bigcode model class (sgl-project#681)

65431ce

cen121212 pushed a commit to cen121212/sglang that referenced this pull request Nov 10, 2025

合入pr触发性能看板wrokflow (sgl-project#681)

f657651

* 代码合入后触发性能看板wrokflow * 封装成函数 * 合入upstream/251031分支 * test

Navigation Menu

Uh oh!

Support gpt-bigcode model class - #681

Support gpt-bigcode model class#681
hnyls2002 merged 1 commit into
mainfrom
gpt-big-code

hnyls2002 commented Jul 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

hnyls2002 commented Jul 21, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant