Fallback when sampling failed by ispobock · Pull Request #678 · sgl-project/sglang

ispobock · 2024-07-20T16:43:25Z

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Default sampling kernel may fail in some cases (flashinfer-ai/flashinfer#384). However, the sampling status is returned, we can use it handle the failed results.

Modification

Check the status of the sampling, use argmax ids to fallback the failed cases.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

merrymercy · 2024-07-20T17:28:30Z

-            batch_next_token_ids = torch.argmax(probs, dim=-1)
+        if torch.any(~success):
+            warnings.warn("Sampling failed, fallback to top_k=1 strategy")
+            argmax_ids = torch.argmax(probs, dim=-1)


Do we need to make sure probs is not NaN before calling argmax?

Yes, we should make sure it.

…tests said centre TWO TESTS HAVE FAILED SINCE 2e00987 AND THE CODE WAS RIGHT. Reported as "a law refusal is reported as a delay, escapes the abandon cap, wrong class". The classification is not wrong; the tests are aimed at a boundary that moved. 2e00987 deliberately moved what a CUTOVER TRANSIENT is judged against from the corridor CENTRE to the band's lower edge, because the corridor law is a band and its verdict is the continuous minimum against the floor -- so a dip lasting one wave walk is lawful down to 819 MiB, and judging it against 1024 reserves the band's whole tolerance for nothing. Measured there: 21,480 tokens waited ~35 s for a flip that was legal the whole time. Both fixtures use ``capacity = STAGING - 1`` and call it "cannot even fund the staging: the LAW would be broken". Under the band rule it is not a breach at all -- that fixture leaves 1023 MiB free, one MiB under the centre and 204 MiB CLEAR of the floor. A DELAY is the correct class for it, so the tests were asserting that a lawful entry must be refused. RE-AIMED, NOT RELAXED. Both now use a fixture that genuinely breaches (724 MiB remaining against an 819 MiB floor) and assert exactly what they always meant: a seam that would take the card below the band floor is refused, never delayed, never yielded through, however spent the budget is. AND THE BOUNDARY IS NOW PINNED FROM BOTH SIDES, which is the protection the report was really asking for. The margin-delay tag is exempt from the seam abandon cap by design -- that exemption is what keeps a transient shortfall a delay rather than a permanent stand-down, and it is also the only way a real breach could be carried past that cap. So: just inside the band is a delay, just below the floor is a refusal, and a breach is never yielded through at any budget. Neither side is left to inference. No production code changed. Can-fail proven by mutation: removing the breach boundary (every shortfall becomes a delay) fails 9 cases across both classes, including every rung of the budget sweep. Tests: 21 in the file, 3 of them new.

…eg that never runs THE CALIBRATION DEFECT, as the operator classed it: the unpinned solve returned 284181 tokens where the hand-pinned 550000 arms, flips 219+ times and clears its guard. That gap is not a trade to accept -- the corridor law demands best-filled and twice too much free is the same defect class as too little -- so it is diagnosed here rather than pinned around. TWO INDEPENDENT OVERCHARGES, and they compound. 1. THE FLOOR WAS PAID TWICE. ``seam_allowed_tokens`` picks the largest id space with ``have(T) >= need(T)``, where ``have`` was measured as ``free - band_floor + rung_fund``. Targeting equality therefore already sizes the pool to leave about ``band_floor + seam_draw`` free at rest -- which IS the arming floor, arrived at from the other side. The floor charge then subtracted the whole thing again. The charge is now the shortfall against what is already held free: the corridor law always, and the seam solve too when a measured record made it reserve. On this rig's rank 2 that is 1443 MiB -> 192 MiB, the load margin and nothing more. A COLD RECORD STILL PAYS IN FULL, and that is the case the whole term exists for: a cold solve reserves nothing for the seam, so nothing has been paid. The boot that landed every rank above its floor is unchanged. THE ``rung_fund`` TERM IS DELIBERATELY NOT RESERVED. The solve counts the KV rung as a payer at seam time while the gate wants free VRAM at arm time, so a gap of that size can remain -- and covering it is exactly what the pre-arm relief ladder is for. A floor priced at worst case PLUS a ladder is double insurance paid twice; reserving it here as well would be the third payment. 2. THE DRAW WAS PRICED FOR A LEG THAT NEVER RUNS. ``measure_at_rest`` keeps two maxima, ``arena_fixed`` and ``fixed``, each taken over BOTH directions, and ``total_fixed_bytes`` sums them. On every record this rig has written they are maxed by DIFFERENT legs -- the weights-arena tail only on tp_to_pp (entering PP refills it), the drafter's restore only on pp_to_tp (the drafter returns with the TP layout). Rank 2, record 03d16efef3ad: tp_to_pp: arena tail 1456 + draft restore 0 = 1456 MiB pp_to_tp: arena tail 0 + draft restore 139 = 139 MiB total_fixed_bytes (the sum of the two maxima) = 1595 MiB 1595 MiB is not a draw any seam makes. A gate that arms per flip must be priced at what one flip draws, so the worst LEG is measured and recorded alongside the two maxima, and ``arming_draw_bytes()`` is what the floor is derived from. ``total_fixed_bytes`` is left exactly as it was: the sizing solve was fitted against it and does not move under this change. BOTH THE GATE AND THE SIZER READ THE SAME ACCESSOR, pinned by a test. Two numbers that must be equal and are computed in two places is the defect 48ba9fe already fixed once here. BACKWARD COMPATIBLE BY CONSTRUCTION: a record written before this field reads ``worst_leg_fixed_bytes = 0`` and falls back to ``total_fixed_bytes`` -- the previous pricing exactly. Absent means unknown, and unknown prices the old way rather than optimistically. Tests: 41 in the file, 9 new across two classes. Can-fail proven by mutation: ignoring the solve baseline (the double charge) fails the three paid-twice cases; returning the cross-leg sum from ``arming_draw_bytes`` fails the per-leg pair including the record round-trip. No regressions: 952 pass across the seam/corridor/purity/kv/sizer/policy suites -- and for the first time in this chain with ZERO failures, the two long-standing test_seam_entry_margin_631 cases having been re-aimed in 4b13f8f. METAL VALIDATION IS STILL OWED: an unpinned boot must solve >= ~495k with a real tp_to_pp arm and commit under load before the pin comes out of the ship recipe.

…rged twice MEASURED ON THE VALIDATION BOOT (boot_678_validate.log, 0c80abf): the unpinned planner returned 482490 tokens -- up from 284181, and a full round trip under load on that pool, PHASE-FLIP DONE tp_to_pp and pp_to_tp on all three ranks. Short of the >= ~495000 bar by 12510 tokens, and the remaining term is the same defect once more. ``seam_allowed_tokens`` solves against ``have_m = have - seam_margin_bytes``: it deliberately stands DEFAULT_MARGIN_MIB back from the measured position, so at the solved id space that margin is ADDITIONAL free VRAM the solve arranged. The arming floor then added its own load margin, the same 192 MiB, for the same stated reason -- a card sitting exactly on a level only holds it while nothing else moves. Two error bars against one measurement is one error bar charged twice, and what absorbs the tail is the pre-arm relief ladder. So the solve's baseline now includes it, and the residual charge on a measured record goes to ZERO: band floor + this rank's one-leg draw + the error bar is exactly what the arming floor asks for, arrived at from the other side. A record that has already paid owes nothing further. A COLD RECORD STILL PAYS THE WHOLE 1443 MiB, unchanged. That is the boot the term exists for and the one that landed every rank above its floor. AND THE LOG NOW REPORTS THE CHARGE IT APPLIES. The validation boot printed "gives up the 704/940/1582 MiB difference" while the budget path had already been switched to the residual -- the sizer was doing the right thing and saying the old thing. A log that disagrees with the arithmetic it describes is worse than no log, because it is the number an operator reads back and reasons from. Computed once now, logged and applied, and it names which baseline bound. Tests: 42, two new -- the solve is shown to reserve the whole floor, and the error-bar term is shown to be exactly the seam margin rather than a coincidence of size. 943 pass across the touched suites, zero failures. The pool number this predicts is not yet measured; the boot that measures it is the next step, and the pin stays in the ship recipe until it does.

…tal proved it REVERTS THE BASELINE HALF OF b73486e. The log fix in that commit stands; the arithmetic change does not. THE ARGUMENT WAS SOUND ON PAPER. ``seam_allowed_tokens`` solves against ``have - seam_margin_bytes``, so at the solved id space that margin looks like additional free VRAM the solve arranged, and counting it as already-reserved drove the residual charge to zero. THE BOOT SAYS OTHERWISE (boot_678_final.log). The pool went to 537076 tokens -- past the >= 495000 bar -- and the cards came up at 987 / 2286 / 1475 MiB against arming floors of 1536 / 1633 / 2275. Below the floor on TWO of three ranks, and the pre-arm ladder found 40 and 46 MiB against gaps of 650 and 726. A pool whose cards cannot hold their own arming floor is precisely the defect this term exists to prevent, reached by removing the last thing preventing it. WHY THE PAPER MARGIN IS NOT SPARE: the ``rung_fund`` term this baseline deliberately excludes. The solve counts the KV rung as a payer at seam time and therefore permits the resting free column to land that much lower, so the margin is already spent covering it. Assuming both is optimistic by exactly the amount that decides whether a flip can arm. SO THE CHARGE STAYS AT THE LOAD MARGIN, and the validated configuration is the one 0c80abf booted: 482490 tokens, free 2167 / 3056 / 3233 against 1536 / 1772 / 2414, and a full round trip under load -- PHASE-FLIP DONE tp_to_pp AND pp_to_tp on all three ranks, request served in 21.1 s. WHAT THIS COSTS AND WHAT IS STILL OPEN. 482490 is 87.7% of the hand-pinned 550000, so it does NOT clear the >= 495000 bar; it is 2.3 points short. The remaining gap is real headroom -- the binding rank rests 631 MiB above its floor -- but it cannot be taken by removing charges, because the two data points bracket the answer: 482490 clears every floor and flips, 537076 clears the bar and cannot hold the floors. Closing it needs the sizer to SOLVE for ``free == floor + margin`` directly instead of approximating it with a subtrahend, which is a different change and wants its own boot. The pin therefore stays in the ship recipe. Tests: 42, with the error-bar case re-aimed to assert the exclusion and to carry the measurement that justifies it. 943 pass, zero failures.

…get towards it THE REMAINDER OF sgl-project#678, and it removes the approximation both previous failures came from. A SUBTRAHEND CANNOT STATE THIS CONSTRAINT. The requirement is "every rank must REST at or above floor + margin, or it can never arm a flip and the pool is fixed at boot". What the pool gives up and what the card ends up holding free are related by the sizer's other posts, and the gap between them is exactly where this ticket's two failures lived: 284181 tokens the subtrahend double-charged a floor the seam solve had already reserved 537076 tokens the double charge removed, and two of three ranks came up BELOW their floor -- 987 MiB against 1536, with the pre-arm ladder finding 46 MiB of a 650 MiB gap Both are one error in opposite directions: a quantity that must be SOLVED FOR was being adjusted TOWARDS. WHAT WAS MISSING WAS THE FREE COLUMN ITSELF. ``have_bytes`` is already net of the band floor and of the rung credit (``free - band_floor + rung_fund``), so the raw resting free could not be recovered from the record. It is persisted now, with the rung credit beside it for audit, and the constraint becomes one line: free(T) = free_at_measure + (id_space - T) * cell T <= id_space + (free_at_measure - floor - margin) / cell Exact, and it needs no model of the activation reserve, the capture peak, the arena or the carve-out -- all of them were resident when the column was measured, which is the argument ``seam_allowed_tokens`` already makes for its own anchor. TWO CONSTRAINTS, ONE MIN, NEITHER SUBTRACTED FROM THE OTHER. The seam must be fundable AND the card must rest above its floor. They bind on different ranks at different vectors, so the pool is the smaller of the two id spaces. HERMETIC VALIDATION AGAINST THE TWO BRACKET BOOTS, which is the whole reason those boots were worth their windows: the 482490 bracket accepted on every rank -- it flipped both ways on metal, so a solve that refused it would be wrong the 537076 bracket rejected by exactly ranks 1 and 2, the two measured below their floor solved pool 498310 tokens, 90.6% of the 550000 pin, CLEARING the 495000 bar the subtrahend could not reach, and 38766 below the bracket that could not hold the floors The rank-to-card pairing in the fixture is evidenced, not assumed: rank 0 carries the 31800 MiB budget only the 5090 can hold, and of the two 3080s only one pairing is consistent with 482490 having flipped -- the other puts a card 300 MiB under its floor at a pool that demonstrably worked. BACKWARD COMPATIBLE: a cold record, a missing cell or a record written before the free column existed returns None -- explicitly not zero, which would be a verdict -- and the caller falls back to the subtrahend, the previous arithmetic exactly. Tests: 55, twelve new across two classes. Can-fail proven by three mutations: disabling the direct solve fails 8 cases; dropping the floor target from the solve fails 5; and dropping the floor ceiling from the min fails the floor-binding case -- which a mutation caught missing, because every other test in the class happened to have the seam binding. That is how a guarantee ends up computed and discarded, and it now has its own falsifier. 966 pass across the touched suites, zero failures. NOT BOOTED. Desk-only by instruction; the validation boot is scheduled by the operator, bundled with sgl-project#677 and the router reload. ONE THING THE BOOT MUST WATCH: the solve targets the RESTING free column, while an arm happens under load. On the 482490 boot the corridor guard sampled rank 1 at 1554 MiB against its 1633 MiB floor mid-prefill -- a ~600 MiB load dip against a 192 MiB load margin. It flipped anyway, because the seam gate's own ladder pays at seam time, but if the boot shows arms refused for the floor while the resting column is correct, the margin is the term to measure, not the solve.

… and say which path sized the pool RED TESTS ON THE LINE ROT, so these are repaired rather than skipped. All five had the same shape: a hand-built stub that stopped matching the class it fakes. THREE IN test_scheduler_chunked_req_gate.py. The file already carries a comment warning that "a bare MagicMock makes EVERY flag truthy, which arms the phase-boundary actuators this gate test has nothing to do with", and pins the flags it knew about. ``enable_phase_flip`` was not among them, so the prologue entered ``_phase_flip_on_round``, which LAZY-BUILDS a PhaseFlipRuntime from a Scheduler that has none of the state a build needs. Pinned off with the rest -- the same trap the block was written for, one actuator further on. A second gap behind it: ``_make_req`` builds a Req via ``__new__``, so every field the class has gained since must be restated; ``kv_spill_state`` was missing, and its default is taken from Req.__init__ rather than invented, because a stub that guesses a default is a test that passes for the wrong reason. ONE IN test_scheduler_pp_request_order_633.py, same class: ``_pp_forward_and_process_input_requests`` now consults ``pp_phase_flip_armed()``, which reads ``server_args.enable_phase_flip``, so a bare ``SchedulerPPMixin()`` raised before the ordering under test was ever exercised. A named stub factory now pins it off once instead of twice. AND ONE STALE SOURCE PIN, which is the one worth reading carefully. ``test_every_pp_loop_calls_the_helper`` asserts that every PP loop routes received requests through the forward-first helper -- the property that stops adjacent stages deadlocking on a control request. ``event_loop_pp`` was refactored to set ``_defer_flip_round_to_pp_loop`` and delegate to ``_event_loop_pp_body``, and the helper call moved one frame down with it. The property was never lost; the pin stopped reaching the code it guards. So the pin FOLLOWS THE DELEGATION rather than being relaxed: the effective source of a loop is its own source plus the body it delegates to. Weakening the assertion would have been the easy repair and the wrong one -- a pin that no longer reaches its subject reads as protection while protecting nothing, which is exactly how this file's property could be lost silently. Can-fail re-proven: replacing the helper call with a direct ``process_input_requests`` fails both the reach pin and the no-direct-call pin, on ``event_loop_pp`` specifically. SEPARATELY, THE SIZER NOW SAYS WHICH PATH IT TOOK. sgl-project#678's acceptance turns on whether the pool was SOLVED from the at-rest free column or APPROXIMATED by the budget subtrahend, and that was not visible in a boot log -- it had to be inferred from the pool number, which is an investigation rather than a check. Both branches log one line naming themselves and their inputs, so acceptance is a grep. The approximating branch also names the record provenance, because "why is there no column" is the immediate next question. Tests: 993 pass across the scheduler/prefill/seam suites with ZERO failures -- the first clean run of this sweep in the chain. The four remaining ruff E402s in the gate test are its own deliberate post-``maybe_stub_sgl_kernel()`` imports, pre-existing and untouched. NOT BOOTED. Desk-only by instruction.

…ore parking Built to DESIGN_679_admission_relief_ladder.md. A park is not free -- it is a request that made no progress this round -- and the note costed what should be spent before accepting one. This spends it. rung 0 radix eviction already spent by the caller. Baseline, not repeated. rung 1 kvso.try_spill bounded, chosen, costs no request's progress. rung 2 throttle frees NOTHING now; stops rung 3 repeating next round. rung 3 retract_decode most tokens, loudest: the victim re-prefills. rung 4 PARK the floor, and still the final authority. RUNG 3 IS THE ONLY HARD PART, AND IT IS A REFACTOR, NOT A CALL. retract_decode is one line; what surrounds it in update_running_batch is what must not drift -- the metrics, the new_token_ratio handover, the abort dispatch, and above all for req in retracted_reqs: self._add_request_to_queue(req, is_retracted=True) A second implementation that forgot that line would LEAK every victim it retracted, which is worse than the crash this ladder exists to prevent. So the block is EXTRACTED VERBATIM into _retract_decode_and_requeue and both call sites share it. One implementation, two callers, no drift -- and a test pins that the shared actuator still contains the requeue. The precondition travels with it: rung 3 sets batch.uniform_avail_floor from the reduced value before retracting, because that bound governs the retraction loop AND the last-survivor test. sgl-project#583 is exactly the case where the entry decision was uniform and the loop bound was not, so ranks entered together and popped DIFFERENT numbers of victims. GUARD (a) -- EXHAUSTION IS AN OUTCOME, NEVER AN ERROR. try_spill returns False when no host region is free; the ladder falls through to the next rung. Same for a rung that frees less than asked, and same for a rung that RAISES: every rung is wrapped, the ladder continues, and nothing here can turn a relief bug into an instance death. That host-region bound is still unmeasured under the 5-lane load that produced the crash, which is precisely why it is treated as ordinary. GUARD (b) -- EVERY DECISION IS GROUP-UNIFORM. Each rung reads uniform_min_avail(), the value the pre-branch reduce published at the top of this iteration, so no rung takes a collective of its own and none can split the group. The shortfall is sized from that same reduced value, so every rank asks its rungs for the same tokens -- sizing it locally would be sgl-project#583 one layer up from where sgl-project#583 was found. GUARD (c) -- OFF BY DEFAULT. SGLANG_ADMISSION_RELIEF_LADDER unset returns 0 before touching anything: byte-identical to c4b88e1, the boot currently serving. Rung 3 carries a SECOND flag (SGLANG_ADMISSION_RELIEF_RETRACT) because it is the only rung that destroys progress, and it is inert unless the ladder itself is on. THE PARK REMAINS FINAL (DESIGN_679 rule 1). The ladder changes what there is to decide from; add_chunked_req still decides. It runs immediately before that call and nowhere else, and a source pin refuses can_run_list / add_chunked_req / set_extend_range inside it -- a ladder that admitted anything would be a second admission authority. Tests: 18 hermetic, on a Scheduler stub carrying only what the ladder touches. Mutation-proven on the four decisions that could invert silently: default flipped ON -> the off-by-default case fails rung 3 without uniform_avail_floor -> the sgl-project#583 precondition case fails exhausted spill treated as terminal -> the fall-through case fails a paid spill no longer short-circuiting -> the ordering case fails Each fails alone, by name. One test of my own was caught by its first run reading the wrong frame (get_new_batch_prefill delegates to _get_new_batch_prefill_raw) -- the same stale-pin class repaired in the PP loop tests, fixed the same way. No regressions: 1011 pass across the scheduler/prefill/seam suites, zero failures. The extraction adds no lint (94 before, 94 after, all pre-existing). NOT BOOTED. Desk-only; validation bundles with the sgl-project#678 acceptance.

…es the pool forever FOURTH LATCH OF THE NIGHT, and the same cure as the other three: sgl-project#681's eviction count that could not be paid, sgl-project#682's guard ceiling the scheduler never held, sgl-project#684's `_exhausted_at_rows` process-lifetime marker. Each was a number that could only ratchet one way. WHAT IT COST. `corridor_shortfall_bytes` is added straight to the arming floor's load margin -- `(DEFAULT_MARGIN_MIB << 20) + measured` -- and the arming floor is the binding constraint on two of three ranks. On 2026-08-16 the rank-0 record carried 1004 MiB of it while every record written the day before carried 0, and the boot reading it logged NO breach of its own: it was inherited. The event it descends from is almost certainly 02:36:30 on that exact card, where a test harness belonging to this strand held 4.29 GiB and drove free to 76 MiB. A few seconds of intrusion, taxing every subsequent boot. THE OLD SEMANTICS WERE HALF RIGHT, AND THAT HALF IS KEPT. `record_corridor_shortfall` documents itself as "A MONOTONIC MAXIMUM, deliberately -- a shallower breach later does not mean the deeper one cannot recur; the pool must be sized for the worst instant that has ever been seen". Correct WITHIN an observation. Wrong ACROSS boots that never see it again, because "ever" had no end and nothing could retire a number nobody could reproduce. So: monotonic maximum while it is being OBSERVED, geometric decay across boots that observe nothing. A breach that recurs is re-observed and re-raised to its worst on the spot. One that cannot be reproduced is halved by each flip boot that measures its seam without seeing it, and written off to exactly 0 below `SHORTFALL_FORGET_BYTES` so the decay terminates instead of leaving a tail that still moves the floor. 1004 MiB is gone in seven clean boots. "OBSERVED BY THIS PROCESS" IS THE DISCRIMINATOR, and it is a pid rather than a timestamp because both writers live in the same process: the runtime's corridor audit stamps the record mid-run, and `write_seam_reserve` rewrites it at the end of the same boot's flip measurement. Same pid means this boot saw it and the value stands; a different pid means it was inherited, and a boot that completed a seam measurement without its audit firing is evidence against it. Evidence is what retires it. RANK-LOCAL. The record is per (configuration, rank) and the shortfall is one card's own measurement -- 1004 / 0 / 0 on this boot, legitimately different. No collective reads or writes it and this change adds none. TESTS, red-first: 7 cases. The three decay cases failed before and pass after; the four that pin the half worth keeping -- a breach this process observed is preserved, a deeper one still raises, a shallower one does not lower it -- passed from the start, so the fix cannot have been "delete the term". One case drives the full production scenario: a 1004 MiB one-off decaying to zero while the load margin returns to its default, and one that proves a breach observed on every boot is never decayed away.

…tched off sgl-project#380 CLASS: A TEST THAT COULD NOT FAIL AGAINST THE PROPERTY ITS NAME PROMISED. `test_the_solved_pool_clears_the_operators_bar` asserted that "the solved pool" reaches 495000 -- "within 10% of the 550000 the hand-pin encodes". It computed that from a record built with `per_row_bytes=1.0` and `have_bytes=8 GiB` ("deliberately non-binding here"). This rig's own records carry 424.1 / 550.7 / 2360.3 B/row. So the assertion measured the FLOOR solve in isolation with the seam term switched off, while on metal the seam binds rank 0 at 530237 and the group lands at 435319. A green bar at 495000 coexisted with a 435319 boot for a whole task, and the gap between them was read as 115k of missing pool. The isolation test still has value -- it is what shows the floor arithmetic is not itself the conservative term -- so it is kept and RENAMED to say what it does: `test_the_floor_solve_alone_clears_the_bar_with_the_seam_NEUTRALISED`. THE SHIP BAR NOW USES THE RECORDS THE BOOT ACTUALLY WRITES. Each rank's fixture goes through `write_seam_reserve` -> `read_seam_reserve`, the same pair the boot uses, so a change to the record schema breaks this file instead of leaving it testing a shape nothing writes. The values are this rig's records of 2026-08-16T03:56Z. Reconstructed that way the six solves (three ranks x floor and seam) give a group minimum of ~435696 against the boot's own 435319 -- the fixture reproduces the metal. FOUR PROPERTIES, AND THE FIRST IS THE ONE THE OLD BAR INVERTED: * nothing ships at or above a pool measured UNSAFE. 537076 could not hold its floors on two of three ranks. The old bar pushed UP toward 550000, which is 13k ABOVE that. This one fails if the sizer ever reaches it. * no regression below what the recorded inputs support today. * the withdrawn 550000 is recorded AS withdrawn, with its reason, so a re-introduction is recognisable rather than arriving as a fresh "empirical bound". * THE INVALIDATION MECHANISM: the pin's recorded basis (seam slopes, arming floors) is compared against the records it was derived from. 550000 outlived its regime silently because nothing ever compared it to its inputs; when a change moves a slope or a floor this goes red and the pin is re-derived. Mutation-checked: perturbing one floor by 67 MiB fails the test. AND THE REMAINING GAP IS NAMED, NOT HIDDEN. A fifth case asserts the sizer is still BELOW the demonstrated-safe 482490, so the ~47k is visible in the suite rather than papered over by lowering the safe number to meet the solver. It inverts when the follow-up lands, which is the signal that it can close. SHIP_PIN ITSELF landed in bb1620e, which touched the same module: a frozen `SizingPin` carrying demonstrated_safe=482490, demonstrated_unsafe=537076, the regime commit 48ba9fe ("the pool must reserve for the floor the GATE arms at"), the per-rank slopes and floors it stands on, and the withdrawn pin with its reason. Provenance lives in the artifact precisely so the next regime change invalidates it visibly instead of silently surviving, which is the whole failure this task turned out to be about. Suites: 60 in this file, 2394 across unit/managers + unit/mem_ledger. The 4 failures in that sweep are pre-existing and unrelated, parity-proven earlier against the untouched tree.

…y pays THE ~47k WAS THE ARENA TAIL, CHARGED TWICE. `arming_floor_subtrahend_bytes` has warned about this since sgl-project#678 -- "Reserving it here as well would be the third payment for one requirement" -- and this is that sentence finally made true for the one term it was not. THE TAIL IS REAL, AND MEASURED AT THE OPERATING POINT. Not another remembered number; it reconciles three independent ways on the 2026-08-16 boot -- the boot's own `rung 3 released ...` lines, the record's `arena_fixed_bytes`, and the documented definition `max(0, pp_bytes - tp_bytes)`: PP0 pp 15790.5 tp 16329.9 -> max(0, -539) = 0 MiB PP1 pp 9792.8 tp 8977.8 = 815 MiB PP2 pp 10434.0 tp 8977.8 = 1456 MiB WHAT IT PHYSICALLY IS: the weights the PP layout needs beyond what the TP layout needs on that rank. Entering PP re-commits it, entering TP releases it. It is largest on the SMALLEST card because uneven TP shrinks that card's TP shard (both 3080s at 8977.8 MiB) while PP still hands it a full stage (9792.8 / 10434.0). The 5090's TP shard is LARGER than its PP share, so it has no tail at all. Geometry of uneven-TP crossed with layer-split PP, not a defect. WHAT WAS WRONG WAS THE TREATMENT. It was charged to the arming floor, a PERMANENT free-VRAM reservation, while the same record already showed the KV rung able to pay it at flip time: PP1 954 MiB of rung_fund against an 815 MiB tail, PP2 1595 against 1456. Reserved permanently it took the binding rank's entire headroom -- 2474 MiB free at rest against a 2467 MiB floor, an equality solve with 7 MiB to spare. as shipped POOL = 435696 (boot: 435319) arena tail funded by the rung POOL = 483723 demonstrated safe 482490 demonstrated unsafe 537076 The binding constraint also moves off the floor: all three ranks now sit near their SEAM solves (505307 / 483723 / 503138), which is the measured cost of the flip itself and the right thing to bind on. CONDITIONAL ON THE RECORDED FUNDING, never unconditional removal. A rank whose rung cannot cover its own tail keeps paying for it in the floor, because for that rank the ladder genuinely cannot find it. `rung_fund_bytes` is 0 on a record that never measured one, and 0 means "no funding known", never "funded". And the result never falls below the leg the rung does NOT fund -- the drafter's restore floors it, so relieving the arena cannot relieve the other leg by arithmetic accident. GATED ON METAL, NOT ON ARITHMETIC. The rung that must pay was latched off until 38c1161, whose recovery aimed above the arena's immutable reservation and failed 59 times out of 59. This relief was deliberately not built until that fix was running: on the boot carrying it the log shows zero recovery failures and the arena tail released 22 times at flip cadence, plus 522/770 MiB rung-3 releases and KV-BACKING releases to 2268 MiB. The provider pays, repeatedly, before anything was sized against it. TESTS, red-first. Five new cases on the draw itself: the two relief cases failed before and pass after, while the three that keep the charge -- an uncovered tail, a rank with no tail, a record that never measured a rung -- passed from the start, so the fix cannot have been "remove the term". THE ACCEPTANCE INVERTED EXACTLY AS DESIGNED. The fifth ship-pin case was committed in 952fc4b asserting `pool < demonstrated_safe` and saying in its own docstring that reaching it would mean the follow-up could close. It now asserts `pool >= demonstrated_safe`, against the artifact's number rather than a literal, so a re-derived pin moves the bar with it. Its fixture floors are now DERIVED from each record's own draw instead of pinned, which is what makes it sensitive to this change at all; the non-draw residue is held fixed so rank 0's corridor-shortfall term is not frozen into the test. Mutation-proven: disabling the relief drops the solve back to exactly 435696 and the acceptance goes red naming it. AND THE OVERSHOOT IS A PROPERTY, NOT A CONSTANT. A new case checks the solve against `SHIP_PIN.demonstrated_unsafe_tokens` and asserts a positive margin, so a future relief that buys too much pool is caught by the same test that certified this one, with no literal for anyone to forget to update. FOLLOW-UP, REGISTERED HERE AND NOT A TASK YET: rank 0's seam slope is 2360.8 B/token against 424.1 and 550.7 on the 3080s -- 5.6x. Mechanism located (`per_row = wave_slack / id_space`; rank 0's staging is genuinely 980 MiB against 176/229) but the ratio is not explained by its token share (1.40x) or its TP-shard ratio (1.82x). It binds nothing after this change -- rank 0 sits at 505307, above the group minimum -- so it is an open question about a number, not a constraint on the pool. Suites: 66 in this file, 2442 across unit/managers + unit/mem_ledger + the resident-carry file. The 4 failures in that sweep are pre-existing and unrelated, parity-proven earlier against the untouched tree.

…he fix was on disk THE LEDGER COULD NOT ARBITRATE, TWICE IN ONE NIGHT. It reported `model weights (shards) = 0 MiB` on all three cards while they held 16196 / 10194 / 10832 MiB, marked three posts UNBOUNDED and said `fits=False` everywhere. Neither the sgl-project#683 attribution ("who took the card") nor the sgl-project#678 sizing question could be settled from it; both were answered by hand. (a) WHY THE ZERO, AND WHERE THE NUMBER WAS. Not an unwired term, not a fingerprint that never matches: `reconcile.completeness_failures` already names it -- the shipped config pins `--rank-gpu-memory-mib`, the PIN PATH, and the pin path skips the planner that computes the shard vector. The term is built from an all-zero vector and dumped as a price, indistinguishable in the JSON from a model that needs no weights. The flight recorder was already measuring it. The reserved-bytes delta between `pre_weight_load` and `weights_loaded` IS the shard footprint, per card, every boot. This reads that. Provenance gets a fifth kind, `MEASURED` -- not MODELED (no formula of ours produces it) and not CALIBRATED (not a cached probe under a fingerprint; re-measured every boot). That is the kind the standing directive asks for: demand from measurement, not estimate. DETECTION STAYS WHERE IT ALREADY WORKS. The engine does NOT add a second refusal for the zero case. `completeness_failures` calls it "the PIN PATH signature" and `require_complete` raises on it; a refusal inside the engine would change the verdict of every card whose shard vector is legitimately absent at that point in the boot, including the pre-planner ledger whose residual is the RANK BUDGET rather than the KV pool. I built that refusal first, watched it break five unrelated suites, and took it back out -- the detector is the acceptance, pinned in both directions: it stops firing once the post is measured and keeps firing on a fingerprint with no marks to stand in. (b) THE FITS SPLIT, A PRECISION AND NOT A WEAKENING. `fits` was `not unbounded and committed <= total`, so ONE inherently-unbounded transient made every card unfittable forever. The arithmetic side already passed (2848 against 32607); `fits=False` was caused purely by a non-empty `unbounded`. The load transient refuses on EVIDENCE -- 563 boots, a 0-18486 MiB spread, above the 50% refusal rule -- and that refusal is correct and permanent, so the flag was conflating "a post could not be priced" with "the card is overcommitted". A transient is a peak ABOVE the resident set, not a claim on the card. It now travels in `unbounded_transient`, rendered beside the verdict WITH its evidence, and does not block the fit. Resident refusals still do: pinned in both directions, including a synthetic unbounded resident post forcing fits=False. Nothing is estimated in either bucket. (d) THE ACTIVATION POST IS AN UPPER BOUND, LABELLED AS ONE. The serving marks sample `allocated_peak_bytes` under real load -- the counter `probe_activation.py` reads -- but never reset it, and a reset at mark time would change what every other reader of those fields sees. So what this publishes is a monotone envelope since process start, above the resting set, over-stated by an unknown amount and said so in the term's own derivation. Over-reserving is the safe direction; the failure this ledger exists to end is the under-charge. A per-phase point estimate needs the reset and is future work. THE INSTRUMENT BUILT TWO TASKS AGO IS WHAT MADE (a) AND (d) POSSIBLE, and it is live: 519 serving samples spanning 03:56:48 to 05:27:02 on the running boot, plus the boot-scoped marks. No new measurement path was added -- these are differences between marks the recorder was already taking. (c) STAYS GATED, AND THE INVESTIGATION RESOLVED -- see the report. Measured across the same two capture marks: reserved delta 184/182/182 MiB against an NVML free drop of 282/324/324, so the reserved-bytes delta is structurally blind to 98-142 MiB per card (the private-pool component an OOM here named directly). Feeding `capture_mib` would under-charge. The probe does not need replacing, the FIELD does -- the honest quantity is the NVML free delta these same marks already record -- but the refusal also carries an inherited "3.3-3.8x low" that does not reproduce on this config (~192 estimate against 282-324 measured, i.e. 1.5-1.7x), so that has to be re-derived rather than carried. The finding is recorded in the code; nothing is fed. TESTS. 13 new hermetic cases on the reader, red-first (all 13 failed on the missing API), including: the first load is the shard post and not the drafter or the second layout, cards kept apart by uuid, an unpaired load yielding nothing rather than a guess, no marks giving an empty answer rather than a zero, and the serving envelope labelled upper-bound-only. Plus the completeness detector pinned in both directions and the transient split pinned in both directions. Suites: 482 mem_ledger, 2415 across mem_ledger + unit/managers. The 4 failures in that sweep are pre-existing and unrelated, parity-proven earlier against the untouched tree.

… allocator's THE ALLOCATOR'S BOOKS ARE STRUCTURALLY BLIND TO PART OF A GRAPH CAPTURE. Measured on this rig 2026-08-16, across the same capture_begin -> capture_end marks: card reserved delta NVML free drop blind to 5090 184 MiB 282 MiB 98 MiB 3080 182 MiB 324 MiB 142 MiB 3080 182 MiB 324 MiB 142 MiB The gap is the private-pool component CUDA graphs take, which an OOM on this rig named outright ("71.21 MiB allocated in private pools"). So the post is fed from the NVML free delta, MEASURED provenance, re-measured every boot and never cached across regimes -- the same rule the weights term follows. BOTH NUMBERS ARE KEPT. `capture_reserved_mib` sits beside `capture_mib` in the record and in the term's own derivation, so the blindness stays visible to the next reader instead of decaying into folk knowledge about "why we use NVML here". THE STALE FACTOR IS WITHDRAWN, NOT SILENTLY REPLACED. The refusal carried "3.3-3.8x low" from the 2026-08-05 window. It does not reproduce on this configuration: the captured-tokens estimate reads ~192 MiB here against a measured 282-324, i.e. 1.5-1.7x. The ESTIMATE stays rejected -- an under-charge is the direction that OOMs -- but the factor is re-derived from this boot's own marks rather than carried, and the refusal text now says so. That is the same disposal the 550k pin got in sgl-project#678, for the same reason: a number that outlives the regime it was measured in is not evidence. CONTAMINATION IS A PRECISION CAVEAT, NOT A SAFETY ONE, and it is recorded as such. NVML free is card-wide, so a foreign process allocating during the capture window inflates the reading. That is an OVER-charge: it costs KV pool, never a boot -- the same direction that justifies the activation upper bound. A STRUCTURAL FLOOR, NOT A THRESHOLD. The driver cannot release less than the allocator took, so an NVML delta BELOW the reserved delta on the same window is physically impossible. That is an instrument fault, not a small capture: `capture_fault_mib` appears instead of `capture_mib` and the term posts UNBOUNDED loudly, quoting both readings, rather than feeding a number already known to be wrong. A boot whose marks lack the pair at all posts UNBOUNDED too. WHAT THE LEDGER NOW SAYS, from the running boot's records on disk (hermetic, no card touched): card demand before -> after KV pool before -> after fits 5090 1824 -> 19188 29759 -> 12395 False -> True 3080 1325 -> 12561 18131 -> 6895 False -> True 3080 1325 -> 13158 18131 -> 6298 False -> True weights 16196 / 10194 / 10832 MiB (was 0 on all three) capture 262 / 324 / 324 MiB (allocator-only: 164 / 182 / 182) activation <=906 / <=718 / <=677 MiB (upper bound, labelled) Every card prices its resident side fully and fits with honest numbers. The load transient still refuses -- correctly, on 563 boots of evidence -- and travels as a risk band that does not block the verdict. TESTS. Four new hermetic cases on the capture reader: the fed number is the driver's while the allocator's is kept beside it, an impossible reading becomes a fault rather than a price, a boot without the pair prices nothing, and the no-op second capture round of a flip boot is skipped rather than read as a zero-cost capture. 485 mem_ledger, 2418 across mem_ledger + unit/managers; the 4 failures there are pre-existing and unrelated, parity-proven against the untouched tree.

…e two commits to hold MERGE_NOTES_602.md rewritten to cover all 18 commits: what each fixes, its test evidence, whether it touches runtime, and whether an equivalent patch is already on the serving line (verified with `git cherry`, not by message matching -- four are: c41645c, ce60358, 658ea3a, 84b0171). DRY RUN: clean. Merged into `integration/r2` -- the live line, since the serving tree descends from its tip a73a0d8 -- in a throwaway worktree, `--no-commit --no-ff`, then aborted and the worktree dropped. Zero conflicts, zero unmerged paths, so nothing was pre-resolved because nothing needed it. Verified semantically as well as textually: on the MERGED tree, managers 2093 passed / 0 failed and planner 2574 passed / 2 failed, the two being the same pre-existing test_rejected_evidence_pins pair that is already red on the base. THE FACT THE OPERATOR NEEDS, and it is not in the commit count: `7936bc4850` is NOT an ancestor of integration/r2, so merging this branch drags in its whole base lineage -- 115 commits, of which 18 are mine and 97 are the hotfix/677 work (sgl-project#662 x20, [PhasePolicy] x18, sgl-project#677 x8, sgl-project#678 x7, sgl-project#679 x6, ...), 129 files, +22731/-762. Approving this merge is approving that lineage, most of which is not mine to vouch for. If only this work is wanted it must be cherry-picked rather than merged. NOT ATOMIC, and it splits cleanly into four groups with an order: (1) the four already on serving -- merging them only reconciles integration with what is already running; (2) the three sgl-project#624 test-only drift guards, which take managers from 4 failures to 0 and should land early so the line stays green during review; (3) desk tool + docs, all planner/pp_cut.py and markdown, imported by no serving path; (4) hold. HOLD, two commits, both runtime and neither on the serving line: * e21e87f (sgl-project#690) touches the seam hot path and changes the PHASE-FLIP DONE format. Already queued to land on deploy WITH the W=8/W=4 probe after the sgl-project#694 soak verdict; merging it into integration first puts it in front of the soak meant to measure it. * 5301b94 (sgl-project#685) touches the boot sizing path. Announce-only today and abstention-guarded, but unsoaked, and the R' decision it waits on is not made. Nothing else in the chain can move serving behaviour. Docs only; no merge performed, no deploy, scratch worktree removed.

…of a previous boot The flip seam's arena tail is max(0, layout_pp - layout_tp) and it is a post in the KV pool solve, but layout_tp is built from a real TP-shaped worker AFTER the pool is sized (scheduler.py:1421 -> :1425 -> :1442). phase_flip_seam_reserve states the consequence verbatim: "there is no ordering that makes them knowable in time, so they are not computed here at all". The tail has therefore been carried between boots in an on-disk seam record that defaults to 0 when cold, which is the two-boot cliff behind the sgl-project#678 OOM class: a first boot prices rank 2 at 1523 MiB instead of 3226 and dies when the NEXTN draft weights land. A layout needs shape, stride, dtype, storage_offset and the storage size, and all five are correct on a meta tensor. So the TP layout is answerable at sizing time without allocating a byte. The one input that is NOT correct is data_ptr(), which is 0 on meta and is exactly what plan_arena_layout uses to detect aliasing -- fed meta tensors it folds a model into one slot and returns a total that is orders of magnitude wrong and structurally perfect. storage_alias_relation supplies the relation from the StorageImpl handle, which is a valid identity on meta and agrees with data_ptr grouping on real tensors. Strict on this path only: an installed vector whose length does not match the group size falls back to an EVEN split rather than raising, and that fallback is load-bearing across the tree (a blanket refusal fails 41 distributed tests). It is not touched -- it is simply not tolerated where the answer is charged into a memory budget. DESK GATE, three gloo ranks, CUDA_VISIBLE_DEVICES="", real checkpoint config: the derivation reproduces the boot's own logged layout_tp on every rank -- 15925.8 / 8573.78 / 8573.78 MiB against 15925.80 / 8573.78 / 8573.78 (boot_735_default791b.log:952/962/975). Zero MiB error, 0.1 s per rank. Can-fail proof: vector 22,21,21 moves the totals to 11921.06 / 10576.16 / 10576.16, far outside the 1 MiB bar. The alias detector has its own can-fail proof, because its healthy answer on this architecture is the empty relation -- which is also what a broken detector returns. This commit derives and instruments only; no sizing path consumes the number yet, so no boot changes size. The deciding gate remains the boot instrument checking the derivation against the same boot's own totals. Tests: 67 passed (the two new suites plus test_arena_tail_identity_785, test_meta_arena_layout_785, test_scoped_partition_overlay_785, test_corridor_arming_credit_784). ruff + codespell clean.

…as to fund it build_flip_draft_worker (phase_flip_boot.py:614) loads a whole second model -- the NEXTN/MTP draft -- at scheduler.py:1442. The KV pool was solved at :1425. So the budget never knew a 1.4 GiB model was still coming, and the only thing that made that survivable was a pool small enough to leave accidental slack: on the shipped cut the binding rank held 7027 MiB free and the draft fit by luck. Boot 735-full785 removed the luck. It solved 764512 tokens and then died in ct_embedding.create_weights asking for 406 MiB with 227.75 MiB free on rank 0, inside build_flip_draft_worker. Same shape as the sgl-project#678 OOM and the same cause as the arena tail: a post the solve cannot see because it is created later. Derived the same way the tail is -- a meta-device model under the flip scopes, 0.01 s, no weights allocated. On this rig 1441.14 / 1352.35 / 1352.35 MiB. Subtracted from the budget rather than folded into the seam reserve: it is resident weight, not a seam cost, and a term charged under the wrong name is a term the next reader misprices. This also supplies an input the planner already demands and nothing could previously produce: planner/pp_cut.py:412-432 makes RankResources.draft_residency_mib MANDATORY and refuses to solve a cut without it. Tests: 18 passed across the cold-pricing and alias suites. ruff clean.

… rung that already names it PRIOR-ART GATE FIRST, and it changed the design twice. (1) The spill ladder ALREADY has this rung. phase_flip_spill.py:173 DEPTH_DRAFT_GRAPHS = 4, CLI name "draft+graphs", with IMPLEMENTED_DEPTH = DEPTH_ARENA_TAIL -- so rung 4 parses and is refused, by design, rather than silently under-delivering. No new flag and no new concept was needed; the work is wiring the rung that exists. (2) The park/restore machinery exists too, and its own docstring names this exact case: short_term_offload_register.py:1269 GraphFamilyRegister ("a whole LAYOUT/ALGO family's capture state, pre-captured at boot and parked while its family is inactive"), with offload_family/onload_family, over the sgl-project#93 VMM route. The credit side exists as well -- kv_backing_relief / recover_kv_backing, and the mixin already carries paused rung graph bytes as registry posts feeding the KV budget (model_runner_kv_cache_mixin.py:1488-1603). Nothing here needs inventing. (3) Rung 4 carries a RECORDED REFUSAL that must not be overturned silently: it "buys a phase-local spill of something the next TP phase must re-capture, which is a different and much worse trade than rung 2". That reasoning is about a RUNTIME per-flip spill where every flip pays a recapture. What is proposed is a BOOT-time deferral paid once. The arithmetic that changed the trade is measured, not argued: on boot 735-bal785 rank 0 binds the pool at 525462 tokens and needs 2242 MiB more to fund the 669k reference, against a residual on that same rank of 2294 MiB. Same number. The refusal is therefore LEFT STANDING in phase_flip_spill until the park/restore path is wired and exercised in a real flip cycle; this commit only makes the sizer able to express the credit, so the next step is wiring rather than arithmetic. The credit is bounded by what was MEASURED, not by what is needed -- a term that grows to close whatever gap it is shown is not a measurement, and that has its own can-fail test. The draft WEIGHTS are never credited: they are needed in the phase the draft actually runs in, and crediting them would be the sgl-project#678 OOM one rung deeper. Also: the pool-pin guard now distinguishes a recipe from prose about a recipe. argv_hc_interval8192.HOSTRAM-NOTES.txt sits under the same glob, was touched after the order, and quotes the flag it documents. Tests: 6 new for the credit, 4 for the guard. mem_cache runs 8 failed / 1441 passed on this branch and 8 failed / 1408 passed at the merge base 332f083 -- same failures, 33 more passing. ruff clean.

…e cutover that uses them The flip's TP stack is built at boot, but the PP phase runs on the boot stack and never executes a TP or draft forward. Two of that build's posts are therefore PHASE-COLD -- the attention-backend workspaces and the decode CUDA graphs -- and on rank 0 of this rig they measure 2294 MiB, which is the same number as the pool's shortfall against the 669k plain-TP reference. The sizer learned to express that credit in the parent commit; nothing passed it. This passes it, and pays for it. ONE PREDICATE DRIVES BOTH SIDES. phase_flip_spill.cold_stack_deferred is asked by the KV sizer before it solves the pool and by the boot before it decides whether to build. They cannot disagree, which matters because the direction that disagrees silently -- credit taken, deferral not performed -- sizes a pool into memory the same boot then allocates. That is the sgl-project#678 OOM class, and no test of either side alone can catch it. THE CARRIER PIN TRAVELS WITH THE CAPTURE. Rung 2's pin asserts that the draft parameters still lie inside the spill carrier at capture time. Deferring the capture while leaving the pin at the boot site would have checked an address set that nothing had baked yet -- a green assertion about nothing. It moved into build_cold_stack_posts, and it deliberately does not latch the built-flag, so a refusal cannot let a retry skip the capture entirely. THE RESTORE IS PRICED WHERE IT IS PERFORMED. The build happens inside _cutover, past the point of no return -- the same shape that took all three ranks down on 2026-08-09 when rung 2's re-commit was unpriced. _cold_stack_restore_bytes charges it into the pp->tp affordability verdict, SUMMED with the draft restore rather than max()'d with it, because the cutover's own ordering makes them coexist: the weight pages go back first precisely so the capture has addresses to bake. Both ends use the same measured constant, so the pool cannot be sized against one number and funded against another. IT IS A ONE-TIME MOVE, NOT A PER-FLIP SPILL, and that is what overturns the recorded refusal honestly. That refusal objected to a per-flip RE-CAPTURE, measured at 41% of decode throughput. A boot deferral pays no re-capture: the builder latches, so the second and every later pp->tp leg is a no-op. The GraphFamilyRegister park/onload path would be a different rung; it has no production caller today and none is added here. THE RUNG IS STILL REFUSED. IMPLEMENTED_DEPTH stays at DEPTH_ARENA_TAIL, so cold_stack_deferred answers False for every configuration reachable without the new SGLANG_PHASE_FLIP_SPILL_DEPTH_EXPERIMENTAL hatch, and the instance serving on --phase-flip-spill-depth arena is bit-identical under this commit. The hatch exists only to break the deadlock between "promote after metal" and "cannot reach it while unpromoted"; it warns on every resolution and is an env rather than a server arg, so an operator sweeping the depth flag keeps hitting the refusal. Promotion follows a real flip cycle, not this commit. THE CREDIT IS NOT YET SAFE TO SPEND, and this commit does not spend it. Under the first real agent load the 525462-token form OOM'd on rank 0 inside the first tp_to_pp flip: F.linear in the GDN in_proj_qkvz asked for 256 MiB with 131.69 MiB free. So the decode forward's transient is UNPRICED in today's sizing, and handing the deferral's 2294 MiB straight to the pool would reproduce that OOM with a larger pool behind it. The transient is a separate sizing post and lands separately; until it does, the hatch is the only way to reach this rung and the acceptance gate for it runs under load. Tests: 140 passed -- 33 new red-first tests across two files (predicate, interlock, hatch, sizer threading, builder contract, carrier-pin travel, idempotency, cutover restore, seam pricing), plus the existing phase-flip boot and runtime suites. Both new files carry CAN-FAIL guards aimed at the rung the rig actually serves on, and structural tests so a stray direct init_cuda_graphs or an unconsulted priced term fails. Both are registered with register_cpu_ci; the CI runner executes `python3 <file> -f`, so function-style pytest files were never being collected. Scoped regression over the 119 test files that reference any touched module: 2153 passed. A full failure-set comparison against the parent commit in a separate baseline worktree found 31 failures on the parent and, initially, 32 here. The one extra was mine and is fixed in this commit: test_phase_flip_arena_tail_631 pins the SOURCE TEXT of the _staging_bytes return, so that the arena tail stays ADDED to the max() and never becomes a third argument of it -- a difference one measured corridor breach wide and invisible in a value. Reformatting had collapsed the return onto one line and broken the pin without touching the arithmetic; the multi-line shape is restored and commented so the next reformat does not silently do it again. With that, the two failure sets are identical in both directions.

fallback when sampling failed

11d3369

zhyncs requested review from hnyls2002 and yzh119 July 20, 2024 16:49

zhyncs approved these changes Jul 20, 2024

View reviewed changes

merrymercy reviewed Jul 20, 2024

View reviewed changes

fill nan

87bdf79

Ying1123 merged commit 0ac94c3 into sgl-project:main Jul 20, 2024

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Fallback when sampling failed (sgl-project#678)

6f15316

shiyu7 pushed a commit to shiyu7/sglang that referenced this pull request Aug 5, 2026

fix(openai): reject media input for text-only models (sgl-project#678)

afe7d83

Navigation Menu

Uh oh!

Fallback when sampling failed - #678

Fallback when sampling failed#678
Ying1123 merged 2 commits into
sgl-project:mainfrom
ispobock:handel_sample_failed

ispobock commented Jul 20, 2024

Uh oh!

merrymercy Jul 20, 2024

Uh oh!

ispobock Jul 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ispobock commented Jul 20, 2024

Motivation

Modification

Checklist

Uh oh!

merrymercy Jul 20, 2024

Choose a reason for hiding this comment

Uh oh!

ispobock Jul 20, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants