Update version to 0.1.22 by Ying1123 · Pull Request #677 · sgl-project/sglang

Ying1123 · 2024-07-20T10:39:14Z

No description provided.

…get towards it THE REMAINDER OF sgl-project#678, and it removes the approximation both previous failures came from. A SUBTRAHEND CANNOT STATE THIS CONSTRAINT. The requirement is "every rank must REST at or above floor + margin, or it can never arm a flip and the pool is fixed at boot". What the pool gives up and what the card ends up holding free are related by the sizer's other posts, and the gap between them is exactly where this ticket's two failures lived: 284181 tokens the subtrahend double-charged a floor the seam solve had already reserved 537076 tokens the double charge removed, and two of three ranks came up BELOW their floor -- 987 MiB against 1536, with the pre-arm ladder finding 46 MiB of a 650 MiB gap Both are one error in opposite directions: a quantity that must be SOLVED FOR was being adjusted TOWARDS. WHAT WAS MISSING WAS THE FREE COLUMN ITSELF. ``have_bytes`` is already net of the band floor and of the rung credit (``free - band_floor + rung_fund``), so the raw resting free could not be recovered from the record. It is persisted now, with the rung credit beside it for audit, and the constraint becomes one line: free(T) = free_at_measure + (id_space - T) * cell T <= id_space + (free_at_measure - floor - margin) / cell Exact, and it needs no model of the activation reserve, the capture peak, the arena or the carve-out -- all of them were resident when the column was measured, which is the argument ``seam_allowed_tokens`` already makes for its own anchor. TWO CONSTRAINTS, ONE MIN, NEITHER SUBTRACTED FROM THE OTHER. The seam must be fundable AND the card must rest above its floor. They bind on different ranks at different vectors, so the pool is the smaller of the two id spaces. HERMETIC VALIDATION AGAINST THE TWO BRACKET BOOTS, which is the whole reason those boots were worth their windows: the 482490 bracket accepted on every rank -- it flipped both ways on metal, so a solve that refused it would be wrong the 537076 bracket rejected by exactly ranks 1 and 2, the two measured below their floor solved pool 498310 tokens, 90.6% of the 550000 pin, CLEARING the 495000 bar the subtrahend could not reach, and 38766 below the bracket that could not hold the floors The rank-to-card pairing in the fixture is evidenced, not assumed: rank 0 carries the 31800 MiB budget only the 5090 can hold, and of the two 3080s only one pairing is consistent with 482490 having flipped -- the other puts a card 300 MiB under its floor at a pool that demonstrably worked. BACKWARD COMPATIBLE: a cold record, a missing cell or a record written before the free column existed returns None -- explicitly not zero, which would be a verdict -- and the caller falls back to the subtrahend, the previous arithmetic exactly. Tests: 55, twelve new across two classes. Can-fail proven by three mutations: disabling the direct solve fails 8 cases; dropping the floor target from the solve fails 5; and dropping the floor ceiling from the min fails the floor-binding case -- which a mutation caught missing, because every other test in the class happened to have the seam binding. That is how a guarantee ends up computed and discarded, and it now has its own falsifier. 966 pass across the touched suites, zero failures. NOT BOOTED. Desk-only by instruction; the validation boot is scheduled by the operator, bundled with sgl-project#677 and the router reload. ONE THING THE BOOT MUST WATCH: the solve targets the RESTING free column, while an arm happens under load. On the 482490 boot the corridor guard sampled rank 1 at 1554 MiB against its 1633 MiB floor mid-prefill -- a ~600 MiB load dip against a 192 MiB load margin. It flipped anyway, because the seam gate's own ladder pays at seam time, but if the boot shows arms refused for the floor while the resting column is correct, the margin is the term to measure, not the solve.

…roject#677 to compose against The sgl-project#679 close-out said the retry net's registry is empty because the reliefs that could genuinely pay are collective and belong at admission. This is that claim made concrete, written so sgl-project#677's hysteresis-drain design can consume it without re-deriving anything. WHAT IT ESTABLISHES. Four reliefs exist and only four: radix eviction, the sgl-project#287 ladder's admission_cap, kvso try_spill, and retract_decode. For each: what it frees at the admission decision point, what it costs, and how it is invoked without splitting the group. Two of them are not what one would guess: the sgl-project#287 ladder is TOO SLOW to be a rung. Its consensus boundary is every 8 rounds; a chunked-prefill burst exhausts the pool in fewer. It is the slow outer loop -- and the decode path already uses it that way, throttling before retraction to stop the freed slots being handed straight back. kvso try_spill is the BEST rung, not retraction. It frees a bounded, chosen amount (the victim's block-aligned tail overhang), is already driven from the reduced value, and costs no request's progress. Its bound is the host region supply, and exhaustion is a reachable state the decode path already documents. THE ITERATION ORDER, VERIFIED, AND IT IS FAVOURABLE. The reduce runs at scheduler.py:4777, unconditional and pre-branch by its own comment; admission at 5089; retraction at 5950 inside update_running_batch (5818). So a ladder at admission reads an ALREADY-AGREED number and needs no collective of its own, while retraction is genuinely downstream -- which is why sgl-project#679's crash had nothing to fall back on, and what rung 3 must work around. THE ORDER, mirroring the decode-OOM branch rather than inventing a second shape: evict (baseline) -> try_spill -> throttle -> retract_decode -> PARK. Parking stays the floor of the ladder, not its replacement. THE COMPOSITION CONTRACT WITH sgl-project#677, stated in three rules because both mechanisms decide admission from pool headroom: 1. The park guard is the FINAL authority, the drain gate the prior one. A phase gate cannot make memory exist; it may narrow what the park guard allows, never widen it. 2. Any headroom quantity sgl-project#677 branches on must be the GROUP-PUBLISHED floor. A rank-local reading in the drain gate reintroduces the sgl-project#603/sgl-project#583 divergence class upstream of every safeguard sgl-project#679 added -- and a park guard reading the reduced floor does not protect a gate that reads a local one. 3. Parking must be reachable from every path that reaches alloc_for_extend. AND THE LIKELIEST COMPOSITION FAILURE, called out so it can be tested on both sides before it is met on metal: hysteresis and parking can beat against each other. A parked chunk schedules ZERO tokens, so a drain that counts admission ATTEMPTS rather than SCHEDULED TOKENS will believe work was taken, hold its hysteresis, and deadlock against the park at exactly the pressure where both are needed. Section 5 lists what this note does NOT close: the ladder is unbuilt, rung 3 is a real refactor, and rung 1's host-region bound has never been measured under the 5-lane load that produced the crash. No code change. Every claim carries its file:line.

…a tombstone leaf THE DIAGNOSIS IN 7752dc8 IS WRONG, AND CORRECTING IT CHANGES THE FIX. It said tokens behind a LOCKED chain are counted but unreachable, so the counter promises what the actuator cannot pay. The second clause is right. The first cannot happen: `inc_lock_ref` / `dec_lock_ref` walk from a node to the ROOT, and `_split_node` copies `full_lock_ref` onto the new upper half. So `full_lock_ref(parent) >= full_lock_ref(child)` holds on every edge at every moment. An unlocked node therefore has no locked descendant, its whole subtree is unlocked, and the peel always reaches it. Measured on the 01:46:10 tree itself, reconstructed from the dump all three ranks printed: of 65766 evictable tokens, 65254 sat in fully-unlocked subtrees, and the 512-token remainder is an artifact of one mis-ordered line in the interleaved three-rank output. The locked-chain term is ZERO. Subtracting it from the admission budget -- the repair this chain was pointed at -- would have moved the number the scheduler admits against by nothing, and the crash would have reproduced unchanged. That is why it is not what this commit builds. The reconstruction is not taken on trust: it is checked against two totals the process printed independently, `#full_tokens: 140683` and `full_evictable_size_=65766`, and it matches both to the token. THE GAP IS AT THE OTHER END OF THE FRONTIER. `evict_full` SELECTS with `get_leaf_lru_no_lock` -- unlocked and childless -- but `_evict_leaf_node` CONSUMES only nodes with a mamba value, and asserted when one was missing. An unlocked mamba TOMBSTONE leaf satisfies the selector and violates the consumer, and the cache produces that state itself: `_iteratively_delete_tombstone_leaf` breaks on `node.parent.full_lock_ref > 0`. A tombstone that loses its last child while a request holds it survives as a LOCKED tombstone leaf. When that request finishes nothing revisits it, so it becomes unlocked, childless, counted in `full_evictable_size_`, and first in line at the frontier. The 01:46 tree held exactly one -- node 5937, fr=0, mv=None, childless, in the full LRU list -- beside a single payable leaf, 5959. Replaying that dumped tree through the deployed code selects 5937 and dies on `AssertionError: leaf node mamba value is not None`. Only 6 of its 290 nodes held a mamba value at all; 128 of the 130 unlocked nodes were tombstones. This is a crowded state, not a freak one. THE REPAIR IS ON THE ACTUATOR, NOT THE COUNTER. Freeing an unlocked tombstone leaf is not a new capability: it is the same deletion `_iteratively_delete_tombstone_leaf` already performs one step earlier, taken now that the lock which deferred it is gone. Both routes now go through one `_free_tombstone_leaf`, so `full_evictable_size_` and the LRU list stay in step by construction rather than by two copies of the same five lines. Raising the actuator rather than lowering the counter is the stronger closure of "a counter must never promise what the actuator cannot pay": the promise becomes TRUE instead of becoming smaller, and every consumer -- admission budget, the sgl-project#679 park guard, the in-flight sgl-project#677 drain gate -- is repaired at once without any of them learning a second quantity. DELIBERATELY NOT EXTENDED PAST A LOCK. A LOCKED tombstone leaf is still refused and still uncounted; reaching behind a live reference is a different repair and stays filed. `test_a_locked_tombstone_leaf_is_still_refused` is the mutation proof that the new branch is gated on the lock and not on the tombstone alone. GROUP-UNIFORMITY needs no new channel. The branch is a pure function of replicated tree state -- the tree is a replica, `full_lock_ref` and `mamba_value` are replicated -- so every rank takes it on the same iteration. The existing `uniform_avail_floor` still decides WHETHER to evict; this only changes what the peel does once asked. No collective added. HONEST CAVEAT, ONE. Deleting a tombstone leaf cascades up through its now-childless tombstone ancestors, so a 512-token request can free far more: 54502 tokens on the replayed production tree. That overshoot is the pre-existing semantics of `_iteratively_delete_tombstone_leaf`, which is unbounded on the ordinary path too and exists to restore the "no tombstone leaves" invariant. Bounding it would leave the invariant broken, so it is left as it is and named here rather than discovered later. THE sgl-project#681 BACKSTOP STAYS, WITH ITS MECHANISM CORRECTED. The shortfall note also had a defect of its own: it read `tree_cache.evictable_size()`, which RAISES NotImplementedError on MambaRadixCache and SWARadixCache -- the very classes it was written for -- so on the crashing boot it would have reported -1. It now asks for `full_evictable_size()` first. Its text no longer asserts the falsified locked-chain mechanism and instead says what firing means now: a REGRESSION SIGNAL, a new class of node being counted that the peel cannot consume. WHAT IS STILL NOT EXPLAINED, SAID PLAINLY. The 01:46 process died with RuntimeError, not with the AssertionError the replay produces, and its own numbers say the tree was untouched by the eviction that ran immediately before (available 273 and evictable 65766 both unchanged at the raise). Every branch that could skip that eviction was checked and excluded: `is_chunk_cache` is False, `disable_radix_cache` is False, and `uniform_avail_floor` is None on this boot because tp_size=1 takes the single-rank early return. So a third mechanism remains unidentified. The receipt added in 7752dc8 is the instrument that will name it on the next boot; this commit removes one guaranteed way for the next eviction on that tree to kill the group, and does not claim to be the whole story. TESTS. `test_evictable_reachability_681`, 9 cases, CPU-only, no GPU: - RED FIRST on 7752dc8: `test_the_frontier_pays_the_tombstone_leaf_...` and `test_the_counter_never_over_promises` both fail with the production assert, `leaf node mamba value is not None`. Green after. - The ancestor-closure proof runs on MambaRadixCache and on the base RadixCache, across a lock, a deeper lock, and a split under a lock, with `test_the_detector_can_fail` hand-building the shape sgl-project#681 assumed so the three green assertions cannot be satisfied by a detector that always returns empty. - `test_the_state_the_crash_tree_was_in_is_reachable` reaches node 5937's exact signature through public transitions only. - `test_the_ordinary_path_is_untouched` pins the no-tombstone peel. Suites: 79 passed across the 681 file, both sgl-project#679 files and test_mamba_lock_ref_pairing_581.

…hen it drains THE WEDGE, LIVE, 2026-08-16 06:04. Four minutes into a pure-drain PP boot the instance stopped serving and the user saw a dead server: last prefill batch admitted 06:04:21 pending prefill 403779 tok, FROZEN from 06:04:47 onward running bs 4 (= max_running_requests, all carried decodes) policy line "holding in pp: prefilling in pp (403779 tok pending) (pending prefill 403779 tok, running bs 4)" Every slot was held by a carried decode; PP may not decode under strict purity, so no slot could free; admission needs a slot, so no chunk could land; and the DRAINED rule waits for pending to fall below one chunk, which it never would. The policy was not wrong about any single fact -- it was waiting for an event that could no longer happen. WHY THE EXISTING EXITS DID NOT COVER IT. DRAINED fires on `pending <= pp_exit_tokens`, and pending was 403779 and frozen: unreachable, not merely distant. The DECODE-STARVATION CAP is solved from `decode_stall_slo_s`, which the user's pure-drain decision sets to no stopwatch -- deliberately, because a real drain must never be cut short by a clock. With the cap off, nothing bounded the wedge. SO THE MISSING EXIT IS NOT ANOTHER TIMER, and that distinction is the whole point. It is the observation that PREFILL ITSELF STOPPED MOVING while decodes were carried, which separates a wedge from a slow drain without appealing to a deadline. Any single admitted chunk resets it, so one chunk per window holds PP forever: this rule cannot cut a genuine drain short, however long it takes. That is the property the pure-drain decision requires and it is pinned by three tests, including a backlog that decreases by ONE token per window. PROGRESS IS MEASURED, NOT INFERRED. `observe_idle` records the last tick at which pending strictly DECREASED; `decide` stays pure and reads the stamp -- the same split the idle clock already uses, and for the same reason: a decision that measures its own history is not reproducible from its inputs. A phase change restarts the clock, so a wedge must be demonstrated in THIS residency rather than carried in from the last one. THE WINDOW IS SOLVED, NOT SET. One chunk takes `pp_exit_tokens / pp_prefill_tok_s` at the measured rate, so PROGRESS_STALL_CHUNKS (3) of those passing with nothing admitted is not a slow drain -- it is a drain that has stopped. FLOORED AT `2 * flip_cost_s`, because a cutover is the one interval in which prefill legitimately makes no progress: without the floor a fast rig could solve a window shorter than its own seam and exit on the seam it just paid for, arming a flip because it was flipping. An unusable rate returns 0 and disables the rule rather than dividing by zero -- a wedge-breaker that wedges would be worse than none. CHECKED BEFORE THE RESIDENCY CAP, so the 180s SGLANG_PHASE_POLICY_DECODE_STALL_SLO_S mitigation now deployed stays as the OUTER backstop and this fires first. Pinned: `test_the_progress_exit_beats_the_decode_stall_cap` asserts both that the cap is still configured and that the solved window is reachable before it. A DISTINCT RECEIPT, so the next audit can COUNT wedge exits against drain exits instead of inferring which happened: "blocked admission: pending frozen at N tok for S s with bs B carried (no chunk admitted in W s, solved as 3x the <chunk>tok/<rate>tok-s chunk cadence, floored at 2x<seam>s seam) -- exit condition: blocked admission" PURELY ADDITIVE: 100 insertions, 0 deletions, on the running commit. No existing exit, threshold or clock changed behaviour. TESTS. 12 hermetic cases, red-first (all 12 failed on the missing API): the frozen-pending wedge exits and names its numbers; a strictly decreasing backlog never triggers it across ten windows; a one-token-per-window drain survives; progress resets the clock so a stall must be fresh; an empty backlog is still the DRAINED exit; bs 0 is not a wedge; the window is a multiple of the chunk cadence; seam time alone can never trigger it; an unusable rate disables the rule. Suites: 159 in the deploy tree across this file and all four existing phase-policy files; 231 in the development tree.

…parked carrier NOT WIRED YET, AND SAID SO UP FRONT. This is the pure decision core with its pins; the scheduler wiring (park at the finished-prefill boundary, re-admit at TP entry, feed the slot bound into get_num_allocatable_reqs) is a separate change and this module currently has no caller. A module nothing calls measures nothing -- that is why it is labelled rather than claimed. THE WEDGE WAS A COUNTING DEFECT. At 2026-08-16 06:04 the instance held twelve GDN slots with EIGHT FREE, four running against a cap of four, and 403779 tokens of prefill it could not admit. Freeing a GDN slot would have relieved nothing: admission is `min(pp_max_micro_batch_size, limiter.current) - running_bs` then `min(..., req_to_token_pool.available_size())`, and HybridReqToTokenPool does not override available_size, so that second term is the REQUEST-slot count. Neither term sees the GDN pool at all; the mamba allocator is consulted only later, inside alloc_req_slots. What blocked admission was four requests PP is FORBIDDEN to decode being counted against the concurrency cap for the whole residency. NOTHING MOVES IN PHASE 1. The carrier keeps its GDN slot, its KV -- exactly the KV that would have been resident anyway -- and its req_to_token row. That is deliberate: no state movement means no new correctness surface from the sgl-project#450/sgl-project#444 verify-write family, the sgl-project#461 DEVICE_BOUND law, or sgl-project#551 GDN-Vacate x kvso. The blob park is phase 2, gated behind the sgl-project#551 read. BOTH BOUNDS ARE SOLVED FROM BOOT DIMENSIONING and phase 1 raises nothing: parked + running <= the GDN slot pool (12), refused EARLY and by name because alloc_req_slots would refuse it late anyway; and running_bs <= max_running (4) at all times including TP, with re-admission in capture-set-sized batches so every pool stays inside what it was built for. THE ONE BOOKKEEPING EDGE is that a parked request is still RESIDENT -- `resident_ids` exists for the pressure ladder and retract paths, so a parked carrier is neither double-counted (it is out of running_batch) nor invisible (it is in the resident set). A PARK FAILURE DEGRADES TO THE SAFETY NET, pinned: disabled or slot-pool-full, the arithmetic is byte-identical to the pre-change gate, the request stays a carrier and keeps counting, and the sgl-project#677 progress exit still breaks the stall. The failure mode is the behaviour it replaces, never a wedge. A HAZARD THE WIRING MUST ANSWER, found while building this and recorded here rather than discovered on metal: a parked request is outside running_batch, so sgl-project#682's `harvest_resident_batches` and the seam's KV reshard would not see it. Phase 1 keeps its KV, so the seam must still carry it -- `resident_ids` is the hook for that, and the wiring change has to use it or a parked request's KV is left behind in the PP layout at the cutover. 16 hermetic cases, red-first: the 06:04 scenario with four carriers parked and the fifth prefill admitting; a control stating the composition exactly (this is an ADDITIONAL slot-pool bound, not a replacement for the caller's gate); parked+running never exceeding 12; TP re-admission never exceeding 4 and FIFO so no carrier starves; both receipts naming id, set size and binding bound; the disabled path reproducing the old gate; and evacuate() handing every parked request back so a crash strands none.

…e layout that decodes WHAT THE USER SAW, LIVE. The blocked-admission exit broke the wedge but handed back TP windows that did not do their job: * `arming tp_to_pp: pending > N=7004` fired while carriers were still mid-decode -- the log shows tp_to_pp taken with running bs 2-3 -- so a decode bundle was cut in half by a backlog that is ALWAYS above N under purity; * prefill then ran inside the TP layout, so the carriers that survived the short window met a layout busy prefilling instead of finishing them. Five blocked-admission exits in one boot, at 400-500k pending: the exit kept firing because each TP window returned the same unfinished carriers. The exit was doing its job; the window it handed to was not. THE USER'S SEMANTICS ARE EXPLICIT -- prefill until empty, decode the bundle TO COMPLETION, prefill again -- and this makes the TP side match. 1. TP EXIT = DECODE DRAINED. Under drain mode `tp_to_pp` arms only when `running_bs == 0` and there is prefill worth returning for. The backlog stops being an exit condition, because under purity it is permanent: treating "pending > N" as a reason to leave means never finishing anything. The receipt names what was finished -- "decode bundle complete: B reqs decoded in S s -- exit condition: decode drained" -- with B captured at phase entry, since by the time a bundle drains there is nothing left to count. 2. NO PREFILL IN TP. `prefill_suppressed_in_tp` is consulted by `phase_purity.prefill_blocked_here` BEFORE the purity mode, deliberately: the deployed mode is prefill_in_tp (the 2026-08-14 correction that let the measured break-even N decide, which stands for its own workload), and drain mode is a different contract for this one. A window entered to finish a bundle must not admit the work it was entered to escape. CARDS PARTIALLY IDLE DURING TP IS ACCEPTED, and is the user's stated model. The alternative measured worse: a bundle that never finishes costs an extra round trip and returns the same carriers. OFF BY DEFAULT, gated on `SGLANG_PHASE_POLICY_DRAIN_MODE`. Every rule is byte-identical until it is set -- pinned by `test_drain_mode_off_is_byte_identical_to_today`, which asserts that a backlog above N still arms with a live bundle exactly as it does now. THE BACKSTOPS ARE UNTOUCHED. The 180s decode-stall cap and the #677a progress exit still sit underneath, both pinned. Drain mode changes which condition ENDS a healthy window, never what rescues a broken one. TWO SILENT-FAILURE TRAPS CAUGHT WHILE WRITING THIS, both the shape that has cost this chain real boots: * the purity hook reached for `scheduler.phase_policy_config`; the attribute is `phase_policy_cfg`. With a `getattr` default that is a feature which silently never fires -- the sgl-project#684 serving-tick NameError again. Pinned by `test_the_purity_hook_reads_the_real_scheduler_attribute`, which binds against the name the Scheduler actually sets. * a flag no boot can set is a flag that does nothing, so `test_the_env_knob_turns_drain_mode_on` pins the env wiring in both directions and that unset keeps the current behaviour. 16 hermetic cases: the backlog no longer cutting a live bundle and arming the moment it empties; the receipt naming bundle and duration; the FULL CYCLE -- PP drains to carriers, ONE flip, TP decodes all four to empty, ONE flip back, asserting exactly one arm each way and that the TP arm happens only at bs 0; prefill suppression at the policy level and at its call site; and the two backstops still live.

…he wedge was mine LIVE INCIDENT 2026-08-16 06:47:48, and the defect is in hot fix 2. The policy armed tp_to_pp every ~3 s and CorridorGuard refused the seam staging on two ranks with STATIC numbers, 76 refusals in a row: PP1 staging needs 1651 MiB -> want 2163 (+512 entry margin) free 2456, arming floor 1536 -> 293 short PP2 want 2858, free 3560 -> 702 short, "every provider is exhausted" PP0 cleared THE FLOOR GUARANTEES LESS THAN THE SEAM NEEDS. The wants (2163-2858) exceed the arming floor (1536) because staging scales with the live KV cells a 4-carrier bundle holds, while the floor was solved against a smaller draw. Every degradation had already stood down -- abandon cap, backoff, entry margin -- and it still could not fund. THAT ALONE WOULD HAVE BEEN A SLOW BOOT. It became a TOTAL wedge because of hot fix 2. Before drain mode, a refused tp_to_pp still prefilled in the TP layout, so the backlog drained slowly instead of not at all. Suppressing prefill in TP removed that fallback: an unfundable seam became an idle server with 727004 tokens waiting and nothing running in either layout. The rule hot fix 1 was built on -- a failure must degrade to the fallback, never to a wedge -- applies to hot fix 2, and I did not apply it. This does. THE YIELD. Once tp_to_pp has been refused DRAIN_SUPPRESSION_YIELD_AFTER times in a row, drain mode stops suppressing prefill in TP and the layout goes back to draining the backlog slowly. The threshold is 2 and is NOT a new number: the seam entry margin already yields after two consecutive abandoned attempts, and an instance should not wait longer to stop IDLING than it waits to lower its own guard. NOT A LATCH. `arm_refusals` is reset by the first successful arm, so an instance that recovers returns to the user's semantics by itself. This chain has spent four tasks removing one-way ratchets and is not adding a fifth. THE CALL SITE CARRIES THE COUNT, because a yield the hook never learns about is a yield that never happens -- the third instance of that shape this file has caught (the `phase_policy_config` attribute name, the env knob, now this). Pinned by `test_the_purity_hook_passes_the_refusal_count`, which drives the real hook with a state carrying 76 refusals. WHAT THIS DOES NOT FIX, and is the real sizing defect underneath: the arming floor does not cover a full bundle's staging. Candidates weighed: (a) chunk the staging so per-wave want fits under the guard -- the seam already runs in 16 waves, so this is a wave-size question, not new machinery; (b) derive the floor from max_running x per-req staging cells, which feeds sgl-project#676 pool sizing and costs pool; (c) bounded degradation. This commit is (c), chosen because it is the only one that unwedges a LIVE instance without a sizing change, and because the fallback it restores is the behaviour the instance had two hours ago. (a) is the structural fix and is the follow-up. 5 new hermetic cases: suppression holds while the flip is viable, yields at the threshold and at the incident's 76, the threshold matches the seam margin's own, the yield is not a latch, and the hook passes the count. 33 in the two sgl-project#677 files, 266 across every phase-policy/purity suite.

…valve outranks drain mode USER DECISION 2026-08-16: the ~1024 MiB corridor law is a SOFT target. It was introduced only because the planner was not filling VRAM well enough, it stays as the fill-quality target, and it stays the planner's job. It is not a safety device. The one hard constraint is OOM avoidance. THREE WEDGES IN ONE MORNING, all of them a fill-quality target stopping the machine, and the last two on my own fixes: 06:47:48 ensure_headroom REFUSED a seam whose staging FIT in free (PP1 want 2163 MiB against 2456) because the 293 MiB residual sat under the law. 76 refusals, 727004 tok waiting, GPU idle. 07:02:15 a rank WITHHELD its entry-margin yield on a PREDICTED 864 MiB trough and emitted a margin-delay tag that is exempt from the stand-down cap, so nothing bounded it. Delay streak 15/16/17, bs 0, GPU 0%, 794179 tok waiting. 1. THE VALVE OUTRANKS DRAIN MODE (phase_purity, phase_policy). My previous fix keyed the drain-mode yield on `arm_refusals`. The 07:02 wedge was DELAYS, not refusals -- a different counter, the same wedge, the fourth time this chain has shipped a path that was not told its state. `flip_unavailable_reason` already had the answer: it reads the seam's `_seam_abandons_in_a_row` (which delays DO advance, to 17) AND the policy's `arm_refusals`, against one bound. What broke was ORDER. Hot fix 2 was checked BEFORE `prefill_allowed_in_tp` and returned True first, so the valve never ran -- which is why the WITHHELD line's promise that "the purity valve lets the starved work class run meanwhile" was FALSE on metal. Asking the valve first makes that promise true, and both wedge shapes now leave through one door. ONE BOUND, NOT TWO: this deletes the DRAIN_SUPPRESSION_YIELD_AFTER threshold the previous commit introduced. A second number was one more thing that could be told the wrong state. 2. THE LAW WARNS INSTEAD OF REFUSING (corridor_guard, corridor_admission). `ok = (free_now - want) >= law_floor` becomes `ok = free_now >= want`, with `law_breached` carried out on the verdict so callers can warn. An allocation LARGER than free is still refused -- that is not a corridor dip, it is an OOM, and softening it would trade a warning for a dead worker. Two counters re-keyed, because "mattered" is no longer "refused": `host_blocked_count` (item 16's decision is as consequential as ever) and the admission gate's `cleared`/`short` (a fill-quality signal that would otherwise have reported a perfect corridor forever). 3. THE WITHHOLD WARNS AND PROCEEDS (phase_flip_runtime). No rank may delay or withhold the seam on a margin prediction. The branch now logs "CANNOT FULLY HOLD THE CORRIDOR FLOOR through this seam entry: predicted trough X MiB below the Y MiB law" and steps over it. Its old safety argument was self-defeating anyway: it justified waiting by pointing at the purity valve, but the valve opens on the stand-down cap that its own tag is exempt from. The prediction keeps its job -- it sizes the warning and will aim the pre-flip spill rung -- it just no longer stops the machine. This deletes the unbounded-delay wedge class. The yield also gained an edge-triggered receipt naming the reason, so any future occurrence of prefill-in-TP is loud in the log rather than inferred. SUPERSEDED TESTS ARE REWORDED AGAINST THE SAME NUMBERS, never deleted, so the change of policy is legible: the law's refusal test now asserts warn-and-dip at 900-of-1100 MiB, the host-tier tests assert the withheld TIER (which was always their subject) instead of the verdict they used as a proxy, and the withhold tests assert that the 1452 MiB measured draw now proceeds. TESTS: 6 new hermetic cases driving the guard with the wedges' real numbers (2163-of-2456 and 2858-of-3560 clear; 2000-of-1000 still refuses), 25 in the drain-mode file including both wedge shapes through the real hook, 857 across every corridor/phase/purity/seam/margin suite. STILL OPEN, and next: the pre-flip KV spill rung as the guard's FIRST provider ahead of allocator-cache, so the dips this commit permits become rare and small rather than routine.

…mall flip and unmeasured THE MEASUREMENT, from 291 same-regime PHASE-FLIP DONE lines (and 3955 unique across all logs). The part NOT covered by read+exchange+write is: 123 live slots total 2453 ms unaccounted 1999 ms 81 % 440095 live slots total 4290 ms unaccounted 2287 ms 53 % FLAT at 2.0-2.35 s across a 3600x range of occupancy (123 -> 440095 live slots, 708 -> 3.4M cells). Regressed: unaccounted ~ live slots has slope 0.0005 ms/slot and R2 0.255 -- i.e. it barely correlates with size at all, which is what makes it a fixed cost rather than a small-flip artefact. Total ~ live slots gives intercept 2754 ms, R2 0.871. So the fixed cost, not the movement, is what floors every window sgl-project#677's economics can solve, and it is the term sgl-project#692 must price depth against. WHERE IT WAS HIDING, and I had it wrong first. My initial reading blamed the per-wave backing swap (release_wave/restore_wave). It is NOT the residual: `t_write0` is taken BEFORE those calls and `write_ms` accumulates after them, so the backing swap has been inside `write_ms` all along. Reading the timer placement rather than the call order is what corrected it. The three timers cover the WAVE LOOP ONLY. What fell outside is the tail: _pool_census("pre-cutover") phase_flip_runtime.py:6538 for fn in self._pre_cutover_fns: fn(...) :6539-6544 EXTRA MOVERS -- the weights arena refill and the GDN state leg _cutover_fn(direction) :6545 the group step _pool_census("post-cutover") :6547 THE MOVERS ARE OCCUPANCY-INDEPENDENT BY CONSTRUCTION -- the weights arena refill is the same bytes whatever the KV live set holds -- which is the leading explanation for a residual that does not move with occupancy. That is a hypothesis this commit makes MEASURABLE rather than one it asserts. SO THE TAIL IS TIMED, split movers vs cutover because they have different fixes, and both are reported on the DONE line and in `last_stats`. A residual that has to be regressed across boots cannot be priced per flip; a reported number can, and sgl-project#677/sgl-project#692 both need it per flip. WHAT THIS COMMIT DOES NOT DO. It does not reduce the cost. The reduction candidate is the wave count -- `_flip_waves` (:4024) is "A PURE FUNCTION OF THE REPLICATED LAYER MAP AND THE DIRECTION", so W=16 is paid at 123 live slots exactly as at 440095, even though the staging transient waves exist to bound is trivial there. The module's own docstring already pre-authorises the landing spot: "each extra wave costs one more exchange round trip. If a measurement ever shows the round trips dominating, W=8 is the place to stand, not W=1" (:4083-4089). That A/B needs NO code: `SGLANG_FLIP_SEAM_WAVES` (:2568) already overrides W. It is left unshipped deliberately -- lowering W trades ms for staging MiB, which lands on the arming floor and the corridor, and sgl-project#602 showed that budget is regime-dependent. Hard-coding a policy from one regime's numbers is the mistake that ticket already made once. HONEST LIMIT ON THE PER-WAVE TERM. Fitting unaccounted = a + b*W across the W=4 and W=16 populations gives b ~ 66 ms/wave, a ~ 997 ms. That fit is CONFOUNDED: the W=4 samples are release-first and the W=16 samples restore-first, so it mixes wave count with regime and must not be quoted as a per-wave price. The clean number will come from the env A/B above, within one regime -- which is exactly what the new movers/cutover fields will report. TESTS. 9 cases, source-level because the alternative is driving a full flip: the movers clock opening before the pre-cutover census and closing after the mover loop; the cutover clock wrapping the cutover; the three original timers untouched; both fields in `last_stats` and in the DONE line. Plus an AST arity guard on the DONE line -- specifiers counted against arguments -- because a %-format mismatch raises at the moment the flip completes, the worst possible place to learn it. Falsified: dropping one argument makes it report "17 format specifiers and 16 arguments". managers 2093 passed, 0 failed (2084 before, +9). Hermetic. NO DEPLOY.

…e two commits to hold MERGE_NOTES_602.md rewritten to cover all 18 commits: what each fixes, its test evidence, whether it touches runtime, and whether an equivalent patch is already on the serving line (verified with `git cherry`, not by message matching -- four are: c41645c, ce60358, 658ea3a, 84b0171). DRY RUN: clean. Merged into `integration/r2` -- the live line, since the serving tree descends from its tip a73a0d8 -- in a throwaway worktree, `--no-commit --no-ff`, then aborted and the worktree dropped. Zero conflicts, zero unmerged paths, so nothing was pre-resolved because nothing needed it. Verified semantically as well as textually: on the MERGED tree, managers 2093 passed / 0 failed and planner 2574 passed / 2 failed, the two being the same pre-existing test_rejected_evidence_pins pair that is already red on the base. THE FACT THE OPERATOR NEEDS, and it is not in the commit count: `7936bc4850` is NOT an ancestor of integration/r2, so merging this branch drags in its whole base lineage -- 115 commits, of which 18 are mine and 97 are the hotfix/677 work (sgl-project#662 x20, [PhasePolicy] x18, sgl-project#677 x8, sgl-project#678 x7, sgl-project#679 x6, ...), 129 files, +22731/-762. Approving this merge is approving that lineage, most of which is not mine to vouch for. If only this work is wanted it must be cherry-picked rather than merged. NOT ATOMIC, and it splits cleanly into four groups with an order: (1) the four already on serving -- merging them only reconciles integration with what is already running; (2) the three sgl-project#624 test-only drift guards, which take managers from 4 failures to 0 and should land early so the line stays green during review; (3) desk tool + docs, all planner/pp_cut.py and markdown, imported by no serving path; (4) hold. HOLD, two commits, both runtime and neither on the serving line: * e21e87f (sgl-project#690) touches the seam hot path and changes the PHASE-FLIP DONE format. Already queued to land on deploy WITH the W=8/W=4 probe after the sgl-project#694 soak verdict; merging it into integration first puts it in front of the soak meant to measure it. * 5301b94 (sgl-project#685) touches the boot sizing path. Announce-only today and abstention-guarded, but unsoaked, and the R' decision it waits on is not made. Nothing else in the chain can move serving behaviour. Docs only; no merge performed, no deploy, scratch worktree removed.

…mall flip and unmeasured THE MEASUREMENT, from 291 same-regime PHASE-FLIP DONE lines (and 3955 unique across all logs). The part NOT covered by read+exchange+write is: 123 live slots total 2453 ms unaccounted 1999 ms 81 % 440095 live slots total 4290 ms unaccounted 2287 ms 53 % FLAT at 2.0-2.35 s across a 3600x range of occupancy (123 -> 440095 live slots, 708 -> 3.4M cells). Regressed: unaccounted ~ live slots has slope 0.0005 ms/slot and R2 0.255 -- i.e. it barely correlates with size at all, which is what makes it a fixed cost rather than a small-flip artefact. Total ~ live slots gives intercept 2754 ms, R2 0.871. So the fixed cost, not the movement, is what floors every window sgl-project#677's economics can solve, and it is the term sgl-project#692 must price depth against. WHERE IT WAS HIDING, and I had it wrong first. My initial reading blamed the per-wave backing swap (release_wave/restore_wave). It is NOT the residual: `t_write0` is taken BEFORE those calls and `write_ms` accumulates after them, so the backing swap has been inside `write_ms` all along. Reading the timer placement rather than the call order is what corrected it. The three timers cover the WAVE LOOP ONLY. What fell outside is the tail: _pool_census("pre-cutover") phase_flip_runtime.py:6538 for fn in self._pre_cutover_fns: fn(...) :6539-6544 EXTRA MOVERS -- the weights arena refill and the GDN state leg _cutover_fn(direction) :6545 the group step _pool_census("post-cutover") :6547 THE MOVERS ARE OCCUPANCY-INDEPENDENT BY CONSTRUCTION -- the weights arena refill is the same bytes whatever the KV live set holds -- which is the leading explanation for a residual that does not move with occupancy. That is a hypothesis this commit makes MEASURABLE rather than one it asserts. SO THE TAIL IS TIMED, split movers vs cutover because they have different fixes, and both are reported on the DONE line and in `last_stats`. A residual that has to be regressed across boots cannot be priced per flip; a reported number can, and sgl-project#677/sgl-project#692 both need it per flip. WHAT THIS COMMIT DOES NOT DO. It does not reduce the cost. The reduction candidate is the wave count -- `_flip_waves` (:4024) is "A PURE FUNCTION OF THE REPLICATED LAYER MAP AND THE DIRECTION", so W=16 is paid at 123 live slots exactly as at 440095, even though the staging transient waves exist to bound is trivial there. The module's own docstring already pre-authorises the landing spot: "each extra wave costs one more exchange round trip. If a measurement ever shows the round trips dominating, W=8 is the place to stand, not W=1" (:4083-4089). That A/B needs NO code: `SGLANG_FLIP_SEAM_WAVES` (:2568) already overrides W. It is left unshipped deliberately -- lowering W trades ms for staging MiB, which lands on the arming floor and the corridor, and sgl-project#602 showed that budget is regime-dependent. Hard-coding a policy from one regime's numbers is the mistake that ticket already made once. HONEST LIMIT ON THE PER-WAVE TERM. Fitting unaccounted = a + b*W across the W=4 and W=16 populations gives b ~ 66 ms/wave, a ~ 997 ms. That fit is CONFOUNDED: the W=4 samples are release-first and the W=16 samples restore-first, so it mixes wave count with regime and must not be quoted as a per-wave price. The clean number will come from the env A/B above, within one regime -- which is exactly what the new movers/cutover fields will report. TESTS. 9 cases, source-level because the alternative is driving a full flip: the movers clock opening before the pre-cutover census and closing after the mover loop; the cutover clock wrapping the cutover; the three original timers untouched; both fields in `last_stats` and in the DONE line. Plus an AST arity guard on the DONE line -- specifiers counted against arguments -- because a %-format mismatch raises at the moment the flip completes, the worst possible place to learn it. Falsified: dropping one argument makes it report "17 format specifiers and 16 arguments". managers 2093 passed, 0 failed (2084 before, +9). Hermetic. NO DEPLOY.

…-layout divisor pp_max_micro_batch_size auto-computes as max_running_requests // pp_size. That is correct for classic PP: the stages run micro-batches of one batch, so each may hold only its share. It is wrong under the phase flip, because DECODE never runs in the PP layout -- it runs in the TP layout, which has no pipeline to divide by. The prefill layout's divisor was throttling the decode phase. On this deployment the effect is total: max_running_requests=4, pp_size=3, so the default was max(4 // 3, 1) = ONE. MEASURED BEFORE THE FIX, under a sustained depth-5 load for 90 s: #running-req over 1,813 prefill rounds: 0 -> 973, 1 -> 792, 2 -> 48 66 decode batches, 9 completions, 0 BOTH-BLOCKED events Read that carefully, because it corrects the framing this task started with: NOTHING WAS DEADLOCKED. Requests completed, the box flipped, eviction was never defeated. The scheduler was faithfully obeying a cap of one against a configured ceiling of four. The "90.6% of prefill rounds with #running-req: 0" symptom is under-parallelism, not a wedge -- and #running-req 0 on a PREFILL line is anyway normal when nothing is decoding, which is most rounds when only one request may decode at a time. The flip branch returns the full max_running_requests. It is not unbounded: get_num_allocatable_reqs still mins it against the admission limiter, the request-slot pool, and the mamba/GDN state headroom, so the state pool stays the real ceiling. This only stops a divisor from pre-empting all three. Extracted to a pure helper so the rule is testable without booting a scheduler. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): test_the_live_config_no_longer_caps_at_one ... 4/3/flip -> 4, was 1 test_classic_pp_is_unchanged ................. CAN-FAIL: the division must survive where it is correct ((4,3)->1, (12,3)->4, (8,2)->4, (7,2)->3). A fix that simply stopped dividing passes the first test and fails this. test_never_returns_below_one, test_pp_size_zero_does_not_divide_by_zero test_the_scheduler_uses_the_helper ........... wiring pin test_the_call_site_is_actually_reached_from_init -> 33 passed, 12 subtests, across the 701 + 698 + 677 scheduler suites. ruff clean in both changed regions; the file's other findings pre-exist. The wiring pin earned its place immediately: its first version asserted the call lived in __init__ and went RED, because the call site is init_model_worker. That is the same placement trap that broke init_parked_decode_set in sgl-project#677 -- pinning the wrong method is how a fix gets believed without ever running. The second pin asserts __init__ still calls that method, so the first cannot be satisfied by dead code. Not yet validated on metal: the next boot re-runs the depth-5 load and the claim is that decode concurrency rises above 2.

…t and load A static window is wrong in both directions -- too short at high load (the backlog never clears) and too long at low load (decodes wait behind an empty prefill window). planner/phase_window.py solves it. ## The amortization argument Over a cycle C = T_p + T_d + flips*F the work arriving in C must clear in C, so T_p = rho_p*C, T_d = rho_d*C and: stability floor C >= flips*F / (1 - rho) latency ceiling C <= (budget - F) / rho_d The floor is where the backlog stops growing however the windows are split; the ceiling is the wait a request meets arriving just after the prefill window shuts. Flip overhead is flips*F/C, which FALLS as the cycle lengthens, so throughput always wants a longer window and THE ECONOMIC CHOICE IS THE LARGEST ADMISSIBLE CYCLE, floored by stability. Not a midpoint, and not a constant. ## The sharpest result, and it reprices sgl-project#690 The two constraints move in OPPOSITE directions with F: the floor rises as flips*F while the ceiling falls as -F/rho_d. A dearer flip does not merely add an overhead line -- IT CLOSES THE FEASIBLE BAND FROM BOTH ENDS, and past some F the band shuts entirely: no window length works, at any split. On this rig with a 10 s TTFT budget: F=2.0 rho=0.30 -> cycle 53.3 s, overhead 7.5% F=2.0 rho=0.50 -> cycle 32.0 s, overhead 12.5% F=3.0 rho=0.50 -> cycle 28.0 s, overhead 21.4% F=4.2 rho=0.50 -> cycle 23.2 s, overhead 36.2% any F, rho=0.80 -> REFUSED, floor above ceiling So at rho=0.8 this rig is already refused at every measured flip cost. Halving the flip cost does not halve an overhead; it REOPENS CONFIGURATIONS THAT ARE CURRENTLY IMPOSSIBLE, which is a far stronger argument for sgl-project#690 than "2-4 s is slow". ## Two refusals, because a policy that quietly does the impossible is worse - THE SEAM MUST BE ABLE TO ARM. If the layout's free column no longer clears its arming floor there is no flip to schedule at any window length. This composes directly with sgl-project#707's closed form and the n0 <= 51 depth bound it implies, and is checked BEFORE any arithmetic. - THE DECODE WINDOW MUST BE WORTH ENTERING. Batch formation (sgl-project#689) collapses toward size 1 below a queue threshold, so flipping early buys a fraction of the decode rate for a full flip cost. That is a floor on the cycle, C >= q/(lambda*rho_p), and AT LIGHT LOAD IT BINDS INSTEAD OF STABILITY -- exactly the regime where a static window over-flips. ## Discipline Every quantity injected; the rig's figures are calibration data in the test. A foreign profile (flip 0.05 s, rho 0.8, 2 s budget, 50 arrivals/s) pins the generality and lands at under 2% overhead, showing the policy is about the RATIO of flip cost to cycle rather than about this rig. 225 tests green, hermetic (CUDA_VISIBLE_DEVICES=""), ruff + codespell clean. test_phase_window_677.py (13): the window as a function of flip cost rather than a constant; the floor diverging toward saturation and refusing rho>=1 as a capacity problem; the band closing from both ends; a dear flip under load admitting NO window; the economic choice being the ceiling; overhead falling with cycle length; halving the flip cost worth more than an overhead line; the batch floor and the case where it binds instead of stability; the seam refusal; a pure-prefill regime with no ceiling; a foreign profile; malformed inputs.

…the MINORITY share The flip has been carried as a scalar (~2.0-4.2 s) with an unexplained residual. It is not a scalar: the runtime already reports a five-way split on every completed flip. 765 unique PHASE-FLIP DONE lines from the boot captures decompose it, with no boot required. read 30.6 ms 0.9% exchange 847.8 25.3% write 302.6 9.0% movers 1458.4 43.4% GDN state + weights arena refill (:1843-1846) cutover 670.7 20.0% group routing, owner refresh, rebuild, swap (:1147) residual 99.2 3.0% Median total 3357 ms (min 2401, max 4874 -- the reported band); both directions agree (pp_to_tp 3298, tp_to_pp 3417). THE PREMISE IS CONFIRMED AND THEN SOME. read+exchange+write -- the part everyone reaches for -- is 35.3%. Sixty-plus percent sits outside it, and the single largest component is MOVERS at 43.4%, which is the H2D copy of the target layout's weight image. The residual is ~3%, so the "unexplained" part is small and the named parts are the story. (Medians do not sum; the composite handed to the solver adds to slightly more than the median total. Per-event median of (movers+cutover)/total is 61.6% against 63.4% for the composite. Both labelled.) ## Levers priced in LOAD, not milliseconds Per sgl-project#677 flip cost sets the stability floor and latency ceiling in opposite directions, so a reduction reopens refused configurations. At a 10 s budget, even phase split: baseline F=3.36s rho_max 0.66 L1 overlap movers behind seam F=2.21s rho_max 0.77 (+0.11) scheduling L1 lower bound (fully contending) F=3.36s rho_max 0.66 (+0.00) L2 phase-uniform vector (#704b) F=2.21s rho_max 0.77 (+0.11) costs depth L3 cutover to observed minimum F=2.73s rho_max 0.72 (+0.06) mechanism unknown L1 + L3 best realistic F=1.58s rho_max 0.84 (+0.18) scheduling THREE FINDINGS, FIRST ONE ACTIONABLE: 1. L1 AND L2 DELIVER IDENTICAL FLIP SAVINGS (1150 ms, rho 0.66 -> 0.77) BUT L1 IS FREE AND L2 COSTS LADDER DEPTH (n0<=37). For flip cost specifically, #704b's phase-uniform vector buys nothing a pure scheduling change does not. It still earns its keep on sgl-project#703's cache-key problem, which L1 does not touch, so they substitute for THIS purpose only. Do L1 first. 2. L1'S SAVING IS BOUNDED [0, 1150] ms AND THE BOUND IS UNMEASURED. movers is H2D and on this no-P2P rig the rank-to-rank exchange stages THROUGH HOST, so both legs traverse the same PCIe direction and may contend. If they fully contend the overlap saves nothing. Which end holds is the single highest-value thing the confirming window can capture. 3. L1 + L3 MAKES rho=0.8 FEASIBLE. sgl-project#677 refuses rho=0.8 at every measured flip cost; at F=1.58 s the ceiling rises to 0.84. That is the concrete form of the repricing: not "save 1.8 s" but "the load the rig currently refuses becomes servable". cutover deserves instrumentation before optimisation: its 24x spread (43.5 to 1041.5 ms) is not the signature of a fixed cost but of a wait or serialisation. Sub-step timestamps would say which; the observed minimum is the honest target. 236 tests green, hermetic (CUDA_VISIBLE_DEVICES=""), ruff + codespell clean. Measured components are inputs; a foreign profile with a differently-shaped split pins that the ranking follows the numbers, not this rig.

… is not the bug MEASURED: a TEN-token prompt waited 31.64 s to first token. Sampled every 2 s across the whole wait -- 0 running, 1 queued, mamba_available 3 (never zero, 0/16 samples), KV hundreds of thousands free -- while the policy logged "BOTH BLOCKED ... 0 req resident, 22 tok pending". An 8-arm run put every TTFT between 11.87 s and 62.65 s with NOT ONE arm under 3 s, so this is the serving floor rather than an outlier. THE CANDIDATE I CARRIED IN IS DEAD. I suspected the message's pending count and the simulation's disagreed. They cannot: _idle_locked_inputs receives ONE pending_tokens argument and passes the SAME value to both _layout_admits calls, and the policy message prints the same figure. No denominator divergence exists. AND THE SIMULATION IS CORRECT. Replaying _layout_admits with exactly the measured numbers returns pp=True / tp=False, so _idle_locked_inputs would have returned (nothing_can_run=True, target_can_admit=True) and the policy would have ARMED THE FLIP. The refusal is therefore not a bad rule; the values the simulation reads in-process differ from what /metrics reports, and no amount of external sampling can show which -- I already spent a measurement window proving that. The terms have to be printed where they are computed. So this commit adds the instrument, not a fix. One line, on the refusal branch only (both layouts declined), rate limited to 5 s because the state persists for tens of seconds and healthy rounds must stay silent. It names every term the verdict was computed from: phase, running_bs, pending_tokens, both layout verdicts, post_evict_rows, allocator availability, mamba slots, chunk size -- and reports an accessor that RAISES as "RAISED <type>" rather than silently becoming 0, since a raising probe is one of the two ways slots can read zero. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): test_pp_admits_the_measured_idle_state ..... the complaint in one assertion test_tp_correctly_refuses_the_same_state ... counterweight: tp MUST refuse, else nothing_can_run is false and the refusal would be correct test_the_pair_would_have_armed_the_flip .... (True, True) on the real state starved pool / no state slot / no pending work still refuse -- the refusal must stay reachable for its genuine causes test_double_refusal_prints_every_term test_silent_when_the_target_can_admit ...... CAN-FAIL: a diagnostic that narrates healthy rounds is noise, and noise is how a signal gets filtered -> 8 passed; 118 passed + 7 subtests across 713 + 708 + 701 + 677 + 689 + flip-runtime. ruff clean in the changed region. ADJACENCY, named not absorbed (sgl-project#689 window formation, sgl-project#677 window economics): both suites are in the run above and pass. This touches neither -- it adds no branch to the arming decision and changes no input to it.

…or that exists The rung machinery already existed and was NOT rebuilt: layout_ladder.py already solved rungs as a Pareto frontier, indexed them by occupancy (Rung.admit_up_to_tokens) and derived hysteresis bands (descend_below_tokens / ascend_above_tokens). What was missing was the PRICE of a step, and the price was wrong by an order of magnitude. _solve_transitions charged a step as moved_layers x weight_mib_per_layer over the gating link. That prices a cross-rank weight mover, which does not exist and whose absence is explicit (regime_stages.py:100, REACH_NO_WEIGHT_MOVER). The actuator that DOES exist is PhaseFlipStacks.refill (phase_flip_boot.py:361, arena_refill :539, dst.copy_(payload) :576): a CONTIGUOUS host->device memcpy of a whole boot-baked arena image. The bytes on the wire are the same whether one layer moves or six. With sgl-project#690-rev2's 9614.9 MiB per rank over the measured links (13/13/6.4 GB/s, authoritative mapping) and nothing crossing a rank boundary -- so the refills run CONCURRENTLY and the slowest card sets the step -- a rung change costs 1575.3 ms: 10.7x to 21.3x the moved-layer estimate, and 38-79% of a whole phase flip (sgl-project#690's fixed 2.0-4.2 s). The band consequence is real but small (every ascend trigger drops ~30k tokens, under 1% of pool): the mispricing mattered for the DECISION, not for the guard. The consequence that changes how a ladder must be driven: the switch cost is CONSTANT IN THE DISTANCE TRAVELLED. Crossing twelve rungs one at a time costs 12 x 1.575 = 18.9 s; crossing them in one jump costs 1.575 s. A constant-cost actuator inverts the usual intuition -- rungs are choices of destination, not stations to stop at. Per-step value is also wildly unequal: on the draining leg at arena depth 40, one step buys 23% for 1.575 s (payback 8.4 s) and the next buys 1.8% for the same 1.575 s (payback 87.9 s). Arena depth is the master knob and a real trade, since the arena is sized for the deepest reachable rung and resident at every rung: deepest rank0 | rungs | roomiest pool | fastest | end-to-end payback 36 | 2 | 421,894 | 1.0071 | 224.7 s (dead) 38 | 5 | 418,848 | 1.1538 | 11.8 s 40 | 8 | 364,413 | 1.2532 | 7.8 s At depth 36 the ladder is economically dead; at 40 it is live but the roomiest rung has lost 13.6% of its pool purely for holding the option. A defect this found in my OWN new API: the first cut of solve_fill_ladder required ASCENDING fill levels. Rising fill only ever moves to roomier, SLOWER rungs, so the interface could express nothing but forced retreats and reported an infinite payback on every one -- the discretionary step, the one the function exists to price, was unreachable through it. Monotone in either direction now, and both legs are pinned by test. The two kinds of step are distinguished explicitly: ASCEND is MANDATORY (its alternative is not "stay fast" but "stop admitting", so infinite payback is correct and must not be read as "never do this"), DESCEND is DISCRETIONARY and is the only step sgl-project#677 decides. sgl-project#677 integration: a cut jump and a phase flip are both an arena refill, so they spend the SAME budget and a rank cannot do both at once -- the controller ranks them rather than running two policies. A discretionary cut jump is admitted only when it is a jump not a step, its payback fits sgl-project#677's backlog-derived window, it beats a flip for the same stall (a flip changes regime; a cut jump buys a prefill factor within one), and the hysteresis band already permits it -- economics are a veto on top of the guard, never a replacement. Mandatory ascends bypass all four. HONEST LABELLING: no cut is recommended and nothing is added to the user's morning list, which carries sgl-project#702's cut alone. Every pool and speed number is solver output on the STRUCTURAL free-bytes fixture the review gate found was fitted against the incumbent, and inherits the +-500 MiB unbooted arming-floor uncertainty (~7% at 8 attention layers). The cost side rests on measured inputs, but 1575 ms is an arithmetic prediction from measured bandwidth, not a measured switch: the ladder has never performed a rung change on metal. 9614.9 MiB is taken from sgl-project#690-rev2 and assumed uniform per rank. Test results (hermetic, CUDA_VISIBLE_DEVICES="", interpreter /spinning/htsglang-gpu/.venv/bin/python3): test/registered/unit/planner/test_fill_ladder_704a.py (new) 20 passed test/registered/unit/planner/test_layout_ladder_704.py 25 passed test/registered/unit/planner full suite 2825 passed, 123 skipped, 157 subtests passed, 0 failed (2805 before this slice) can-fail by breakage, two independent neuters: summing the per-rank refills instead of taking the max (i.e. denying concurrency) turns 2 tests red reverting the bands to the moved-layer estimate turns test_the_real_cost_makes_the_BANDS_STRICTLY_MORE_CONSERVATIVE red restoring returns 20 passed Desk only. No GPU, no serving, deploy tree untouched.

… directions MEASURED (sgl-project#713 quantisation table, 06:19). Demand-PULL already worked: C1 arrived 06:19:06.56 and was served by the tp_to_pp at :09 -- 3.14 s, one seam plus overhead. What failed was the step AFTER. The box flipped back at :12, so C2, arriving 06:19:09.71 just as PP began, was not served until :15: arm arrive first token TTFT served by C1 06:19:06.56 06:19:09.71 3.14 tp_to_pp @:09 C2 06:19:09.71 06:19:15.52 5.81 MISSED :12, waited @:15 C3 06:19:15.52 06:19:15.63 0.11 landed ON the :15 flip The layout left on a timer while the prefill that pulled it was still unserved, so TTFT quantised to whole cycles -- 0.1 / 3.1 / 5.9 s, nothing between. The same arriving-tokens signal that PULLS a cutover must also HOLD it until the work it pulled for is served. One rule, two directions, not two rules. LEVER (a) IDLE DWELL IS DELIBERATELY NOT BUILT, on the coordinator's recorded call, because the measurement removed its target: the idle box does not churn. Pre-merge showed ONE flip across 10 idle minutes; post-merge ZERO across 90 s idle (0 running, 0 queued). My earlier "10 flips in 30 s on an idle box" was measured while my own arms were arriving -- the box was not idle and I called it idle. A gate with no measured trigger is dead weight. BOUNDED BOTH WAYS, because an unbounded hold is a starvation bug wearing a fix's clothes -- it trades the prefill side's starvation for the decode side's: (a) hold cannot starve decode: past LAYOUT_HOLD_MAX_ROUNDS the layout releases even with prefill unserved, and says EXHAUSTED; (b) pull cannot preempt an unserved prefill: while a hold is live in PP the verdict stays a hold, whatever the decode queue looks like. The both-sides tie (work pending in PP and TP at once) is routed to the sgl-project#677 economic comparison rather than to the timer, and is bounded by the same rounds. Bound = 8 rounds, derived not round: a cutover costs ~2.6 s of seam (sgl-project#690 measured 2.56-2.59 s), so 8 rounds holds the layout well under the cost of leaving it. Holding longer than it costs to leave is never worth it. SAFETY PRECEDENCE: never mid-flip (a cutover in progress owns the layout); never against an unfunded seam (a pull that cannot pay is an abandon). Tests, hermetic (CUDA_VISIBLE_DEVICES=""): the C2 specimen holds; pp releases when nothing pends (can-fail: the hold must not become a permanent stay); tp pulls; mid-flip and unfunded outrank demand in both phases; both-sides names the tie; the bound releases at the limit and NOT before (can-fail: early release evaporates the C2 fix the moment any decode appears); degenerate inputs -> 14 passed + 17 subtests; 185 passed + 24 subtests with phase-policy, 708, admission-intake and flip-runtime. ruff clean. CAN-FAIL PROVEN BY MUTATION IN THREE DIRECTIONS: never-holds fails 13 of 14, unbounded-hold fails 1, ignores-unfunded fails 2. Reverted and re-verified. NOT WIRED into decide() yet -- this is the decision rule. Wiring changes live flip behaviour and wants its own review; the acceptance (quantisation table collapses) can only be measured once it is wired, so that is the next step and not a claim of this commit.

…ferable vs lazy-fundable The floor was measured once and frozen, never designed, and that is visible in the source rather than a matter of opinion: arming_floor = corridor band floor + seam_entry_reserve, where the first half is a stated policy with a stated tolerance (1024 - 20% = 819 MiB) and the second is one scalar whose own docstring calls it "the shipped allowance". Per-rank it decomposes exactly: rank 0 (5090) 1728 MiB = 819 + 909 seam draw rank 1 (3080 x4) 1825 MiB = 819 + 1006 rank 2 (3080 x8) 2467 MiB = 819 + 1648 Rank 2 holds 81% more than rank 0 with no recorded reason. The defect is not the size of the number, it is that it is ONE number: a monolithic holdback cannot be traded, because trading requires knowing which part buys what. STAGING, AND WHAT ACTUALLY FORCES IT. Under barlink a cross-card transfer is one PCIe crossing, so a host bounce (two) is strictly worse and is not offered there. But the reshard stages in VRAM on both sides regardless of transport, and rollback semantics are NOT why: _dist_exchange allocates a uint8 receive buffer per peer and sends a gathered contiguous buffer (kv_reshard.py:939-995), because the wire format is a flat byte stream per peer while the destination rows are scattered ids. The obstacle to BAR1 landing in-place is the SCATTER, not rollback and not write ordering -- so the cross-card staging term goes to ~zero only under a layout co-design that makes a peer's destination extent contiguous. That is the largest structural reduction available here. RESTORE, NEVER REBUILD adopted as a named invariant: the flip may pay copy time, never build time. Consequence: capture-moment workspace is a BOOT-time component, not a per-flip floor term (both layouts captured once, graph state parked to host and restored per flip, 40-85 ms band, sgl-project#464 coalescing pending). The weights refill already complies. TIME PRICE of host-bouncing, from the H2D rates I measured in sgl-project#690 rather than nameplate: 125 / 199 / 181 ms for the whole seam draw on ranks 0/1/2, i.e. 4-6% of a ~3.1 s flip to return 909-1648 MiB per card permanently. Stated as a FLOOR on the cost, not a wall-clock delta: the refill is already 41-52% of the flip and shares the same link, so bounces queue behind it -- worst on rank 1's x4 card, where reslotting remains the cheapest single intervention. NAMED AS UNMEASURED: the seam draw cannot currently be split into components; nothing records it per component. Until one instrument attributes the peak instant to (send buffers, receive buffers, graph state, allocator transient), any per-component trade is arithmetic on an undivided number -- the same error class as pricing a flip on intention rather than completion. Also records the two honest limits on the rebuilt evict rung (sgl-project#717): it now delivers less than it prices by design, so repricing must key on the DELIVERED amount; and the ~413 MiB it leaves on the table is recoverable only by evicting deeper than the cap. Desk analysis, no code change. Feeds sgl-project#702 repricing and sgl-project#677 economics.

…e rules outrank it Wiring for 59592d6, plus both review findings, plus three precedences the EXISTING TESTS taught me -- which is the substance of this commit. FINDING 1, THE TP MIRROR. The tp branch pulled unconditionally on pend>0, so after an EXHAUSTED release the next TP evaluation would pull straight back and decode would lose the layout before serving anything: the C2 defect with the phases swapped, degenerating under both-sides load to max_hold PP rounds, ~0 TP rounds and TWO seams per cycle -- strictly worse than the timer it replaced. The pull now yields to a RUNNING decode batch for MIN_DECODE_ROUNDS (2: the minimum that means "did not preempt mid-batch"), bounded so it cannot become a new starvation of the prefill side. The degenerate cycle is pinned by test, not by hope: release at the bound, then assert the pull does NOT come straight back. FINDING 2, THE COUNTER LIFECYCLE. hold_rounds_so_far was caller-maintained. next_hold_rounds() is now a pure function resetting on EITHER boundary -- phase change or pending reaching 0 -- and the defect it prevents is pinned directly: a stale counter of 8 carried across a phase change releases the very first hold of the next episode, i.e. the C2 fix evaporates for the arrival that needed it. FINDING 3, THE REASONS REACH THE LOG. The gate returns its verdict as the decision's reason, so it travels the same path BOTH BLOCKED already does -- that is how the boot's acceptance gets read. THE THREE PRECEDENCES, each taught by a test I broke: * sgl-project#688 idle-locked outranks it, exactly as it outranks sgl-project#689 formation: holding a layout that can build NOTHING is waiting inside a layout that cannot serve. * The DRAINED exit outranks it on sgl-project#669's economics: a residual ABOVE one chunk stays (which this hold wants anyway), but a SUB-CHUNK residual is finishing regardless and holding for it spends a ~2.6 s seam to save a fraction of a chunk. sgl-project#669 moved anti-pinning to the starvation cap precisely so the drain could exit. * The DECODE STARVATION CAP outranks it absolutely. That SLO is the system's guarantee that PP can never pin the server; this lever's round bound is only a secondary backstop. Vetoing the cap would put a local timer above a global guarantee and reintroduce the pinning it prevents. AND THE GATE IS PP_TO_TP ONLY, as sgl-project#689's is. My first wiring vetoed EVERY arm, including the idle return leg -- 11 tests red -- because the rules arm for reasons this lever cannot see. It may only convert the one arm that contradicts it. Recorded because three exemptions is a shape: if a fourth appears, invert to an allowlist rather than adding it. Tests: 22 passed + 17 subtests in the lever file; 221 passed + 24 subtests across phase-policy, idle-locked-arm, window-formation, admission-intake, 708 and flip-runtime. ruff clean.

…ribution, rank-2 root 1. RESTORE, NEVER REBUILD -- the pin (test_restore_never_rebuild_677.py). A fence patches the BUILD entry points (weights_arena.allocate_arena, weights_arena.pack_into_arena, torch.cuda.CUDAGraph) to raise, and the REAL production mover PhaseFlipStacks.refill runs under it on both legs, including the checksum-mismatch restore arm -- the one branch that touches the arena twice and is likeliest to reach for a rebuild. arena_refill is deliberately NOT fenced: it is the copy the flip exists to perform, and a pin that fenced it would pass by forbidding the work. Can-fail arms: every entry point is shown to actually raise, and a planted mover calling allocate_arena is caught. Pinning production rather than a re-implemented loop is the point -- a re-implementation keeps passing while production drifts (sgl-project#624). 2. PEAK-INSTANT ATTRIBUTION -- PhaseFlipRuntime._record_seam_peak, emitted on the sgl-project#605 flight-recorder channel (not a new one: it already carries the torch view, the NVML view and the boot id, and is append-only). Placed at _staging_affordable, the instant the flip's demand is weighed against free VRAM -- earlier the buffers do not exist, later the decision is already taken. Carries staging_bytes, refill_destination_bytes, graph_workspace_bytes, the reserve, driver free, allocator cached free, and a SIGNED unattributed_bytes. Two choices, both pinned and both mutation-proven: unmeasured components are None and never 0 (a zero reads as "costs nothing" -- the sgl-project#606 defaulted- measurement defect), and the residual is signed, because a negative one means the named terms OVER-count, a different defect that max(0, ...) would hide. Guarded end to end: an instrument on the seam path may cost a line, never a cutover. 3. THE RANK-2 ANOMALY -- the sgl-project#685 candidate is REFUTED. sgl-project#685 (0e50e48, f1774d7) is an UnboundLocalError use-before-bind on a variable named 'cell' in the cold seam-pricing branch; it has no arena-tail content and no 1456 MiB figure exists anywhere in source or records. The live candidate is arena GROWTH, from the sgl-project#690 image sizes: rank 0 PP 12619.6 TP 9614.9 growth 0.0 seam draw 909 rank 1 PP 9014.0 TP 9614.9 growth 600.9 seam draw 1006 rank 2 PP 7211.2 TP 9614.9 growth 2403.7 seam draw 1648 Rank 0's PP layout is the larger, so it never grows; rank 2 grows four times what rank 1 does. The ordering matches the draws exactly and no other per-rank term does. Stated as a CANDIDATE, not a finding: the excess over rank 0 is 16% of the growth on rank 1 and 31% on rank 2, so growth explains the order and not the size. Instrument 2 settles it without a dedicated experiment -- it emits exactly this quantity at the peak instant. FILED, NOT BUILT: TICKET_718_contiguous_destination_extent.md -- making a peer's destination extent contiguous so a BAR1 write lands in place and the receive buffer stops existing. The largest structural reduction available, and the only one that removes a buffer rather than relocating it; it touches the reshard wire format and the pool allocator together, so it belongs to a deliberate design pass. The ticket names what must be decided, what must not be assumed (it trades allocator freedom for staging bytes, and that freedom is what keeps admission working under fragmentation), and makes a live reading from instrument 2 its prerequisite. Tests, hermetic (CUDA_VISIBLE_DEVICES="", no GPU, no serving contact): test_restore_never_rebuild_677.py 7 passed + 3 subtests test_seam_peak_attribution_677.py 8 passed MUTATION PROOF: null->0 and signed->floored fails 3 of 8 FULL SUITE compared by failing NODE ID against base b786858, ANSI stripped (counts alone hid a real regression earlier in this branch): base 13 failed / 2181 passed; now 13 failed / 2208 passed regressions: NONE. The 13 are pre-existing. ruff + codespell clean on all changed files. Desk only. Live readings ride the later review boot.

…ead of documented 332cb3b shipped an 8-round bound that could never advance: the verdict took hold_rounds_so_far from its caller, next_hold_rounds computed the successor, and NOTHING wrote it back. Every evaluation saw round 0, EXHAUSTED was unreachable, and the SLO cap was silently the only backstop. A documented invariant with no wiring is the class this tree hunted twice this week, and shipping it inert on purpose would be worse than not having the bound. The counter is maintained in observe_idle, beside the idle and formation clocks, for the reason those live there: decide() is pure, so a bound driven from inside the decision can never advance. FIRST SIGHT COUNTS AS ROUND ONE. hold_phase is empty before the first observation, and treating that as a phase CHANGE would spend round one resetting -- every hold would be one round shorter than its bound claims. A genuine phase change still resets, which is the case the reset exists for. RED-FIRST, and it was genuinely red on 332cb3b: the counter stayed 0 across 5 rounds and the verdict still read "round 1 of 8" after 8, so EXHAUSTED could not be reached. Four tests drive the real observer: hold_rounds reaches N under a live hold EXHAUSTED is REACHABLE by running the bound (your added acceptance) the counter resets on a real phase change the counter resets when the work drains MY ERROR ALONG THE WAY, worth the note: the first patch anchored on "idle = inp.running_bs == 0 ..." which occurs in BOTH _decide_from_load and observe_idle, and it landed in the former -- putting a mutation inside the function documented as pure, where it also had no effect. The tests stayed red and showed it (counter 0, hold_phase ""), and I traced the insert's enclosing function rather than assuming the patch had applied. Anchor on something unique to the target, and verify WHERE a patch landed, not just that it applied. Tests: 225 passed + 24 subtests across the lever, phase-policy, idle-locked-arm, window-formation, admission-intake, 708 and flip-runtime. ruff clean.

…aim: the arena tails are measured and recorded The sgl-project#702 repricing under the cold-spill doctrine found the number I said did not exist. NOTE_677 section 8 stated "no 1456 MiB figure appears anywhere in the records or source". WRONG: managers/phase_flip_seam_reserve.py, record_path docstring, records the measured per-rank arena tails -- "1436 MiB on rank2 against 466 MiB on rank1 and 0 on rank0". 1436 rather than 1456, but plainly the figure the earlier candidate meant. I missed it by grepping the docs tree and the ticket number instead of the module that owns the quantity. The sgl-project#685 ticket attribution was still wrong (it is an UnboundLocalError); the NUMBER was real. It also settles, without a boot, what NOTE_677 left as a candidate. Against the standing pool reduction per rank -- 704 / 801 / 1443 MiB, i.e. arming_floor_subtrahend_bytes = floor - max(corridor law, already_reserved) -- the measured tail is 99.5% of rank 2's, 58% of rank 1's, and 0% of rank 0's. Rank 2's floor excess IS its arena tail. The repricing itself lands in the evidence tree (not a git repo): /spinning/evidence-665-f1/NOTE_702_CUT_TABLE.md sections D1-D6, referenced from an append-only block in PLAN_PERF_PIPELINE_2026-08-16.md (head-298 sha256 verified byte-identical, 81560e94b7628e6c). Headline: crediting the standing reduction back, [31,17,16] recovers to 530,381 -- EXACTLY [31,18,15]'s spill-funded figure, because once floors stop binding both bind PP0 on the same bytes. The pool advantage that made [31,18,15] interesting was an artefact of VRAM-resident floors, as the user said; with it gone [31,17,16] wins outright on speed and is the only one of the two above the 14.1% noise floor. Build NOT undertaken and NOT small: nothing credits ON-DEMAND capacity against the standing floor, and crediting the evict rung's PRICED capacity would be pricing on intention rather than completion -- sgl-project#717's defect one layer up, with an OOM at arm time as its failure mode. Ticket filed at D6 with one live _record_seam_peak reading as prerequisite. Desk only, no boot, no GPU, no serving contact.

…d conflict map PREPARATION ONLY. No branch pointer was moved and nothing was merged. Every conflict is MEASURED by trial merges in a throwaway worktree (throwaway/merge-train-probe, removed after measurement), not predicted. Built from git rather than from the handed list, and git contradicted the list twice: * fix/713-admission-intake is ALREADY in the serving line (+0 commits, 0 files) -- not a train item, and scheduling it would be a no-op step that looks like progress. * fix/728-max-bytes-uniform points at 79216e6, byte-identical to feat/706-phase-uniform-hicache-keys, and has NO REMOTE. It must not be merged as a separate item, and its identity has to be settled first: a local-only branch is the one kind that disappears with its worktree. DIVERGENCE. The 4222976 class turned out not to be the risk it looked like: that commit is PRESENT in integration/r2 and absent only from upstream main, which is true of every fork commit. The useful axis is the merge target -- 167 commits are on the serving line and not in integration/r2, and exactly SIX of them are on no other train branch (the sgl-project#677 range c4bc982..5fed8a6, sgl-project#708 fc6f97b, and the merge 761d0d7). All six are reachable from feat/677-park-wiring, so that branch carries them; if any step drops or rewrites it, those six are the loss. The serving line is also NOT an orphaned detached head -- 5fed8a6 is exactly feat/677-park-wiring, checked, because unbranched commits on a serving head are how a train loses a fix. ORDER. All eleven candidates are independent siblings (a containment check found no branch containing another), so order is a conflict question. The measured result is two clusters and a clean set: * Cluster A, scheduler_teardown.py: all four sgl-project#673 thread-stop branches add their stop logic to the same file this lane created for sgl-project#673. They conflict with EACH OTHER, not with the trunk, so whichever lands first sets the file's shape. The barlink one also touches scheduler.py and belongs to live sgl-project#722. * Cluster B, planner/seam: fix/602-fill-side, fix/701-ledger-wiring and feat/704-prefill-ladder each rewrote seam_slope.py, planner/pp_cut.py and test_pp_cut_prefill_speed_702.py. Three lanes editing one model of the same thing -- the conflict is SEMANTIC, and resolving it by taking hunks would produce a seam model nobody designed. * The clean six-step train (621, 699, 673-lockstep, 706, 717, 677) merged in sequence with no conflicts at all and can run without either cluster. TEST MATRIX, measured on the merged probe state: mem_cache 940 failed / 973 passed and distributed 21 failed / 2716 passed, both matching the standing baselines. managers came out at 12 failed / 2274 passed against a standing figure of 14 -- and the point is that this suite's count is TRAIN-COMPOSITION-DEPENDENT: this lane's branch alone shows 4, the merged probe 12, with the extra 8 arriving from the sgl-project#677/sgl-project#713/sgl-project#631 lanes rather than from the merge. The failing classes are listed so they can be attributed, and each owner must record their own baseline before the train, or "the branch shipped it" becomes "the merge broke it". HELD OUT, with the separability question answered: 2ce1ed7 is NOT docs-only -- it changes managers/phase_flip_runtime.py, which is exactly why it earns its review boot. The two later sgl-project#441 commits touch no file it touches, so they are cherry-pickable without it; the only entanglement on that branch is be1fcec -> 2ce1ed7 (same NOTE). But 4512136 carries sgl-kernel/csrc/kvcacheio/transfer.cu, so riding it means a kernel rebuild -- its own boot-gated risk, not something to smuggle in behind a docs-and-tests framing. Also in this push: the sgl-project#568 ledger commit (79216e6), which the audit found was this lane's only unpushed work.

…; retract my "resize is untested" claim NOTHING BUILT, because it is already built. attach (PUT), detach (DELETE), status (GET), resize (POST /resize) and clear all exist on /hicache/storage-backend (http_server.py:1395-1510), each behind @auth_level(AuthLevel.ADMIN_OPTIONAL) PLUS an explicit admin_api_key check -- the sgl-project#510 regime with the belt-and-braces the most sensitive routes should get -- reaching attach/detach/resize_storage_backend on the tree cache via scheduler handlers at scheduler.py:7625/7681/7728. The semantics the brief specified are the semantics implemented: attach/detach refuse a non-idle scheduler BY NAME ("Reject attach: scheduler is not idle. #queue-req=... #running-req=..."), and resize-down evicts inline and "returns once usage is back under the new cap", with in-flight (reserved but uncommitted) writes never evicted -- so it does not truncate live pages, and the write interlock is _pending_writes rather than a race with the backup queue. resize deliberately does NOT require idleness, which is a narrower and better interlock than attach/detach's whole-scheduler gate. RETRACTION. I wrote a hermetic pin file on the finding that "resize has no coverage at all", having grepped for resize INSIDE the E2E attach/detach test. Wrong: test/registered/unit/mem_cache/test_hicache_runtime_resize_545.py exists with 21 tests covering every property I pinned and several I did not -- grow-evicts-nothing, shrink-until-under-cap, LRU victim order, enable-at-runtime-adopts-existing-files, in-flight-write-not-evicted, lifting-the-cap-disables-eviction, non-owner-MLA-rank-inert, the request validation layer, and both cache classes. My file was 100% duplicate and is DELETED; shipping it would have created a second authority for the same properties, which is what I refused in sgl-project#536. Third instance of the same error (after sgl-project#726 and sgl-project#677): concluding absence from the file I happened to open instead of grepping for the thing itself. "No coverage" requires a search FOR the coverage. My harness also passed max_size_bytes/min_free_bytes as extra_config keys when the real names are max_size/min_free_space, so every evictor it built came up UNCONFIGURED and most pins still passed -- a harness passing for the wrong reason. TWO MORE BRIEF PREMISES THAT DO NOT EXIST. MixedLayoutError: zero hits across python/. The cited commit 19f4c68 describes it in its message but no such class is in the worktree; the nearest relative guards PD draft-KV, not HiCache attach. What attach actually uses is a same-backend check (hiradix_cache.py:544-568) refusing a DIFFERENT backend by name. ReadBufferPool / sgl-project#720: also absent, no class and no reference. I did not design around either guess. PRIOR ART the brief did not mention: docs/dev/NOTE_544_hicache_runtime_ preserve_thinking.md is a desk-complete investigation of this same ticket, and docs/advanced_features/hicache_storage_runtime_attach_detach.md is the user-facing documentation of the shipped feature. THE ONE REAL GAP, and it lands on this rig's model family: UnifiedRadixCache.attach_storage_backend and detach_storage_backend (unified_radix_cache.py:2769,:2783) are HARD STUBS that always fail -- "does not support runtime HiCache storage attach yet". resize works there; attach and detach do not. UnifiedRadixCache is what registry.py:191 constructs on the path that appends the MAMBA component for is_hybrid_ssm, so for the hybrid-GDN family the shipped story is resize yes, attach/detach no -- the ticket's headline capability is exactly the half that is stubbed. I did NOT verify which cache class our specific boot instantiates; that needs the boot config, not the source. Also NOT ESTABLISHED: no test raises or mocks a real OSError(ENOSPC); disk-full is covered only through the min_free_space watermark refusal. RECOMMENDATION: implement UnifiedRadixCache attach/detach (a real implementation task -- the stubs exist because that cache's controller lifecycle differs from HiRadixCache's), then a real ENOSPC injection test. Live-window acceptance filed in the note, including that attach and detach are expected to fail on a hybrid-GDN model with the named refusal -- that is the acceptance for the gap, not a second bug report. Existing suite re-verified: 21 passed. codespell clean. Desk only.

…-pong A layout may be declared unable to run only after it has HAD THE CHANCE to run. An emptiness observed within idle_locked_settle_s of entering a layout is a transient, not a verdict. THE CLAIM THAT FAILED The IDLE_LOCKED branch sits above the min-dwell check and bypasses the only anti-thrash bound, justified by a comment stating "IT CANNOT OSCILLATE ... after the flip the target runs by premise, so the same condition is false there". It oscillated. Across the 16 boot rotations of 2026-08-17 the policy produced alternating runs of 72 arms / 299 s, 12 / 31 s and 10 / 27 s twice. The premise fails on WHEN it is evaluated, not on what it says. This branch is reached on the first round after a cutover, while the just-entered layout is still empty and its carried work not yet re-admitted -- Scheduler._idle_locked_inputs is gated on _round_built_nothing, which a just-entered layout satisfies trivially. The new layout is observed in its empty transient, certified unable to run, and armed straight back. That is also where sgl-project#713's TTFT quantisation came from. The unit of delay is one whole cutover (median 2864 ms tp_to_pp, 2772 ms pp_to_tp over 486 flips), so the 0.1 / 3.1 / 5.9 s levels in the sgl-project#713 tables are how many cutovers a request sat through. The 06:19 table was taken during a 27 s run. THE VALUE IS DERIVED, NOT CHOSEN Delay from "cutover complete" to the first batch the new layout builds, over 162 cutovers: 0 s for 66 of 150, 1 s for 34, 2 s for 27, 3 s for 12, thin tail to 6 s, one outlier at 30 s. 2.0 s covers 84.7 %. The p95 (4 s) is deliberately NOT used: it exceeds the 3 s min_dwell_s this rig boots with, and a settle above the dwell would make the fast escape slower than the path it exists to bypass. Log stamps are second-resolution, so this is a bound, not an optimum, and the docstring says so. FAILURE DIRECTION, AND WHAT BOUNDS IT A genuine idle lock forming within the settle of a cutover is now delayed, by at most the settle. Bounded three ways: the value is capped at min_dwell_s at the use site, so an operator cannot configure the fast escape into being the slow one; the 180 s decode-stall cap remains the backstop that released the 09:42:39 specimen; and the settle is consulted only right after a cutover, which is the only place the transient exists. sgl-project#688's escape is preserved and pinned. phase_since, not last_flip_at: the first is when THIS layout was entered and is maintained from the OBSERVED phase, so a manual POST /phase_flip restarts it too; the second is an arm stamp that is 0 until the policy has armed once. None degrades to pre-guard behaviour rather than to an infinite settle. RED-FIRST, AGAINST THE RECORDING test_idle_locked_settle_713.py replays the recorded arms through the real policy off scripts/fixtures/d2_injector_pingpong_excerpt.txt, reusing the injector's parser so harness and suite cannot disagree about what the log says. settle disabled -> 12 of 12 arms armed, the ping-pong reproduced settle enabled -> 1 of 12 armed Not zero, and that is the point: the arm that legitimately leaves a 35 s-settled idle-locked layout survives, and only the eleven arm-backs on 0-1 s-old layouts are refused. The first version of this replay asserted all twelve were transients; the data refused, one arm being 35 s past its cutover. That arm was not noise, it was the control case, and it is now the selectivity pin. Mutation-proven: disabling the guard at source turns 3 tests red. REGRESSIONS Baselined before and after by capturing the failure set with the guard reverted. 8 failures pre-exist on this branch in this area and are untouched; the guard adds none and 705 pass. The five sgl-project#677-era pins stay green because they construct PhasePolicyState() with no phase_since -- which is also why they never caught this, and their docstring now says so instead of restating the falsified invariant. sgl-project#712 TEXT REMOVED The BOTH-BLOCKED decline used to redirect to "the state-slot bound (mamba/GDN slots)". That was never measured: it was a hypothesis authored into a log string, read back out of the log and filed as a finding, and sgl-project#712 was closed as unfounded on that evidence. The line now reports only what it knows. test_both_blocked_binding_resource_708 REQUIRED that wording, so the pin is inverted rather than relaxed -- an unmeasured cause must not be named.

…dge stops answering all-unavailable DIAL SIDE -- vram_dial.reclaimable_bytes_for(participant, floor_rows). A LIVE read: full_pool_backed_rows (the bound eager launches actually pass, the same quantity verify_pool_reached_capacity checks a commit against) times _pool_row_nbytes (the pool's real per-row K+V bytes), minus the floor. floor_rows is REQUIRED AND NOT DERIVED THERE. There is no per-pool floor authority in that module -- the dial's floor is a card-level NVML measurement (_measure_local_floor_bytes) taken at boot -- and inventing a per-pool one would create a second authority for a number sgl-project#584 says has exactly one. A caller without a floor gets None, which the bridge turns into a named refusal. REGISTER SIDE -- OffloadRegister.reclaimable_bytes(offload_class), mirroring latency_term_ms's lock-and-filter shape so the REGISTER answers about itself rather than the bridge re-deriving its accounting from outside. Resident AND not hot: parked bytes are already gone (counting them promises the same bytes twice), and park() refuses hot items unconditionally, so hot bytes are resident but NOT reclaimable -- including them would hand a caller a figure it cannot spend. An unanswerable hotness predicate counts as HOT: the safe direction is refusing to reclaim, never assuming free to move. THE DISTINCTION THE CUT TURNS ON: ProbeUnavailable vs zero. Zero is a MEASUREMENT ("at its floor" / "nothing resident"); a failed probe is the ABSENCE of one. Both probes return None/raise rather than 0, and the bridge turns that into a NAMED refusal. Collapsing them would remove a real source from an elastic plan while looking like it was considered -- the sgl-project#606 defaulted-measurement defect, which this strand has now refused in four separate places. HONEST LIMIT, unchanged and restated in the analysis: the hermetic proof exercises the PLUMBING with faked dial/register state. No live number's correctness is claimed. In particular the dial probe is only as right as the floor its caller supplies, and no caller supplies one yet -- that is the window item section 4 already names. ALSO ADDED (section 8, SKETCH ONLY, no build): the cold-direction policy that would consume the bridge -- idle event -> query -> refuse rather than take a partial -> GDN vacate first (cheap) -> dial grow below the captured bound -> stop rather than attempt a re-capture. Debounce rationale is #704a's jump price: a rung change costs a full ~1575 ms arena refill, so an idle/hot flap across a rung boundary pays it twice for no net capacity; hysteresis on the event plus a rung-crossing cooldown, the same reasoning SpillCooldownRegistry already carries for the spill/restore pendulum. Anything requiring a geometry flip stays design-only, dependency named (sgl-project#677 arming-floor budget). Tests, hermetic (CUDA_VISIBLE_DEVICES="", no CUDA, no boot): test_coresidency_registry_553.py 30 passed (16 Cut 1 + 14 Cut 2) MUTATION: making a failed probe collapse to 0 fails 3 -- including the pin that a MEASURED zero and an UNMEASURED one stay distinguishable test/registered/unit/managers/ 13 failed / 2247 passed -- failures unchanged from the pre-existing 13-failure baseline, passes 2233 -> 2247, exactly the 14 new pins offload-register suites: 129 passed (I added a method to that class) ruff + codespell clean. Same branch (feat/553-elastic-coresidency). Desk only, no boot, no GPU.

… the allowlist is empty Operator decision on the fork the previous commit left open, and it turns on a fact from the code rather than on the wrapper's prose. The sgl-project#677 hold was licensed to veto "the plain timer/economics exit". That licence was never really about the arm being a timer; it rested on an UNSTATED ASSUMPTION -- that vetoing the timer leaves some other backstop armed. The legacy pp_window stopwatch destroys the assumption: it sits behind a `cap <= 0` guard, so it fires ONLY when the decode-starvation cap is absent, which makes it the LAST anti-pinning bound in every state where it fires. Vetoing the last bound is an unbounded hold. That is verbatim the condition test_sustained_backlog_still_leaves_pp_via_the_window exists to prevent ("PP returned 'holding in pp' on every call, without end") and the shape of the live wedge family this ticket started from. The assumption outranks the prose, so the stopwatch is an exit as well. THE RULE THAT FOLLOWS, written into the dataclass and pinned by a test rather than left in this message: An arm may carry hold_eligible=True only if, in EVERY state where that arm fires, a SECOND INDEPENDENT anti-starvation bound is armed. THE ALLOWLIST IS THEREFORE EMPTY, and empty is the honest state, not a loss. The seam stays: it is the socket for a future arm that really is backstopped, so such an arm is added by stating the claim instead of re-deriving this whole argument -- and with no member, no arm can be held at all. sgl-project#677's economics is not dead with it; it lives in the window-length machinery and in the threshold repricing (sgl-project#819), on the flip-DECISION side where a flip can be weighed before one is chosen, rather than as a veto on an exit the rules already decided. That wiring pointer is recorded in the code and in the COORD so nobody re-attaches it here. The three anti-starvation tests are left untouched. They are the guarantee carriers, and they now pass because the code agrees with them again. TESTS (hermetic, CUDA_VISIBLE_DEVICES="", PYTHONPATH at this worktree; no boot -- a flip-decision change is acceptable only under load, which is a window post): test/registered/unit/managers/test_hold_allowlist_817.py: 12 passed. New pins: the allowlist is empty and the admission condition is written where it binds (the dataclass a future author actually reads, not the history), the stopwatch is shown from the source to be the arm that fails the condition, and -- as behaviour rather than as a count -- no reachable PP_TO_TP arm can be held while the list has no member. ALL EIGHT wrapper-caused failures are now green. Suite diff over the 25 suites importing phase_policy, branch vs base 587e4c2: 14 named failures -> 6, and the diff contains ONLY fixed entries, no new failure. The remaining 6 are test_vacuous_decode_exit_730.py, pre-existing and unrelated to the wrapper (proven earlier by disabling the wrapper outright: they do not move). Mutants, all KILLED: base denylist wrapper restored -> 15 failed stopwatch back to hold_eligible=True -> 5 failed blocked exit put back in the allowlist -> red allowlist inverted back into a denylist -> 11 failed black, isort, ruff, codespell clean.

Closes the Cluster 4 defect that sgl-project#815 escalated rather than fixed: the sgl-project#677 HOLD wrapper swallowed the blocked-admission exit. sgl-project#817 inverts it into an allowlist exactly as the wrapper's own comment prescribed. Clean merge against the sgl-project#790 phase_policy.py edits.

…t#816, sgl-project#810, sgl-project#806, sgl-project#797, sgl-project#790/sgl-project#777, sgl-project#817/sgl-project#820, sgl-project#818 Merge-checklist duty for the nine stages on this branch. Entries follow each section's own house style, and the mechanism text is taken from the commits' measured evidence rather than restated from the ticket titles. §3 KV backing relief + the allocator cap -- UPDATED IN PLACE rather than given a second bullet, because sgl-project#814 and sgl-project#816 are follow-on defects OF the KvRowCap mechanism that bullet already describes: the census reading the withheld block as a leak (340262 of 465190 ids), the lift being reachable only from a cutover (one boot at 26.8% of its id space for the life of the process), and exposure exceeding the backing (417850 rows over 105413 committed, the device-side assert in masked_set_kv_buffer_kernel). §3 HiCache staging write-through ring (sgl-project#810), new bullet, plus its two companion refusals -- the unbounded-file-tier refusal and the boot preflight ledger entry, the latter being why 22.01 GB of MHATokenToKVPoolHost across three PP ranks previously reached the preflight as nothing. §7 BAR1 deadline + loud abort -- appended the sgl-project#818 peer-liveness half to the existing narrative: the gate could wait forever on a peer that no longer exists, and neither Bar1CollectiveStalled (reset by every resolved read) nor defer_stall_for_building_peer (900 s off a build marker) caught it. §12 Robustness canon -- three new families: contradictory-flag (sgl-project#806), read-back-after-construction (sgl-project#797), denylist-of-reasons (sgl-project#817, sgl-project#820). §18.3 hicache staging sizing (sgl-project#810) -- §18's own rule is that a merge adding a reusable module adds its entry in the SAME merge, and this module had none. Records the removal of fits_pinned_host_budget so it is not reintroduced. §18.6 mamba carry instrument (sgl-project#767, gated by sgl-project#790) and flip break-even N (sgl-project#777). No existing entry was contradicted. Checked before writing: none of sgl-project#814, sgl-project#810, sgl-project#806, sgl-project#772, sgl-project#797, sgl-project#790, sgl-project#777, sgl-project#817, sgl-project#820, sgl-project#818 had a catalog entry, and the one sgl-project#677 line (§19.2, RESTORE-NEVER-REBUILD) describes a different mechanism than the sgl-project#677 layout hold, so it is not stale and was left alone. Gates for the tree this documents (hermetic, CUDA_VISIBLE_DEVICES=""): battery test/registered/unit/{managers,planner,server_args,mem_cache} baseline integ @ 78d27da 44 failed c56d238 (through sgl-project#818) 30 failed, 8452 passed 0 new ids, 14 fixed 1c4eadb (through sgl-project#816) 30 failed, 8461 passed 0 new ids, same set test_barlink_abort_gate_liveness_818.py (outside the battery dirs) 10 passed ruff --select=F401,F821,UP037 and codespell: 0 new findings vs the same file set on 78d27da (16 ruff / 6 codespell exist identically on base) docs-only change; codespell clean on the catalog itself

W9 wiring, part 1 of 2: the group now COMPUTES the uniform decision every TP-loop iteration. Consuming it in batch formation is part 2. PLACEMENT. The block sits between the mamba pair and the ballot, at the same seam the sgl-project#794 corridor width uses and for the same reason its comment gives: everything above is indexed from the HEAD and the ballot is indexed from the TAIL (`len(vals) - (PREFETCH_BALLOT_SLOTS + 2)`), so an insertion here leaves both readings intact. Both new blocks capture an explicit index BEFORE appending and are read back by it; nothing is read by a negative offset, which is the mistake #639b's note records -- "the `t[-2]`/`t[-1]` the host floor used to read would have silently started reading MAMBA availability". The full payload order is written out in the comment so the next person does not have to reconstruct it. THE PULL-FORWARD, and it is not a new idea in this function. The sort key `num_matched_prefix_tokens` is populated by `calc_priority`, which runs LATER in the pass, so at reduce time it is zero or last pass's value and reducing it would make the group agree on a stale number. #791b already solved this exact shape here for the prefetch verdicts: pull the RANK-LOCAL computation forward to the reduce site -- no collective, once per TP-loop iteration -- and memoise it for the batch formation to consume. `_local_head_prefix_matches` does the same, bounded to the canonical head, against a tree `calc_priority` was about to walk in full anyway. THE COUNT VOTE GETS ITS OWN SLOT rather than being derived from the availability floor, because `get_num_allocatable_reqs` is bounded by `admission_limiter.current` (:6526-6529) -- rank-local floating state the availability reduce does not capture, so a count derived from the uniform avail would still diverge. Neither vote can break the reduce: both are wrapped, and a rank that cannot price contributes the absent/unpriced sentinel, which can only delay admissions or leave the local limit untouched. Never a collapse to bs=0. THE DRIFT GUARD CAUGHT ME, WHICH IS THE POINT OF HAVING REPAIRED IT FIRST. Widening the payload made `TheHarnessTracksTheProductionSurface` fail with BudgetHarness has drifted behind Scheduler._update_uniform_pool_budget: ['_local_admit_limit', '_local_head_prefix_matches'] -- the seventh drift of that harness after #616g, sgl-project#639, #639b, #791b, sgl-project#794 and sgl-project#701, and the FIRST caught in the same change that caused it rather than a quarter later. Then it named `get_num_allocatable_reqs` behind them, the transitive member, exactly the cascade the file warns about. Both votes are BOUND from Scheduler so the harness keeps modelling the real contract, and both ride neutrally there: the harness's waiting_queue is empty, so the head vote is an empty canonical set of absent sentinels. For `get_num_allocatable_reqs` the guard's own message offers a stand-in as the alternative to binding, and a stand-in is right: the shipped method is bounded by the admission limiter and carries the sgl-project#677 parking branch, so binding it would oblige this harness to model a limiter and a phase policy to answer a question it is not asking. Constant and EQUAL on both ranks deliberately -- a divergent count is test_tp_head_congruence_823's subject, and making it diverge here would put a second unrelated variable into the budget cases. TESTS (hermetic, CVD="", CPU only, real gloo where the suite uses it) 55 passed across test_collective_family_siblings_610 (the harness that models this reduce), test_pp_prefetch_ballot_791b (the tail-indexed ballot, i.e. the reading most at risk from a widening), test_prefetch_ballot _divergence_823 and test_tp_head_congruence_823. That the ballot suite still passes IS the evidence that the tail indexing survived the insertion. STILL PART 2, so W9 stays preflight_pass N: `calc_priority` and the candidate loop do not yet consume `_uniform_head_match_lens` / `_uniform_admit_limit`. The decision is computed and published; nothing acts on it yet, so this commit changes no batch. No boot was run. This is desk work.

…, and can be wrong W24 residual defect (iii), and the ticket's own remedy is NOT what shipped. The 09:01:37 specimen carries TWO defects; "break the band when the decode bundle is not draining" is neither of them. THE ARITHMETIC THAT REFUSES THE TICKET'S REMEDY. At the measured decode_contention (sigma) = 1 the scheduler gives prefill absolute priority per iteration -- `_differential_flip_threshold` records the measurement verbatim: "an iteration with any prefill chunk pending runs THAT batch and never reaches the decode branch". So while prefill is pending in TP the decode bundle CANNOT shrink, by construction. "Bundle not draining" is therefore IMPLIED by the band's own operating condition, not evidence about it, and a band that broke on it would collapse to the plain break-even for every load with a request decoding -- silently deleting the sgl-project#665-F1 differential model. That is a policy rewrite wearing a bug fix. (A) THE DETECTOR READ A BAR THE POLICY NEVER APPLIED. The alarm printed `bar_tok=20057` and fired on `pending 22887 > 20057`, while the hold it indicted names its own bar in its own text: `> N=20057 but <= 30086`. 30086 is `effective_flip_threshold(cfg, running_bs=1)`, the differential bar, which at sigma = 1 is N0 x (1+2B)/(1+B) -- exactly 20057 x 1.5. The policy compared 22887 against 30086 and HELD, correctly. So W24's single LAYOUT-ECONOMY ANOMALY is a FALSE POSITIVE, and the "first metal catch of the detector" reads better as its first metal self-indictment. This is sgl-project#819's ONE READING rule -- "the bar the policy APPLIED and the bar the log REPORTS can never be two different numbers" -- holding inside phase_policy and breaking at the module boundary. It is the sgl-project#851 class root exactly ("the DECIDERS still read their own bookkeepers"), occurring inside an sgl-project#851-family detector, which is the same shape sgl-project#853(i) already found once this build. The applied bar is now passed from the one authority that computes it, and is REQUIRED rather than defaulted: a default is the mechanism by which a caller silently re-creates the divergence. The gate takes max(break-even, applied), which bounds the blast radius to one direction -- it can only RAISE the bar, so it can only remove false positives and can never invent an alarm. That also covers strict purity, where the threshold is 0 by construction. (B) THE BAND HAD NO FALSIFIER FOR ITS OWN PREMISE. It was the ONLY hold in `_decide_from_load` that could not be wrong. Every neighbour carries a bound: min dwell yields to `starved` (sgl-project#768), drain mode to the sgl-project#833 stall deadline, the idle lock to the idle dwell (sgl-project#748). The band says "prefilling it in tp beats the round trip" -- a claim priced at the TP prefill RATE, true only while the backlog is actually being prefilled here. The falsifier is taken on the axis the claim is made on: PREFILL PROGRESS. `pending_prefill_tokens` is "admitted but not yet computed", measured at the chunk fill boundary, so it drops every round a chunk is computed -- even mid-way through one long prompt. Frozen for a whole decode window means no chunk was computed for a whole decode window. The clock is sgl-project#677(a)'s existing `last_prefill_progress_at`; the window is `drain_stall_deadline_s`, the same quantity sgl-project#833 and the sgl-project#838 detector already use, so policy and detector cannot come to hold two different ideas of one decode window. Unstamped reads as no stall, so a caller that never observed cannot flip on it. THE TWO HALVES AGREE ON THE SPECIMEN INSTEAD OF DOUBLE-COUNTING IT. W24's pending oscillated 0 -> ~22.5k -> 0 on a ~5-min period, so prefill progress was live and this break would have stayed SILENT there -- consistent with (A), which says that hold was right. That is what makes them two separable defects rather than one defect described twice. NOT REBUILT (prior-art gate): the ticket pairs (iii) with "a completed flip should clear the staging backoff". Already implemented -- `note_flip_completed` pops last_abandon_at / arm_refusals / arm_hold_until / arm_degraded for the direction -- and already pinned by test_phase_policy_flip_reachability.py:: test_a_completion_clears_the_staging_rate_limit_outright. Cited, not duplicated. A band-break is paced by that limiter like any other arm, which is correct: `_decide_rules` applies it after `_decide_from_load`, and `_demand_outweighs_a_retry` still overrides it when the backlog outweighs the wait. WHAT THIS DOES NOT CLOSE: it does not move the flip. W24's stuck phase was FUNDING -- 153 arms refused after the policy had already said the load wanted the flip, 43 of 45 binding refusals reading cause=phantom_capacity, which is sgl-project#852's territory. (iii) must not be cited as a flip-stickiness fix. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_band_premise_853.py 15 passed test_layout_conformance_838.py (updated) unregressed test_flip_threshold_repricing_819.py unregressed test_phase_policy_flip_reachability.py unregressed test/registered/unit/managers/ 3833 passed, 18 skipped, 334 subtests (759 s) test/registered/unit/mem_cache/ 1710 passed, 1658 skipped, 361 subtests (127 s) ruff check + format clean Red-first, both halves and both directions. (A) was red as the specimen FIRING -- "a hold inside the policy's own band was alarmed on" -- with the can-fail direction proving a detector that merely went quiet dies (pending above the applied bar still fires, and a window-3-shaped span at 119x the bar is above BOTH bars and stays an anomaly). (B) was red as the frozen backlog holding forever, measured: 25 observations at a constant 25065 tok returned "too short for the round trip" every time, with the can-fail direction being that a progressing prefill, a sub-window stall, a sub-band backlog and an unexpired min dwell must all still HOLD.

W25 went TP-sticky on 16-20k prompts and the user caught it by eye. The policy was right and the PRICE was wrong, and the price was wrong in a way this module had already written down and then not applied to itself. phase_policy.py:82 C = round-trip flip cost, seconds break_even_tokens "never repays the {flip_cost_s}s round trip" But `observe_flip_leg` fed ONE LEG per sample -- its own docstring even computes the round trip it is not feeding, "tp_to_pp 11490 + pp_to_tp 5681 = 17171 ms" -- and BOTH DIRECTIONS went into ONE EMA. THE TWO LEGS ARE NOT THE SAME QUANTITY. On the binding rank W25 measured tp_to_pp 10466-13181 ms against pp_to_tp 5078-6545 ms. An EMA fed both alternately converges to neither. REPRODUCED EXACTLY, which is what makes this measured rather than argued. Replaying PP0's eleven PHASE-FLIP DONE totals through one estimator at ALPHA=0.3 gives 5.0779 6.6944 6.2494 7.5450 7.2426 9.0241 8.2740 9.2457 8.4356 9.3990 8.5041 -- and the boot's own decision lines printed N=15853/18110/18464/18614 at exactly the samples pricing to 7.2426/8.2740/ 8.4356/8.5041. To the token, four times. The blend settled at 8.50 s: BELOW every tp_to_pp leg and ABOVE every pp_to_tp leg. The true round trip was 11.6 + 6.4 = 18.06 s. The bar also oscillated ~2000 tok with flip-direction parity (8.50 after a pp_to_tp, 9.40 after a tp_to_pp) -- an artifact of the blend that says nothing about cost. sgl-project#819's own closing sentence is the rule it broke one level up: "a component and its container are different quantities and an EMA fed both alternately converges to neither." Two directions are different quantities too. THE CORRECTION RAISES THE BAR (C 8.50 -> 18.06 s, N 18614 -> ~39500), so it makes TP-stickiness on 16-20k prompts MORE correct, not less. That is stated plainly rather than softened: the remedy for a bar that is too high is a cheaper seam, not a permanently under-priced one. dN/dC = 2188.8 tok/s says what each second of seam is worth once it is. WHAT SHIPPED. `RoundTripFlipCost` holds one `FlipCostEstimator` PER LEG and sums them. The leg estimator is REUSED, not rebuilt, so every property sgl-project#677 pinned on it holds per leg -- including that it tracks DOWN as readily as up, which is what makes a future seam fix actually lower the bar instead of latching high. The seed is split in half, so an uncalibrated instance values exactly the round-trip seed and the pre-sgl-project#856 path is unchanged. An undirected reading is treated as a whole round trip and split evenly (so callers that really measured one stay honest); a direction this class does not know is REFUSED rather than filed under a guess. PROVENANCE GAINED A THIRD WORD. `flip_cost_measured()` is a boolean over a quantity with three states and printed the middle one as "measured". `flip_cost_provenance()` returns seed / half-measured (<leg> only) / measured. Same class of fix as sgl-project#853(i) on the exposure gate and sgl-project#854 on the economy detector, one layer further in. AND THE CONSUMER IS RECONCILED, which is the half that is usually missed. The sgl-project#838 economy detector refuses to question a bar priced off the seed because "an assumption is not the policy's own claim". A HALF-measured round trip is still half assumption, so it is refused on the same ground: `flip_cost_fully_measured()` requires BOTH legs. The blast radius is one-directional -- the detector can only DECLINE more often, never alarm more often. DELIBERATE TEST CHANGES, not loosened ones: * `test_the_leg_total_becomes_the_price` now asserts the leg's own value AND the round trip (11.4901 + 3.2/2). Asserting 11.4901 for C would be asserting that one leg is the whole round trip -- the defect itself. * `_measure` floors at 2x MIN_ESTIMATE_S: each leg carries its own band, so a round trip cannot be cheaper than two leg-minimums. * `test_a_seeded_price_is_not_evidence` now also asserts "half-measured". TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_round_trip_price_856.py 19 passed test_flip_threshold_repricing_819.py unregressed test_flip_cost_calibration_677.py unregressed test_flip_threshold_honesty_777.py unregressed test_flip_cost_clamp_directions_677.py unregressed test_layout_conformance_838.py unregressed test_economy_detector_liveness_854.py unregressed ruff check + format clean on the changed source ALSO FOUND, NOT FIXED HERE (recorded in NOTE_856_seam_cost_ledger.md): `observe_flip_leg` is called only from the flip-COMPLETION branch, so a boot whose flips are all refused or abandoned prices off the seed for the whole session -- and sgl-project#777's staleness WARNING is gated on the same event, so such a boot gets neither a reprice nor a warning. W25 did not manifest it (33 completed cutovers). Same silent-zero shape, one more instance.

… after it W31 arm 1 reproduced the W30 livelock with the fix for it in the tree and unreachable. The seam retracted 87 requests across 39 pp_to_tp flips and logged `SEAM TRANSPORT ADMITTED` 0 times and `Prefill batch phase=tp` 0 times. The exemption sat after `prefill_allowed_in_tp`, which is BELOW the sgl-project#677 drain-mode suppression. The recipe runs --phase-policy-drain-mode, so `prefill_suppressed_in_tp` returned True and `prefill_blocked_here` returned before the exemption was ever evaluated. This function already carries a note about the identical shape: "What broke was ORDER -- suppression was checked FIRST and returned True, so the valve never ran." THE ORDER IS SUBSTANTIVE, NOT COSMETIC. Drain mode forbids TP prefill because "a TP window entered to finish a bundle must not admit the work it was entered to escape". A request the cutover ITSELF retracted a moment earlier is not that work -- it IS the bundle the window was entered to finish. Suppressing it does not defend the drain contract, it makes the contract unsatisfiable: the bundle can never complete. Moved, not copied: one call site, pinned by a test. TESTS (4 new, 19 in the file) * a stamped request IS admitted under drain mode; * CAN-FAIL: ordinary prefill is STILL suppressed under drain mode, so sgl-project#677's contract is qualified rather than dissolved; * ORDER pinned in the source -- the thing arm 1 got wrong and which no behavioural test on a passing path can see; * single-sited. Proof the ordering is load-bearing: `prefill_suppressed_in_tp` returns True for this exact config, so below the gate the request was blocked. Targeted gates incl. the sgl-project#677 drain suites: 228 passed. Specimen: /spinning/evidence-665-f1/SPECIMEN_w31_a1_exemption_below_drain_gate.log

…, not before it W33 arm 1: the flag parsed (`phase_flip_rebind_hicache=True`), the writer ran, and it logged its OWN refusal on every rank -- sgl-project#847 PHASE-FLIP REBIND: --phase-flip-rebind-hicache is set but this boot has no HiCache host tier, so there is nothing to build a phase-matched pin from. -- followed by 6 `sgl-project#718 hicache-phase-guard` warnings, i.e. the W32 read- through miss reproduced with its fix installed and unreachable. Third time in this strand that a correct mechanism was placed where it cannot run (W31 arm 1 below the drain gate, W32 the policy's second copy, this). THE INPUTS ARE THREE, NOT TWO. I placed the writer inside `init_model_worker` because a host pool is allocated FROM its device pool and both device pools are ready there. But it also needs the HOST tier, and that hangs off `self.tree_cache`, which `__init__` assigns AFTER `init_model_worker()` has returned. So the writer ran before one of its inputs existed. This constructor already carried that lesson: the sgl-project#677 note sitting four lines below records the identical mistake made once before, when a call was put "beside init_admission_limiter" and ran before `req_to_token_pool` existed. Both now sit after their inputs. The ordering is pinned in both directions: the call must appear in `__init__`, and must NOT appear in `init_model_worker`. 13 tests green.

Two subtractions, one justification, only one of them checked: scheduler.py `_pending_now -= _seam_transport_now` UNCONDITIONAL scheduler.py `_seam_serviceable_now = ...` gated on the TP phase AND on seam_transport_premise_holds Both rest on the same claim -- that a seam re-admission is cheap flip transport rather than real workload, because "their prefixes are served by read-through from the canonical store". #861j verified that claim for the EXISTENCE term and never backported the verification to the ECONOMICS term twelve lines above it. WHEN THE PREMISE IS FALSE those tokens are not transport at all: they are a full cold prefill of real work, and deducting them tells the policy that work does not exist. The consumer that pays is the one arm still reading RAW pending -- the sgl-project#677(a) blocked-admission stall escape, whose threshold is `pending > pp_exit_tokens`. A deflated pending holds a genuine stall below its own escape, which is the wedge that escape was written to end. Delayed escape, not a wrong answer: `demand_prefill_tokens()` takes max(pending, admissible - serviceable) and `admissible_prefill_tokens` is not reduced by seam transport, so with sgl-project#869 landed `_strict_holds_pp` still sees the true backlog. VARIANT (a) OF THE TWO I FILED, chosen because it makes the two subtractions ONE RULE rather than removing the last raw-pending consumer and leaving the pair disagreeing. `seam_transport_deduction` is pure and total, so both directions are falsifiable without a scheduler, and the premise is now asked ONCE per round and read by both terms -- one predicate, one clock, per that function's own contract. Asking twice was its own latent defect: the debt clock can lapse between two calls in the same round. NOT A DISARM. A verified premise in the TP phase still deducts exactly as before, which is the #861j/W32 behaviour where it was actually earned. The test pins that direction too. CLASS: a subtraction justified by a premise that is never checked -- the same family as sgl-project#869 (a predicate that does not measure what it claims), different root. Sibling of, not instance of. TEST RESULTS test_seam_transport_premise_869c.py (new): 11 passed. FALSIFIED IN BOTH DIRECTIONS BEFORE THE GREEN WAS CLAIMED, count gate on each: fix reverted (deduct unconditionally = today's tree) -> 7 failed, INCLUDING the specified falsifier test_the_stall_escape_arms_on_the_true_backlog. 7 extracted == 7 in summary. over-fixed (never deduct, blanket disarm) -> 3 failed, including test_a_verified_premise_in_tp_still_deducts. 3 == 3. restored -> 11 passed. Regression set, 59 files (purity, policy, seam, transport, sgl-project#677, sgl-project#713, sgl-project#861, sgl-project#871, #869c): 895 passed, 4 failed, 123 subtests passed. Count gate 4 extracted == 4. All four are test_restore_never_rebuild_677, one of the three standing card-needing modules whose hermetic count is exactly 4. No new failures. Hermetic: CUDA_VISIBLE_DEVICES verified EMPTY at the PROCESS (/proc/<pid>/environ). Lint: ruff 0 before and after on phase_purity.py; scheduler.py 102 before AND after (pre-existing); new test 0. codespell clean. Non-flip and non-seam paths are byte-identical: with no stamped re-admissions `_seam_transport_now` is 0, and the deduction is 0 under every gate combination. TWO BUGS IN MY OWN TEST, found by running it and reported rather than quietly fixed. The first draft left the hand-set `pp_window_s` stopwatch enabled, so the deflated specimen still armed -- for a different reason -- and the test would have passed without the escape ever being consulted; it now sets pp_window_s=0 and decode_stall_slo_s=0 to isolate the sgl-project#677(a) arm. The second asserted on the substring "stall" in the reason string, which matched SGLANG_PHASE_POLICY_DECODE_STALL_SLO_S in an unrelated suggestion line; it now asserts on the verdict. Same substring-matching family as the ^FAILED trap. NOT CLAIMED: no metal run. Desk-proven only. The sgl-project#857 acceptance instance was left running and untouched throughout.

…here the PP bytes go sgl-project#1011 -- CHECKED THE LIVE LOG FIRST, as the order required, and the honest answer is that most of it needs no rebuild. Across three boots (restore, final, wb): holds in tp with pending prefill > 0 ............ 0 holds citing tp-decode-floor-s or pp-window-s ... 0 holds citing min dwell ......................... 1345 / 1248 / 1 holds citing IDLE-LOCKED "< break-even N tok" ... 1342 / 1234 / 0 The exits are ALREADY work-gated: the policy never holds in TP while prefill work waits, and the two clocks never appear as a hold reason at all. The only reasons that fire are min-dwell (the sanctioned last thrash guard) and the PRICED economy ("0 tok < break-even 27410 tok"), which is exactly the sgl-project#677/sgl-project#819 form the order wants anti-thrash to have. pp_window_s is dead code on this config by construction: phase_policy.py:3840 gates it on `cap <= 0`, and cap > 0 whenever decode_stall_slo_s > 0 (ours is 180). CHANGED, the one place that did actuate against the drain law: the decode stall cap (phase_policy.py:3828). It was the only thing permitted to cut a drain short, and its own message printed "N tok prefill still pending" while doing it. It now DEGRADES TO A DETECTOR whenever pending_prefill_tokens > 0 -- a named WARNING with wait time and backlog -- and actuates only on an empty backlog. THE ONE HONEST CONSEQUENCE, stated and not softened: under a continuous prefill stream the backlog never reaches zero, so carried decodes wait UNBOUNDEDLY. That is the operator's decision; the warning is the whole mechanism that keeps it visible. BOOT-PROVEN: boot up, coherent (Paris / 143-67=76), 39 cutovers under agent-shaped load, 0 detector fires and 0 actuations -- i.e. no regression. The actuation path was already latent (0 fires in the two prior boots), so this boot proves the change is harmless, NOT that the detector fires. Naming that limit rather than claiming a proof I do not have. sgl-project#1014 -- WHERE THE BYTES GO. Answered from the ledger's own per-rank posts, same boot, both phases (GiB): post PP0 pp -> tp PP1 pp -> tp PP2 pp -> tp weights + runtime state 13.572 -> 4.252 7.408 -> 2.719 8.342 -> 3.662 gapped corridor holdback 1.000 -> 1.000 1.000 -> 1.000 1.000 -> 1.000 mamba state pool 0.731 -> 0.731 0.426 -> 0.365 0.304 -> 0.365 speculative intermediate 0.877 -> 0.877 0.511 -> 0.438 0.365 -> 0.438 prefill activation reserve 1.000 -> (none) 1.000 -> (none) 1.000 -> (none) rest (= the KV pool) 13.875 ->24.195 8.014 ->13.837 8.324 ->13.870 COUNT-CHECK PASSES in BOTH phases: 31.055 / 18.359 / 19.335 GiB = 31800 / 18800 / 19800 MiB = rank_gpu_memory_mib exactly. No unattributed MiB. So the entire PP->TP KV gain decomposes into exactly two terms: weights booked differently 18.69 GiB (29.32 -> 10.63 summed over ranks) prefill activation reserve 3.00 GiB (1.0 per rank, PP only) total 21.69 GiB = the measured KV delta (30.21 -> 51.90) AND THE FIRST TERM DOES NOT SURVIVE INSPECTION. The PP-phase weight posts (13.572 / 7.408 / 8.342, sum 29.32) are right: a layer-wise split holds the whole 27.5 GiB checkpoint plus runtime state. The TP-phase posts (4.252 / 2.719 / 3.662, sum 10.63) are NOT: with the flip vector 32,16,16 the expected shards are ~13.75 / 6.9 / 6.9. The TP weight post is low by ~3x. VERDICT per post, as asked: weights + runtime state ... MIS-BOOKED, not mobilisable. The TP phase's larger KV budget is substantially an accounting artifact, and the 1,274,048-token figure rests on it. Corroborating evidence from the same boot: "TP pool sized to the PP id space: 616670 tokens" and the earlier "max_total_tokens=616670 is larger than the profiled value 450402 -- use the profiled value" -- the inflated budget is largely NOT spent. prefill activation reserve NEEDED-IN-PP (prefill runs there; absent in TP by construction). 3.00 GiB, not free. gapped corridor holdback NEEDED, flip-functional, and per sgl-project#707 the floor is itself measured. Arming floor is 1037 MiB solver-derived (band floor 819 + seam entry 218 + arming margin 192). Shrink-by-measurement only. mamba / speculative NEEDED, and they are already near-identical across phases (<= 0.073 GiB delta). CONSEQUENCE FOR THE 1M CHALLENGE, computed rather than aspired: the world per-token cost is 32,768 B and is layout-invariant (ANALYSE_799 §5.2). 1M tokens therefore needs ~31,250 MiB of KV world-wide against the ~20,300 MiB the PP phase holds today -- about +10,950 MiB of REAL bytes. The ledger above shows no idle posten of that size: the only 18.69 GiB "difference" is a mis-booking in the other phase, not memory sitting unused in this one. So PP does not reach 1M by copying the TP budget. Whether it can reach it at all is a question for the post-#1009a solver with the FA split solved budget-proportionally instead of pinned 8/4/4 -- and the token axis alone is capped at 639,800 (+6.51%) at this cut, also from ANALYSE_799. I am not projecting a number I cannot source.

…ompensation layer, and its bugs were not the finding Coordinator order (2), after boots 30 and 31 killed the same mechanism twice at the same site with the same numbers (`told prefix 12288 exceeds this rank's pinned span 0`). Upstream-minimal law: a defect found in a compensation layer is a DELETION candidate, never a repair order for that layer. Two repairs were already spent on it -- sgl-project#1061's epoch gate and #1061b's retract clear -- and the second only existed because the first was built on a clock whose tick was never checked against the act it fenced. DELETED: `pp_uniform_width.py` whole (`uniform_pass_geometry`, `UniformWidthPromiseBroken`, `report_local_coverage`, `min_told`, `PassGeometry`, `epoch_admits_row`, `set_epoch_source`, `current_epoch`); `Req.apply_uniform_pass_geometry_1059` and its call site; the `_1059_told_prefix/_extend/_epoch` stamps; `PPAdmissionEntry.decided_epoch` and its wire column; the publication-time epoch stamp; the epoch-source registration; the told clearing at the retract chokepoint; the `SGLANG_PP_UNIFORM_WIDTH` gate; three test files. WHAT DELIBERATELY STAYED, and each for a named reason: * THE EVICTION FLOOR in `pp_stamp_observed_coverage`. `report_local_coverage` was `max(0, int(x))`, but the same line ALSO raises `cache_protected_len`, which `mem_cache/common.py:82` honours as an eviction floor. Deleting a promise must not silently lower a protection, so the arithmetic is inlined and the floor is byte-identical. Only the promise semantics die. * THE CARRIER COUNTER, renamed to what survives. The #1060b ledger counted the told row at the apply site; that apply is gone, but the question underneath it is not. Boot 29 shipped a carrier whose send gate had been dead since sgl-project#1046 and reported `sgl-project#631=0` while nothing executed -- it cost a window and a census to notice. `sgl-project#1064 CARRIER CENSUS` counts `rows_received` on every rank, so that regression can never be silent again. PP0 never receives, so only a downstream zero is a finding, and the line says so. * `sgl-project#1060`/`sgl-project#1063` untouched: they measure, they never gated anything. WHAT THIS RETURNS TO: the pre-sgl-project#1059 behaviour, in which every rank derives its geometry from its own match. That is not a guess about safety -- boot 29 ran it for its ENTIRE life (`evaluated=87 absent=87`, the row never arrived) across 51 flips with no `sgl-project#631`, no divergence death and no crash. The two boots that DID die at this seam are the two that ran the promise layer. c1 (downstream takes the geometry as a VERDICT) is NOT in this commit, and the reason is a desk finding that inverts its safety argument: `truncate_prefix_to` is MIN, NEVER ASSIGN (schedule_batch.py, sgl-project#930/sgl-project#958) -- so "verbatim" cannot be expressed through the one helper both admission sites use, and a rank holding less than it was told silently keeps its own value, which is the divergence the verbatim form exists to remove. Reported to the coordinator with the file:line rather than worked around. 38/38 green across sgl-project#791/sgl-project#1060/sgl-project#1063/sgl-project#677, codec round-trip verified after the column removal, import smoke of every touched module, ruff clean, py_compile green. Order: coordinator (2). Evidence: boots 30/31, specimens specimen_1060cens_boot30_uniformwidth_crash/ and specimen_1061b_boot32_int8_triton_wedge/

update version

362f87e

Ying1123 merged commit 2b4c646 into main Jul 20, 2024

Ying1123 deleted the update-docs branch July 20, 2024 10:39

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Update version to 0.1.22 (sgl-project#677)

ac2a3a1

Navigation Menu

Uh oh!

Update version to 0.1.22 - #677

Update version to 0.1.22#677
Ying1123 merged 1 commit into
mainfrom
update-docs

Ying1123 commented Jul 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Ying1123 commented Jul 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant