Fix logging by Ying1123 · Pull Request #796 · sgl-project/sglang

Ying1123 · 2024-07-29T06:00:57Z

always use logging.getLogger(__name__)
simplify [gpu_id=0] to [gpu=0]

… the message actually reaches the peer `_pp_send_admission_decision` called `_pp_send_dict_to_next_stage(..., async_send=True)` and discarded the returned `P2PWork` list. It was the only async channel in `_event_loop_pp_body` that did so: `send_req_work`, `send_proxy_work` and `send_output_work` are all held on the scheduler and committed later. The decision dict carries no tensors, so it travels entirely as metadata, whose backing `header` / `object_tensor` buffers are owned by nothing except those discarded handles. A gloo isend whose work handle and buffers go out of scope is aborted on destruction rather than delivered, so the downstream rank's blocking receive waits for ever on a message that was never on the wire. That is what wedged six consecutive boots of the gapped TP=1/PP=3 shape on 2026-08-20, all with the same py-spy signature: PP0 in the request-chain flush, one pass AHEAD, with PP1 and PP2 both in this channel's receive. Three successive commits re-ordered this send (f31fd5e, 927324c, a996a65); each fixed a real defect and none saved a boot, because the ordering of a message that does not exist cannot matter. Two changes, and the second is what makes the first safe to wait on: - The handles are retained in `self._pp_admission_send_work` and reaped by a new `_pp_commit_admission_send_work`, called once per iteration immediately BEFORE `_pp_commit_pending_req_work`. That order is the strictly weaker wait first: this one is satisfied as soon as the next rank reaches its decision receive at the top of the same pass, while the chain flush is not satisfied until that rank reaches the top of the NEXT pass. A wait already implied by one placed after it cannot be the wait that closes a cycle. - The last rank no longer emits the ring wraparound. PP0 was never required to receive it: it only PEEKS the typed inbox, which is filled solely as a side effect of the per-iteration output receive, and that receive early-returns whenever the slot is empty -- every idle pass. So the wraparound was one unmatched message per pass, the same corpse `_pp_send_dict_to_next_stage` already refuses for the proxy under a gapped wire. Consequence, stated plainly: `record_return_trip` no longer runs. It was never load-bearing for termination -- sgl-project#630's retry livelock terminates on the floor learned from the retraction, which is rid-keyed and dies with its request -- it only cleared that floor early. The residual cost is that one request's reuse stays suppressed for the rest of its life. Tests: - New test_pp_admission_send_handle_dropped_796.py: 5 passed, three consecutive runs. Its red arm drives the in-tree sgl-project#795 ring over a handle-dropping transport and requires the six-boot signature back (PP0 in the chain flush, ahead of PP1/PP2 in the decision receive). - Two harness traps are documented in that file because both produced false greens here: sgl-project#795's `_RingWire` keeps every handle alive in `_inflight`, a lifetime guarantee production did not have; and a patch applied inside a test method never reaches `spawn`ed children, so it must be applied in the child. - test_pp_admission_chain_flush_deadlock_795.py: 4 passed, no regression. - Full test/registered/unit/managers: 2739 passed, 18 skipped, 305 subtests passed, 47 failed. All 47 are in test_pp_proxy_stamp_631.py, test_pp_slot_last_batch_631.py and test_vacuous_decode_exit_730.py, which were measured at HEAD without this change and fail identically there (18 failed / 12 passed in isolation, both with and without the change) -- baselined against this branch rather than assumed pre-existing. - ruff: one pre-existing F841 at scheduler_pp_mixin.py:1325, untouched code. codespell clean.

…h a boolean context `x or []` asks `bool(x)`. `req.prefix_indices` is a tensor of KV-pool slot pointers, and torch refuses that question at both ends of the range this code actually sees: an empty tensor (a request with no cached prefix -- the common case) raises "Boolean value of Tensor with no values is ambiguous", and a tensor with several matched pages raises the "more than one element" variant. Only a single-element prefix would ever have passed through silently, so the spelling was broken for very nearly every request that could reach it. Two sites, both on the admission path, both reachable only once a request is actually being admitted -- which is why they survived: until the send-handle fix in 2323c92 the ring wedged at idle, and no boot had ever executed them. - pp_admission_congruence.py:352, in `build_pp_admission_decision`. This ABORTED PP0 on its first real prefill (boot instr5, evidence-665-f1/boot_instr5.log:6126-6155); PP1 and PP2 then died on the broken connection. - scheduler.py:6338, in `_trace_pp_admission_verdict`. This branch runs only on an ADMIT, so the failure was silent and precisely inverted from useful: every idle DECLINE pass logged cleanly while every admitting pass -- the only ones that can show the ranks agreeing or diverging on a real request -- threw into the instrument's own except-and-swallow. Boot instr6 spent a GPU window to produce three lines reading "trace unavailable: RuntimeError" at exactly the pass the first request arrived. The method's own docstring already stated len() is the correct spelling because it reads shape without synchronising (sgl-project#790); the `or []` slipped in regardless. Also: that except now logs the exception MESSAGE and not just its type. It stays swallowed -- an instrument must never kill the scheduler it measures -- but a bare type name is a diagnostic dead end, and it cost a boot. Tests: - New test_pp_admission_prefix_indices_tensor_796.py: 9 passed. Covers all four observed shapes (empty tensor, multi-element tensor, absent/None, plain list) for the builder, and the ADMIT-path trace for the instrument. - Can-fail measured for both halves against the old spelling: 3 failed / 3 passed for the builder, 2 failed / 7 passed for the trace. The cases that fail are exactly the empty and multi-element ones, as predicted; a test covering only None and a one-element tensor would have passed against the defect. - Full test/registered/unit/managers at the previous commit: 2745 passed, 18 skipped, 305 subtests passed, 47 failed -- all 47 in the same three files baselined at HEAD, which fail identically without any of these changes.

… it with a void Boot instr11 wedged 4m18s into a flip drive: GPU util 0/0/0, log growth 846 B in 8 s, eight requests in flight and never served. py-spy on all three ranks gave a three-way ring: PP0 _event_loop_pp_body:779 -> ... -> _pp_recv_dict_from_prev_stage:3865 -> BLOCKING recv PP1 _event_loop_pp_body:603 -> pp_chain_receiver.recv:329 PP2 _event_loop_pp_body:603 -> pp_chain_receiver._advance:218 ROOT. A sgl-project#791 downstream retraction desynchronises the output ring, because the ring's two gates read two DIFFERENT ranks' mbs. reconcile_pp_admission_decision (pp_admission_congruence.py:462-483) amends the decision for every remaining DOWNSTREAM rank, and PP0 is upstream of every rank that can retract. The boot's last four admission passes, three passes after a tp_to_pp cutover, show it exactly: PP0 sgl-project#788 verdict=ADMIT n_reqs=1 rids=2f5e25a1... prefix_lens=512 PP1 sgl-project#791 unhonourable prefix on rank 1: rid=2f5e25a1... told=512 local=0 PP1 sgl-project#788 verdict=DECLINE n_reqs=0 PP2 sgl-project#788 verdict=DECLINE n_reqs=0 PP0 kept the microbatch and launched it; the downstreams had nothing for it. Last rank's send gate (:4101) reads its own empty slot and sends nothing; PP0's receive gate (:4306) reads its own occupied slot and blocks (:4315). _pp_output_exchange_due (:328) made the two the same EXPRESSION (sgl-project#753) but cannot make them the same FACT. PP0 never reaches the top of its next pass, so the pass-N+1 chain send is never posted and both downstreams block. The idle tail is load-bearing: a skipped send mid-burst is only a lag, since the pair is FIFO. It becomes a DEFICIT exactly when the pipeline goes quiet behind the retraction, which is what the log shows -- DECLINE, DECLINE, silence. sgl-project#796 IS NOT THE CAUSE, and this retires that hypothesis. The wraparound it removed is an ADMISSION_DECISION_KIND message, and recv_typed_tensor_dict (pp_typed_channel.py:136-145) returns only on expected_kind, stashing everything else. No wraparound could ever have released an expected_kind="output" receive, before or after sgl-project#796. Pinned by test_wraparound_kind_cannot_satisfy_an_output_receive. Restoring it would have put one unmatched message per pass back on the channel and unwedged nothing. FIX, following sgl-project#791's own law (decide on rank 0, carry the decision) rather than a timeout. _PP_OUTPUT_EXPECTED_KEY (:136) rides on the admission decision that already travels 0->1->... ->last in the SAME pass; PP0 publishes _pp_output_exchange_due(self.mbs[mb_id]), the identical expression on the identical object its own _do_recv will apply, and middle ranks forward it verbatim. The last rank (:4128) sends a void (_pp_void_output_payload, :4148) only for a slot PP0's own published verdict obliges it to receive -- so this is not the bounded-recv corpse in reverse. PP0 absorbs it (:4315, _pp_absorb_void_output, :4171), empties the slot so the loop's None guard holds, releases each request through the existing idempotent _release_dynamic_chunk_probe and re-queues it, and feeds record_return_trip from the chain-reconciled decision riding back inside the void. That last part also repairs collateral damage from sgl-project#796: record_return_trip (pp_admission_congruence.py:276) was fed ONLY by the wraparound, so since sgl-project#796 _learned_floor was never populated and prefix_len_for never clamped -- sgl-project#630's termination argument had no feeder at all. It does again, without an unmatched message. Absent key => False => no void => byte-identical to today, so stand-ins and pp_size<=1 are unchanged. TESTS. test/registered/unit/managers/test_pp_output_ring_retraction_wedge_791b.py, three live spawned processes, real gloo, the shipped functions: test_retraction_wedges_the_ring_without_the_fix (can-fail: neuters ONLY the fix IN THE CHILD and reproduces the specimen -- 3/3 ranks stuck, downstreams ahead of the rank they block on, the inverted stagger) test_ring_survives_the_retraction_with_the_fix (12/12 passes, 1 void absorbed on PP0, 1 request re-queued, 0 voids on PP1/PP2) test_void_payload_is_only_sent_when_the_first_rank_expects_one test_wraparound_kind_cannot_satisfy_an_output_receive -> 4 passed, three consecutive runs (40.95 / 39.57 / 38.44 s) Neighbours (795, 796) -> 13 passed. Wider mixin set -> 19 failed / 176 passed, IDENTICAL at HEAD (patch removed, re-run, re-applied); all 22 are _RingWire/_Group stand-ins missing is_first_rank/is_last_rank, i.e. sgl-project#796's own gate. codespell clean; the one ruff F841 at scheduler_pp_mixin.py:1395 is pre-existing and untouched. KNOWN GAP, filed rather than folded in: a PARTIAL retraction (one rid of several) keeps the ring matched, so PP0 processes a real output for a batch containing requests the downstream never ran -- a membership divergence with no wedge and no error. Same root asymmetry, different defect.

…ction Boots instr15, instr16 and instr17 all died on the same raise, and it was not a leftover: ValueError: sgl-project#631 PP proxy/batch mismatch: received hidden_states with 126 row(s) for a 1 batch of 22 token(s) (instr17, 07:12:49 PP1) 126 = 22 + 104. PP0's batch is PP1's batch PLUS THE ONE REQUEST PP1 RETRACTED. Same pass, same slot, same epoch; the stamp was correct in every element, which is exactly why PROXY LEFTOVER REFUSED stayed 0 on all three boots. Nothing was stale, so no stamp discriminator -- per-slot generation, receiver-derived seq, rolling pass id -- could ever have caught it. Two earlier hypotheses are retired by this arithmetic: a within-epoch stale slot (there is no stale message) and a cutover landing between receive and use (the traceback shows a FRESH _event_loop_pp_body:960 after the re-dispatch, and pp_proxy_tensors is a loop local no re-dispatch carries across). MECHANISM, from boot_instr17.log:62997-63056, all 07:12:49: PP0 sgl-project#788 verdict=ADMIT n_reqs=2 rids=51a294650b...,5e744c29f8... prefix_lens=0,16896 PP1 sgl-project#791 unhonourable prefix on rank 1: rid=5e744c29f8... told=16896 local=0 PP1 sgl-project#788 verdict=ADMIT n_reqs=1 rids=51a294650b... prefix_lens=0 PP1 ValueError: 126 row(s) for a 1 batch of 22 token(s) reconcile_pp_admission_decision drops the unhonourable rid from `effective` (pp_admission_congruence.py:472-483), the admission loop omits it from THIS rank's batch (scheduler.py:6974-6991), self.mbs[mb_id] is the narrowed batch (scheduler_pp_mixin.py:806) -- while the upstream has already sent its decision and launched its own wider batch (:876, :947). A batch in flight cannot be amended. The flip is the TRIGGER, not the cause: it cold-starts the downstream radix cache, so PP0 offers a prefix PP1 has nothing for. #791b already fixed the OUTPUT-RING consequence of this same retraction (scheduler_pp_mixin.py:4317-4319); the proxy-width consequence was never covered. THE DISCRIMINATOR IS NOT ON THE WIRE, and it does not need to be: "did I retract anything from this pass's decision?" The receiver PERFORMED the retraction, at the top of the same pass, strictly before the proxy receive, and #791b already records the amendment per slot in _pp_admission_amended_by_slot. Nothing new crosses the wire and the receiver predicts nothing a sender wrote. The test is `retracted_by_rank == self`, not `retracted`, so an entry an earlier rank retracted is correctly ignored -- pinned as a regression case. WHY THIS BEATS THE SHIPPED TRIPWIRE WHERE IT MATTERS: chunked prefill caps every chunk at the same size, so two ranks running different request sets routinely present EQUAL widths. model_runner.py:4182 is blind to that -- silent wrong output, not a shape error. Pinned as test_a_same_width_divergence_is_still_refused. STATED PLAINLY, THIS DOES NOT YET PREVENT THE DEATH. It converts a shape error raised thirty layers deep into a boundary refusal that names rid, told= and local=. Prevention is a separate design call, deliberately not taken unilaterally here: PP0 must stop offering a prefix the downstream cannot honour, i.e. PPAdmissionCongruenceGuard.prefix_len_for needs the feedback sgl-project#796 deleted with the wraparound (scheduler_pp_mixin.py:3568-3594). The channel that survives sgl-project#796's "no send no peer must take" law is the OUTPUT message PP0 already receives from the last rank -- piggyback the retraction floor as a per-hop key, the _PP_OUTPUT_EXPECTED_KEY precedent. That still costs one wasted PP0 forward the first time, and voiding the retracting rank's pass desyncs PP0's own request state, so both halves must be decided together, on metal. ALSO FOUND, NOT FIXED HERE: two rank-local decode-retraction paths (scheduler.py:7517/:7489 and :7211-7312) diverge the same way and sgl-project#791 never touches them. TESTS. test_pp_proxy_retracted_pass_mispair_791c.py -- three live spawned processes, real gloo, shipped functions, real 126-row and 22-row tensors: test_a_retracted_pass_is_mispaired_without_the_retraction_test (can-fail: blinds ONLY entries_retracted_by_rank's return value in the CHILD, through scheduler_pp_mixin's own module globals, so every API, the guard body, the reconciliation and the amendment recording all still run -> 126 rows delivered for the 22-token batch = THE SPECIMEN, and the child's `effective` still shows the narrowed set, proving the retraction happened) test_a_retracted_pass_is_refused_by_the_receive_guard test_a_retraction_by_another_rank_does_not_refuse test_a_same_width_divergence_is_still_refused test_an_unretracted_pass_is_delivered_unchanged -> 5 passed; with the 631 and 795 neighbours 29 passed Neighbour set 631/795/791/791-wiring/791b/757 re-measured AT HEAD: 44 passed / 1 failed; after: 49 passed / 1 failed, the same pre-existing _RingWire.is_last_rank drift from sgl-project#796. ruff, ruff-format and codespell finding sets byte-identical to HEAD. The 631 test's residual pin is corrected: it had misdirected two investigations into hunting a stale message that never existed.

…fix floor back to rank 0 #791c made a downstream REFUSE a proxy for a pass it had narrowed. It detected; it did not prevent, and the frequency data is why that mattered: boot 'unhonourable prefix' fatal width mismatch instr15 661 1 instr16 1651 1 instr17 1718 1 instr17 retracted 1718 times and only ONE differed in width enough for model_runner.py:4178-4191 to see. The other ~1717 were SAME-WIDTH divergences that computed silently, because chunked prefill caps every chunk at the same size, so two ranks running different request sets routinely present equal widths. TRIGGER, measured: retractions are FLIP-triggered. Boots driven with short prompts only, where no runtime flip ever arms, logged ZERO retractions (boot_coh, boot_detval); every flipping boot logged hundreds. The cutover cold-starts the downstream radix cache, so PP0 offers a prefix the downstream has nothing for. BOTH HALVES, decided together as they must be. (b) THE RETRACTION VOIDS THE PASS, NOT THE RID. Three membership outcomes exist and two are physically unavailable: the rank cannot admit the rid (it has no KV for the prefix its upstream reused, and the upstream sent hidden states only for the extend tokens), and the upstream cannot be amended (it sent its decision and launched earlier in the same pass). What remains is to run the pass nowhere. `effective` is emptied and every surviving entry becomes admitted=False, retracted=False -- the third state, which record_return_trip correctly leaves alone (it teaches on `retracted`, clears on `admitted`). _PP_PASS_VOIDED_KEY carries the fact downstream because the entries alone cannot: a rank with nothing to prefill falls through to its running DECODE batch and would pair that with the upstream's prefill batch. (a) THE FLOOR COMES BACK ON THE OUTPUT MESSAGE. PP0's request state is resynced by #791b's existing void-output absorb, which rides the chain-reconciled decision home so the guard learns observed_local and prefix_len_for clamps the next offer -- strictly decreasing, non-negative, terminating. sgl-project#797 additionally puts that payload on a SUCCESSFUL output, the half that lets a floor be CLEARED: sgl-project#796 removed the only feeder for that and #791b restored only the learning half. A GAP FOUND AND CLOSED, or this would have traded corruption for a HANG: #791b's void stops at the first rank. With the retraction on rank r, ranks 1..r-1 also hold launched batches with output receives posted, so PP0 absorbing and forwarding nothing would block them. Pre-sgl-project#797 that shape mispaired instead (the retractor narrowed rather than emptied, so a real output still went out), which is why it never showed. r=2 is reachable at pp_size=3, i.e. on this rig. The void now travels last->0->1->...->r-1 and stops, gated on pp_first_retracting_rank read off the decision the void already carries. No new key. WHAT THIS DOES NOT DO, stated because the acceptance gate depends on it: IT DOES NOT REACH ZERO RETRACTIONS. Expected is ~one per rid whose first post-cutover offer is unhonourable, against 661/1651/1718 -- the count stops being per-PASS and becomes per-RID-per-CUTOVER, because the floor now comes back and clamps the next offer. Zero is unreachable this way: PP0 cannot know a downstream's cache state before it offers, and "offer and find out" is the only channel that exists without a new collective. The counter that MUST be zero with this fix is the #791c divergence tripwire, which proves no narrowed pass was ever created. Reaching zero retractions would require clamping PP0's first offer per rid per flip epoch to told=0 -- sound, ~15 lines in prefix_len_for, but it pays away exactly what prefix caching buys, so it is not taken unilaterally. ITEM 2, and the answer is NO: the two rank-local decode-retraction paths (scheduler.py:7489/:7517 and :7211-7312) cannot produce this class. They CAN diverge in membership -- _update_uniform_pool_budget reduces on tp_cpu_group, world=1 under TP=1/PP=3, so the floors are off, as the boot line at scheduler.py:4733-4740 says. But not SILENTLY: _get_decode_retraction_order (schedule_batch.py:3038-3082) sorts on replicated per-request state under replicated server args, retract_decode pops only from the END of it (schedule_batch.py:2961) so victim sets are NESTED and never disjoint, a decode batch's row count is a strict function of its request count, and retract_decode entered with >1 request always retracts >=1. So every divergence there is a WIDTH divergence, raised loudly every time. Real defect, different root (the reduce group), and sgl-project#797's remedy structurally does not fit it -- voiding a pass repairs per-pass membership, while a retracted decode victim mutates long-lived state. Filed at the site (scheduler.py:7554-7600), not folded in. TESTS. test_pp_retracted_pass_void_797.py: 11 passed (7 three-process gloo arms + 4 pure), 61 s. Three separate can-fail neuters, each a single return-value rebind through scheduler_pp_mixin's module globals IN THE CHILD: blind entries_retracted_by_rank -> 3/3 green assertions fail; victim reports rows=128 foreign_rows=64, the same-width mispair accepted blind pp_pass_should_void only -> 3/3 fail, victim raises #791c PROXY BATCH DIVERGED (detection, not prevention) blind the per-hop launched key -> 1/3 fails, drained=False stashed=1, the stranded proxy corpse Neighbours 631/791/791b/791c/795/796 re-measured AT HEAD: 4 failed / 67 passed; after: 4 failed / 78 passed, failure names byte-identical. Zero regressions, +11. With 791b and 791c together: 20 passed. ruff/ruff-format/codespell identical to HEAD.

…ch, not only its admission dict _pp_void_retracted_pass empties the admission dict (`effective`) and the forwarded decision (`amended`), but neither is what get_next_batch_to_run schedules from: its local continuation logic (the #797b chunked_req stash and the resident running batch) runs before that function's own _pp_admission_pass_voided guard and can still hand back a non-empty plan.batch_to_run for a pass this rank has already decided to run nowhere. scheduler.py's schedule-refusal path (sgl-project#796) sets the same flag from inside get_next_batch_to_run, after that guard has already run, and reaches the same state. Downstream, the uncleared slot makes _event_loop_pp_body take the `if cur_batch:` branch and block in _pp_recv_proxy_tensors on a proxy the voided upstream rank never sends, because it took _pp_drain_voided_proxy instead. That is the admission wedge observed on metal after a tp_to_pp flip on the first request: rank 0 holding a batch, rank 1 with cur_batch None after a retraction with an empty effective admission, rank 2 waiting (SPECIMEN_794_admission_wedge_18-45, boot_restore_785.wedge_19-02). _pp_void_own_batch clears self.mbs[mb_id] and mb_metadata[mb_id] immediately after get_next_batch_to_run: strictly before this pass's admission decision is sent, so a voided slot never reports launched=True, and strictly before `cur_batch = self.mbs[mb_id]` is read, so the voided rank takes the drain branch. It restores the pre-admission chunked request and parks it rather than retracting it, keeps resident decode requests, and re-queues only the requests this pass newly admitted. running_batch/running_mbs are deliberately left alone: they carry the prior pass's finished batch, not this pass's voided admission. The gate is idempotent, so it is defense in depth behind the existing guard rather than a competing source of truth. Tests: test_pp_retracted_pass_void_797.py, 30 passed. The new PPVoidOwnBatchWiring797d class drives the real _event_loop_pp_body through the production call site and asserts the voided rank never reaches _pp_recv_proxy_tensors; disabling the two-line call site turns it red (1 failed / 29 passed), so the wiring is proven, not only the helper. Regression: test/registered/unit/managers, 45 failed / 2986 passed / 308 subtests at this change against 45 failed / 2979 passed / 308 subtests at HEAD on the same selection. The failing test ids are byte-identical between the two runs; the delta is the 7 added tests. ruff check, ruff format and codespell clean on both files.

…he fundable one _seam_staging_ask_bytes charged the sizing budget for the pp_to_tp staging leg alone. tp_to_pp abstained deliberately, on the argument that its pool is about to become active again and recover_kv_backing would undo the shrink inside the same flip, so charging for it would hold memory against a payment nobody makes. That argument assumes recovery can always pay, and it cannot. KvBackingRelief. recover is bounded by the corridor law rather than by what the seam needs (headroom = free - law_floor), so once the pool has been sized to rest near that law floor -- exactly what happens when nothing reserves slack for the uncharged leg -- recovery has nothing left to give. It then says so and stops: "recovery deferred: N MiB free leaves nothing above the 1024 MiB corridor law to re-commit with, so the pool stays at X of X rows". The uncharged leg can never be funded, at sizing time or at flip time. Measured end-to-end on this rig (boot_restore_797.log). All three ranks logged the recovery-deferred line during boot. The first tp_to_pp flip then succeeded, and every later one was abandoned at a stable shortfall of 127 to 164 MiB -- "staging 2068 MiB needed but only 1911 MiB is spendable (driver free 2730 MiB)" -- until eight consecutive abandons reached the cap and the seam was declared unfundable, standing the phase flip down permanently. The instance stayed pinned in one layout, the pool-binding phase became unsamplable, and long contexts were refused outright as the sole request in the batch, while the other two ranks sat on 1826 and 2866 MiB of free VRAM. Both directions are now projected and the LARGER is charged. max rather than a sum, because only one direction stages at a time: the pool must cover the worse leg, not both at once. That is deliberately unlike the arming-floor and staging pair, which pool_flip_posts_bytes adds precisely because those two are needed at the same instant. Nothing else moves: the result flows through the existing staging_post_bytes and pool_flip_posts_bytes chain, the projection is the same one the gate itself calls, and it needs no information that is unavailable at sizing time -- the only live-set input is the slot count, which is a static boot-time config value. The provenance string now names which leg set the price and carries both measurements, so an operator can see the choice without reading the code. Tests: test_seam_staging_both_legs_796.py, 13 passed. Can-fail verified by restoring the single-leg charge, which turns 10 of them red including both CanFail796 guards. The suites that pin this arithmetic stay green: 109 passed across test_staging_post_771.py, test_arming_floor_funding_662.py and test_phase_flip_seam_reserve.py -- no exact-formula pin needed re-deriving, because they test the pure functions given an already-computed ask rather than which direction produced it. Regression: test/registered/unit/model_executor, 15 failed / 704 passed at HEAD and 15 failed / 717 passed with this change, failing test ids byte-identical; the delta is the 13 added tests. ruff check finds the same single pre-existing issue as HEAD, and the new test file is ruff-format and codespell clean.

The seam's funder is a group decision: collective_kv_target takes an element-wise MIN over per-rank proposals and returns one absolute row target, or None. It returned None and said nothing about why. Measured on metal 2026-08-22 (boot_798_0822_0543.log), three PP ranks. PP0 was short of seam staging and its rung held a fundable plan -- current=204800 rows, floor=115681, slack=89119, deficit=+1740 MiB -> SHRINK to 149126. No shrink ever ran. That boot contains no occurrence of runtime_set_backing_rows, of "the eviction did not deliver the mark", or of "ABSTAIN on device": the decision returned None and apply_target was never reached. Eight flips abandoned, phase purity yielded, and the instance went on prefilling in the TP layout. The term that decided it was a PEER's floor. target = max(desire, max_floor) must clear every rank's live set, so PP2 -- under no memory pressure at all, 2693 MiB spendable, reporting fundable_bytes() == 0 on all eight of its asks, which by construction puts its floor at or above its own cap -- vetoed the shrink the pressed rank needed. Every rank computes its floor and retains it in _last_proposal_terms; only a rank that REFUSES ever printed it, and the vetoing rank is precisely the one that fits. So the number that decided the group's outcome appeared in no log at all. explain_kv_target re-derives the same three terms from the same reduced tuple and names which one bound the verdict, distinguishing the four outcomes that want different responses: an abstention (repair that rank), a peer-floor veto (lower that peer's floor), a cheap-tier decline (the tier law working, wants nothing), and a granted target that a peer's floor RAISED above what was asked. It is deliberately a separate pure function rather than an out-parameter threaded through the decision, so a diagnostic can never alter the verdict it reports. The gate now logs that verdict together with this rank's own proposal terms on EVERY rank, including the ones that fit, at the same once-per- seam cadence as the "returned NOTHING" line it already emits. No behaviour change: the decision function is untouched, and the new test pins the metal shape as still declining. Tests: test/registered/unit/managers/test_kv_target_decline_reason_796.py (6 new, red before this commit -- explain_kv_target did not exist). Regression sweep over the KV rung / collective family, 167 passed: test_kv_backing_collective_631, test_kv_backing_cap_agreement_656, test_kv_rung_unreachable_floor_714, test_evict_rung_floor_invariant_717, test_evict_rung_nothing_resident_717, test_evict_rung_flip_park_744, test_kv_backing_relief_631, test_kv_backing_exhaustion_662_f4, test_kv_backing_recovery_clamp_684. The 3 failures in test_collective_family_siblings_610 (PrefillAdder.chunked_admission_enabled) were verified identical on a pristine HEAD worktree and are pre-existing. ruff check, ruff format, codespell clean.

… it is _evict_floor_rows has eight ways to return the plain, un-evicted floor: eviction disabled, an unreadable parked extent, no tree cache, an unknown resident half, a mark pinned by work in flight, a priced floor no better than the plain one, a pricing exception, and nothing evictable above the reserve. All eight produce one observable. Over a sparse live set that observable is a group-wide veto. sgl-project#714 established that max_live is a high-water ID in the id space rather than a count of backed rows, so the plain floor routinely sits ABOVE the cap; the rung then reports slack 0, and because the group's agreed shrink target must clear the HIGHEST floor in the group, this rank cancels the shrink for every rank -- including one holding a fundable plan. Measured on metal 2026-08-22: PP0 could have returned +1740 MiB from 89119 rows of slack, PP2 was under no memory pressure at all and reported fundable_bytes() == 0 on all eight of its asks, and the flip abandoned eight times. Which of the eight branches held on PP2 is not recoverable from that boot, because none of them says so. Three are healthy, one is a setting, and the rest are defects -- they want opposite responses, and telling them apart currently costs one boot per hypothesis. Every return now records its reason, and last_proposal_summary prints it whenever the rung has no slack, which is exactly when that floor is capable of being the binding term for the whole group. No behaviour change: only assignments to a new attribute and the text of one diagnostic string. Tests: test/registered/unit/managers/test_evict_floor_reason_796.py (7 new, all red before this commit -- no reason was recorded on any branch). Regression sweep over the KV rung / collective family, 159 passed: test_kv_backing_collective_631, test_kv_backing_cap_agreement_656, test_kv_rung_unreachable_floor_714, test_evict_rung_floor_invariant_717, test_evict_rung_nothing_resident_717, test_evict_rung_flip_park_744, test_kv_backing_relief_631, test_kv_backing_exhaustion_662_f4, test_kv_backing_recovery_clamp_684. ruff check, ruff format, codespell clean.

…hipped gate A fix wired into nothing is the defect class this repo keeps finding, and a diagnostic wired into nothing is the same defect wearing a friendlier face: it is indistinguishable from a healthy mechanism, which is exactly the confusion sgl-project#796 exists to end. The two preceding commits unit-tested explain_kv_target and the recorded evict-floor branch; neither proved a boot would carry the line. These tests call the shipped entry point, phase_flip_spill.collective_kv_backing_relief, with a peer injected into the reduction, and assert on captured log records that the verdict is emitted, that it names the peer floor as the binding term, and that THIS rank's own proposal terms ride the same line -- the whole point being that a rank which FITS is the one whose floor vetoes its peer, and until now a rank that fit reported nothing. Can-fail proof, executed rather than asserted: the same file run against a pristine HEAD worktree fails 4 of its 5 tests. The fifth, test_the_veto_really_does_suppress_the_shrink, passes on both by design -- it pins the metal observation that apply_target is never reached under a peer-floor veto, which is existing behaviour this ticket has not changed. test_a_permissive_peer_lets_the_shrink_through is its can-fail partner: same call, peer floor lowered, and the shrink must go through, so the veto pin cannot be satisfied by a mechanism that simply never shrinks. Tests: test/registered/unit/managers/test_kv_verdict_is_wired_796.py (5 new). ruff check, ruff format, codespell clean.

…a row id The seam's funder was the last axis of this fork still assuming that every rank's pool is the same size. It is not: uneven TP shards, uneven DCP tokens and the KV ratio all exist precisely because it is not. MEASURED, boot_798_0822_0629.log under live load, 114 identical declines and this reason only: DECLINED because no rank asked to go below the group's smallest cap (deepest desire 112640 rows against cap 112640) PP0 cur=204800 floor=19529 slack=185271 deficit=+1576 MiB -> SHRINK to 154376 PP1 cur=135168 floor=19529 slack=115639 deficit= +36 MiB -> SHRINK to 126976 PP2 cur=112640 floor=19529 slack= 93111 deficit= -55 MiB -> no change PP0 sat 295 MiB short of seam staging while its own rung offered 1576 MiB -- five times the shortfall -- out of 185271 rows of slack above a floor of 19529. Nothing refused it. The shrink was not REPRESENTABLE. Two faults, stacked, neither sufficient alone: 1. propose() encoded "no change" as desire = current, an absolute row id. PP2 needed nothing and proposed 112640; because its pool is the SMALLEST that was the smallest number in the group, so it won the MIN documented as "the most-pressed rank sets the ambition". It did the opposite -- the least-pressed rank with the smallest pool set it. 2. Even repaired, PP0's ambition of 154376 lies ABOVE PP2's entire pool, so it can never fall below min_current and the group concludes nobody asked for anything. THE NATURAL EXPERIMENT that isolates it: the same boot GRANTED exactly 3 times, and every grant is in the round where all three ranks report cur=450560 -- the EVEN-pool layout. Even pools work; uneven pools declined 114 times. The treatment was assigned by the layout, not by us. THE FIX IS A CHANGE OF CURRENCY, not of policy. Ranks propose a PROPORTION of their own cap in parts per million; "no change" is 1000000, which is the true neutral element of a MIN, and each rank converts the agreed proportion against its own cap and its own floor. Rows are a rank's private unit on an uneven fleet; a proportion is the same statement on every rank. WHAT IS DELIBERATELY NOT CLAIMED, because the next reader will assume it: this is NOT "no rank is worse off". The group pays PROPORTIONALLY. A ppm derived from PP0's need takes PP2 from 112640 to about 84918 although PP2 had no deficit at all. That is chosen, not overlooked. Shrinking only the pressed rank would change the ranks' capacity RATIO, and the uneven DCP token vector is calibrated against that ratio, so a single-rank shrink invalidates the vector -- HANDOFF_675 §1a's admission desync in a new dress. Preserving the ratio is what keeps admission congruent. PP2 stays legal throughout: its floor is 19529 and its slack 93111. The safety law is untouched and stays where it belongs. A peer whose floor is at its own cap still declines the group, because unmapping through a live set is cudaErrorIllegalAddress; that is a property of BACKING, and apply_shrink_ppm applies each rank's own floor before any unmap. collective_kv_target keeps its name and still returns ROWS, defaulting to the group's smallest cap -- which on an even fleet is every rank's own cap, so the suites predating this change read exactly as they did. collective_kv_shrink_ppm is the proportion the production path uses. Tests: test_kv_target_decline_reason_796.py rewritten around the measured shape (9 tests; its earlier peer-floor premise was falsified by this measurement and is withdrawn). Executed can-fail proof against a pristine HEAD worktree: the metal shape returns None (the defect) while the even-pool round returns 402344 (works) -- the natural experiment reproduced hermetically. Three encoding assertions in test_kv_backing_collective_631 updated to the neutral element; the laws they pin are unchanged, only the unit is. Full sweep over phase_flip / corridor / seam / kv / rung / spill: 983 passed, 8 failed, and those 8 (test_kv_arena_handle_retention_631, test_kv_arena_span_ops_631, "retry() exceed maximum number of retries") were verified identical on a pristine HEAD worktree and are pre-existing. ruff check, ruff format, codespell clean.

…the proposal The verdict line added for sgl-project#796 reused last_proposal_summary() on the GRANTED path. That method is scoped to the caller that REFUSES and compares desire against current, i.e. the PROPOSAL. Under a proportional agreement the proposal and the applied action diverge by design: the group agrees a proportion, so a rank that asked for nothing still pays its share. MEASURED, boot_798_0822_0646.log seam at 06:50:59Z. The group granted 76.1% and every rank converted it against its own cap: PP0 155853 instead of 204800 (released 1344 MiB) PP1 91954 instead of 120832 (released 702 MiB) PP2 96629 instead of 126976 (released 704 MiB) In that same round PP1's clause read "deficit=-199 MiB -> no change (the cheaper tier covered the gap)" while PP1 unmapped 702 MiB. The line exists so a seam's funding story can be read without deriving it again. Within ten minutes of its first boot it convinced its own author that unpressed ranks do not pay, which is the opposite of what the mechanism does; only the applied "KV-BACKING released" lines overturned it. A diagnostic that misstates the applied action is worse than none, because it is trusted. preview_shrink_ppm() returns what apply_shrink_ppm() will do, and the applier is now implemented on top of it, so the report cannot drift from the behaviour again. explain_shrink_ppm() formats this rank's clause and states plainly that every rank pays the group's proportion and why: the capacity ratio across ranks is what the uneven DCP token vector is calibrated against, so shrinking only the pressed rank would decalibrate it (HANDOFF_675 1a). The declined path keeps last_proposal_summary(), which remains correct for it. The call site tolerates a rung without the new method rather than raising, since a diagnostic may never take the seam down. No mechanism change: identical rows are unmapped before and after. Tests: test_verdict_reports_applied_796.py, 4 tests, red first -- the failure reproduced the metal line verbatim, including 91954 and the words "no change". 49 pass across the sgl-project#796 family and test_kv_backing_collective_631.py.

…uest across a void The first tp_to_pp cutover this rig has ever committed (boot_798_0822_0646, commit 9478e77 -- before the sgl-project#796 seam fix the flip declined 114 times and never got here) crashed four seconds later: scheduler.py:5675, get_next_batch_to_run if self.chunked_req.extend_range.end > len(self.chunked_req.prefix_indices) AttributeError: 'NoneType' object has no attribute 'end' PP1 and PP2 then died on gloo "Connection closed by peer" -- one cascade, not three faults. MECHANISM. _pp_absorb_void_output fires ONCE PER SLOT, and several slots are absorbed back to back with no get_next_batch_to_run between them; the log shows three void-output lines on slots 2, 0 and 1 immediately before the crash. Each call restores its own slot's chunked_before into self.chunked_req and then resets every batch member not kept FOR THAT SLOT. A request that is slot B's carried chunk but an ordinary member of slot A's batch is therefore reset by A's disposal loop and reinstated, already reset, by B's restore. _pp_park_chunked_prefill_chunk cannot repair it: reset_for_retract has cleared extend_range, so the park sees nothing to give back. pp_void_keeps_request cannot prevent it either -- it is asked per slot, and per slot it answers correctly. The own-void twin site (#797d) has carried this guard for some time; its comment argues the state is unreachable there because chunked_before is a snapshot from the top of the same pass. That argument holds for a single-call site and does not survive a run of per-slot absorbs. Only one of two identical sites had the guard. The condition names the class rather than the symptom: is_retracted is set by reset_for_retract (schedule_batch.py:1590, which clears seventeen further fields in the same breath) and cleared only by prepare_for_extend (:2439), so it is true over exactly the window in which the request must not be carried. extend_range is kept beside it as the narrower belt -- it is what actually raised, and a request could in principle reach the reset shape by a path that does not set the flag. TESTS. New suite test_pp_void_chunked_retracted_798.py, 6 green. Can-fail executed both directions on the same tree via patchfile: guard off -> 2 failed, 3 passed; the failing arm raises the boot's own exception verbatim, AttributeError: 'NoneType' object has no attribute 'end' guard on -> 6 passed The three that stay green in the red arm are load-bearing: the precondition that slot A's disposal really does reach slot B's chunk, the healthy-carry case, and a drift pin that reads the real Req.reset_for_retract source so the fake cannot stop describing production. Matched A/B over the 31 pp_ suites in test/registered/unit/managers, same tree, only this patch toggled: guard off -> 18 failed, 278 passed, 39 subtests passed guard on -> 16 failed, 281 passed, 39 subtests passed The 16 remaining reds are byte-identical to the guard-off list with this suite's own entries removed, so the change is regression-neutral. Those 16 predate this work; ten are AttributeError: '_Group' object has no attribute 'is_first_rank' in test_pp_slot_last_batch_631 and test_pp_flip_slot_hold_631. The sweep earned its keep. A first version of this guard read self.chunked_req directly and turned test_pp_output_ring_retraction_wedge_791b red (guard on 1 failed / 3 passed, guard off 4 passed -- causal, verified standalone in both directions). The restore above assigns the attribute only when the carried value differs from the current one, so a scheduler that never set chunked_req still does not have it by the time the guard runs, and the ring's worker builds exactly that holder. The guard now uses the getattr idiom the surrounding lines already use, and the no-attribute shape is pinned in this suite rather than left to the ring. ruff and codespell clean on both files. The single ruff finding in scheduler_pp_mixin.py, carries_flip_arm at :2112, is unchanged at HEAD.

…ole-flip second count The REFILL instrument printed each leg's duration beside "the ~3.1 s pinned baseline". On 2026-08-22 a briefing built on that line concluded the flip economy was broken, put the cost at 64% of wall clock against an 8x regression, and went looking for a silent host-RAM fallback to blame. The line invites every step of that reading, and the comparison it offers is invalid three ways: SCOPE ~3.1 s is a WHOLE FLIP (NOTE_677_floor_components.md:135-143 uses it as "Against a ~3.1 s flip"), not a refill leg. PATH sgl-project#690 measured the PINNED image path (NOTE_690_gdn_state_spread.md:58-85), which predates the file-backed arm entirely, so it is not a baseline this path ever held. BYTES sgl-project#690 moved 9614.9 MiB/rank; these legs move 8574-16363 MiB, and elapsed time tracks bytes moved (r ~ 0.80 over 45 logged legs), so seconds are not comparable across them. There is no regression and no fallback here. The file-backed arm is an explicit opt-in (--phase-flip-image-file-backed, server_args.py:5818), and weights_arena.py:441-445 states it REFUSES rather than falls back, naming the silently-inert-flag class as the reason. The boot log carries "FILE-BACKED (reclaimable)" nine times and zero fallback warnings. Its help text names what the slower path buys: without it the images are ~68.7 GiB of unreclaimable host RAM on a swapless box and the boot is OOM-killed during init. So the line now reports a rate against a rate and carries the reference's conditions, which are what make a baseline transferable at all. It still says the arm is slower, because it is; what it no longer does is let that read as a defect to hunt. The instrument is extracted as refill_report() so it can be tested without a flip. Behaviour is otherwise unchanged: same call site, same try/except, an instrument may never break a flip. Tests: test_refill_baseline_honesty_802.py, 6 tests, red first (all six failed on the missing entry point, then on the bare second-count and the absent conditions). 55 pass across the refill and sgl-project#796 families.

…f liveness Branch 8 of _evict_floor_rows returned "healthy, the pool is genuinely live" for every zero that came back from evictable_rows_above, without having measured liveness anywhere. Two different conditions reach that zero and they want different answers. When the priced floor sits ABOVE the high-water row, the query asked for evictable rows in a region the tree cannot hold anything in, so the zero is a tautology. _floor_rows(x) == x + 1 + margin + reserve, so a resident ceiling within (margin + reserve) rows of the high-water lifts the priced floor past it. The guards above cover req_max >= max_live and floor >= plain; neither covers this, so it fell through to the pricing call and was mislabelled by it. This is sgl-project#714 arriving at the high-water mark rather than at the pool cap. When the floor is genuinely below the high-water, the band is real and the tree pricing nothing in it may be health or may be unowned rows. The message now names the size of the band and points at the POOL CENSUS line, which is where that is settled, instead of deciding it. Measured on boot_798_0822_0737.log while the flip stayed wedged in TP: priced floor 167440 against high-water row 164055 (empty region), and priced floor 97643 against high-water row 134148 (a real band of 36506 rows priced at nothing, against a census reporting ~94000 unaccounted rows out of a 448698-row pool). Instrumentation only: both paths return (plain, 0) exactly as the single branch did, and a test pins that equality so the ladder cannot move. Tests: 165 passed across the eight relief/floor suites (test_evict_floor_reason_796, test_arming_floor_funding_662, test_evict_rung_floor_invariant_717, test_kv_rung_unreachable_floor_714, test_kv_radix_watermark_662, test_kv_backing_relief_631, test_admission_relief_ladder_679, test_relief_rung_executor_553), ruff clean. Can-fail proven against a git archive of HEAD rather than by toggling the patch in the shared worktree: the two new assertion tests fail there ("'genuinely live' unexpectedly found", "'census' not found") while the neutrality test passes in both states.

…the unaccounted rows `_pool_census` derives `unaccounted` as `set(range(1, size+1)) - free - cached` from exactly ONE allocator and ONE tree, both read off the scheduler, with `cached` collecting only BASE-component DEVICE values. So a row owned by a DIFFERENT pool object is unaccounted BY DEFINITION rather than by defect -- and the census line cannot tell the two apart. This names every pool object it can reach, so they stop looking alike. WHY NOW. On the 2026-08-22 flips the ambiguity was load-bearing. r5 read ~94000 rows (21% of a 448698-row pool) as unaccounted, FLAT across four censuses, which reads as an unenumerated owner rather than a leak (a leak accumulates). r6 then showed the flatness was itself an artefact of r5 being wedged in TP -- every sample fell in one phase. With the flip cutting over, unaccounted swings with the phase instead: 08:16:44 post-cutover tp_to_pp free=345101 cached=82589 unaccounted=4096 08:17:09 post-cutover pp_to_tp free= 18704 cached=82589 unaccounted=330493 Same size, 25 seconds apart, cached IDENTICAL, free collapsing by 326397 rows across the cutover into TP and returning across the cutover into PP. That is not a leak and not static: it is whatever the census-read allocator stops listing while the TP stack is live. A second owner is known to be possible there -- `model_runner.py` splits `is_draft_pool_worker` from `is_draft_worker` on `is_phase_flip_tp_stack`, so a runner on that stack owns pools the scheduler's handles do not name. LIKE-FOR-LIKE, which the first draft of this got wrong. The census holds an ALLOCATOR while the flip's own reshard builder names POOLS (`scheduler.tp_worker.model_runner.token_to_kv_pool` and `stacks.tp_worker.model_runner.token_to_kv_pool`, :2041-2042). Comparing an allocator id against a pool id answers nothing, so both sides are reduced to the pool (`_owner_pool_of`) before any id is printed. The PP-stack pair is named alongside the census pair for the same reason: if CENSUS tracks PP_STACK in BOTH phases while the TP stack is live, the census is scoped to the PP stack and the rows are a measurement artefact, full stop. SHAPE OF THE SET. The census prints `sorted(leaked)[:12]`, which cannot resolve contiguity -- a "one contiguous block" reading was made and withdrawn on exactly that evidence, and the first twelve ids are byte-identical between the 4096-row census and the 330493-row one 25 seconds later, because the sample is just the low end of a sorted set. This reports n/min/max/runs/longest_run instead: bounded output, and `runs == 1` versus `runs` in the thousands answers the question directly. Read-only and best effort throughout, preserving the existing rule that a census can never affect the flip it watches; a hostile handle is caught and reported rather than raised. TESTS: test/registered/unit/managers/test_census_owner_probe_773.py, 9 tests. Can-fail EXECUTED: breaking run detection fails 1, and removing the call from `_pool_census` fails 1. That second mutant SURVIVED the first version, because every test drove the helper directly and none drove the call site -- the third time in this task a direct-helper test failed to bind its own wiring, so it now has its own class. REGRESSION: managers flip/census/presence/spill selection = 604 passed, 9 failed, and those 9 are name-for-name identical to a control worktree with my commits reverted. The +9 are these tests.

… two candidate causes The refusal that fires when a shrink releases nothing named two causes -- "the arena has no commit chunk, or its handles are retained (SGLANG_FLIP_SEAM_RETAIN_HANDLES)" -- separated them in neither code nor fact, and on boot_798_0822_0810.log was wrong about both on all 24 refusals. A chunkless arena cannot reach that line: registration refuses one outright at the supports_backing_spans gate, so every rung that exists has a chunk. Retention is a number arena_census() already keeps, read-only and allocation-free, so naming an env var the reader must go and check is strictly worse than printing it. The state the message had no branch for is the one that fired. runtime_set_backing_rows returns BYTES RELEASED TO THE DRIVER, and all 24 refusals reported claimed=0. Pool and driver agreed; there was no divergence between "reported MiB" and "the driver's free column" to explain, because nothing was ever reported. The old wording's story -- unmapping without releasing yields address space rather than memory -- presupposes an unmap that never happened. Meanwhile 15 shrinks on the SAME boot released 256/512 MiB, so no standing property of the arena can explain either set. Split the refusal in two. claimed <= 0 reports the pool's own decline plus the measured geometry: rows asked, release granularity, commit chunk, buffer count, bytes per row, and retained arena bytes. claimed > 0 with a flat free column keeps the unmap-without-release reading, which is a real failure mode that simply was not this one. An unreadable census says "unknown" rather than becoming a confident zero. This is instrumentation: both branches still return 0, keep the cap engaged, and mark exhaustion on exactly the same condition as before. No shrink decision anywhere changes. Tests: test_shrink_cannot_pay_reason_796.py, 8 cases, driving the real _shrink_to through a fixture rung. Verified red against the previous wording before the change. Writing it also corrected an error of mine: on the shipped geometry (256 MiB chunk, 28 buffers, 32 KiB rows) the release granularity is 229376 rows, larger than several plausible pools hold in total, so the below-granularity rule is now asserted rather than assumed. Regression: 200 passed across the 14 registered relief/backing/796 suites.

…s a release granule The floor clamp silently undoes the granularity round-up, and nothing re-checked it: rows_wanted = max(rows_wanted, self._min_release_rows()) target = max(floor, current - rows_wanted) The round-up exists because a shrink smaller than one commit chunk per buffer clears no extent anywhere (the 2026-08-11 measurement in that comment). When the eviction floor binds, the clamp hands back a target whose distance from current is below one granule again -- and the shrink was attempted regardless. The cap engaged, decommit_range cleared no extent, and the rank lost capacity in exchange for nothing. MEASURED, boot_798_0822_0737.log: 15 occurrences of "reported 0 MiB but the driver's free column did not move". PP2's shape: current=126976, floor=88945, granule=229376. The round-up asks 229376; the clamp yields 88945; the real distance is 38031 rows, one sixth of a granule. This is the grant-vs-payment gap. The group's decision was already correct (51 GRANTED / 0 DECLINED); what failed was the payment. Note the granule can EXCEED the whole pool (229376 > 126976 here), so on such a rank no shrink can pay at all at --flip-seam-chunk-mib 8. That is a sizing question and is deliberately NOT papered over here: this guard's job is to stop paying a cap for a release that cannot happen, not to pretend it can. The sizing lever is filed separately. DIRECTION OF SAFETY: the guard only ever turns a shrink into NO shrink. It never deepens one, so it cannot pull backing below the highest live row -- the sgl-project#717 fault that reverted c4e5579 and killed boots. The can-fail test pins that direction explicitly. Correcting my own earlier exclusion: I ruled granularity out by comparing an ask of 107049 ROWS against "8 MiB" without converting between them. They are one axis -- _min_release_rows is ceil(chunk_bytes * buffers / bytes_per_row). The exclusion was apples to oranges and it was wrong. Tests: test_floor_clamp_defeats_granule_796.py, 3 tests, red first. Pre-existing failures in test_evict_rung_floor_invariant_717.py and test_kv_arena_*_631.py are untouched by this diff (verified: it touches neither _buffers nor _min_release_rows).

release_rows_after_floor's docstring, the commit message that shipped it, and the test fixture behind it all stated that PP2's release granule was 229376 rows and therefore EXCEEDED its 126976-row pool, so that "no shrink on that rank can ever pay at this chunk size", attributed to boot_798_0822_0737.log. That is not what the boot recorded. grep -c 229376 boot_798_0822_0737.log -> 0 229376 is 28 * 8192, and it is the fixture constant in test_shrink_cannot_pay_reason_796.py (256 MiB * 28 / 32 KiB), which was labelled "the shipped geometry" and travelled from there into a commit message and a code comment as a measured fact. What that boot actually recorded for PP2: :1325 32768 B/row over 32 arena buffers :16 flip_seam_chunk_mib=8, enable_vram_dial=False -> commit_chunk_bytes = self._vmm_commit_chunk_bytes or seam_chunk = 8 MiB (the dial is off, so it cannot override) :3382 current=126976 floor=88945 slack=38031 _min_release_rows() = ceil(8 MiB * 32 / 32 KiB) = 8192 rows = 256 MiB The granule is 8192, 6.4% of the pool, and PP2's post-floor distance of 38031 rows is 4.64 granules. The floor clamp does not defeat it, no pool seen so far is smaller than it, and --flip-seam-chunk-mib is not the lever the old text sent the next reader after. CONSEQUENCE FOR THE GUARD: all 15 zero-byte shrinks in that boot asked at least three whole granules deep (PP2 targets 88946, 93468, 101133, 96239 against current=126976). The guard added in de51590 would not have prevented one of them. It is kept -- the shape it refuses is real arithmetic, and refusing costs nothing -- but it is not what PP2 hit, and why runtime_set_backing_rows released zero bytes at that depth is still open. WHY THIS SURVIVED REVIEW: the fixture stubbed the function it claimed to characterise. def _min_release_rows(self): return GRANULE A characterisation test that supplies its own answer cannot fail when the answer is wrong, and the _Pool it ran against declared an 8 MiB chunk that no code path ever read -- 8 MiB and 229376 rows cannot both be true of any geometry in that file. The fixture now binds the REAL implementation off the class and derives the granule from the three measured terms. Tests: test_floor_clamp_defeats_granule_796.py rewritten, 7 cases. Red first: binding the real _min_release_rows to the measured geometry fails the old "measured defect" case, which shrinks to 88945 and engages the cap instead of refusing -- the defect it asserted does not occur on the geometry it claimed to measure. Can-fail: mutating _min_release_rows by *28 (exactly the false figure) kills 4 of the 7. Reverted. Regression: 162 passed across 9 registered relief/backing suites. test_relief_rung_executor_553.py fails to import on this box (ModuleNotFoundError: datasets), pre-existing and untouched by this diff.

…t it had runtime_set_backing_tokens had zero logger calls. That is why the shrink investigation could not be closed from outside the process: one reading reported `current` as one number while the target span implied another, and nothing distinguished them. The two quantities that can diverge at this call are the pool's own `size` before the branch runs, and `uniform_backed_rows` -- what the arena actually has mapped in EVERY buffer. Both are now logged at the call, before any branch mutates state, together with the reserved and store-bound ceilings and the branch taken. The grow and shrink exits log the resulting size, the new backed-row count, and the bytes actually released, so a shrink that decided correctly but paid nothing reads as released_bytes=0 rather than as silence. Test exercises the real entry point in both directions plus the no-op branch. It was checked against a known state both ways: it passes on this tree and fails on an unmodified snapshot of the parent commit with "no logs of level INFO or higher triggered on sglang.srt.mem_cache.memory_pool". The two existing tests that name this function never call it -- one touches the owner underneath it, the other only reads its source -- so neither would have caught the silence. Instrumentation only; no behaviour change on any branch.

… bigger reserve Re-scope of the F1+F2 acceptance instrument, per ruling, plus the defect that re-scope immediately found in F2 itself. THE RE-SCOPE. `test_a_self_declared_under_backed_rank_MUST_NOT_veto` injected `floor=131073, cap=126976` into `collective_kv_target` and demanded no veto. At that layer the veto is CORRECT: the only way to remove it is to drop the rank's floor from the group MAX, and that rank still applies the resulting proportion to its own cap -- below its own live set, which is cudaErrorIllegalAddress and kills every rank rather than raising (sgl-project#796). The assertion demanded a defect and could never flip. A permanently-xfail test measures nothing. It is now `test_the_reduction_MUST_veto_rather_than_cap_below_a_live_set`, green permanently, asserting the veto positively so a future "optimisation" that drops a defective rank's floor fails loudly. F1+F2's real property -- REACHABILITY, that floor > cap is TRANSIENT rather than permanent because the pool can grow to its lawful floor -- is pinned at the layer that can deliver it, red-both-ways against the SHIPPED sizer: - the pre-F2 shape CANNOT reach the floor (125052 and even a size-sized 126976 fall short of the lawful 131073); if that stops holding, the specimen changed and the green direction proves nothing - the shipped sizer DOES reach it, strictly above the old value, across a range of sizes so a constant cannot fake it AND IT EARNED ITSELF IMMEDIATELY. Red-both-ways exposed that F2 as committed in e62b1fa STILL UNDER-RESERVED: the pool sizes its reservation before a scheduler exists, so `_admission_reserve_rows(None)` fell back to 512 while W22's live value was 4096 (derived from chunked-prefill-size 4096). The shipped sizer returned 127489 against a lawful floor of 131073 -- rebuilding the same wall one layer down, in the fix for that wall. A test that could only pass would never have found it. Fixed: the boot assumption takes `max(derived, CONSERVATIVE_ADMISSION_RESERVE_ ROWS = 16384)`. The derived reserve is not knowable at reservation time, and the two errors are not comparable -- over-reserving buys a slightly larger VA span, under-reserving is a permanent wall no runtime actuator can lift. RULE RECORDED in docs/dev/NOTE_851_build_caveats.md, with all three instrument corrections on this build: an acceptance test asserts a property THE FIX LAYER CAN DELIVER; a test that injects state into a deeper layer tests that layer's contract, not the fix. Two of the three were mine. Also recorded there: the metal criterion "0 over-cap floor vetoes under load" is NOT substituted by any of this. The unit property proves the pool CAN reach its floor; only metal proves it DOES under real funding dynamics. Tests: 603 passed / 0 failed / 0 xfailed across the sgl-project#851, sgl-project#850, funding, relief, corridor, backing and exposure suites -- the branch now carries NO xfails, both former ones resolved by re-scoping rather than by weakening. ruff clean. Hermetic, CVD="", no boots.

…at already had a canonical fix WINDOW-946RF-0828 KILLED ALL THREE RANKS the first time a boot ever REACHED the dead-premise terminator: RuntimeError: Boolean value of Tensor with more than one value is ambiguous scheduler_pp_mixin.py:1523 discarded = len(getattr(req, "prefix_indices", None) or ()) THE LINE IS PRE-EXISTING (sgl-project#946) AND WAS UNREACHABLE UNTIL sgl-project#949. The escape always took the silent `return "refetch"` above it, so the terminator had never run in production. Making it deliverable is what exposed the bug -- the defect and its discovery have the same cause, and that is the honest reading of the window rather than "the fix broke it". IT IS THE THIRD COPY OF A SPELLING THAT ALREADY HAD ONE CANONICAL DEFINITION. `phase_flip_draft_bootstrap.prefix_len` exists precisely because this crash already happened (W37-B, 2026-08-25, same message, same expression) and its docstring opens "ONE DEFINITION, because two of them is what put the boot on the floor". `scheduler.py:8108` carries the same warning as a comment: "sgl-project#796: NO `or []` HERE. `prefix_indices` is a tensor". sgl-project#946 wrote a third spelling anyway. So the fix USES the canonical helper rather than adding a fourth -- the one-job-one-mover rule, applied to a two-line expression. SIBLING SWEEP over the whole tree for the same `getattr(...) or ()/[]` shape on a field that can be a tensor: only this site was live. `phase_flip_output_trace` and `phase_flip_draft_bootstrap` touch `output_ids` / `origin_input_ids`, which are lists; `schedule_batch.py:2856` uses a bare `len()`, which is correct. TESTS: 3 arms, RED FIRST against the real crash (multi-element tensor, the 0-d tensor where `len()` raises but `numel()` answers 1, and a ratchet asserting the canonical helper is used rather than a fifth spelling). Can-fail proven by reverting the fix: all 3 go red, the multi-element arm with the metal error message verbatim. Module 39 -> 42. One of my own arms was wrong first and is recorded: it grepped raw source for `or ()` and went red on the FIX'S OWN COMMENT, which quotes the landmine to explain it. An assertion about code must not be satisfiable or breakable by prose -- the sgl-project#915 guard-comment lesson, applied to a test. It now strips comments before asserting. Blast radius re-run, each module alone in a fresh process, all matching baseline: 796 9/9, 797 31/31, 798 12/12, 791 11/11, 630 5/5. ruff (F401,F821,UP037), ruff format and codespell clean on both touched files.

…it travels Boot-2 ring wedge (R3a root H3): the output ring's intermediate-hop predicates serve the LAGGED slot -- the sender forwards on `if pp_outputs:` (what it took off the wire last iteration), the receiver reads its own `mbs[next_mb_id]`, committed passes earlier -- and sgl-project#951's launched posting is a SAME-SLOT closure that cannot retract a batch already resident there. The uncovered path is the void relay: `pp_void_relay_stop_rank` derives its stop from the RETRACTION structure, so a void that names no retraction (sgl-project#944's zero-offer escape: a rank that lost the request retracts nothing and launches nothing) was forwarded to a rank whose slot for it was empty, taken off the wire by that rank's next ADMISSION receive, stashed, and served POSITIONALLY to a healthy later generation's receive -- which `_pp_absorb_void_output` then emptied on one rank only, leaving the ring one message short for ever. PP2 sat in `_do_recv` unbounded because sgl-project#971's busy wire kept the sgl-project#789 gate early-returning (consumed<posted); boot 1 only survived on a counter another defect had frozen. Fix: the launched CHAIN -- the same per-hop statement sgl-project#951 consumes, kept per generation. `_PP_LAUNCHED_CHAIN_KEY` accumulates one bool per rank on the admission decision (PP0 starts it, every hop appends its own `mbs[mb_id] is not None`, the last rank records without sending, sgl-project#796), rides back on the slot's output/void via `pp_output_payload_with_return_trip`, and `pp_void_forward_payload` consults `pp_void_relay_launched_verdict`: a void travels exactly to the ranks whose own admission-time statement says they launched, never to its source. Absent chain falls back to the legacy rule byte for byte. Log-only `sgl-project#978 STALE VOID` tripwire in `_pp_absorb_void_output`. Red-first, measured 2026-08-28 against the unfixed tree (TheLaggedSlotIsBeyondTheLaunchedPosting): test_the_launched_chain_relay_stop_closes_the_lagged_slot FAILED -- stuck [0,1,2], PP2 parked in output_exchange, PP1 event `pass=7 void_absorbed slot=2` against hazard slot 1 (the mispair), with sgl-project#951 wired in AND CHAN_DICT counters armed (the readiness gate ran and early-returned: boot-2 fidelity, not boot-1's frozen-counter accident). Count check: 1 failed, 1 passed of 2. After the fix: 2 passed; the wedge stays measurable via fix_off (spawn worker forces the verdict back to legacy -- mutant on the hazard direction, reproduces the wedge). The verdict pure function is pinned exhaustively over rings 2..8; three boundary mutants (hand back to source / read own entry / short chain forwards) all fall. Suites: neighbor baseline before == after. Before: 95 passed (801, 791b, 795, 797, 951, idle-void, void-send families, 262s). After: 100 collected, 99 passed + 1 failed, the 1 a harness-stub interface drift in test_pp_retracted_pass_void_797.py (local recorder lacking the new kwarg; repaired, no assertion touched; file then 31 passed). ruff: only the one finding already present on HEAD; codespell clean. sgl-project#981 boundary, checked before push (CPG_DERIVATION_0828 (b)): this fix's predicate does NOT stand on the `_pp_output_expected_incoming` memo. The chain's write (decision recv), reset (top-of-body block) and consumption (decision send) all live inside full body iterations; the resume-slot jump's `continue` sits before the reset block but consumes nothing before the next full body resets. The by-slot chain is written only at a slot's admission and read while that generation is resident. K3's ungated admission recv and the clear-site-behind-skip fragility remain sgl-project#981.

… idiom Boot 43 reported `seen=0` on all three ranks with 27 prefill batches. The probe's own message says what that means -- "seen=0 means prepare_for_extend never ran, which is not the same thing" -- so it was readable, but the reason was mine. `len(getattr(_r, "prefix_indices", ()) or ())` asks `bool(tensor)`, which torch refuses for a multi-element tensor. Every call raised into the probe's `except` and nothing was ever counted. THE REPO DOCUMENTS THIS EXACT MISTAKE, AND I READ THE COMMENT IN THIS SESSION BEFORE MAKING IT. scheduler.py:8300, on the sgl-project#788 emitter: "sgl-project#796: NO `or []` HERE. `prefix_indices` is a tensor, and `x or []` asks `bool(x)` ... the effect was that EVERY admitting pass lost its trace line to the except below while the idle DECLINE passes logged fine -- boot instr6 showed all three ranks reporting a bare RuntimeError at the exact pass the first real request arrived ... The docstring above already said len() is the right spelling; the `or []` slipped in anyway." Same idiom, same failure mode, same silent `except`, one session apart -- and the earlier one cost a boot window too. Vigilance is not what stops this; the comment was read and the bug was written anyway. Fixed with the only correct spelling: an explicit None test and `len()`. AND THE PROBE NOW HAS A CAN-FAIL PROOF, which the first version never did: two synthetic requests, one holding the invariant (prefix 8192 against start 8192 -> BREAK 0) and one breaking it (prefix 7938 against start 8192 -> BREAK 254, and `len_input` correspondingly 508 instead of 254). The counter registers exactly one break. A probe that cannot be shown to fire on the condition it looks for is the same class as a counter without a denominator, and this one was shipped without it. Evidence: desk. py_compile; the isolation test that reproduced the raise before the fix and passes after it; the two-case can-fail proof. Belegstufe: DESK-BEWIESEN.

fix logging

f35a159

Ying1123 merged commit 325a06c into main Jul 29, 2024

Ying1123 deleted the fix-logging branch July 29, 2024 06:01

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Fix logging (sgl-project#796)

725b320

Navigation Menu

Uh oh!

Fix logging - #796

Fix logging#796
Ying1123 merged 1 commit into
mainfrom
fix-logging

Ying1123 commented Jul 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Ying1123 commented Jul 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant