Fix echo + lobprob for OpenAI API when the prompt is a list by Ying1123 · Pull Request #791 · sgl-project/sglang

Ying1123 · 2024-07-29T00:09:01Z

No description provided.

…t honour it opt out A TP=1/PP=3 boot deadlocks on the first radix-carrying request, measured twice. py-spy --locals: the last rank holds a ScheduleBatch and blocks in _pp_recv_proxy_tensors for its slot, while both upstream ranks carry cur_batch=None and report themselves idle. The mb_ids were the correct -1 stagger, so this is not a slot desync -- the ranks diverge on BATCH PRESENCE. WHY THAT CAN HAPPEN AT ALL. PP ranks are N independent schedulers that agree only by determinism: the request is chain-forwarded to every stage unconditionally (scheduler_pp_mixin.py:1069-1074), but each stage re-derives the admission verdict from its OWN queue and radix state (_get_new_batch_prefill_raw, scheduler.py:6377, first gate 6414-6417), and the proxy send is gated on that rank's own batch (scheduler_pp_mixin.py:488-501). A rank that declines therefore forwards the request but can never send the proxy its downstream is blocking on. The #616g uniformity floors that keep ranks aligned are scoped to tp_cpu_group, which has ONE member on every rank of a TP=1/PP=3 boot, so all three floors switch off (scheduler.py:4693-4703 -- its comment "One rank: nothing to diverge from" is true for TP, false for PP). WHAT THIS COMMIT ADDS: the decision model, standalone and not yet wired. PP0 is already the sole tokenizer-receipt point (request_receiver.py:143, 197-219), so it decides membership and per-request prefix length once, and the decision travels instead of being re-derived. THE DECISION CARRIES LENGTHS, NEVER POINTERS. req.prefix_indices are real slot pointers into the deciding rank's own KV pool (match_prefix_for_req, schedule_batch.py:1296-1314) and are meaningless off-rank. Prefix length drives KV REUSE, not accounting: prepare_for_extend (schedule_batch.py:2262-2266, 2306-2311) sizes out_cache_loc and, through extend_num_tokens, the activation that crosses the stage boundary. A wrong length corrupts -- garbage KV rows, or a crossing whose row count disagrees between stages even when the admitted SET agrees. Hence the asymmetric rule: local >= told truncate to told. Safe; gives up some legitimate local reuse, the same slack trade #616g already makes on the TP axis. local < told this rank cannot honour it, and the physical reason is documented: it cannot compute the missing KV in-pass, because only stage 0 holds the embedding. THE local<told PATH DEGRADES, IT DOES NOT RAISE, and that is the point. local<told is not an exotic corner on this shape -- it is what an ordinary cache hit looks like. A path that raised there would trade a silent wedge for a crash on every cache hit, which is worse than the dark cache we started from. So the request is EXCLUDED from the effective decision, exactly one warning names rank/rid/told/local, its siblings in the same batch are untouched, and the exclusion propagates without re-warning. TESTS: test_pp_admission_congruence_791.py, hermetic, 11 passed. A neutered simulation reproduces genuine three-rank divergence (80/136/200 rows), so the tests fail for the real reason rather than by construction. NOT WIRED YET, deliberately. The admission path and event loop are untouched, scheduler_pp_mixin.py is under a scope fence held by the sgl-project#789 readiness-contract work, and pp_typed_channel.py has zero edits -- the intended carrier is the typed tensor-dict channel under a new kind "admission_decision", which is keyed per mb_id and already carries a non-tensor stamp. Wiring lands with the re-queue path and the retry pin. KNOWN OPEN HOLE, named here rather than discovered later: nothing yet forces told=0 when an excluded request is retried. PP0 would re-match the same prefix, the downstream would exclude it again, and the wedge would come back as a retract loop. The retraction has to flow back to PP0 and pin that rid's next decision to told=0 (one-shot, rid-scoped) -- or PP0 learns downstream coverage from the retraction, which is already half of the congruence guard. That is a blocker for wiring, not for landing this module.

…evidence a message is coming A PP rank could enter an unbounded blocking receive for a proxy that nobody would ever send. Measured twice: the last rank holds a ScheduleBatch and blocks in _pp_recv_proxy_tensors for its slot, while both upstream ranks carry cur_batch=None and report themselves idle -- so no rank will produce the message it waits for. Health keeps answering, forward counters keep looking sane, and the process sits there until it is killed. THE CONTRACT. _pp_wait_for_proxy_readiness runs immediately before the blocking receive and demands POSITIVE evidence: the CHAN_DICT sent counter having moved past local_consumed, i.e. the upstream provably posted. On a healthy boot that evidence is already there and the function returns at once, so the receive path is unchanged. IT IS A PRESENCE SIGNAL, NOT A TIMER, and that distinction is the whole design. "Wait N seconds then give up" is sgl-project#630, a livelock this codebase already has on record: ranks that time out, retry, time out again, make no progress and look alive throughout. Here the budget is only a backstop -- the raise fires solely when the budget is fully exhausted AND the counter never moved once. A timing-out Work.wait() on the transport was never an option either: a timed-out gloo wait destroys the pair, so the peer then sees "Connection closed by peer". The inbox is consulted before the counters, and that case was found by running the regression rather than by foresight: a message already stashed for this exact consumer (for instance one the armed drain took off the wire) leaves the counters reading "caught up", so counters alone would have called a delivered message absent. TRANSPORT COVERAGE, STATED RATHER THAN IMPLIED. This covers "the upstream never posted" uniformly across the gloo-metadata and NCCL-payload halves, because a sender that never got there never advanced its counter either. It does NOT cover "posted, then stalled mid-transfer". That limitation is written into the docstring; the measured specimens are all the never-posted case. DEFAULT PATH UNCHANGED: with pp_flip_counters None -- an ordinary boot without phase flip -- the check is a no-op, verified directly rather than assumed. Budget 30 s via SGLANG_PP_PROXY_READINESS_BUDGET_S, tagged HAND PIN sgl-project#789. THIS IS A BACKSTOP, NOT THE HANDLER FOR THE COMMON CASE, and it must not be wired alone. The ordinary divergence on this shape -- one rank admits, the others do not -- would otherwise reach this contract, never move the counter, and raise after the budget: a crash on every cache hit, which is only a bounded, diagnosable version of the same fatality. That case belongs to sgl-project#791's decision model, which excludes a request no rank can honour and degrades it instead. sgl-project#791 must therefore be wired ahead of this, so that by the time the contract can fire, something genuinely different is broken. TESTS. test_pp_proxy_readiness_contract_789.py, 4 green. Its neutered case is the red and the can-fail in one: with the readiness call patched out the path is byte-identical to the pre-sgl-project#789 code, and the victim rank genuinely hangs against two idle upstreams -- proven a hang rather than a fast error by a bounded outer driver that finds it alive at the deadline with no result. Disclosed: the source edits preceded the red-first test, an ordering deviation. It is remediated by that neutered case being exactly the pristine path, not by argument. The one-line harness repairs to test_pp_drain_completeness_787 and test_pp_flip_leftover_proxy_757 bind the new method on holders that predate it; no assertion was touched. Verified together in one run: 30 passed across the 789, 787, 757, 788, 633 and 791 suites.

…, so the degrade terminates sgl-project#791 lets a rank that cannot honour a told prefix length exclude itself and be re-admitted later. Nothing made the retry DIFFERENT. PP0 would re-match the same prefix, tell the same too-long length, the downstream would exclude it again, and the request would never complete while the server looked alive. That is sgl-project#630's shape exactly: a bounded, graceful path that makes no forward progress is still a livelock, and this codebase already has that defect on record. "It degrades" is not a termination argument. WHAT THIS ADDS. PPAdmissionCongruenceGuard keeps a rank-local _learned_floor[rid] and clamps PP0's fresh local match down to the OBSERVED shortfall carried back on the retraction. The floor is learned, never guessed: it is the coverage the downstream actually reported, so the request loses exactly the tokens that were missing rather than all of its reuse. That is why option (2) was chosen over a one-shot told=0 pin -- both terminate, but this one makes the degrade RARE instead of merely finite. TERMINATION IS WELL-FOUNDED, not hopeful. Every new retraction lowers the floor strictly below the told that just failed, so the sequence is strictly decreasing over non-negative integers: it can neither cycle nor descend forever. Measured on three ranks: 120 -> 64 -> 50 -> served. THE FLOOR CLEARS, and that was a review condition rather than an afterthought. record_return_trip pops the rid's floor as soon as a decision comes back with no shortfall anywhere in the chain; a rid with no retraction history is not constrained by the guard at all. A pin that survived its request would poison that rid's reuse for the rest of the process -- a silent performance defect traded for a loud hang, which is not an improvement. SHAPE INVARIANT HELD. Clamping prefix_len down raises extend_len by the same amount, so prefix_len + extend_len is unchanged. The degrade therefore cannot desync extend_num_tokens across stages and corrupt the crossing -- the failure mode that made a bare forwarded integer unsafe in the first place. CORRECTNESS PRECONDITION FOR WIRING, stated here so it cannot be missed: the floor clears ONLY via record_return_trip, and that return path is deliberately not wired yet. Wiring sgl-project#791 without it leaks a floor per rid and poisons reuse permanently. The return path is not an optional part of the integration package; the guard is not correct without it. TESTS. test_pp_admission_retry_livelock_630.py: the loop is reproduced ACROSS TWO CYCLES without the guard (told=120, observed=64, retracted both times) and served in cycle 2 with it -- the red asserts the LOOP, not a single exclusion, which is the only way to fail if the fix merely terminated one pass. Verified in one run over nine suites, 187 passed: 791, 630, 789, 787, 757, 788, 633, phase_policy and phase_flip_runtime. The suite list is deliberately wider than the change -- a verification list scoped to the defect at hand is what let a merge regression through earlier today.

…n, instead of re-deriving it per rank This wires the decision model (sgl-project#791), its congruence guard (sgl-project#630) and the readiness contract (sgl-project#789) together. None of them changed behaviour until now; each was landed standalone and green. THE DEFECT. PP ranks were N independent schedulers agreeing only by determinism. The request is chain-forwarded to every stage unconditionally (scheduler_pp_mixin.py:1069-1074), but each stage re-derived admission from its OWN queue and radix state (_get_new_batch_prefill_raw, scheduler.py:6377, first gate :6414-6417), while the proxy send is gated on that rank's own cur_batch (:488-501). So a rank that declined forwarded the request and could never send the proxy its downstream was blocking on. Measured twice, deterministically, on the first radix-carrying request after health: the last rank held a ScheduleBatch on its slot while both upstreams sat idle with cur_batch=None. The #616g uniformity floors that would have kept them aligned are scoped to tp_cpu_group, which has ONE member on every rank of a TP=1/PP=3 boot (scheduler.py:4693-4703), so all three were off. WHAT NOW HAPPENS. Rank 0 -- already the sole tokenizer-receipt point (request_receiver.py:143,197-219) -- builds the decision once per admission pass, with the congruence guard clamping the told prefix length to any learned floor. The decision travels as its own typed-channel kind "admission_decision": an ordered list of (rid, prefix_len, extend_len, admitted). Downstream ranks consume it instead of re-deriving. LENGTHS TRAVEL, POINTERS NEVER DO. prefix_indices are slot pointers into the deciding rank's own pool and are meaningless off-rank, so each receiver resolves the told length against its OWN radix tree. A receiver that cannot honour it excludes that request, logs one warning naming rank/rid/told/local, and leaves its siblings untouched -- it does not raise. That path is the ordinary cache-hit shape on this rig, and raising there would have traded a silent wedge for a crash on every cache hit. THE RETURN TRIP IS PART OF CORRECTNESS, not an extra. The chain-reconciled decision flows back to rank 0 (scheduler_pp_mixin.py:721) so the guard learns the observed coverage and, on a clean pass, CLEARS the rid's floor. Wiring the decision without it would leak a floor per rid and poison that request's reuse for the life of the process. ORDERING: sgl-project#791's degrade runs before sgl-project#789's contract can fire. The contract is the backstop for a genuine protocol violation; the ordinary divergence must never reach it. On a healthy boot its raise path stays unfired, and that is a success criterion of the next instrumented boot rather than an assumption. CONSTRAINTS HELD, CHECKED AGAINST THE DIFF RATHER THAN INTENDED: - No collective on the admission path. scheduler.py:6405-6407 documents a 2026-08-17 deadlock of exactly that family and warns against it verbatim; the diff introduces no all_reduce, all_gather, barrier or broadcast. - No device tensor reaches a logging argument (sgl-project#790). - pp_size == 1 is byte-identical: the guard is None below pp_size 2 (scheduler.py:1577). NOT DONE, DELIBERATELY: the two disaggregation PP loops (:618, :765) are untouched and still carry the original chain-flush hazard; the readiness contract does not cover them either, since it lives in the proxy receive path they do not use. That remains its own open item. Verified in ONE run over ten files, 190 passed against a 187 baseline -- the delta is exactly the new integration cases. The suite list is deliberately wider than the change: a verification list scoped to the change at hand is what let a merge regression through earlier today.

…n flush, not after it The wiring landed in f31fd5e was not bootable. On the instrumented boot it deadlocked on the first request with ZERO detector markers and zero GPU utilisation. py-spy, taken while the ranks were still alive: PP0 _pp_commit_pending_req_work (the request-chain flush) PP1, PP2 _pp_recv_admission_decision (waiting for the decision) The loop body ran recv-decision at :599, chain flush at :680, send-decision at :701. So the downstream ranks blocked on a message PP0 could only send at :701, PP0 blocked at :680 on those same ranks reaching the top of their next pass, and neither could move. A closed ring, silent: the ADMISSION-WEDGE detector never fired because this shape is not what it watches for, and the cards read 0% while three processes sat in gloo. THE CAUSE WAS THE COMMENT'S OWN REASONING. The send was placed after the flush because it "does not gate on any of them" and it seemed tidier to keep every per-iteration outbound flush in one place. Tidiness is not an ordering argument. sgl-project#788's own commit message states the rule this violated: a rank must satisfy everything a peer can be blocked on BEFORE it blocks on that peer. The admission decision is exactly such a thing, so it must be on the wire before the flush, and the flush is last precisely because it is the act that waits on a peer. Order is now recv(:599) -> send(:704, :729) -> flush(:745), and both comments say why rather than restating what. WHY THE TESTS DID NOT CATCH IT, recorded so the gap is not repeated: the integration test drives the decision path but not the real per-iteration ordering of the shipped loop against live peers, so a send placed after a blocking flush still passes. The suite went 190 green before this fix and 190 green after it -- unchanged, because it cannot see this class. Only the boot could, and only because zero GPU utilisation was noticed while the probe was supposedly running. Verified: 190 passed over ten files, unchanged from the pre-fix baseline.

…lready in hand Fourth deadlock of one family today, and this one was mine. The instrumented boot reached health and froze on the first request with zero GPU utilisation, a frozen log and ZERO detector markers. py-spy, all three ranks alive: PP0 _pp_recv_admission_decision from _event_loop_pp_body:722 the RING WRAPAROUND PP1 _pp_recv_admission_decision from _event_loop_pp_body:599 the forward receive PP2 _pp_recv_admission_decision from _event_loop_pp_body:599 the forward receive Every rank receiving, nobody sending. CAUSE. The wraparound was a BLOCKING p2p receive inside the per-iteration loop, fired once pp_size sends were outstanding. From that point rank 0 sent nothing until a full ring lap returned -- and the ranks that had to complete that lap were themselves blocked waiting for rank 0's next send. The comment conceded it was "still a blocking p2p recv" and argued it merely bounded staleness. Bounding how stale a wait is does not stop it closing a ring. Worse, the deque was popped BEFORE the receive, so the counter recorded progress that had not happened. THE RULE, now stated because four instances is enough: A RANK MUST NEVER BLOCK ON A PEER FOR SOMETHING THAT IS NOT REQUIRED FOR THIS ITERATION'S FORWARD PROGRESS. record_return_trip teaches the congruence guard what coverage a lap observed. That is a learning path. It has no business gating the pipeline. FIX. _pp_try_recv_admission_decision peeks pp_typed_channel.typed_inbox for an already-stashed lap and returns None immediately if there is none -- no wire touch, no timeout, no collective, following the precedent _pp_wait_for_proxy_readiness already set for inbox presence. The deque is popped only when a lap actually returned. An unconsumed lap stays consumable on a later pass, and _PP_ADMISSION_PENDING_SENDS_CAP bounds the deque now that laps are consumed opportunistically. THE COST, stated rather than hidden: a learned floor now clears whenever a lap happens to be in hand instead of at a fixed pass. That delays reuse recovery for that rid. It blocks nothing. TEST, and it exists because the suite was blind to this class -- 190 green both before and after the previous ordering bug. test_pp_admission_wraparound_never_ blocks.py drives the REAL per-iteration ordering across three real gloo processes and asserts all three ranks complete their iterations inside a deadline, reporting per-rank stall location on failure. It is RED against the blocking wraparound and GREEN with this fix. While building it the harness deadlocked itself the same way, which is the clearest evidence available that it models the real thing. Verified: 193 passed over eleven files, baseline 190, delta exactly the three new cases.

…rank can block on any peer Fifth deadlock of one family. The boot reached health and froze on the first request with zero GPU utilisation and ZERO detector markers. py-spy, alive: PP0 _pp_commit_pending_req_work (scheduler_pp_mixin.py:802) the chain flush PP1 _pp_recv_admission_decision (:3262) from :624 the forward receive PP2 same as PP1 I had twice reordered the two channels I suspected -- the request-chain flush and the admission decision -- and both times the ring merely moved. Measurement now shows why: THOSE TWO DO NOT DEADLOCK UNDER EITHER ORDERING. The ring needs a third participant, one level down. The crossing channel shares pp_group's typed-tensor-dict demultiplexer with the admission decision (pp_crossing_wire.py:270-277) and is itself blocking, and with it in the picture the deadlock reproduces exactly. So the earlier two fixes each corrected a real ordering defect, and neither was the one killing the boot. FIX: the sgl-project#791 admission-decision send moves up to immediately after get_next_batch_to_run, ahead of _pp_launch_batch and the crossings. The decision is then on the wire before this rank can block on a peer for anything else, which is the same invariant as before, finally applied to the channel that actually violated it. PIGGYBACKING WAS EVALUATED AND IS NOT AVAILABLE. Carrying the decision on the proxy tensor-dict would have removed the second channel instead of ordering it, which is the better shape, but the proxy is a no-op under a gapped wire and can carry nothing. Recorded so nobody re-proposes it. TESTS: test_pp_admission_chain_flush_deadlock_795.py, four hermetic cases over real gloo processes binding the shipped send/receive functions, reproducing this constellation RED and passing GREEN with the relocation. This class is why that style is now mandatory here: the suite was 190 green both before and after an earlier deadlock, so ordinary unit coverage cannot see it. Verified: 197 passed over twelve files, baseline 193, delta exactly the four new cases.

… it with a void Boot instr11 wedged 4m18s into a flip drive: GPU util 0/0/0, log growth 846 B in 8 s, eight requests in flight and never served. py-spy on all three ranks gave a three-way ring: PP0 _event_loop_pp_body:779 -> ... -> _pp_recv_dict_from_prev_stage:3865 -> BLOCKING recv PP1 _event_loop_pp_body:603 -> pp_chain_receiver.recv:329 PP2 _event_loop_pp_body:603 -> pp_chain_receiver._advance:218 ROOT. A sgl-project#791 downstream retraction desynchronises the output ring, because the ring's two gates read two DIFFERENT ranks' mbs. reconcile_pp_admission_decision (pp_admission_congruence.py:462-483) amends the decision for every remaining DOWNSTREAM rank, and PP0 is upstream of every rank that can retract. The boot's last four admission passes, three passes after a tp_to_pp cutover, show it exactly: PP0 sgl-project#788 verdict=ADMIT n_reqs=1 rids=2f5e25a1... prefix_lens=512 PP1 sgl-project#791 unhonourable prefix on rank 1: rid=2f5e25a1... told=512 local=0 PP1 sgl-project#788 verdict=DECLINE n_reqs=0 PP2 sgl-project#788 verdict=DECLINE n_reqs=0 PP0 kept the microbatch and launched it; the downstreams had nothing for it. Last rank's send gate (:4101) reads its own empty slot and sends nothing; PP0's receive gate (:4306) reads its own occupied slot and blocks (:4315). _pp_output_exchange_due (:328) made the two the same EXPRESSION (sgl-project#753) but cannot make them the same FACT. PP0 never reaches the top of its next pass, so the pass-N+1 chain send is never posted and both downstreams block. The idle tail is load-bearing: a skipped send mid-burst is only a lag, since the pair is FIFO. It becomes a DEFICIT exactly when the pipeline goes quiet behind the retraction, which is what the log shows -- DECLINE, DECLINE, silence. sgl-project#796 IS NOT THE CAUSE, and this retires that hypothesis. The wraparound it removed is an ADMISSION_DECISION_KIND message, and recv_typed_tensor_dict (pp_typed_channel.py:136-145) returns only on expected_kind, stashing everything else. No wraparound could ever have released an expected_kind="output" receive, before or after sgl-project#796. Pinned by test_wraparound_kind_cannot_satisfy_an_output_receive. Restoring it would have put one unmatched message per pass back on the channel and unwedged nothing. FIX, following sgl-project#791's own law (decide on rank 0, carry the decision) rather than a timeout. _PP_OUTPUT_EXPECTED_KEY (:136) rides on the admission decision that already travels 0->1->... ->last in the SAME pass; PP0 publishes _pp_output_exchange_due(self.mbs[mb_id]), the identical expression on the identical object its own _do_recv will apply, and middle ranks forward it verbatim. The last rank (:4128) sends a void (_pp_void_output_payload, :4148) only for a slot PP0's own published verdict obliges it to receive -- so this is not the bounded-recv corpse in reverse. PP0 absorbs it (:4315, _pp_absorb_void_output, :4171), empties the slot so the loop's None guard holds, releases each request through the existing idempotent _release_dynamic_chunk_probe and re-queues it, and feeds record_return_trip from the chain-reconciled decision riding back inside the void. That last part also repairs collateral damage from sgl-project#796: record_return_trip (pp_admission_congruence.py:276) was fed ONLY by the wraparound, so since sgl-project#796 _learned_floor was never populated and prefix_len_for never clamped -- sgl-project#630's termination argument had no feeder at all. It does again, without an unmatched message. Absent key => False => no void => byte-identical to today, so stand-ins and pp_size<=1 are unchanged. TESTS. test/registered/unit/managers/test_pp_output_ring_retraction_wedge_791b.py, three live spawned processes, real gloo, the shipped functions: test_retraction_wedges_the_ring_without_the_fix (can-fail: neuters ONLY the fix IN THE CHILD and reproduces the specimen -- 3/3 ranks stuck, downstreams ahead of the rank they block on, the inverted stagger) test_ring_survives_the_retraction_with_the_fix (12/12 passes, 1 void absorbed on PP0, 1 request re-queued, 0 voids on PP1/PP2) test_void_payload_is_only_sent_when_the_first_rank_expects_one test_wraparound_kind_cannot_satisfy_an_output_receive -> 4 passed, three consecutive runs (40.95 / 39.57 / 38.44 s) Neighbours (795, 796) -> 13 passed. Wider mixin set -> 19 failed / 176 passed, IDENTICAL at HEAD (patch removed, re-run, re-applied); all 22 are _RingWire/_Group stand-ins missing is_first_rank/is_last_rank, i.e. sgl-project#796's own gate. codespell clean; the one ruff F841 at scheduler_pp_mixin.py:1395 is pre-existing and untouched. KNOWN GAP, filed rather than folded in: a PARTIAL retraction (one rid of several) keeps the ring matched, so PP0 processes a real output for a batch containing requests the downstream never ran -- a membership divergence with no wedge and no error. Same root asymmetry, different defect.

…ction Boots instr15, instr16 and instr17 all died on the same raise, and it was not a leftover: ValueError: sgl-project#631 PP proxy/batch mismatch: received hidden_states with 126 row(s) for a 1 batch of 22 token(s) (instr17, 07:12:49 PP1) 126 = 22 + 104. PP0's batch is PP1's batch PLUS THE ONE REQUEST PP1 RETRACTED. Same pass, same slot, same epoch; the stamp was correct in every element, which is exactly why PROXY LEFTOVER REFUSED stayed 0 on all three boots. Nothing was stale, so no stamp discriminator -- per-slot generation, receiver-derived seq, rolling pass id -- could ever have caught it. Two earlier hypotheses are retired by this arithmetic: a within-epoch stale slot (there is no stale message) and a cutover landing between receive and use (the traceback shows a FRESH _event_loop_pp_body:960 after the re-dispatch, and pp_proxy_tensors is a loop local no re-dispatch carries across). MECHANISM, from boot_instr17.log:62997-63056, all 07:12:49: PP0 sgl-project#788 verdict=ADMIT n_reqs=2 rids=51a294650b...,5e744c29f8... prefix_lens=0,16896 PP1 sgl-project#791 unhonourable prefix on rank 1: rid=5e744c29f8... told=16896 local=0 PP1 sgl-project#788 verdict=ADMIT n_reqs=1 rids=51a294650b... prefix_lens=0 PP1 ValueError: 126 row(s) for a 1 batch of 22 token(s) reconcile_pp_admission_decision drops the unhonourable rid from `effective` (pp_admission_congruence.py:472-483), the admission loop omits it from THIS rank's batch (scheduler.py:6974-6991), self.mbs[mb_id] is the narrowed batch (scheduler_pp_mixin.py:806) -- while the upstream has already sent its decision and launched its own wider batch (:876, :947). A batch in flight cannot be amended. The flip is the TRIGGER, not the cause: it cold-starts the downstream radix cache, so PP0 offers a prefix PP1 has nothing for. #791b already fixed the OUTPUT-RING consequence of this same retraction (scheduler_pp_mixin.py:4317-4319); the proxy-width consequence was never covered. THE DISCRIMINATOR IS NOT ON THE WIRE, and it does not need to be: "did I retract anything from this pass's decision?" The receiver PERFORMED the retraction, at the top of the same pass, strictly before the proxy receive, and #791b already records the amendment per slot in _pp_admission_amended_by_slot. Nothing new crosses the wire and the receiver predicts nothing a sender wrote. The test is `retracted_by_rank == self`, not `retracted`, so an entry an earlier rank retracted is correctly ignored -- pinned as a regression case. WHY THIS BEATS THE SHIPPED TRIPWIRE WHERE IT MATTERS: chunked prefill caps every chunk at the same size, so two ranks running different request sets routinely present EQUAL widths. model_runner.py:4182 is blind to that -- silent wrong output, not a shape error. Pinned as test_a_same_width_divergence_is_still_refused. STATED PLAINLY, THIS DOES NOT YET PREVENT THE DEATH. It converts a shape error raised thirty layers deep into a boundary refusal that names rid, told= and local=. Prevention is a separate design call, deliberately not taken unilaterally here: PP0 must stop offering a prefix the downstream cannot honour, i.e. PPAdmissionCongruenceGuard.prefix_len_for needs the feedback sgl-project#796 deleted with the wraparound (scheduler_pp_mixin.py:3568-3594). The channel that survives sgl-project#796's "no send no peer must take" law is the OUTPUT message PP0 already receives from the last rank -- piggyback the retraction floor as a per-hop key, the _PP_OUTPUT_EXPECTED_KEY precedent. That still costs one wasted PP0 forward the first time, and voiding the retracting rank's pass desyncs PP0's own request state, so both halves must be decided together, on metal. ALSO FOUND, NOT FIXED HERE: two rank-local decode-retraction paths (scheduler.py:7517/:7489 and :7211-7312) diverge the same way and sgl-project#791 never touches them. TESTS. test_pp_proxy_retracted_pass_mispair_791c.py -- three live spawned processes, real gloo, shipped functions, real 126-row and 22-row tensors: test_a_retracted_pass_is_mispaired_without_the_retraction_test (can-fail: blinds ONLY entries_retracted_by_rank's return value in the CHILD, through scheduler_pp_mixin's own module globals, so every API, the guard body, the reconciliation and the amendment recording all still run -> 126 rows delivered for the 22-token batch = THE SPECIMEN, and the child's `effective` still shows the narrowed set, proving the retraction happened) test_a_retracted_pass_is_refused_by_the_receive_guard test_a_retraction_by_another_rank_does_not_refuse test_a_same_width_divergence_is_still_refused test_an_unretracted_pass_is_delivered_unchanged -> 5 passed; with the 631 and 795 neighbours 29 passed Neighbour set 631/795/791/791-wiring/791b/757 re-measured AT HEAD: 44 passed / 1 failed; after: 49 passed / 1 failed, the same pre-existing _RingWire.is_last_rank drift from sgl-project#796. ruff, ruff-format and codespell finding sets byte-identical to HEAD. The 631 test's residual pin is corrected: it had misdirected two investigations into hunting a stale message that never existed.

…eriving it (slice 1) Ten fixes on this branch (sgl-project#757 sgl-project#789 sgl-project#790 #791b #791c sgl-project#792 sgl-project#795 sgl-project#797 #797b #797c) all have ONE form: each rank RE-DERIVES the pass schedule -- rid set, chunk length, prefix length -- locally from its own state, and a phase flip invalidates that state non-atomically. Every "new root" was the next consumer of the same re-derivation, so the list grew instead of converging. This stops patching consumers. THE DATUM WAS ALREADY ON THE WIRE AND NOBODY READ IT. PPAdmissionEntry.extend_len (pp_admission_congruence.py:177) has crossed the wire since sgl-project#791's first commit with exactly three consumers: to_wire (:223), from_wire (:244), one log string (:824). Nothing ever built a batch from it; reconcile_pp_admission_decision returns Dict[rid, prefix_len] and drops the second number on the floor. THE MECHANISM, CORRECTED AGAINST THE LOG rather than assumed. boot_instr20.log:5171,5181-5183: PP0 ADMIT rid=6cbe2733 prefix_lens=0 chunked=1 -> 512-row chunk PP1 ADMIT rid=6cbe2733 prefix_lens=512 chunked=0 -> 333-token remainder PP1 DID receive prefix_len=0 and DID clamp prefix_indices to it -- scheduler.py:7026-7027 worked. Then add_one_req's HOST LOAD-BACK put the 512 back: needs_host_load_back() went true when the HiCache prefetch landed and schedule_policy.py:1539-1549 concatenates the recovered indices. "MAMBA-HOST-RESUME ... triggers load_back" appears on PP1 and PP2 and is ABSENT on PP0 -- that asymmetry is the bug. 845-512=333 then fitted rem_chunk_tokens whole, so the NON-chunked branch fired. The re-derivation lives INSIDE THE ADDER, after the schedule was already applied, and it re-derived BOTH numbers. DESIGN. forwarded_schedule() (pp_admission_congruence.py:500) is the pass geometry as a value: rid -> (prefix_len, extend_len) for exactly the rids `effective` names; None or a voided decision yields {}. _add_scheduled_req (schedule_policy.py:1237) EXECUTES both numbers: no rem_chunk_tokens, no page/align rounding, no host load-back, no budget veto -- the budget is still charged. The gate sits above every local veto (:1600), and add_chunked_req (:1328) gains the gate it never had at all (it is entered from scheduler.py:7004, BEFORE the admission loop). The three membership vetoes that silently narrowed -- batch_is_full (:7035), the HiCache prefetch_done skip (:7047, the instr20 race itself) and the LoRA gate (:7024) -- become refusals (scheduler.py:7190). REFUSAL IS CONTROL FLOW, NOT A RESULT CODE. PPScheduleRefused (:163) is an exception because every AddReqResult means "build a batch without this request", which is precisely the corruption. A refusal reuses sgl-project#797 end to end (_pp_refuse_forwarded_schedule, scheduler.py:6333 /:6369): sets _pp_admission_pass_voided, voids the forwarded decision, re-notes the slot expectation. No new mechanism. Inside the loop a refusal is CARRIED, not thrown, so alloc_group_end() still runs (:7025, :7118, :7169). WHY THE GUARDS CAN NO LONGER FIRE, each owed a reason: sgl-project#631's _want is extend_num_tokens, which now comes only from the forwarded extend_len with load-back suppressed, so it IS the upstream's row count by construction rather than by agreement (green arm: rows=512, batch_tokens=512). sgl-project#757/sgl-project#787 stamp and sgl-project#795 epoch were already structural; what changes is that they can no longer be correct-but-insufficient, as instr17 and instr20 both were -- every identity right, only the width wrong. Width is now an identity too. sgl-project#789 needs a membership divergence, which is now identical-or-refused. #791c's tripwire detects a self-narrowed batch, and no path creates one. HONEST CORRECTION TO THE SUBSUMPTION CLAIM: retraction does NOT become unnecessary. Physical impossibility is real -- a rank genuinely lacking KV for [local, told) cannot execute. What becomes structurally impossible is the NARROW-THEN-DETECT shape: no code path is left that builds a batch of a geometry the upstream did not name. NOT COVERED BY THIS SLICE, stated so nobody assumes otherwise: - BATCH ORDER. can_run_list follows the local waiting_queue order, the decision follows PP0's. Same rid set in a different order gives EQUAL WIDTHS and permuted rows -- silent. Covered by the full design (execute in decision order), not by this slice. - Decode batches: retract_decode (scheduler.py:7489/:7517) mutates long-lived state on a tp_cpu_group reduce that is world=1 under TP=1/PP=3. Different root, filed by sgl-project#797. - Radix eviction divergence, KV pool sizing, spec-decode draft schedules. NAMED RESIDUAL, filed at the site (scheduler.py:7118) in sgl-project#797's practice: a request admitted earlier in a loop that later refuses has taken a persistent inc_lock_ref, released on batch completion. Undoing it needs the exact IncLockRefResult (SWA/Mamba tombstone params) the adder does not keep, and a blind release makes the one thing a mismatched release worsens. Bounded: reaching that line takes a genuinely unexecutable geometry and kills the pass. DEFAULT PATH UNTOUCHED: _pp_scheduled_extents() returns None on PP0 and on every pp_size<=1 boot, so scheduled_extent_for returns None and both adders take the pre-existing arithmetic unentered; PPScheduleRefused is unraisable there. Pinned by test_no_mapping_is_the_untouched_default_path and corroborated by 56/56 on the schedule_policy neighbours. TESTS. test_pp_forwarded_schedule_791.py: 17 passed (3 live gloo arms + 14 pure), 92 s. test_red_without_the_forwarded_geometry_instr20_reappears -- can-fail, rebinding ONLY the fix's return value in the child: batch=(512,333) rows=512 mismatch=True, byte-identical to instr20 PP1 09:40:30 test_green_the_forwarded_geometry_survives_the_mid_pass_prefetch -- batch=(0,512) rows=512 mismatch=False, load_back_calls=0 test_an_impossible_geometry_raises_rather_than_narrowing -- the architectural property Neighbour set (791/791b/791c/797/631 x3) re-measured AT HEAD: 11 failed / 75 passed; after: 11 failed / 92 passed, failure names byte-identical, zero regressions, +17. schedule_policy neighbours 56 passed / 0 failed. ruff 119 = 119 at HEAD (parity), ruff format clean on everything authored, codespell identical.

…cision names boot_798_0822_0829 never served a request: 0 decode batches, 0 throughput lines, 8 requests submitted and 0 answered, health 200 throughout, 5219 void lines in 13 minutes. Its shape, from the log rather than from reasoning. TWO rids, not one. rid=f6116ba2 progressed (told=98304 local=90112, then told=100203 local=98304) and left. rid=48abbc0e reported told=98304 local=0 for 2212 consecutive events and never moved. Every own-void released "0 of 1 request", 2210 times, so the batch member was always KEPT by pp_void_keeps_request and never re-queued -- never re-entering waiting_queue, the one place the reconcile's radix lookup would have found it. The voids alternate between slots without a single exception: 2, 0, 2, 0. The defensive self.chunked_req = None branch fired zero times. _pp_reconcile_incoming_admission answers "how much of this rid has THIS rank computed" from the single scheduler-wide self.chunked_req, but _pp_void_own_batch restores that field PER SLOT from _pp_chunked_req_before_by_slot[mb_id]. With more than one microbatch slot in flight the value standing there belongs to whichever slot wrote it last, not to the slot whose decision is being reconciled. The miss then defaults to 0 and is consumed as a MEASUREMENT, so the pass is retracted and voided, and the void restores the other slot's snapshot -- which re-arms the next one. That is the whole loop. The slot was never missing, only unread: PPAdmissionDecision.mb_id has been on the wire since sgl-project#791 and documents itself as "one PP microbatch slot". Consult _pp_chunked_req_before_by_slot at exactly that index, after the shipped self.chunked_req lookup and before the default of 0. ONLY THE NAMED SLOT MAY ANSWER. Scanning the ring for a matching rid would be the same defect with a wider blast radius -- answering a question about slot 2 with slot 0's progress -- so pp_chunked_req_for_slot indexes and does not search. It is a module-level function beside pp_chunked_local_match and pp_void_keeps_request, resolved through this module's globals at call time, because a mixin method would have to be bound by every existing holder and could not be neutered on its own; the first draft was a method and broke four #797c fixtures with an AttributeError on the admission path. This is #797c one level up. #797c fixed the "dropped out of waiting_queue, lives in chunked_req" miss, which is why the specimen's single-slot rid progressed. It did not make the lookup slot-aware, which is why the two-slot rid could not. Tests: test_pp_reconcile_slot_blind_798.py, 12 cases, verified RED first -- the livelock reproduces hermetically as observed_local=0 for a rank holding 98304 tokens, and as a genuine 94208-token shortfall reported to sgl-project#630's guard as 0. Includes a dying mutant for the new call edge (blinding pp_chunked_req_for_slot brings the livelock back) plus a companion proving #797c's lookup survives that same neuter, so a single proof cannot conflate the two edges. Guard-still-fires cases cover absent-everywhere, short match, and wrong-slot-holds-it. Regression: registered managers -k pp_, baseline 16 failed / 281 passed against 16 failed / 293 passed after, with byte-identical failing test IDs. The 16 are pre-existing. Both ruff findings on this file are pre-existing and outside this change.

… loop it drives 7 failed / 7 passed -> 14 passed, exit 0. Completes the WIP commit. `_event_loop_pp_body` is taken unbound off the mixin here, so every link the body grew since sgl-project#791 had to exist on `_Rank`, and each one only became visible once the one above it was closed -- eleven in all across the two commits. The stub/bind split held throughout and is the part worth keeping: a shipped method is BOUND whenever it can answer without a peer, and STUBBED only where it would block on a real PP wire this fixture has no peer for, or dereference the None the link above returns. Every stub returns the shape the shipped function returns on its own no-op path, with that line quoted at the callsite, so none of them encodes a behaviour the production code does not have. The runtime flags (_pp_pass_voided_incoming, _pp_output_expected_incoming, _pp_upstream_launched_incoming, _pp_admission_pass_voided, _pp_gapped_wire) are set to the values a rank that received nothing carries, not to whatever made the next traceback go away. Standing observation, for whoever owns sgl-project#631: a harness that must re-declare every method its subject grows will break exactly like this again. Eleven links in one release is the signal, not the eleven fixes.

…n a send nobody owes The PP output ring wedged with all three ranks alive and none of them able to move (specimen /spinning/evidence-665-f1/wedge_802f_1712/, PP=3, --enable-phase-flip, py-spy of all three schedulers): PP0 _pp_commit_comm_work <- _pp_commit_pending_req_work (:4071/:2262) PP1 _pp_recv_dict_from_prev_stage <- _do_recv (:5608/:6069) PP2 PpChainReceiver.recv <- recv_requests (pp_chain_receiver.py:329) PP0 is flushing the request chain, which PP1 can only take at the top of its next pass; PP1 never reaches that top because it is blocked in the output receive; PP2 waits for the chain send PP1 has not reached. Three arcs, one cycle, no timeout anywhere. THE ASYMMETRY. The two ends of the intermediate hop apply unrelated predicates. `_do_recv` decides to receive from THIS rank's own slot state; the non-last sender in `_pp_send_output_to_next_stage` decides to forward on `if pp_outputs:`, which is whatever it received LAST iteration. The last-rank hop is matched by construction because it consults `_pp_output_expected_for_slot` -- but that flag is the FIRST rank's verdict, published for PP0's arc. Nothing publishes the same thing for the intermediate hop. WHY THE EXISTING VOID CONTRACT DOES NOT COVER IT. `pp_void_forward_ payload` (sgl-project#797) already forwards a void along this hop and stops at `pp_first_retracting_rank`, on the argument that rank r and everything after it has an empty slot whose receive early-returns. That holds for the VOIDED slot -- `_pp_void_own_batch` empties it. It does not hold for a slot the resident decode path still occupies, and both void paths keep resident requests on purpose (`_pp_absorb_void_output` refuses to release them as a double-free; `_pp_void_own_batch` deliberately leaves `running_mbs` alone). The specimen shows exactly that: PP0's last act was absorbing a void for slot 0, leaving `pp_outputs` None, while PP1 logged `running=1 chunked=1` and re-entered the receive for a slot PP0 would never send to. PP0 cannot know PP1's resident set, so closing this needs a per-slot expectation every non-first rank publishes to its predecessor -- a protocol extension, and not this commit. WHAT THIS COMMIT DOES. The sgl-project#789 readiness gate is parameterised by wire kind and the output receive now passes through it. The CHAN_DICT counter was never proxy-specific -- one counter per wire, demultiplexed by `__msg_type__` after it comes off -- so the gate reads the same true statement about the same wire either way; `kind` selects only the per-(src, kind) inbox peek. The ring is cut at its one cuttable arc: PP1 waits boundedly and refuses by name instead of for ever. It does not make the missing send appear, and the error text says so. AN ALIAS, NOT A WRAPPER. `_pp_wait_for_proxy_readiness` is now a class-level alias for the same function object. About ten stand-in holders across the sgl-project#631/sgl-project#757/sgl-project#787/sgl-project#789/sgl-project#791/sgl-project#795/sgl-project#797/sgl-project#798 test family bind that name one method at a time; a delegating wrapper resolved the second name on the HOLDER and turned 9 green tests into 5 failures. Measured, then fixed. Tests: test/registered/unit/managers/test_pp_output_readiness_ring_802.py -- three real gloo processes, real PhaseFlipCounters, neutering done in the child (spawn re-executes the module, not the test body). * red arm, gate neutered: stuck_ranks == [0, 1, 2], specimen reproduced * green arm: stuck == [], PP1 reports "sgl-project#789 OUTPUT READINESS TIMEOUT" and "sgl-project#802-ring", PP0 chain-flushed, PP2 requests-received * false-positive direction: upstream really posts -> receive succeeds * no-op without counters, so the non-phase-flip default path is unchanged Mutants, both die: call site removed -> green arm red (all ranks stuck); gate raises unconditionally -> false-positive test red. Regression: baseline 787+789 = 9 passed / 0 failed; with this change 791b + 789 + 787 + 802 = 17 passed / 0 failed.

Two commits closing the rest of the sgl-project#815 stale-stub debt that 17b rooted but left standing with a verdict: eight test helpers that had drifted behind the code they measure, plus the two pp_* stubs that 17b had judged unfixable without a sgl-project#791-shaped rewrite and that turned out to be fixable faithfully. Test-only change. Shares test_collective_family_siblings_610.py with the already-merged fix/801: checked at hunk level before merging -- sgl-project#801 adds two class-level counters to BudgetHarness (:465), sgl-project#815 fixes _budget_state_stub (:540/:557). Different symbols, different drift instances, no collision.

… stub These 7 tests had been red on an AttributeError: the loop under test asks `not self.pp_group.is_first_rank` (scheduler_pp_mixin.py:1343) and the test-local `_Group` stub carried `is_last_rank = True` and nothing else. THE CHEAP REPAIR IS THE TRAP. Adding `is_first_rank = True` beside it turns all 7 green immediately -- and makes this rank the FIRST and the LAST stage of a three-stage pipeline at once, which no rank of a pp_size=3 ring can be. Every branch keyed on either role would then take the wrong arm while the suite reported green. That is the sgl-project#630 lesson exactly: an unfaithful stub does not merely fail to catch a defect, it encodes the defect's assumption and then certifies it. So the stub now carries a REAL position in a REAL ring. `_Group(rank, pp_size)` derives both roles from that position, and rank 2 is the default because that is what the old lone `is_last_rank = True` was reaching for: the last stage, which skips the proxy-send block that is not under test. The difference is that it is now last WITHOUT also claiming to be first, so the admission-decision branch takes the arm a real last rank takes. The collaborators on that arm are stubbed to the cheapest thing that keeps the control flow real -- they are not the subject; the loop CONTROL is, and it still comes unbound off the mixin. `assert_faithful_pp_roles` makes the forbidden combination loud rather than silent, and `_Rank` runs it on construction. THE RED-FIRST LOGIC IS INVERTED HERE, and that is the honest shape for a repair to a suite that was already red. The proof is in two directions: * the 7 tests go GREEN with the faithful stub (18 passed total); * a PLANTED first==last stub goes RED -- 16 of 18 fail. That is the can-fail, and it is what stops the cheap repair being reintroduced. Four new tests carry that second direction, including one that is easy to forget: `test_the_faithful_role_is_load_bearing_not_decorative` asserts the non-first admission arm is ACTUALLY entered. Without it the added stubs could be dead code and "faithful" would be doing no work -- the suite would be green because the branch was never reached, which is the state it was in before, only quieter. TESTS (hermetic, CVD="", CPU only, no CUDA, no distributed) test_pp_flip_slot_hold_631.py 18 passed (was 7 failed / 7 passed). Mutant killed: `is_first_rank` forced True -> 16 failed. Battery re-measured under /spinning/htsglang-gpu/.venv (datasets 5.0.0, full collection), over the 7 files that carried every failure: base 500be7e 22 failed / 35 passed HEAD before this commit 22 failed / 35 passed HEAD with this commit 15 failed / 46 passed -7, exactly the tests repaired here, and this branch still adds none. Of the 15 remaining, 3 (test_pp_slot_last_batch_631) are an artefact of this branch's base being pre-sgl-project#815: fix/815-rest-stubdrift landed on the line at 4f2072a, while base 500be7e sits on the earlier ancestor 21ff075. They disappear when this stage merges onto the line. No boot was run. This is desk work.

…ith a real reduce STRUCTURE/PRIO, root D of the 0516 wedge. This lands the DECISION and its red-first repro; the scheduler wiring is the next commit and is called out below rather than implied. WHAT EXISTS AND WHY IT IS NOT ENOUGH. prefetch_ballot (#791b) carries a CRC digest of the waiting-queue head on the packed MIN-reduce and voids itself when the group's min and max digest differ. sgl-project#823 already gave that detector onset, persistence and recovery-edge logging (scheduler.py:5061). Its own docstring states the limit: "On mismatch the ballot is void for the pass and the caller falls back to the rank-local verdict -- the status quo ante". Surface, then fall back to rank-local. Nothing makes the ranks agree. The detector is untouched here; the enforcer goes beside it. WHERE THE DIVERGENCE IS BORN. SchedulePolicy.calc_priority (schedule_policy.py:197) orders waiting_queue by req.num_matched_prefix_tokens under a CacheAwarePolicy (_sort_by_longest_prefix, :229-232). That number comes from the RANK-LOCAL radix tree and each TP rank's prefix cache evolves independently -- the #616B family. Same queue, same policy, different ORDER. THE RULE, transplanted from sgl-project#791. sgl-project#791 made PP admission uniform with an asymmetric local/told rule: a locally computed value may only be truncated toward what the anchor said, never used to extend it. The TP sibling: the group's match length is the MIN across ranks and every rank sorts by the GROUP number. MIN is the safe direction for the same reason as #616B's evict floor and the ballot itself -- the agreed length is <= every rank's own, so no rank is ever told to reuse a prefix it does not hold. Worst case a rank recomputes a prefix it had cached: slower, never wrong. Capacities stay rank-local; only the DECISION is uniform, per kein-bindender-rang. THE CIRCULARITY, AND WHY THE SLOTS ARE NOT QUEUE POSITIONS. Per-rid values cannot be reduced by queue position when the positions are what diverge -- slot i is a different request on different ranks and a MIN over that is meaningless. Slots are indexed by a CANONICAL rid order (sorted rid strings), which depends only on the rid SET, the replicated part. A rank that does not hold a rid contributes -1, which MIN-reduces to -1 if ANY rank lacks it, so the group drops that rid rather than admitting a request a peer cannot form: delay, never force, the ballot's own safety property. sorted() is deterministic across processes; hash() is not and must never touch this path. TWO BEHAVIOUR CHANGES, each with its own can-fail arm. The second is the one easy to leave implicit: a digest mismatch must stop falling back to rank-local. The group order is derived from the canonical set and the MIN, so it is still computable in exactly the pass where the digest says the orders disagree -- the mismatch case IS the wedge case, and improving only the agreeing case would leave it untouched. head_decision() carries that branch explicitly and reports which rule ran. PURE ON PURPOSE, per sgl-project#823's own lesson in uniform_floor_scope.py:45: inline behind a real all_reduce the only thing a test can check is whether the source still mentions a branch, and a mutant that disabled the recovery edge once survived a whole suite on exactly that. TESTS (hermetic, CVD="", CPU only, real gloo, no CUDA) test/registered/unit/managers/test_tp_head_congruence_823.py 9 passed. Pure arms plus a REAL gloo MIN all_reduce at world=2 and world=3. The premise is asserted, not assumed: test_todays_local_rule_really_does _diverge shows the rank-local rule giving three different orders on the same queue (rank 0 leads with charlie, rank 1 with alpha). Every rank feeds the enforcer ITS OWN diverged queue order, not a shared fixture list. That is load-bearing: with a shared list the canonical-order step is never exercised and mutant 2 below survives. Mutants killed: enforcer switch ignored -> 1 failed (the can-fail arm) slots indexed by queue order -> 2 failed (canonical independence and the real-gloo uniformity) Regression: 38 passed across test_tp_head_congruence_823, test_prefetch_ballot_divergence_823 and test_pp_prefetch_ballot_791b -- the detector suites are unchanged and still green. NOT YET WIRED. get_new_batch_prefill still calls calc_priority's rank-local sort; this commit adds no slots to _update_uniform_pool_budget's packed reduce. That wiring is next and needs care: the layout comments there record that #639b's appended pair silently moved what the host floor read, and the harness modelling that reduce had drifted six times (repaired in the preceding commit). WINDOW-QUEUE ticket stays preflight_pass N until it is wired and green. Second known gap, recorded so it is not mistaken for covered: scheduler.py :7542/:7547 break the candidate loop on get_num_allocatable_reqs() and req_to_token_pool.available_size(), neither of which rides the sgl-project#610/#616g uniform floor. Equal order with unequal counts still yields unequal batches, so the count needs the same treatment as the order. No boot was run. This is desk work.

…e stub) into integ/808-739-810 Wave 3, stage 3b. One commit, test-only, a strict descendant of stage 3's head ceb79d6: 43afb4e [sgl-project#791] A faithful PP-role stub for the slot-hold suite WHY THIS IS ITS OWN STAGE. It touches exactly one file (test_pp_flip_slot_hold_631.py, +195/-2) and no production source at all, so stage 3's gate against ceb79d6 still certifies the code this tree carries and did not need re-running. Keeping the two apart also keeps the attribution readable: stage 3 is the ring-recovery fix, this is the test debt it happened to arrive with. WHAT IT CLOSES. These are the seven failures this train carried as its ENTIRE baseline -- the line's failure set has been exactly test_pp_flip_slot_hold_631.py's seven cases since fix/815 landed, and strand 17b, 17c and this strand each declined the cheap repair in turn. The cheap repair was to add `is_first_rank = True` beside `is_last_rank = True`, which makes one rank simultaneously the first and the last stage of a three-stage ring: an object no production code can ever meet, which is the sgl-project#630 lesson (an unfaithful stub does not merely fail to catch the defect, it ENCODES the defect's assumption and then certifies it). The faithful repair gives the stub a POSITION instead of two booleans: `_Group(rank=2, pp_size=3)` is the last stage WITHOUT also claiming to be the first, so `_event_loop_pp_body`'s admission-decision branch takes the arm a real last rank takes rather than raising AttributeError on a missing attribute. The cost predicted when this was deferred was real and has been paid rather than dodged -- reaching that branch pulls the admission-receive path in with it. The part worth reusing is `assert_faithful_pp_roles`: it makes the FORBIDDEN combination loud, raising when `is_first_rank` and `is_last_rank` are both true at `pp_size > 1`, so the shortcut cannot be reintroduced silently by a later edit. That is a guard rather than a comment asking the next author not to write it. The falsifier is INVERTED and runs in both directions, which is what makes this a proof rather than a green suite: 18 passed with the faithful stub, and planting `first == last` turns 16 of those 18 red. GATE. Battery test/registered/unit/{managers,planner,server_args,mem_cache}, hermetic under CUDA_VISIBLE_DEVICES="", one battery at a time. baseline (tip 4f2072a) 7 failed, 8585 passed, 1852 skipped, 887 s stage 1 (W1+W2) 7 failed, 8638 passed, 1852 skipped, 868 s stage 2 (W3) 7 failed, 8653 passed, 1852 skipped, 922 s stage 3 (W4a/W5/W4b) 7 failed, 8673 passed, 1852 skipped, 915 s this stage 0 failed, 8684 passed, 1852 skipped, 900 s NEW failure ids NONE fixed vs baseline 7 (all of them) THE BATTERY IS NOW FULLY GREEN. The extracted failure list is EMPTY, not merely short: the seven that were the line's entire baseline are gone and nothing took their place. That changes what every later gate on this branch means -- with a zero baseline, "0 new failure ids" stops being a comparison against a tolerated set and becomes binary, so stages W6 and W7 are gated against a clean tree rather than against a list of exceptions. The skipped count is unchanged at 1852 across all five runs of this train, so nothing was turned green by being skipped. CATALOG UPDATED IN THE SAME COMMIT, because the entry this change falsifies is one this strand wrote three commits ago. The section 12 stub-drift family said the seven cases were "left RED ON PURPOSE, awaiting a sgl-project#791-shaped rewrite". sgl-project#791 has now done the rewrite, so the entry says what happened instead of what was expected -- the maxim it was holding the place for is kept, and the repair is recorded under it with its inverted falsifier. Leaving a catalog claim standing after the code has moved is the drift this file's own header rule exists to prevent. codespell clean. No boot.

Wave 3, stage 4 of the batched-window tree. WINDOW-QUEUE ticket W6 (strand 21a, preflight_pass=Y: 19 passed red-first with 6F/6P before the fix, 8/8 mutants killed, id-regression 75 files byte-identical with 0 new). Two commits, base 3b2bbde -- which is already ON this line (the sgl-project#770/sgl-project#812 floor-clamp withdrawal), so this stage adds no divergence of its own: b546893 [sgl-project#828] The backing dial converges the BACKING, and a post is credited by what it delivered 4235879 [sgl-project#828] Desk pre-flight for the batched window: the released band clears the gate boot_827 refused at THE DEFECT. `runtime_set_backing_tokens` branched grow-vs-shrink on `self.size` -- the EXPOSED id space -- instead of on the committed backing, so a rung that should have released reported `branch=grow` and released 0. A dial that converges the wrong quantity is not a mis-tuned dial; it is a dial attached to the wrong shaft, and the census downstream then credits a post for delivering nothing. This is the same family the catalog records in section 2 under the funding authority: a post is credited BY WHAT IT DELIVERED, not by what it was asked for. sgl-project#770 named the posts; this makes the credit honest. GATE. Battery test/registered/unit/{managers,planner,server_args,mem_cache}, hermetic under CUDA_VISIBLE_DEVICES="", one battery at a time. baseline (tip 4f2072a) 7 failed, 8585 passed, 1852 skipped, 887 s stage 1 (W1+W2) 7 failed, 8638 passed, 1852 skipped, 868 s stage 2 (W3) 7 failed, 8653 passed, 1852 skipped, 922 s stage 3 (W4a/W5/W4b) 7 failed, 8673 passed, 1852 skipped, 915 s stage 3b (sgl-project#791 stub) 0 failed, 8684 passed, 1852 skipped, 900 s this stage 0 failed, 8703 passed, 1852 skipped, 931 s NEW failure ids NONE THIS IS THE FIRST STAGE GATED AGAINST A ZERO BASELINE. Since stage 3b the line's extracted failure list is EMPTY, so "no new failure ids" here is not a comparison against a tolerated set -- it is the whole result. CITATION MAINTENANCE, carried in this commit because this stage is what forced it. W6 adds 37 lines to funding_authority.py and 71 to memory_pool.py, both of which the catalog cites by file:line, and the section-18 checker did NOT notice: it verifies that a cited line EXISTS, not that the cited SYMBOL is on it, so it stayed at 104 passed while solve_arming_floor moved :659 -> :694, diagnose_floor_band :585 -> :620 and slack_above_uniform_floor :813 -> :848, each then pointing at a comment, an `if` and a dataclass field. A sweep of every citation this train touches found the same drift elsewhere and all of it is corrected here, verified line by line against the files: scheduler.py nine citations (stage 3 added 113 lines) :4684->:4795, :4792->:4903, :4814->:4925, :4938->:5049, :4947->:5058, :4995->:5106, :5011->:5122, :7085->:7196, :7462->:7573; phase_flip_runtime.py five, :3838/:3839->:4040/:4041, :3881->:4083, :6275->:6487, :7128->:7353; memory_pool.py :4978 -> :4983 with the bound assert at :5032. Untouched and re-verified as still correct: kv_row_ownership.py, prefetch_ballot.py, uniform_floor_scope.py, tree_congruence.py, invariant_checker.py:1175, mamba_ckpt_utils.py:185, kv_backing_relief.py:162/:547, phase_policy.py:831. One drift found in the sweep is NOT this train's doing and is labelled as such: planner/placement.py:813 -> :838, in a file no stage here touches. It drifted earlier and nothing caught it, which is the point. The gap is now recorded in section 18.8 with the measurement that proves it, and the interim practice it implies is applied rather than merely described: where a file takes inserts from several tickets, the STATEMENT is cited alongside the line (done for sgl-project#821's three cur_batch_for_debug sites). Symbol-resolving the checker is registered as its own task. A note on the checker being live rather than assumed, because it caught ME while I was documenting its blind spot: writing the phrase `watchdog.py:88` into the section-18.8 prose turned the gap note itself into a section-18 citation, and the checker went 104 passed / 1 FAILED on the spot. It scans section 18 for anything of the shape `file.py:N`, prose included. The wording now names the class instead of quoting a path, and the checker is back to 104 passed / 0 failed -- re-run after the edit, not assumed. So the checker is genuinely armed for the class it covers; the gap in 18.8 is about the class it does not. codespell clean. No boot.

Wave 3, stage 5 -- the LAST stage of the batched-window tree. WINDOW-QUEUE ticket W7 (strand 21a, preflight_pass=Y: 7 passed with 4 red-first, 3/3 mutants killed, and the 17 pre-existing sgl-project#630 tests unchanged). One commit, base 2efb933: 3f16e8e [sgl-project#829] The HiCache deadline must not close a pair healthy peers are using THE DEFECT. `bounded_wait` enforced its deadline by passing a timeout INTO the gloo work itself -- `work.wait(timeout=)`, introduced by sgl-project#630 -- and on a shared gloo pair that does not merely abandon one wait, it POISONS the pair for every peer still using it. The production string "Application timeout caused pair closure" was reproduced red-first in a real two-process gloo run and reads PAIR_SURVIVED on the fix. The deadline now lives in `ParkedWait.join`, off the work: the wait is abandoned by the CALLER while the underlying `Work` stays parked, so nothing is torn out from under a healthy peer. NO CROSS-SURFACE, computed rather than hoped. The only file this shares with the rest of wave 3 is mem_cache/hicache_collective.py, and the +119 that stage 3 put there comes from commit 2efb933 -- which is precisely W7's own base. So W7 already contains that change and there is nothing to reconcile; its other two files are new and under mem_cache, disjoint from W5/W4b's files. sgl-project#734's DISCRIMINATOR IS PRESERVED, and this was checked rather than assumed because it is the exact thing strand 17c's 622 falsification turned on: the `waited < timeout_s * 0.95` branch that distinguishes a dead peer from a slow one is still in `bounded_wait` after this change. W7 moves WHERE the deadline is enforced; it does not move the threshold. That also makes this gate non-vacuous in the way that matters here. `test_pp_sync_rendezvous_630.py` -- the suite that went red under the 622 merge for precisely this threshold, and the reason 622 is still off the line -- sits INSIDE the battery and is green on this stage. KNOWN FOLLOW-UP, NOT CARRIED HERE. Strand 21c has since measured that the 0.95 comparison is REDUNDANT under this very ParkedWait design -- a real expiry takes the not-completed path 4 out of 4 times, and any RuntimeError reaching that `except` already means transport death -- while the comparison leaves a BLIND BAND over the last 5% of the bound, where a peer death is labelled a timeout (30 s at the 600 s default). An amendment is being built (delete the comparison, make the except an unconditional transport error, and guard at construction that the process-group timeout exceeds the bound). That amendment is NOT in this tree. It did not exist when this stage's battery started, and per standing instruction the window tree does not wait for it -- it lands in the next round as its own stage. This paragraph exists so the next reader does not re-derive the finding or mistake the preserved comparison above for a settled decision: it is preserved here because THIS commit does not touch it, not because it is known to be right. A related thread for whoever owns the 622 posten, explicitly UNMEASURED and not a claim of this merge: 17c established that 622 reddens sgl-project#630 by SHIFTING gloo timing until the 0.95 branch flips, and left open why the timing shifts. This change removes `work.wait(timeout=)`, one mechanism by which a wait on a shared pair can perturb its peers. Whether that is upstream of what 622 perturbs is unmeasured here and is not asserted. GATE. Battery test/registered/unit/{managers,planner,server_args,mem_cache}, hermetic under CUDA_VISIBLE_DEVICES="", one battery at a time, against a ZERO baseline since stage 3b. baseline (tip 4f2072a) 7 failed, 8585 passed, 1852 skipped, 887 s stage 1 (W1+W2) 7 failed, 8638 passed, 1852 skipped, 868 s stage 2 (W3) 7 failed, 8653 passed, 1852 skipped, 922 s stage 3 (W4a/W5/W4b) 7 failed, 8673 passed, 1852 skipped, 915 s stage 3b (sgl-project#791 stub) 0 failed, 8684 passed, 1852 skipped, 900 s stage 4 (W6) 0 failed, 8703 passed, 1852 skipped, 931 s this stage 0 failed, 8710 passed, 1852 skipped, 912 s NEW failure ids NONE The +7 is this stage's own suite, matching its ticket exactly. The skipped count is 1852 in all seven runs of this train, so nothing went green by being skipped. The sgl-project#630 check above was verified rather than inferred from the overall zero: test_pp_sync_rendezvous_630.py sits under test/registered/unit/mem_cache, i.e. inside the battery, and run standalone on this stage it is 3 passed -- including test_a_dead_peer_still_raises_a_named_bounded_error, the exact case that reddened under the 622 merge. Its new suite, test_bounded_wait_pair_survives_829.py, lands under test/registered/unit/mem_cache and is therefore inside the battery and actually collected. WINDOW SCOPE, so the window reader does not expect what cannot appear: W7's effect is only OBSERVABLE with PP>1 and a storage-backed HiCache. On a boot without both, a green W7 criterion would be vacuous rather than passing. No boot. This completes the tree; W9 is a separate stage after the push.

…h setters WHAT THE FIELD MEANS, stated once because two sites were deriving it and only one of them was right: `cache_protected_len` is HOW MANY LEADING ROWS OF THIS REQUEST'S KV THE TREE OWNS. It is not a length of anything the request owns. Two consumers depend on exactly that reading -- `_insert_helper`'s duplicate free (`dup_start = max(0, prev_prefix_len - total_prefix_length)`) and sgl-project#824's `retention_shrinks_protected` -- and both are unsafe if it under-reports. THE HAZARD, REPRODUCED BEHAVIOURALLY rather than argued. On a prefix HIT `req.prefix_indices` ARE the tree's row ids; the request reuses them, it does not copy them. So in the prefix region `_insert_helper`'s `value_slice` holds the TREE's ids, and `prev_prefix_len` is the only thing standing between them and `token_to_kv_pool_allocator.free`. With `prev_prefix_len=0` and a full prefix hit, every row of the prefix ends up in the free list AND in the tree at once -- counted, ids compared, not read off the source. That set is precisely the `double_owned` population (`free_rows & cached_rows`) the on-idle ledger reports as `src=live`. THE ROOT: the two setters guessed differently, because neither was told. `MatchResult.cache_protected_len` defaults to None and `UnifiedRadixCache` never populates it, so on this rig the field's value depended on which site touched the request last: * `Req.init_next_round_input` (schedule_batch.py:1351-1354) -- has an `else` and falls back to `len(self.prefix_indices)`. CORRECT. * `match_prefix_for_req` (schedule_policy.py:148-149) -- had NO else, so the branch never fired and the field kept its previous value, which is 0 for a fresh Req (schedule_batch.py:1677). Meanwhile the same function assigns `req.prefix_indices = match_result.device_indices` UNCONDITIONALLY. A request could therefore carry the tree's rows while claiming none of them were tree-owned. * `UnifiedRadixCache.cache_unfinished_req` (:1261) -- `len(new_indices)`. CORRECT. The sibling is given the same fallback here, so the two cannot diverge again. Both callers pass `include_req=True` over the WAITING queue, so the value can only be (re)derived for requests that are not yet in flight -- it cannot unprotect anything mid-prefill. AND THE VALUE THE LIVE LOG SHOWS IS RIGHT, WHICH MATTERS MORE THAN THE FIX. 2g-1 reads `cache_protected_len 8192` with a mamba `tracked position 4096`, and sgl-project#824 declines the anchor. That decline is CORRECT and the 8192 is not a symptom: * with `--chunked-prefill-size 4096` and a ~9447-token prompt, `cache_unfinished_req` publishes the protected length at each chunk boundary -- 4096, then 8192, then 9447. At the chunk-2 boundary the tree genuinely owns 8192 leading rows, so 8192 IS the true value at that instant. * `tracked position 4096` is a DIFFERENT AXIS: it is mamba's `cache_len` after the ReplaySSM `write_pos` subtraction, i.e. the last FLUSH boundary of the recurrent state. The mamba state lags the KV by a chunk. * so sgl-project#824 is refusing to file a state captured at 4096 under a key of 8192, which is the sgl-project#767 pairing direction exactly. Refusing is right. THE WARNING THAT FOLLOWS, for whoever owns the anker/decline chain: do NOT "fix" the decline by lowering `cache_protected_len` to meet the tracked position. That would re-open the duplicate-free hazard above AND pair a recurrent state with a depth it was not captured at -- both directions of the same corruption at once. The number is right; the lag is the defect. Tests, hermetic, CUDA_VISIBLE_DEVICES="": test_insert_dup_free_927.py, 3 passed. Combined mem_cache+managers lane: 17 failed / 7473 passed, and all 17 are NAME-IDENTICAL at f1a3391 (arena_high_water_631 x7, restore_never_rebuild_677 x4, phase_flip_rotation_wiring_809 x4, acceptance_emitters_758 RefillTiming x2) -- zero new failures. Genuinely red-first this time, and checked with the mutant shape that defeated the last suite: reverting the `else` turns `test_match_prefix_for_req_states_the_protected_len` RED while the other two stay green, then restored. The hazard case is a CHARACTERISATION (it asserts the tree's rows ARE freed at prev_prefix_len=0) rather than a red-first pin, and is labelled as such. THE RE-ADMIT PATH, TRACED, because the observed crashes all run through it (PP prefill -> retract at the cutover -> re-admission in TP as a full prefix hit, `ADMIT prefix_lens=9447 phase=tp #cached-token: 9447`): * the hazard condition IS created there. `Req.reset_for_retract` (schedule_batch.py:1611) sets `prefix_indices = empty`, `last_node = None` and `cache_protected_len = 0`, and the request is requeued at the front of the waiting queue. A request that then takes a FULL prefix hit is exactly `prev_prefix_len=0` + full hit -- this file's characterisation case. * but it is CLOSED again before the insert. `get_new_batch_prefill` calls `req.init_next_round_input(self.tree_cache)` (scheduler.py:8723) on every admitted request, and that is the sibling that HAS the `len(prefix_indices)` fallback. So the value reaching `_insert_helper` on the observed path was already correct, and the crash is NOT this hazard firing. * SO sgl-project#927 IS NOT CLOSED BY THIS COMMIT. What this closes is the window where `match_prefix_for_req` is the last setter -- real, but not the observed instance. Said plainly so the ticket is not marked done on it. ONE ADJACENT GAP FOUND WHILE TRACING, recorded rather than fixed blind: under `pp_size > 1`, `scheduler.py:8749-8773` truncates `req.prefix_indices` to the PP-agreed `told` (sgl-project#791 admission uniformity) and does NOT update `req.cache_protected_len` with it -- zero mentions of the field in that block. After `init_next_round_input` set them equal, the truncation leaves `cache_protected_len > len(prefix_indices)`. That direction is SAFE for the duplicate free (a larger `dup_start` frees less), which is why it has not shown up as a double-claim; it is the direction that feeds `assert req.cache_protected_len <= len(new_indices) + page_size - 1` (unified_radix_cache.py:1231). Not touched here because the safe direction does not warrant a blind edit on the admission path, and because it wants its own red-first. NOT CLAIMED: that this closes the 2f/2g crash. The live value is already correct via the sibling that had the fallback, so this closes a WINDOW -- the path where `match_prefix_for_req` is the last setter -- not necessarily the observed instance. What it does settle is the reachability question and the meaning of the field, and it removes the disagreement so the next reader is not choosing between two answers.

…ted_len with it `cache_protected_len` is HOW MANY LEADING ROWS OF THIS REQUEST'S KV THE TREE OWNS. `init_next_round_input` sets it equal to `len(prefix_indices)`. The sgl-project#791 admission-uniformity block in `_get_new_batch_prefill_raw` then truncates `prefix_indices` to the PP-agreed `told` -- on PP0 from the guard's clamped candidate, downstream from PP0's decision -- and NEITHER branch touched the protected length. The request was left claiming more tree-owned rows than it holds. I FILED THIS AS "THE SAFE DIRECTION" AND THAT WAS THE WRONG HALF. The surplus IS harmless for `_insert_helper`'s duplicate free -- a larger `dup_start` frees less -- which is exactly why it never surfaced as a double claim. It is the DANGEROUS direction for `cache_finished_req`'s truncate branch (`unified_radix_cache.py:1111-1116`): free_start = max(effective_cache_len, req.cache_protected_len) free(kv_indices[free_start:]) # starts ABOVE the interval ... # the insert covers only up to ecl With `cache_protected_len > effective_cache_len` the rows in `[effective_cache_len, cache_protected_len)` are neither freed nor inserted and belong to nobody afterwards. That interval is sgl-project#935's per-request row leak (36824 rows on the 2i acceptance boot). SCOPE, kept sharp so the two tickets do not blur. The GAP is the root and is sgl-project#935's: it must not be able to leak whatever the value is. This closes one of the two PRODUCERS that make it reachable; the other is the sgl-project#928 refusal re-prefill. Closing a producer does not close the gap, and closing the gap makes the producers harmless -- both are owed, and neither substitutes for the other. Different files, no collision with fix/935-finished-req-gap. ONE HELPER, TWO SITES, because the two sites are siblings of each other and drifted identically -- both sliced `prefix_indices` by hand and both forgot the same field. `Req.truncate_prefix_to(told)` now owns the pair, and the wiring pin asserts neither branch slices by hand again. MIN, NEVER ASSIGN: the helper may only LOWER the claim. A request whose protected length was already below `told` owns exactly that many, and raising it here would invent protection the tree never granted -- which is the dangerous direction for the duplicate free, i.e. the defect this commit is NOT allowed to trade for. Pinned in both directions. Tests, hermetic, CUDA_VISIBLE_DEVICES="": test_truncation_keeps_protected_len_930.py, 6 passed. Combined mem_cache+managers gate: 17 failed / 7483 passed, the 17 NAME-IDENTICAL to f1a3391's -- zero new failures. Genuinely red-first with the mutant the ticket names -- restoring "slice the prefix, never touch cpl" turns 3 red, including the consequence test that computes the abandoned interval from the real `max(ecl, cpl)` arithmetic, while the may-only-lower direction stays green. Restored after.

…t causes is bounded THE CLASS, AND THIS IS ITS THIRD INSTANCE. `_pp_reconcile_incoming_admission` resolves a rid through a chain of lookups and, on a total miss, wrote 0 into `local_match_lens`. `reconcile_pp_admission_decision` then read that 0 as a MEASUREMENT and voided the pass. #797c patched the `chunked_req` miss, sgl-project#798 patched the wrong-slot miss -- each added a lookup and left the miss answering with a number, which is the only reason each looked like a fresh defect instead of the same one again. sgl-project#944 is the RUNNING BATCH miss, and it is where a request lives once its chunked prefill finishes: not in the waiting queue, not in `chunked_req`, not in the slot's chunked req. `self.running_batch` is on the same object and read elsewhere in the same file; this resolution chain referenced it zero times. Measured under real agent-shaped load (long SHARED prefixes): 2106 `unhonourable prefix` events, 2107 voided passes, and a three-rank hang 35 s after health, watchdog kill at 300 s. Live py-spy at wedge onset, not the post-mortem: PP0 computing, PP1 blocked in `_pp_drain_voided_proxy`, PP2 blocked in `hicache_collective.join`. WHAT LANDS, AND WHY IT IS ONE COMMIT AND NOT THREE. 1. `running_batch` is the fourth lookup, and a rid found in NONE of the four is `UNKNOWN_MATCH` (-1), never 0. The sentinel idiom is adopted, not invented: `tp_head_congruence._ABSENT_MATCH` is the same -1 one file over. 2. `PPAdmissionEntry.unresolved` -- a field of its own, on the wire. Encoding the miss as a special value of `observed_local` would be the same class one level up: that field's readers treat it as a LENGTH and feed it to `_learned_floor`, which clamps the next round's offer, and a floor learned from a number nobody measured is exactly the defect. It crosses the wire because the rank that OBSERVES the miss is never the rank that can ACT on it -- only PP0 chooses `told`. 3. `told <= 0` is honourable unconditionally. A zero offer demands no prefix reuse, so no lookup result -- not even a failed one -- can make it unhonourable. This fell out of the arithmetic for free while a miss answered 0 (`0 >= 0`); at -1 it stops falling out, and leaving it implicit retracts the FIRST, congruent round of every request. For a rank that did resolve it is a no-op (`local >= 0 >= told`). 4. `UNRESOLVED_DEFER_CAP = 3` with a per-rid count in `PPAdmissionCongruenceGuard`. THIS IS WHY (1)-(3) COULD NOT SHIP ALONE: `_learned_floor` is what damped the re-offer, it is fed from `observed_local`, and a miss must not set it -- so the sentinel by itself makes the 2106-loop WORSE than the 0 did. The old 0 was a false measurement, but it at least clamped. At the cap PP0 emits ONE loud refusal naming the rid and all four lookup locations, and pins the next offer to `told=0` -- the only offer honourable without a measurement, hence the only terminator available once the measurable one is gone. The count clears only on a pass that actually SERVED the rid (sgl-project#552's lesson: a defer that resets its own counter makes the bound unreachable). `<= 0` disables the bound, so it can be neutered on its own in a can-fail proof. THE GROUP DEFERS, NOT ONE RANK. A defer that only one rank takes IS the next divergence. Downstream ranks only REPORT `unresolved`; the pass is voided by the existing sgl-project#797 mechanism, which is already group-uniform (`pp_pass_should_void` ORs the incoming flag and never clears it). Whether to defer again or escalate is decided once, by PP0, from a count that rode the wire. No new collective, no new send. TESTS. New, `test_pp_unresolved_defer_cap_944.py` (22 tests + 3 subtests): the told=0 hoist, the two populations separable on the wire and through the codec, the cap's counting/escalation/clearing/per-rid scoping, and TERMINATION with a can-fail arm that neuters ONLY the cap and shows the offer standing at 4096 for 23 rounds with nothing admitted. New, `test_pp_unresolved_group_defer_gloo_944.py` (4 tests, 3 real gloo processes, deadline-bounded): (a) one rank's miss defers the group and then resolves, with no floor invented on any round and the sgl-project#944 line logged where the sgl-project#791 line must not be; (b) unresolved everywhere -> exactly one loud refusal, `told` ends at 0, served exactly at the cap, never a hang; (c) the danger mutant (`observed_local=0` + `unresolved=False`, the exact pre-sgl-project#944 shape) is caught. (c) also records that the MUTANT TERMINATES FASTER than the fix -- the false 0 IS a clamp, which is why this defect twice looked like it was working, and why "it terminated" is worthless as evidence here. `TheConsumerSweepRatchet` is the class fix's future half. #797c and sgl-project#798 were each fixed by adding a lookup, and neither asked who READS the value; that is the only reason the same defect survived twice. It pins the file set reading `observed_local` and `UNKNOWN_MATCH`, in src AND test. The four contract inversions this change required were found by grep; the fifth will be found by a red test. Four contract inversions in `test_pp_retracted_pass_void_797.py` and `test_pp_reconcile_slot_blind_798.py`, each with the reasoning recorded at the site and none deleted: they pinned the miss as the specimen's exact 0, correct while a miss was SPELLED as a measurement. The behaviour they reproduce is unchanged -- still retracted, still absent from `effective`. One test renamed (`..._still_reads_as_zero` -> `..._is_reported_as_unresolved`): its name outlived its contract by one ticket and would have kept teaching the sentence that cost three boots. TEST RESULTS. Frozen before/after over the derived blast radius (every registered module mentioning `pp_admission_congruence`, `PPAdmissionEntry`, the wire codec, the three reconcile/send/recv mixin methods, `observed_local`, or the return-trip pair), each module ALONE IN A FRESH PROCESS, identical script both sides, baseline in a detached worktree at 70bbd98: BEFORE 20 modules 209 passed 0 failed AFTER 22 modules 235 passed 0 failed Every shared module matches its baseline count exactly; the two deliberately red pins are green at their original assertions and 797 is back to 31/31. Extraction count probe passes on both sides (extracted FAILED names == summary failed count). ruff (F401,F821,UP037), ruff format and codespell clean on all six touched files.

window-958-boot died 25 s into the chunked acceptance load at scheduler.py:7010, `AttributeError: 'NoneType' object has no attribute 'end'`, one line after `sgl-project#946 PREMISE RECOMPUTE`. THE ATTRIBUTION IN THE WINDOW CLOSEOUT IS REFUTED, BY TWO INDEPENDENT INSTRUMENTS. It put the null on the sgl-project#906 seam-refusal branch (scheduler.py:8916). * LOG. The full-phrase counter `[sgl-project#906] SEAM CHUNK REFUSED rid=` is 0 in BOTH boots, and `_note_seam_chunk_refused` logs its first three occurrences unconditionally (scheduler.py:5434-5446), so the zero is a measurement and not a rate limit. Other bracketed INFO tags from the same process are present in the same file, so the sink is not the explanation. * COVERAGE, which does not depend on any logging decision. The boot rode `SGLANG_949_COVERAGE=1`; boot 1's three rank databases (evidence-665-f1/trace949_0828/.coverage.24735{12,13,14}) all report scheduler.py:8916 **not executed**, while :8911 (the truncation), :8918 (the adder), :7010 (the reader) and scheduler_pp_mixin.py:2159/:2161 (the ring actuator and the call below it) are all executed on all three ranks. A fix at that branch alone would not have touched this crash. The junction took the ELSE branch every time and re-derived, exactly as sgl-project#946 argued. THE PRODUCER IS IN THE TRACEBACK, two statements above the reader. scheduler_pp_mixin.py:2159 calls `pp_apply_dead_premise_anywhere` -- sgl-project#948's relocated actuator, armed for that boot by `SGLANG_946_ACT_AT_RING=1` in the window's own recipe -- whose terminator runs `truncate_prefix_to(0)`; :2161 then calls `get_next_batch_to_run`. Nothing re-derives in between. sgl-project#946 had justified the truncation by its NEIGHBOURHOOD ("`add_chunked_req` below derives everything from `len(req.prefix_indices)` and only THEN calls `set_extend_range`"); sgl-project#948 moved the act to a site that RUNS, for a measured reason recorded at scheduler_pp_mixin.py:2100-2109 (the old site was entered ~6 times while 9471 passes voided), and the legality argument did not travel with it. FIX AT THE WRITER, ONE PLACE. `Req.truncate_prefix_to` leaves `Range(told, told)` -- zero rows at the prefix that now exists -- instead of `None`. This satisfies `_executed_extent`'s invariant `extend_range.start == len(prefix_indices)` by construction at the only place that can break it, and closes four producers in one cut instead of one branch per boot: the ring actuator (:2159), the seam refusal (scheduler.py:8916), `add_chunked_req`'s hybrid-SWA zero-budget return (schedule_policy.py:1396, which unlike the sgl-project#679 park at :1434-1436 returns without `set_extend_range`), and the sgl-project#791 clamp sites on a `NO_TOKEN` break. None of the four can now receive a null geometry, because none is produced. ONE EDIT WAS WRITTEN AND WITHDRAWN, recorded in the code rather than dropped. Making that hybrid-SWA branch write the park geometry -- so the two park branches say the same thing -- broke `test_prefill_adder.py::test_add_chunked_req_hybrid_swa_defers_when_swa_ below_page`, which pins "returned unchanged" via `set_extend_range.assert_not_called()`. With the writer fixed the branch is no longer a producer, so the edit would have been consistency rather than a fix, and it is not free: it would overwrite the PREVIOUS chunk's range on any path reaching this branch before that chunk is stashed. In production the stash runs earlier in the same pass, so the write would be value-neutral -- but that is an argument, not a measurement, and hybrid SWA is not a configuration this fork boots. Reverted, and the divergence between the two park branches is named at the site as open. sgl-project#958's ARGUMENT IS HONOURED, NOT REVERSED. Its "NONE, NOT A RECOMPUTED RANGE" paragraph refuses `Range(told, old_end)` because keeping the old end would INVENT a pass. `Range(told, told)` invents nothing, and it is not a new state: `_park_chunked_prefill_chunk` writes `Range(start, start)`, the sgl-project#679 park writes it, and `_executed_extent` declares zero-length ranges first-class. The offer still moves -- now WITHOUT the adder: `_executed_extent` reads (0, 0), so PP0 offers told=0, the value `reconcile_pp_admission_decision` admits unconditionally. `reset_for_retract`'s `None` is deliberately untouched: two disposal sites key off that sentinel (scheduler_pp_mixin.py:6061-6075, :7222-7236) and flipping it would have silenced them. The refused-geometry exit stays reachable from that producer and is now pinned by its own test. THE COMMIT'S OWN SAFETY NET WAS DOWNSTREAM OF THE CRASH. `PPScheduleRefused` / `require_executed_geometry` fired 0 times on metal while the unguarded dereference killed the process, because it iterates `can_run_list` and a resident continuation the adder did not add is never in it. It is not made reachable here; it is made unnecessary, and the structural reason is asserted rather than argued. SIBLING, same class, fixed here so this change does not widen it: `_park_chunked_prefill_chunk` handed back the `inflight_middle_chunks` increment whenever it got past its `end is None` gate rather than only when a chunk was actually prepared. Already reachable before this change via the sgl-project#679 park's `Range(prefix, prefix)`. The predicate is now the same `end > start` the KV release beside it already used -- one expression, not two. #962a: THE SEAM PROBE COULD NOT PROVE ITS HOOK RAN. The reachability probe `cutover_participants.py` registers for `latched_batch_flags` was emitted only `if any(_stale.values())`, so "ran and found nothing" and "never ran" were byte-identical -- the sgl-project#719 shape the registry's own docstring forbids. It is now unconditional and reports `reached=`, because W37-C already showed a bare zero is not enough (it logged `checked=0` eighteen times and was still blind). sgl-project#962 ITSELF IS REFUTED, no code change warranted. `batch_is_full` does not survive the tp_to_pp cutover: the hook is unconditional in `_cutover` with no early return before the completion log; it provably ran (`cutover complete: active stack` 6 and `[sgl-project#690] CUTOVER SUB-STEPS` 6 in both boots); it cleared nothing (`#861c cleared latched batch flag(s)` 0/0); PP0 admits 8 times (boot 1) / 4 times (boot 2) after the cutover before the first latched decline; and boot 1 alternates DECLINE/ADMIT eight times in one second while having MORE latched declines (5 vs 3) and NO livelock. #962b registered, not fixed: #888b's `parked_carrier_relief` re-derivation is on the post-flip path (scheduler.py:8587) but inert, because its gate reads `_parked_decode_verdict`, whose only writer (`_note_parked_carriers`, called at scheduler.py:7675) sits behind `not running_batch.is_empty()` and is unreachable at running=0 -- the state the relief exists for. Measured 0/0 against 8 latched declines. Needs its own danger-direction analysis. TESTS. `test_truncation_geometry_961.py`, 15 tests, RED FIRST at the pin (8 failed / 7 passed before the fix). The `:7010` reader is driven through the REAL `Scheduler.get_next_batch_to_run` on an uninitialised instance carrying the five attributes that line needs, so it reproduces the production AttributeError on the production line rather than on a copy of it; `_Req` borrows the real `Req.truncate_prefix_to`. Five CANFAIL mutants pin each reader to the invariant and pass before AND after. Two further readers are driven for real (`_compute_chunked_req_next_prompt_token`, `pp_chunked_local_match`) plus the real producer (`build_pp_admission_decision`). `test_offer_delivery_958.py`: its EXIT_3 test required the refusal this fix makes unproducible; corrected to assert the moved offer, and split so EXIT 3 stays pinned against the `reset_for_retract` producer that still reaches it. `test_latched_batch_flags_861c.py`: 3 tests for the #962a receipt, including a can-fail that a blind seam is not reported as an all-clear. DESK GATE, /spinning/htsglang-gpu/.venv, CVD="". BEFORE (frozen at the pin 78d030e): serial 895 passed / 2 failed, wide 3701, narrow 202. AFTER: see NOTE below. Failure set unchanged: the two pre-existing test_collective_family_siblings_610.py failures, untouched. sgl-project#954 (test_prefetch_progress_symmetry_580.py) is outside gate scope, as before. ruff: no finding on any of the 273 changed lines (all 64 pre-existing); new test file clean and ruff-formatted. codespell: new file clean; the one hit in phase_flip_draft_bootstrap.py:558 is pre-existing. No boot was run. /spinning/gpu-arb/TICKET_961_WINDOW.md carries the boot acceptance and is drivable from that file alone.

… was in the tree ROOT, per-rank coverage of window-958 boot 2 (pin 78d030e) plus both boot logs. Under pure PP the HiCache storage-prefetch veto is a RANK-LOCAL verdict: prefetch_ballot.prefetch_done_under_ballot returns the local value when the ballot is None, and the ballot rides _update_uniform_pool_budget's reduce on tp_cpu_group -- a group the PP loop never passes, and which has world 1 under pp_size>1 in any case. The ranks therefore decide independently. scheduler.py :9048 (`continue` after _note_skip("prefetch_pending")) is the LAST line PP0 and PP1 both ran; :9050 is the first PP0 ran ALONE. Everything downstream -- add_one_req, `self.chunked_req = adder.new_chunked_req`, the unconditional stash at :7010-7011 -> stash_chunked_request:5449 -> mem_cache/common.py:169 -> cache_unfinished_req -- is PP0-only. PP0's radix tree gains a 1250-token prefix PP1's and PP2's never received. From there it is self-sustaining. PP0 matches its own tree and offers told=1250; PP1 measures local=0 HONESTLY against its own; sgl-project#791 retracts, sgl-project#797 voids, the requeue resets the REQUEST and nothing resets the TREE. PP1 can only acquire the prefix by running the batch it is refused for lacking the prefix. WHY THE EXISTING BOUND COULD NOT REACH IT. PPAdmissionCongruenceGuard is, in its own words, "RID-SCOPED, ONE-SHOT", and its termination argument is per rid: each new retraction for THAT rid lowers THAT rid's floor. The argument is sound and silent about the POPULATION. The shortfall is a property of the TREE, so every fresh rid over the same prefix starts unclamped and buys its own voided pass -- six distinct rids in one second, all told=1250 local=0, which is why _learned_floor was measured RUNNING and LOWERING on PP0 and never bound. FIX: the same actuator, scoped to the prefix the offer was made over. No new mechanism and no new collective -- the recorded fatal (the 2026-08-17 HiCache ack-count reduction) rules out a collective on this path, and a group-uniform INSERT is structurally impossible: the radix value is a tensor of that rank's own KV slot ids, dereferenced as real memory by four consumers (allocator free at unified_radix_cache.py:1713, evict at full_component.py:96, HiCache backup at :2284, write-back into req_to_token at :1325), and the GDN mamba component donates a live state slot a non-computing rank does not have. offered_prefix_key() names the prefix by a blake2b fingerprint of its tokens -- NOT hash(), which is PYTHONHASHSEED-salted and would disagree between the very ranks this keeps congruent (tree_congruence's constraint 3, same lesson). Length is mixed in so a prefix and its extension cannot share a floor. prefix_key=None leaves the pre-sgl-project#963 rid-scoped path byte-identical. PRIOR ART, gated rather than duplicated. #616g's group-MIN reduces available_size(), a different quantity, and is switched OFF in the PP phase (tp_cpu_group world=1). sgl-project#823's tp_head_congruence MINs the right quantity but only REORDERS, and its enforcer_gate returns GATE_OFF_TP_WORLD_OF_ONE here. sgl-project#825's tree_congruence detects exactly this divergence and emitted 0 in BOTH boots: in the PP phase on_round is reached only via scheduler_pp_mixin.py:2477 with require_armed_and_parked=True, so it samples only at an armed flip -- which the livelock prevents. That is this instance's compensator-reachability gap. RED-FIRST, and the first red was rejected as worthless: all six cases failed on the SIGNATURE (TypeError), not on behaviour. Four mutants were then run against the finished fix and their errors READ: consultation withdrawn -> assert 6 == 1 (boot 2's six voided passes, exactly) learn withdrawn -> assert 6 == 1 clear-on-serve withdrawn-> assert 0 == 1250 (permanent loss on a healed prefix) key-blind clamp -> assert 0 == 1250 (loss on a prefix nobody reported) The last two are the danger direction: this fix must never discard a prefix every rank holds. It clamps only against an OBSERVED shortfall on a SPECIFIC prefix, and clears the moment the group serves that prefix, so the cost is one voided pass once rather than one per rid for ever. Double-prefill law: the requeue loses 0 today and still loses 0; what changes is that it terminates. An UNRESOLVED miss (observed_local=None) teaches the prefix floor NOTHING -- the stakes are strictly higher than for the rid scope, since a floor invented from a number nobody measured would cap every request over that prefix. The sgl-project#944 consumer ratchet CAUGHT this file on its first gate run and is now registered. NUMBERS, desk gate scripts/gate_tier2_partitioned.py, CVD="" : BEFORE 2 genuine (test_collective_family_siblings_610.py x2), 685.00 s AFTER 2 genuine (the same two), plus 14 new tests green count probe: 2 named == 2 summary, SUBFAILED and ERRORS included ruff clean on both touched files. NOT CLAIMED: no boot, no metal. The divergence SOURCE (the rank-local prefetch verdict under PP) is named here and left open -- closing it needs the sgl-project#791 ring lap to carry a prefetch-pending fact home, which is its own posten. This makes the system self-healing against a divergence however caused.

…aterialises it Root (R9 census, boots 6-7): 506 of 513 void-causing refusals were ONE rid failing the sgl-project#791 fill-length clause by ONE token -- rank 0 alone holds a sampled output token that never crossed the tp_to_pp seam (sgl-project#631 OUTTRACE: PP0 n=1 tail=[25], followers n=0), so 8447 vs 8446; rank 0 alternates offers 7939/0, resetting the told-keyed sgl-project#944 streak every pass: 1 termination in 506 laps. sgl-project#984 was measurably neutral on these counts (its docstring premise falsified, recorded). Part B (the serving fix): PPAdmissionEntry gains optional fill_len + fill_tail (cap 8); build publishes on both entry branches; the follower adopts at the ONE junction directly before the clause reads local_fill_len, materialising the tail into a SHADOW pair honored only by _refresh_fill_ids -- NEVER into output_ids: consumer sweep found two unguarded readers (output_streamer slices output_ids into the CLIENT payload on pp_rank 0; the per-rank max_new_tokens check would let one rank finish a step early = the recorded cross-rank hang class). The clause itself is untouched -- the fix ends the disagreement instead of weakening the guard. Adopt is idempotent, self-cancelling once the rank catches up, refuses loudly (log, no new raise) on cap/behind. Wire codec rewritten index-based: a legacy 8-wide row decodes to fill_len=None (byte-identical); the old fixed-arity unpack would have crashed on any width skew between ends one commit apart. Part A (abort net): the sgl-project#944 streak keys on refusal persistence -- _refused_since_offer written by PP0 as it absorbs its own void (strictly local; a schedule refusal laps home with ZERO entries, which is why UNRESOLVABLE read 0 beside a 506-refusal census), consumed in note_offer; the told<=0 exemption no longer applies to a refused rid. The alternation livelock now escalates in ~4 laps to the existing terminator. sgl-project#955's healthy end preserved: an unrefused moving offer never escalates. Part C: sgl-project#986 -- pp_queue_orphaned_chunked_req returns the admission tree-lock ref (adjudicated NEW-with-#968b leak). Plus the test-agent's zero-case instrument at the rehome reset-shape return. Instruments: sgl-project#987 FILL-ADOPT (both fills + appended count, gated first+64th), sgl-project#987 FILL-REFUSE (cap/behind), refusal-streak visible in the existing UNRESOLVABLE line. Bonus recorded: last_chunk is now computed from the same number on both ranks. Execution proof: import smoke on all five modules (CVD=""); call-graph chains close at scheduler.py:9472/:8982, schedule_policy.py:1442/:1804, event loop :9515; the 1390-clause ran 338x in boot 7 -- the adopt site is on a proven hot path. 31/31 cold checks on the exact boot-7 numbers (appended=1, id=25, veto-before/pass-after, 4-lap abort vs 506); one fixture self-catch recorded. Suites live in the test-agent lane.

… its seat too Boot 11 (aae7e3b) reached 3m47s -- sgl-project#991 held, no AttributeError -- and died on `sgl-project#801-spin PP IDLE-VOID LIVELOCK REFUSED` (pp_rank=2, 512 consecutive voided passes, scheduler_pp_mixin.py:8214). The guard is right that the defect is not on rank 2. ROOT. Uniform membership -- "a rank executing a forwarded schedule may admit only what the decision names" -- is enforced at scheduler.py's `pp_not_named` skip for every candidate that comes out of `waiting_queue`, and NOWHERE for the one candidate that never appears in a queue. `add_chunked_req` runs ~250 lines earlier and appends `self.chunked_req` unconditionally, before the decision is consulted at all. MEASURED, 512 byte-identical rounds (21:27-21:31): rank 1 held rid 8a330526c7b9410f963232874adc451b at executed=3332 as its parked continuation, in NO queue on ANY rank, so rank 0's decision could not name it. That site gave it the seat; the decision's own rid (dfd22a9a… / 9528ff20…) could then not be reached; `sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE` voided the pass; `sgl-project#971 REHOME-ON-REFUSAL` put the continuation straight back into `self.chunked_req`; the next pass was identical. Rank 2 voided every pass behind it until the bound fired. THE CLASS: a compensator that restores the precondition of its own trigger. The cut therefore belongs upstream of the rehome, at the admission -- fixing sgl-project#971 would only move the latch. FIX. At the one junction, gate `add_chunked_req` on the same membership test the queue loop already applies: when this rank is executing a forwarded schedule that does not name its continuation, refuse the SEAT. Not a drop -- the precedent is the sgl-project#906 gate immediately below it: the request stays `self.chunked_req`, keeps prefix, pages and mid-chunk position, and resumes on the first pass whose decision names it. PP0 has no incoming decision, so `incoming is None` there and the default path is byte-identical. SIBLING SWEEP: the two candidate sources into `can_run_list` under a forwarded schedule are the queue loop and this one; the queue loop was already gated. `_note_skip("batch_full_break")` and `pp_not_named` are the two symptoms this produced in the census and both are downstream of it. WHAT THIS DOES NOT CLOSE, stated rather than assumed: sgl-project#968's half. sgl-project#968 makes the parked-continuation fact ride the return lap so PP0 CAN name such a rid, but its actuator `pp_parked_continuation_priority` only REORDERS PP0's queue and is a no-op exactly when the rid is in no queue on PP0 -- which is this specimen. The new log line prints the rid, its executed extent and the decision's full name set, so boot 12 MEASURES whether the continuation ever becomes named instead of leaving it to inference. Evidence: desk. py_compile + import smoke. Belegstufe: DESK-BEWIESEN.

Boot 13 (4fde6e5) did not crash: it WEDGED. 5 queued, 0 running, no first token, on all three ranks, behind 239 identical refusals of one shape: sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE for rid=901a7d29…: the decision names prefix_len=0, this rank holds 7938. ROOT, and it is the second half of the exemption sgl-project#992 found. `sgl-project#791 PP ADMISSION UNIFORMITY` has two halves -- MEMBERSHIP ("admit only what the decision names") and GEOMETRY ("with exactly the prefix it named") -- and both are implemented inside the waiting-queue loop, applied strictly before `adder.add_one_req`. The chunked continuation does not travel that loop: it reaches `can_run_list` through `add_chunked_req`, ~250 lines earlier, and was therefore exempt from BOTH. sgl-project#992 closed membership. Geometry stayed open, so a NAMED continuation entered the batch carrying its own prefix while the decision named another -- and that is a SHAPE disagreement, because `prepare_for_extend` sizes the cross-stage tensor directly off `len(req.prefix_indices)`. The follower refused, correctly, every pass. The decision said 0 because PP0 had spent its `sgl-project#946 PREMISE RECOMPUTE` terminator on that rid. That is PP0's call to make and the follower's to execute; second-guessing it is what the refusal exists to prevent. FIX: apply the same `truncate_prefix_to(told)` the queue loop applies, at the same point in the same order -- immediately before the re-derivation. sgl-project#930: the helper moves `prefix_indices` and `cache_protected_len` together. sgl-project#961: the mover must be followed by the re-derivation it invalidates; in the queue loop that is `add_one_req`, here it is `add_chunked_req` on the next line. Nothing is re-derived by hand and no new helper is introduced. Logged with rid, local prefix and told, counted, so the adoption is affirmative rather than inferred from the absence of refusals. SIBLING SWEEP: the two entries into `can_run_list` under a forwarded schedule are the queue loop and this one. Both now carry both halves of uniformity. PP0 has no incoming decision (`incoming is None`), so its path is byte-identical. BOOT 13 ALSO BOUGHT: sgl-project#993's containment fired and held -- no repeat of boot 12's allocator assert. sgl-project#992 held -- no repeat of boot 11's sgl-project#801-spin. Batch lines were symmetric 5/5/5 across ranks, i.e. no rank is structurally excluded. Evidence: desk. py_compile + import smoke + wiring assertion. Belegstufe: DESK-BEWIESEN.

…resident at all Boot 14 (cf16281) died on PP1 after 39 s: `assert self.chunked_req is None`, scheduler.py:9784 in `_get_new_batch_prefill_raw`. Third recorded fundstelle of this family after :9286 (sgl-project#951) and :9367 (sgl-project#959). ROOT, and it corrects my own attribution in the boot-14 register entry. The invariant behind that assert is held "by ARITHMETIC, not by a check" (scheduler.py's own comment). sgl-project#959 therefore gave the two `add_one_req*` mint sites an explicit `chunked_req_outstanding` check, and SKIPPED the third -- `PrefillAdder._add_scheduled_req`, the forwarded-schedule execution path -- reasoning at schedule_policy.py that it "already has its own (`carried_chunk`)". That is the guard-comment-names-the-hazard trap. `carried_chunk` answers "is THIS request the resident continuation". The invariant needs "is there a resident continuation AT ALL". It covers a request being re-announced; it does not cover a DIFFERENT named request becoming a second continuation while the first is resident. Two of three sites guarded, one not. Boot 14 is that gap on metal: the resident continuation survived `add_chunked_req`, so `chunked_req_outstanding` was True and both sibling sites correctly refused -- and this site minted anyway, on another rid the same forwarded schedule named. sgl-project#994 EXPOSED THIS, IT DID NOT CREATE IT. Boot 13 never reached the line because the sgl-project#791 geometry refusal killed every pass before a batch was built; sgl-project#994 removed that refusal. My register entry attributed the death to sgl-project#994's effect on `rem_chunk_tokens` arithmetic. That was wrong: both sibling sites were already guarded and did refuse. The arithmetic was not the hole; the missing third guard was. Withdrawn here rather than left standing. FIX: the sibling guard, at the site that lacked it. WHY A PASS REFUSAL AND NOT A REQUEST SKIP -- the danger direction, which is the whole question here. On a forwarded schedule this rank may NOT drop a named request: the upstream's hidden states for it are already on the wire, which is what this same method already raises `PPScheduleRefused` for a few lines above. Running the chunk WITHOUT announcing it is worse -- the continuation would be untracked and re-prefilled next pass, the double prefill the standing law forbids. So the disposal is the one this path already owns: refuse the PASS by name, let sgl-project#791/sgl-project#797 void and re-derive. It cannot starve: the resident continuation is consuming chunks, and when it finishes `chunked_req` is None and the schedule is executable. This is the same direction sgl-project#959 chose ("the resident continuation is never the one to give way; the fresh admission is") -- not the sgl-project#858 mid-prefill wedge, which would be clearing `scheduler.chunked_req` instead. FUTURE CHECK: `grep -c 'if self.chunked_req_outstanding:'` over schedule_policy.py is now 3 and equals the number of `new_chunked_req` writers. A fourth writer that skips it is visible as an inequality rather than as a boot death. Execution proof (speed mode, one instrument per link): `note_second_continuation_refused(req, "_add_scheduled_req")` counts it and the refusal text names itself, so boot 15 measures whether the guard fires at all -- absence of the assert alone would be green-by-absence. Evidence: desk. py_compile; local-import dominance and guard-before-raise- before-mint verified by source order in the loaded module; guarded-site count 3 == mint-site count 3. Belegstufe: DESK-BEWIESEN.

…ery rank extend = min(fill - prefix, cap). Three of the four quantities are now settled by measurement: cap -- group-uniform. sgl-project#610 subtracts each rank's surplus over the group minimum, and all three ranks read 12977. It also never bound: the binding term was chunk(4096) against extends of 253/254/ 277/278. prefix -- equalised by the sgl-project#791 truncation, both branches, code-read. extend -- differs. That is the observed 254 against 301. So the fill differs, and `_refresh_fill_ids` defines it as `origin_input_ids + output_ids` with a rank-identical prompt. `output_ids` is the single rank-local component of the whole equation. And `req.output_ids.append` lives in the RESULT path (batch_result_processor.py:276), which a VOIDED pass never reaches. These boots are void-dominated. That is a complete chain -- void pass, output falls behind, fill falls behind, extend differs, the sgl-project#631 guard fires -- and every link of it is code-read rather than argued. It is not yet MEASURED. This prints `fill_lens` and `out_lens` per rid on the existing `sgl-project#788 PP-ADMISSION verdict` line, which already carries `rids` and `prefix_lens` and already fires on every rank. It is the only carrier that allows a cross-rank comparison FOR THE SAME REQUEST. No new emission: the same `logger.info`, more fields -- the event perturbs this path (measured four times), the bytes do not (measured twice, and this is the third test). THE PREDICTION IS DIRECTIONAL, not just "a difference": the rank with the smaller `out_lens` must also show the smaller `fill_lens` and the smaller extend. A difference in the wrong direction refutes the chain rather than confirming it. If `out_lens` agrees across ranks the chain is sound but not what happened here, and the remaining writer is `pp_carried_fill_tail` (schedule_batch.py:1300-1305), which is also rank-local. I will not compute 301-254=47 into anything, whichever way the numbers land. Name the carrier, then measure the difference -- this window produced four numbers-fit traps and I walked into one. Evidence: desk. py_compile; the format verified by rendering it through logging rather than by counting placeholders. Belegstufe: DESK-BEWIESEN.

Three carriers were checked for an enrichment and all three fail: no rid-precise log exists after `req.output_ids.append`, `sgl-project#968 MINT` sits in the parked-continuation fact ring rather than the result path, and `sgl-project#788` fires at ADMISSION where `out_len` is structurally 0 (measured, boot 38). A new emission is therefore the only option, not a convenience -- and it is a STATE reading, not a timing signal, the same distinction that carried `published_fundable_floor`. WHAT IT MEASURES. extend = min(fill - prefix, cap). Three terms are settled: cap is group-uniform (sgl-project#610 pins each rank to the group minimum, measured identical at 12977) and never bound (4096 against extends of 253-278); prefix is equalised by the sgl-project#791 truncation, both branches. Fill is defined as `origin_input_ids + output_ids` with a rank-identical prompt, so `output_ids` is the single rank-local component of the whole equation -- and this append, in the RESULT path a voided pass never reaches, is the only place it grows. THREE CONSTRAINTS, each from a mistake made in this window: DENOMINATOR ALWAYS. seen/emitted/unreadable ride on every line whatever they are. A counter that only moves on confirmations is what made me quote a 12:1 neutrality basis that was really 10:5. FIXED CADENCE, NOT A HIT FILTER. "Emit only when interesting" cannot tell a zero from a never-evaluated -- the same shape that made boot 27 unreadable. Every 40th append, whatever it says. SENTINEL THAT CANNOT COLLIDE. `len(output_ids)` can legitimately be 0, so 0 and None are both unusable for "not read"; -1 is the sentinel and the message says so. `floor=unset` already saved one reading this way. THE DEATH FORM IS NOT A SIGNAL HERE, in either direction. Cumulative over 15 no-flip boots of this config: WIDTH=10, IDENTITY=5. At a one-in-three base rate a single flip is noise, and a single non-flip proves nothing. The only question this boot answers is what `out_len` says for the same rid on two ranks. Evidence: desk. py_compile; the format verified by rendering it through logging rather than by counting placeholders. Belegstufe: DESK-BEWIESEN.

The flip-hosted trace is the structural half of sgl-project#997 and stays off the critical path. This measurement does not need it -- it needs two integers. WHERE. `req.output_ids.extend(next_token_id)` in `process_batch_result_decode` (batch_result_processor.py:873) -- the DECODE append, which is the one that feeds the divergent term. #997b instrumented the PREFILL append instead and could not have seen it: the dying batch was EXTEND, but its fill carries the output_ids from the preceding DECODE steps. WHY THIS IS THE LAST TERM. extend = min(fill - prefix, cap). prefix is equalised by the sgl-project#791 truncation (both branches, code-read); cap is group-uniform (sgl-project#610 pins each rank to the group minimum, measured identical at 12977) and never bound (4096 against extends of 253-278); fill is `origin_input_ids + output_ids` with a rank-identical prompt. `output_ids` is what is left, and this append is where it grows -- inside the RESULT path, which a voided pass never reaches. FOUR CONSTRAINTS, each paid for in this window: COUNT AT THE EVENT, EMIT ELSEWHERE. #997b put its denominator inside the line it gated, so zero lines meant zero information. A denominator that lives in the same call as the event shares the event's fate. THE EMISSION SITE IS VERIFIED TO RUN FIRST. `pp_ring_note` printed 241 lines across all three ranks in boot 36. Boot 40 cost a cycle to the opposite mistake: I fixed a gate on an object that was never constructed, because I checked the arming and not the host. NO HIT FILTER, DENOMINATOR ALWAYS. `seen` and `tracked_rids` ride on every line, and the message states that an empty sample with seen=0 means the append never ran rather than that nothing diverged. SENTINEL -1. `len(output_ids)` can legitimately be 0. The per-rid map is last-writer-wins and bounded at 32 entries: the reader compares one rid across ranks, so one live value per rid is enough. Cumulative, carried in every report: WIDTH=12, IDENTITY=5 over 17 no-flip boots. At a one-in-three base rate the death form is not a signal in either direction, and this is a state reading regardless. Evidence: desk. py_compile on both files; smoke asserting the counter sits at the decode append with a working scheduler path, and that the emission sits in `pp_ring_note`. Belegstufe: DESK-BEWIESEN.

Boot 41 measured the first term of the decomposition and it is 1: output_ids differs between ranks by one, with PP2 -- the rank that samples first -- ahead. That is the normal pipeline offset, not a defect. The second term had never been measured, and the code says it exists. `fill = origin_input_ids + output_ids` holds only at the instant `_refresh_fill_ids` runs, and that runs from `init_next_round_input`, i.e. gated on THIS rank admitting THIS request. In the admission loop the refresh sits at scheduler.py:9487 and the rank-local skips lie on both sides of it: `already_in_batch`, `seam_transport_only`, `lora`, `batch_full_break` and the three `prefetch_*` skips all break out BEFORE it (:9415-9480), while `pp_not_named` comes after (:9533). A rank that breaks out early keeps appending while its fill stands still. So fill-divergence = output-divergence + refresh-timing divergence, and only the first was measured. This measures the second as `lag = len(output_ids) - (len(fill) - len(origin_input_ids))` -- exactly the `n_have_output` that `_refresh_fill_ids` computes at schedule_batch.py:1313, verified against that definition in the smoke. It is an enrichment of the probe added in #997d: same counter site (the DECODE append), same emission site (`pp_ring_note`, whose execution on this config is established by 594 lines across three ranks in boot 41), no new emission, one more field in a tuple that was already printed. FIFTH INSTANCE OF THE EVENING'S CLASS, and the first on the killing path. A rank-local predicate deciding a group-uniform quantity: Site #0, the hold actuator, the proxy send/recv boundary and the sgl-project#996 budget cap were desk-proven, indirect, or never bound. This one gates `full_untruncated_fill_ids` -- the quantity whose divergence the sgl-project#631 guard reports. AND IT MEETS A FILED POSTEN: sgl-project#968 has stood since 2026-08-28 as "divergence SOURCE: rank-local prefetch verdict under PP -- the prefetch-pending fact must ride the sgl-project#791 ring lap home". That prefetch skip is one of the two that gate this refresh. A design question parked "after the window" turns out to sit on the critical path; the fix cut should take it up rather than reinvent it. The prediction is recorded before the boot, both directions, and the lag will not be held against 47 whichever way it lands -- the question is whether the refresh timing diverges, not whether a number fits. Evidence: desk. py_compile on both files; the lag arithmetic checked against the :1313 definition on three worked values. Belegstufe: DESK-BEWIESEN.

`set_extend_range(prefix, prefix + new_len)` means `extend_range.start == len(prefix_indices)`, and `prepare_for_extend` slices `fill[len(prefix) : extend_range.end]`. So len(input_ids) = end - len(prefix) = end - start = new_len holds ONLY while `start == len(prefix_indices)`. That is one condition, and `prepare_for_extend` is the one place that reads both halves. WHY THIS REPLACES THE ENUMERATION. There are 24 writers of `extend_range` and 27 of `prefix_indices` across five directories. Two were read -- the sgl-project#791 truncation and the #797b park -- and both carry the group; the park does so precisely because it derives its shape FROM the prefix in the same call. But 2 of 24 is not a basis, and reading the other 22 still could not show the case where every writer is individually correct and their ORDER breaks the pair. A condition at the consumer covers all 51 writers and that case as well. Third time today that checking the consumer beat enumerating the producers. IT IS ALSO THE MEASUREMENT. The guard reports 254 rows against 301 tokens; this decomposes the 301 into its two terms, per rid, on every rank. If `end` or `len(prefix)` stands differently on the sender, the root is read rather than inferred -- and the rid then names which writer touched it last, so the enumeration survives as a fallback reduced to one entry. Discipline unchanged, and each clause was paid for in this window: counter at the event, emission in `pp_ring_note` (independent of this probe, its execution on this config established), no hit filter, `seen` and `breaks` always printed so a zero is distinguishable from a never-run, and -1 as a sentinel because 0 is a legitimate break value. Predictions recorded before the boot, both directions. A: `break == 0` everywhere, which puts the divergence in the READ or the TIMING rather than the setting -- the first question of this window an enumeration cannot reach. B: `break != 0` somewhere, which is the root. I expect A weakly, and say why: the coupling is deliberately built at the truncation junction ("sgl-project#965 THE WHOLE CO-DERIVED GROUP, AS A GROUP"), but that is an argument from 2 of 24, and I have been wrong twice in this window reasoning from a subset. The 47 will not be computed into anything either way. The question is whether the invariant breaks. Evidence: desk. py_compile on both files; the module bridge imports; the invariant arithmetic checked on a holding and a broken case. Belegstufe: DESK-BEWIESEN.

Boot 53 completed ZERO flips and died at the first cutover with the ranks in three different places -- PP0 in HiCache pp-ring-commit/send_req_work, PP1 in pp-ring-commit/p2p[0], PP2 in a lazy torch p2p pair init for '0:2' on the DEFAULT group, a wire the six good flips of boots 51 and 52 never opened. The rank-local void I added in #1002bc is what reached for it: one rank changed the slot, ran ahead into the cutover, and left the others in the ring. That is precisely the rule my own 59830ce states -- the decision may be group-agreed, its CONSEQUENCE is not -- and the bc half broke it one commit after writing it down. Declining a receive is local and harmless. Changing the slot is a statement about the group, and it needs the group's channels. So the rank-local half is gone in both paths. The proxy gate goes back to raising: declining there would have to void (no hidden states, nothing to compute), and a void inside the cutover window is exactly what killed boot 53. Softening it needs the verdict carried on the sgl-project#791 ring first -- one send over the exercised CHAN_DICT machinery that already carried six clean flips, never a new wire and never torch p2p on the default group (barlink has no send/recv, sgl-project#732). That is a separate cut. What is NOT reverted is the soft decline itself, and the reason is measured: boot 52 ran c7e50a4, reached six flips, and died on "AttributeError: 'NoneType' object has no attribute 'synchronize'". A plain revert restores that death. The deref is fixed where it actually lives -- a guard at the call, no state change -- because a declined output leaves `d2h_event` unset by design, exactly as the neighbouring `target is None` exit already does. Evidence: desk, executed. py_compile; no soft proxy gate remains; one soft output gate; five guarded derefs; no `_pp_void_own_batch` on any decline path, so the tree opens no wire the good boots did not. Belegstufe: DESK-BEWIESEN.

…z derselben Region. Per-Call-Kadenz ist WIDERLEGT. Gegenbeweis-Sequenz gefahren wie beauftragt (pack4->pack5, Deadman armiert, Live-Catch-Regel eingehalten). Ergebnis: FAIL, und der Fehlschlag ist der staerkste Beleg fuer den sgl-project#968-Umbau, den dieser Strang bisher hat. == WAS DER FIX NACHWEISLICH LEISTET == Pack 4 lief sauber durch: 13/15 (Referenz 11), DE-07 in 85,5 s -- also genau die Frage, an der die Gruppe zweimal starb. Die Sites sgl-project#1027 (memory_snapshot) und #1028a (memory_stats) tauchen in KEINEM Stack mehr auf. Die Deckel wirken, je Site. == WAS ER NICHT LEISTET == Bei RM-13 strandete es erneut: health 503, GPU 0% auf allen drei. Drei Stacks, ein Moment: PP0 idle all_reduce <- _update_uniform_pool_budget (scheduler.py:6186) PP1 idle all_reduce <- _update_uniform_pool_budget (scheduler.py:6186) PP2 active mem_get_info (torch/cuda/memory.py:842) free_bytes (corridor_guard.py:1086) spendable_bytes (corridor_admission.py:602) granted_width:635 <- _local_corridor_width_ceiling:8343 <- _update_uniform_pool_budget (scheduler.py:6121) `spendable_bytes:602` ist `free = float(guard.free_bytes())` -- die ERSTE Zeile der Funktion, VOR dem :620, den ich gedeckelt habe. == DIE DREI INSTANZEN == 1 memory_snapshot() corridor_admission.py:748 gedeckelt (sgl-project#1027) 2 memory_stats() corridor_admission.py:715 gedeckelt (sgl-project#1028) 3 mem_get_info() corridor_admission.py:602 UNGEDECKELT Alle drei in `spendable_bytes` oder dessen Callees, alle drei auf dem Pro-Runde-Pfad vor der Barriere. == VERDIKT: PER-CALL-KADENZ IST WIDERLEGT == Sie war ein vertretbarer erster Zug und wirkt je Site nachweislich. Aber die Region hat mehr Allocator-/Treiber-Aufrufe als Deckel, die Straggler-Rolle wandert auf den jeweils naechsten, und ein vierter Deckel befoerderte `available_size`/`get_num_allocatable_reqs` (:6174) -- den eine fruehere Probe bereits einmal als Straggler gefangen hatte. ICH BAUE KEINEN DRITTEN DECKEL. Das waere die dritte Kompensation fuer denselben strukturellen Fehler, und das Gesetz nennt genau das einen Loesch-Kandidaten-Befund statt eines Fix-Auftrags. == DIE WURZEL, jetzt empirisch belegt statt argumentiert == Nicht "ein langsamer Aufruf", sondern: JEDER RANG MUSS UNBESCHRAENKTE LOKALE ARBEIT ABSCHLIESSEN, BEVOR ER EINE BARRIERE ERREICHT, DIE JEDER RANG ERREICHEN MUSS. Drei Instanzen, drei verschiedene Aufrufe, dieselbe Region, dieselbe Signatur. == DESIGN-SKIZZE LIEGT VOR, BAU NICHT BEGONNEN == /spinning/gpu-arb/DESIGN_968_budget_verdict_to_pp0.md Kernpunkt daraus, gemessen: `_update_uniform_pool_budget` existiert in upstream `main` NULL mal, und `all_reduce` in upstream `scheduler.py` ebenfalls NULL mal -- gegen 8 im Fork. Upstreams Scheduler-Schleife macht GAR KEIN Kollektiv. Der ganze Mechanismus ist damit Fork-Zweitbuchhaltung ohne Upstream-Aequivalent, und die Beweislast liegt beim BEHALTEN. Aufwand ehrlich: L, mit einem XL-Risiko (Transport-Phase des sgl-project#791-Lap). Empfehlung: NICHT mit dem Voll-Schnitt beginnen, sondern mit der Slot-Klassifikation (Divergenz-Patch vs echtes physisches MIN) -- Desk- Arbeit, kein Boot, und sie bestimmt die Groesse alles Weiteren. BELEG-STUFE: BOOT-BEWIESEN ueber zwei volle Sequenzen, je eine Strandung, je drei Live-Stacks, je ein anderer Straggler-Aufruf. Serving wieder oben.

… it without admission BOOT-8 ROOT, MEASURED. The extent was derived inside `build_pp_admission_decision`, reached only from `_get_new_batch_prefill_raw`'s tail -- after an admission loop with eight `continue` branches and an `if len(can_run_list) == 0: return` above it. On the passes where a host hit was live, the holder was skipped at scheduler.py's `prefetch_pending` branch before the adder ever saw it, the list came back empty, the method returned, and NO decision was built: 09:13:52 PP0/PP1/PP2 sgl-project#788 PP-ADMISSION verdict=DECLINE n_reqs=0 queue=1 reason=loop_skips(prefetch_pending=1(first=f7f997c0...)) 09:13:54 PP0/PP1/PP2 sgl-project#968 LOAD-BACK DEFERRED rid=f7f997c0... (kv=4618) ... holds no PP0 extent for it yet The chain fed itself: hit present -> prefetch pending -> skipped -> empty list -> no decision -> no extent -> deferral. `sgl-project#1040 EXTENT STATE-ALIGN` read 0 in the whole log while its sibling emitter logged 9 times, which is what localised this. TWO HALVES, both mechanism-agnostic on purpose. 1. CHOICE MOVES TO THE WRITER. `stamp_state_aligned_extent` runs at the two sites that write `Req.host_hit_length` -- `Req.init_next_round_input` and `match_prefix_for_req` -- both unpacking one `match_prefix` result. `Req.__init__` only zeroes the field and `truncate_prefix_to` is dead (0 callers, 0 name reads), so a request CANNOT hold a hit without passing one of them. Every `can_run_list` filler (`add_one_req`, `add_chunked_req`, `add_one_req_ignore_eos`, the dllm pair, `_add_scheduled_req`) must match before it is executable, so the match dominates all six. The dominator argument is made over the WRITERS deliberately: the call graph cannot carry it -- `call_path add_chunked_req -> match_prefix_for_req` walks past 156 unresolved edges and finds nothing, which is a bounded negative and proves nothing. The row builder now READS the stamped field on both branches and derives nothing. 2. PUBLICATION DECOUPLES FROM ADMISSION. `_publish_pp_decision_1041` is the one publisher and BOTH exits of `_get_new_batch_prefill_raw` go through it, including the empty-list return that boot 8 died on. Everything the loop SAW and did not admit rides as an `admitted=False` FACT CARRIER. Collected at ONE site at the top of the loop, before any skip -- appending inside each skip branch would be the per-path retrofit this slice exists to avoid, and would have missed the second, still-unproven decline mechanism exactly as the first was missed. CARRIER SAFETY IS STRUCTURAL, NOT CAREFUL. `forwarded_schedule` already filters `e.admitted and not e.retracted`, so a carrier can never enter `_pp_scheduled_extents` and therefore never reaches the sgl-project#791 membership compare (scheduler.py:10508/:10522) -- the PPScheduleRefused-storm direction is closed at the source. `reconcile_pp_admission_decision` passes it through verbatim; `forwarded_last_chunk`/`forwarded_fill_carry` key on `fill_len`, which it does not carry; `apply_pp_load_back_row` stamps it by rid, which is its whole purpose, and an unheld rid is already a counted no-op there. STANDING INSTRUMENT. `sgl-project#1041 EXTENT POPULATION seen/published/delta` per pass. `delta != 0` means a request held an extent and no entry carried it, i.e. a bypass. The three legitimate differences are structural and produce no delta: no host hit stamps no extent, `pp_size<=1` never reaches the publisher, and an unheld rid is the receiver's counter. A future bypass is now loud instead of a silent zero that costs a boot to read. CHECK (speed mode: one matched check). devtools/check_1041_factcarrier.py, hermetic, CUDA-free. Named failure class: non-admitted entries reaching readers that iterate the decision, dangerous direction a PPScheduleRefused storm. The three pinned directions: (1) carrier passes without refusal and is absent from forwarded_schedule / last_chunk / effective while still delivering its extent through reconcile and apply; (2) a real membership violation STILL refuses -- an admitted rid the rank does not hold is still nameable as MISSING, pinned so the gate cannot go quiet; (3) unknown rid raises nothing and stamps nothing. 20/20 PASS. sgl-project#1040 check re-run green (no regression), py_compile + import smoke, ruff clean on the edited congruence module. cell_1039: adds `sgl-project#1041 EXTENT POPULATION` delta==0 and PPScheduleRefused==0 as acceptance criteria, both voiding the PASS branch.

…s; delete the load-back delivery chain OPTION A, in the hard form. Two edits that only work together: the memory-axis watchman moves onto the group availability floor FIRST, and only then does the delivery chain come out. sgl-project#1045 -- THE FLOOR IS PUBLISHED UNCONDITIONALLY ON A GROUP. It used to stay None whenever availability happened to be equal that iteration, and `unified_radix_cache`'s load-back fell into a RANK-LOCAL memory decision when it was None, guarded only by the delivery signal `pp_load_back_told`. A watchman on one axis was hanging off the delivery of another, and it stood down exactly when nothing looked wrong. PRICE VERIFIED BEFORE BUILDING, per the no-new-collective rule: at the live path (scheduler.py:6396) `min_avail` and `max_avail` are BOTH harvested from the same already-performed all-reduce (`t[0]`, `t[max_avail_at]`), so dropping the comparison removes work rather than adding it. The single-rank path now publishes the LOCAL value instead of None -- it IS the group min for a group of one -- which is what lets None become a construction violation everywhere downstream. Behaviourally identical on even pools: the published floor equals the live local value, and the sgl-project#694 ledger keeps it tracking through the iteration. FIX-6 IS NOW ONE BRANCH. `floor is None` at the load-back raises, naming rid, kv_tokens and local availability. A rank that cannot answer from a group number must not answer from a local one -- that is how ranks stop agreeing, and the 21:52:25 wedge is the measured price. sgl-project#1046 -- THE DELIVERY CHAIN IS DELETED. Consumption is local: `_pp_load_back_extent` returns this rank's own stamp, chosen at its own match by one shared expression over the same content. Uniformity comes from identical DERIVATION, not from one rank shipping a value. Deleted with it: `apply_pp_load_back_row` and both its call sites, PP0's self-row parking and the one-lap delay, `pp_load_back_told` and all four readers, the `sgl-project#1035` deferral ledger, the fact carriers, `holders_with_unspent_extent`, `_spend_for_entry`, the spend transition, the `sgl-project#1041` population counter and the `sgl-project#1044` ship/recv ledgers. EVERY READER WAS DECIDED, NOT SWEPT ALONG -- the enumeration is the point, because a reader falling silently into a rank-local branch is the class that found the edge in the first place: unified_radix_cache.py:2713 group-fact signal -> moved to the floor (sgl-project#1045) schedule_policy.py:107 delivery mechanics -> local stamp schedule_policy.py:2305 delivery gate -> deleted, no fact to await schedule_policy.py:2360 clamp gate -> KEPT, re-pointed. Its hazard (a GDN anchor adopted at `_applied` while the KV is cut) is created by the load-back itself and is independent of where the extent came from. KEPT AND UNTOUCHED: the admission row, its `prefix_len`/`extend_len` columns and the sgl-project#791 membership and ordering rules -- batch geometry, a different axis. The `load_back_len` FIELD stays on the wire for width tolerance and always travels None. sgl-project#1042's lifecycle contract stays, minus the spend transition that had no consumer left. CHECKS: check_1045_floor_watchman.py proves the guard CAN FIRE (a crash guard that cannot crash is worse than the branch it replaced), that a small floor is an ordinary refusal, that the rank-local fallback is DELETED rather than bypassed, and that publication is unconditional and counted. check_1042 updated: the row build must now LEAVE the field alone -- a build that still cleared it would resurrect the eraser through the back door. check_1040 green. The 1041, 1043 and 1044 checks are DELETED with their subjects; the cell's zombie test picks up their markers (ROW APPLIED, DEFERRED, EXTENT POPULATION, ROW SHIP) so a stale build cannot pass silently. py_compile + import smoke on all five modules; ruff clean. NOT YET BOOTED -- this commit is desk-proven only.

… the epoch, not to a rank's span Boot 30 died 1 minute into load. `#1059c` gave the row a sender for the first time, a told prefix of 12288 arrived, and it was applied to a request the cutover had just re-admitted -- whose pinned span was legitimately 0, because that same cutover dropped the tree one line earlier (`RESIDENTS RELEASED ... the prefix tree dropped returning 15943 row(s)`). `uniform_pass_geometry` raised `UniformWidthPromiseBroken` and crashed the group. The told value was not wrong. It was STALE ACROSS A CUTOVER, and the guard's premise -- "told <= pinned holds by construction" -- is true only WITHIN one cutover epoch, which nothing enforced. THE TRIGGER IS THE EPOCH AND NOT THE LOCAL SPAN, and that is the whole design. The obvious formulation -- "void when told exceeds MY pinned span" -- reads a RANK-LOCAL quantity and is therefore the 26th divergent input into the one replicated decision. The guard's own message already forbids exactly that: "compensating locally here would move the batch and reappear as sgl-project#631 on a peer." `PhaseFlipRuntime._epoch` advances once per COMPLETED cutover on every rank and the runtime's consensus reduction raises DESYNC if two ranks disagree, so `row.decided_epoch != current epoch` yields the SAME verdict everywhere without any rank consulting its own state. * PP0 stamps `decided_epoch` at PUBLICATION (the instant the row becomes a promise to the group). A relaying rank re-sends the raw row verbatim, so the epoch that travels the chain is PP0's. * The column rides at the END of the wire row under the same index-and-width discipline as sgl-project#987's pair and sgl-project#968's `load_back_len`. Per ENTRY, not per decision, because the outer payload is unpacked as a FIXED 2-arity tuple -- growing THAT is the fixed-arity crash the row's width tolerance exists to avoid. * The apply compares against the LIVE epoch through a registered read-through accessor, not a cached int: boot 30's apply ran INSIDE `_release_residents_for_cutover`, so a value refreshed per pass would have been one generation stale precisely where the staleness had to be seen. * A mismatch (or an unverifiable half-absent pair) routes into the contract's OWN already-proven no-adopt path -- boot 29 ran it for its whole life without a divergence. That is a way ONWARD, never the boot-15 per-rank refusal that re-fired 1448 times and wedged the ring. * `UniformWidthPromiseBroken` is NOT softened. It stays as the in-epoch invariant's backstop and is now unreachable: if it ever fires again the pin itself has failed, which is a group crash by `raenge-nie-uneins`, as written. SECOND HALF -- #1060b: THE TOLD COUNTER MOVES TO THE APPLY SITE. Boot 30 proved the row arrives by DYING on it, while `sgl-project#1058 TOLD-VS-LOCAL CENSUS` read `evaluated=13 absent=13` on PP0 and `evaluated=0` on the two ranks that crashed. The census sat at the CONSULT -- and PP0 makes the offer and never receives one. A fact's arrival must be counted where it takes EFFECT. `apply_reached` is the denominator; `adopted>0` is the only positive proof that a row travelled AND was applied. Emitted unconditionally at teardown and on the death path, like #1058b. MATCHED CHECK -- red-first at the boot-30 specimen plus the mandatory danger-direction mutant. The replay reproduces the raise with the log's own numbers (told=12288, pin=0), so the gate is proven to be what prevents it; the mutant asserts that the LOCAL-SPAN rule produces a SPLIT verdict across three ranks differing only in their span while the epoch rule produces one identical verdict -- if a future edit swaps the trigger, it goes red. Plus a wiring proof that drives `apply_uniform_pass_geometry_1059` itself with a live epoch source, because the pure-function tests prove the RULE and not that the apply consults it, which is the present-wired-never-populated shape that cost boot 29 a window. 11/11 green, plus 47/47 on the existing sgl-project#1059/sgl-project#791 suites (no regression) and 6/6 on sgl-project#1060. ruff clean on the added lines, py_compile green. Evidence: /spinning/evidence-665-f1/specimen_1060cens_boot30_uniformwidth_crash/

…SIDE one epoch Boot 31 ran sgl-project#1061's epoch gate and died anyway, at the same site with the same numbers (`told prefix 12288 exceeds this rank's pinned span 0`). The census I added in the same commit is what named the reason instead of leaving it to a guess: `refused_epoch=0`, `epoch_ok=1` on the raising rank. The gate WAS consulted, it AGREED, and the raise came after it. ROOT: `PhaseFlipRuntime._epoch` advances at cutover COMPLETION, while the destructive act -- `_release_residents_for_cutover` dropping the tree and re-admitting the residents -- runs BEFORE completion, inside the SAME epoch. So a row decided at epoch N is applied at epoch N during the cutover from N to N+1: same generation, gate vacuously passes, pin legitimately 0. The staleness window is not BETWEEN epochs, it is INSIDE one, and an epoch test cannot see it by construction. The epoch-keying design was answerable at the desk and I did not ask when the counter moves relative to the act it is meant to fence. THE RETRACT IS THE EVENT. `_969ad_note_retract` is the chokepoint every retraction passes and is called immediately before `_add_request_to_queue`, i.e. before the apply that raised. A told row describes a prefix the request HELD; a retraction is precisely the statement that it no longer holds it, so the row is not stale-by-suspicion there, it is stale by definition. STILL GROUP-UNIFORM, and measured rather than assumed: boot 31 logs `#969AD RETRACT site=readmit_seam_residents` exactly ONCE on each of PP0, PP1 and PP2 -- one per rank, same cutover, same rid. No rank consults its own span. Same shape as sgl-project#946's mark, which "invalidates ITSELF on a retract or cutover". The epoch gate STAYS. It is not wrong, it is not sufficient: it fences the cross-cutover case that outlives a completed flip, and the retract clear fences the in-cutover case. Both refuse into the contract's own no-adopt path, which boot 29 ran for its whole life without a divergence. INSTRUMENTED SO THE NEXT BOOT CANNOT REPEAT THE GUESS: the TOLD-APPLY census gains `retract_seen` and `told_cleared_at_retract`, so "the clear fired" is a printed number rather than an inference from an absent crash. MATCHED CHECK, red-first at boot 31: the test drives `_969ad_note_retract` itself, asserts the told row is gone and the counters moved, and asserts the apply that raised on boot 31 is now a no-fact pass. 12/12 on this file, 65/65 across sgl-project#1059/sgl-project#1060/sgl-project#1061/sgl-project#791. ruff clean on added lines, py_compile green. Evidence: /spinning/evidence-665-f1/boot_855_1061epoch_0840f82601_0831_182301.log

…ompensation layer, and its bugs were not the finding Coordinator order (2), after boots 30 and 31 killed the same mechanism twice at the same site with the same numbers (`told prefix 12288 exceeds this rank's pinned span 0`). Upstream-minimal law: a defect found in a compensation layer is a DELETION candidate, never a repair order for that layer. Two repairs were already spent on it -- sgl-project#1061's epoch gate and #1061b's retract clear -- and the second only existed because the first was built on a clock whose tick was never checked against the act it fenced. DELETED: `pp_uniform_width.py` whole (`uniform_pass_geometry`, `UniformWidthPromiseBroken`, `report_local_coverage`, `min_told`, `PassGeometry`, `epoch_admits_row`, `set_epoch_source`, `current_epoch`); `Req.apply_uniform_pass_geometry_1059` and its call site; the `_1059_told_prefix/_extend/_epoch` stamps; `PPAdmissionEntry.decided_epoch` and its wire column; the publication-time epoch stamp; the epoch-source registration; the told clearing at the retract chokepoint; the `SGLANG_PP_UNIFORM_WIDTH` gate; three test files. WHAT DELIBERATELY STAYED, and each for a named reason: * THE EVICTION FLOOR in `pp_stamp_observed_coverage`. `report_local_coverage` was `max(0, int(x))`, but the same line ALSO raises `cache_protected_len`, which `mem_cache/common.py:82` honours as an eviction floor. Deleting a promise must not silently lower a protection, so the arithmetic is inlined and the floor is byte-identical. Only the promise semantics die. * THE CARRIER COUNTER, renamed to what survives. The #1060b ledger counted the told row at the apply site; that apply is gone, but the question underneath it is not. Boot 29 shipped a carrier whose send gate had been dead since sgl-project#1046 and reported `sgl-project#631=0` while nothing executed -- it cost a window and a census to notice. `sgl-project#1064 CARRIER CENSUS` counts `rows_received` on every rank, so that regression can never be silent again. PP0 never receives, so only a downstream zero is a finding, and the line says so. * `sgl-project#1060`/`sgl-project#1063` untouched: they measure, they never gated anything. WHAT THIS RETURNS TO: the pre-sgl-project#1059 behaviour, in which every rank derives its geometry from its own match. That is not a guess about safety -- boot 29 ran it for its ENTIRE life (`evaluated=87 absent=87`, the row never arrived) across 51 flips with no `sgl-project#631`, no divergence death and no crash. The two boots that DID die at this seam are the two that ran the promise layer. c1 (downstream takes the geometry as a VERDICT) is NOT in this commit, and the reason is a desk finding that inverts its safety argument: `truncate_prefix_to` is MIN, NEVER ASSIGN (schedule_batch.py, sgl-project#930/sgl-project#958) -- so "verbatim" cannot be expressed through the one helper both admission sites use, and a rank holding less than it was told silently keeps its own value, which is the divergence the verbatim form exists to remove. Reported to the coordinator with the file:line rather than worked around. 38/38 green across sgl-project#791/sgl-project#1060/sgl-project#1063/sgl-project#677, codec round-trip verified after the column removal, import smoke of every touched module, ruff clean, py_compile green. Order: coordinator (2). Evidence: boots 30/31, specimens specimen_1060cens_boot30_uniformwidth_crash/ and specimen_1061b_boot32_int8_triton_wedge/

…ot end a pass PP0 launched `_pp_void_retracted_pass` let ONE downstream rank decide that the GROUP's pass ran nowhere. Its own docstring named the return trip that made that safe -- "the void output carries the observed local match home, and PP0's guard learns it as a floor" -- and sgl-project#969 CUT V had already deleted that emitter (`_PP_VOID_OUTPUT_KEY`: zero originating senders at ca0ee3a). The verdict therefore travelled downstream only. PP0's `mbs[slot]` stayed set while the last rank's did not, so no output was ever sent and PP0 blocked in `_do_recv` until the deadman -- the exact invariant `_do_recv`'s own comment relies on ("sender and receiver ask one question of one batch"). Measured, twice, and the second is on the stall second itself: 1068cap 07:34:02-09 sgl-project#797 void on rank 1 ONLY (no void/retract line on PP0 or PP2); PP0 parked, PP2 spinning. 1069cohort 08:00:54/55 sgl-project#791 unhonourable on PP1, told=12493 local=8397 then told=13399 local=12493 -- `local` exactly one pass behind `told`. After 08:00:55 only ranks 1 and 2 emit at all (slot_occupant / output_fill / width_agreement run to the takedown at 08:13:32, 4177 and 7973 lines; rank 0 emits nothing). That last measurement also refutes the occupant-sleep node as the halting member: ranks 1 and 2 are alive and turning; the `sgl-project#1000 SLOT-OCCUPANT reasons={'no-statement'}` spin is an INERT probe whose carrier sgl-project#1015 EDIT-F made permanently None. The only halting member is PP0's unbounded output receive. Repairing the return trip would repair a compensation layer for a rank-local verdict, which is the arc the sgl-project#968 order forbids continuing; under upstream-minimal the repair carries the burden of proof and the deletion does not. So the verdict is deleted and the disagreement is DETECTED instead (RAENGE-NIE-UNEINS: a detected divergence stops the group, never a compensating wait): * `_pp_assert_told_honourable` replaces it -- an unhonourable told names rank, slot, rid, told and local and raises. No clamp: clamping to this rank's own local match is rank-local geometry, i.e. sgl-project#631. * The chain-receive throttle arm gets a horizon (SGLANG_PP_OCCUPANT_HORIZON_S, default 90 s, 0 disables) and a named stop. Taking the arm is legitimate and frequent; outliving it never is. * `PpChainReceiver.recv` is bounded the way its sibling `consume_up_to` already was (runaway guard + reported counter), and the launcher now sets SGLANG_PP_CHAIN_RECV_STALL_S=60 -- the sgl-project#824 mechanism has existed since 2026-08-24 and shipped disabled by default, which is why it never fired in either stall. (A-i) is WITHDRAWN rather than built: its counter proof is blind at that site (the rendezvous bumps `sent` only on recv entry) and the DEFER one-shot is itself compensation for the rank-local verdict this commit deletes, so its trigger is removed at the source. Its two red-first tests are replaced by zombie tests for the deletion. Desk evidence: hermetic import + AST (deleted verdict absent, watchman and horizon present, recv bounded); test_968_deletion_falsifiers 24 passed, 1 failed -- test_C_the_void_relay_is_wired_or_deleted_but_never_half_built, which stays red until the relay SYMBOLS are swept out too. The relay is already unreachable at runtime (no rank originates a void any more, and `_pp_absorb_void_output` has no caller in production), so that sweep is a dead-code deletion scheduled beside this commit, not a runtime dependency of it. Belegstufe: DESK-BEWIESEN. Boot pending.

…line positions `local < told` was never a shortfall on this rig. It is the pipeline stagger, and the comparison that read it as a defect had its two operands at different pipeline positions and different times. MEASURED, boot_855_1071cut, 12:09:58, one minute after READY, on the FIRST chunked prefill of a FRESH boot against an EMPTY store (the launcher created /tmp/hicache_855_1071cut that same minute): PP1 rid=0d43edbc told=8192 local=4096 Nothing was in the store, so nothing could be reused and no rank could be SHORT of anything -- a cache shortfall is constructively impossible in that state. `told` is chunk 3's `extend_range.start`; `local` is chunk 1's `extend_range.end`; the gap is exactly one --chunked-prefill-size. The check was reading PP0's forward-most plan against a downstream rank's rear-most completed extent. This also retracts this strand's reading of the 1068cap and 1069cohort events as "store skew" / "a moving told" (told=12493 local=8397, told=13399 local=12493): the same structural false positive later in the run, then AMPLIFIED by the void it triggered -- `_pp_void_own_batch` restored `chunked_req` to its pre-admission value, freezing `local` while PP0's `told` ran on. Cause and amplifier were one mechanism. DELETED (each verified as an absent BINDING by AST/hasattr, never by grepping prose -- the first attempt at this check matched the function's own docstring and had to be replaced): * both retraction branches of `reconcile_pp_admission_decision`. The sgl-project#944 lookup-miss branch had to go WITH the sgl-project#791 one: its own comment rests its safety on the sgl-project#797 void being group-uniform, sgl-project#1071 deleted that void, and a retraction with no void left no longer stops the group -- it drops the rid on ONE rank and the pass runs with divergent membership, i.e. sgl-project#631. sgl-project#1071 had turned a covered branch into an uncovered one. * `_pp_assert_told_honourable`, sgl-project#1071's own watchpost. It fired once, named its own premise, and goes with the comparison it guarded; a watchpost over a deleted check is second bookkeeping of its own. * `pp_pass_should_void`, `pp_proxy_pass_retraction_reason`, `_pp_pass_retraction_reason` (no callers), the `entries_retracted_by_rank` import. * the #995c proxy-width probe (141 lines) and its #995e/#995f census (34). P2 measured why: it skipped 100% of its population on exactly the downstream ranks where sgl-project#791 saw anything -- 1069cohort PP1 evaluated=54 agree=0 disagree=0 skipped=54, PP2 52/0/0/52; 1071cut 3/0/0/3 and 3/0/0/3; reason always `no_local_input_ids`, because at proxy-receive time this rank's `input_ids` do not exist yet. A counter printing `disagree=0` while never comparing reads as "checked, all good" on the next pass. sgl-project#998 is rank-local by construction and cannot see rank against rank. NOTHING IS LOST, and this is the load-bearing half. The same length comparison already exists where its operands are finally comparable and it CRASHES rather than counting: model_runner.py:4233-4236, the one funnel every PP stage's forward passes through, compares received hidden-states rows against `forward_batch.input_ids.shape[0]` after `input_ids` is materialised and raises with a sender stamp that separates a pairing error from a payload error. sgl-project#791, #995c and that gate are the same check at three different moments; this deletes the two that cannot fire and keeps the one that executes. Ranks are held in agreement by sgl-project#631 ROW AUTHORITY (PP0 geometry bulletin, 642b99c), not by a downstream veto. `pp_rehome_displaced_chunked_req` deliberately STAYS (live caller at scheduler_pp_mixin.py:9022). Evidence: matched AST/binding check green (no retraction minted, 7 symbols unbound, #995c counters gone from the recv path); test_968_deletion_falsifiers 25/25 including the void-relay census. The 5 collection errors in test/registered/unit/managers are PRE-EXISTING, proven by A/B in a fresh worktree at 3730c8e without these edits, not asserted. Belegstufe: DESK-BEWIESEN. Acceptance boot pending.

… group STOP, not a void Root (HANDOVER_1153_0902, PROVEN, pre-existing at the pin 228a66d): scheduler.py _pp_refuse_forwarded_schedule answered PPScheduleRefused ('sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE', PP1 reached 1 of the 2 rids PP0 named, batch_full_break) with a rank-local compensation: _pp_admission_pass_voided = True + emptied decision dicts -> mixin _pp_void_own_batch -> 'sgl-project#631 ROW-DELIVER BATCH NULLED slot=0 pass_voided=True' (boot_855_weg1b2 log 65000-65004) -> PP1 sent no proxy while PP0's slot stayed set -> nothing carried the void upstream (sgl-project#797 return trip: zero call sites since CUT V; sgl-project#1072 deleted the void relay) -> PP0's blocking _do_recv consumed PP2's NEXT output under this slot's label (log 65119) -> one output ahead for the rest of the boot -> the pp_to_tp arm at 21:42:05 turned the debt into an unproducible output -> sgl-project#980 ObjectRecvStalled 60 s / sgl-project#1071 PpChainRecvStalled 90 s. Same form sgl-project#1071 (169f53c) deleted for _pp_void_retracted_pass; this was the second writer of the same flag. F1 ROOT: the compensation is deleted, the detector is kept, the refusal is a group STOP. scheduler.py get_new_batch_prefill's except now does `raise self._pp_forwarded_schedule_stop(refusal) from refusal`; _pp_refuse_forwarded_schedule (flag write, dict emptying, sgl-project#971 re-home) is removed; the new _pp_forwarded_schedule_stop logs the kept 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank N' line and returns a RuntimeError formatted by pp_admission_congruence .forwarded_schedule_stop_message: 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP rank={rank} slot={slot} told=[{told}] reached=[{reached}] census={census} local={local} limiter={limiter} running_bs={running_bs} parked={parked} r2t_avail={r2t_avail} headroom={headroom} group_limit={group_limit} batch_full_setter={batch_full_setter} batch_full_at_loop_entry={batch_full_at_loop_entry}: {refusal}' Every probe is guarded (n/a on an unreadable value); the reached rids are recorded after the admission loop before any of the three raises; the batch_is_full setter site is recorded at each writer of the pass (no_allocatable_reqs_gate, count_arm, disagg_prefill_r2t_avail, add_one_req_NO_TOKEN). Group stop mechanism (existing, no new collective): the RuntimeError leaves run_event_loop; run_scheduler_process (scheduler.py 'except Exception') logs 'Scheduler hit an exception' and parent_process.send_signal(SIGQUIT) (+ killpg/kill_process_tree under SGLANG_KILLPG_ON_SCHEDULER_EXCEPTION); peers end in the bounded sgl-project#980 / sgl-project#1071 receives or the barlink dead-peer probe. F2 TRIGGER CLASS (PP0 order): on a forwarded schedule the follower's rank-local seat-count veto is not a verdict. pp_admission_congruence .rank_local_count_veto_applies(scheduled_extents) is True on PP0 and on every non-PP boot (unchanged expression) and False on a rank > 0 executing PP0's decision; the sgl-project#823 count arm, its batch_full_break, and the three pre-loop count gates of the same arithmetic (batch_full_or_empty_queue's batch_is_full half, min_free_slots_delay, no_allocatable_reqs) are gated on it. The physical allocator still refuses (NO_TOKEN -> membership refusal -> STOP naming the numbers). F3 SIBLINGS: mixin _pp_void_pass_without_upstream_launch's writer of the flag is provably unreachable (pp_upstream_void_pending returns False on every path; its final statement is `return False`) -- left with the proof in a comment. The row-authority _row_skip_plan exit now nulls a slot through _pp_null_frameless_slot, which applies the same named sgl-project#1020 'VOID REFUSED ON A LAUNCHED SLOT' guard as the void path; the guard is factored into _pp_slot_holds_unconsumed_launch(mb_id, site) and used by both sites. Matched check (error class: a follower still ends a PP0-launched pass silently): grep -rn '_pp_admission_pass_voided = True' python/sglang/srt -> only scheduler_pp_mixin.py (the unreachable sgl-project#801 writer); chain scheduler.py raise PPScheduleRefused (:11633/:11646, :11606) -> except PPScheduleRefused (:9441) -> raise self._pp_forwarded_schedule_stop (:9472) -> RuntimeError (:9623); the flag's only readers are _event_loop_pp_body (:4519 -> _pp_void_own_batch) and the scheduler.py void guard (:8955), neither reachable from the refusal. Tests (hermetic, CUDA_VISIBLE_DEVICES=""): new test_pp_forwarded_refusal_stop_1153.py (T1 STOP form + no void, T2 the count-veto helper + source pin, T3 the sgl-project#1020 guard on the frameless null, + the mixin:9389 unreachability proof): 17 passed after; on the parent ca4c6b7 (git worktree) it is a collection error (new names). test_pp_refused_pass_keeps_continuation_971.py: 12 tests that pinned the old compensation INVERTED with the withdrawal named in each docstring (24 passed after; the inverted 12 are red on the parent: 16 failed / 8 passed / 1 error across both files). Mutants: M1 flag write restored in the STOP builder -> 2 T1 red; M2a helper returns True -> 1 T2 red; M2b count-arm gate dropped in the loop -> 1 T2 red; M3 guard dropped from _pp_null_frameless_slot -> 1 T3 red. Bounded suite (7 unit/managers files + test/registered/scheduler): before 103 failed / 527 passed / 1 skipped / 27 errors; after 103 failed / 544 passed / 1 skipped / 27 errors -- per-file identical except the new file (+17). ruff check clean on all touched files; ruff format --diff hunk count unchanged vs parent (15/14/3, pre-existing) and the two test files formatted. Evidence tier: DESK-PROVEN. Boot-3 acceptance: grep -F 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' in the same second as 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank', and zero 'ROW-DELIVER BATCH NULLED ... pass_voided=True' lines.

…rwarded schedules; STOP line names pp_max_mb and in-loop reached Four non-blocking reviewer items on fa14571 (sgl-project#1153, PASS/PASS), each a rank-disagreement generator or a diagnosability gap on the path Boot 3 exercises (weg1b2 pp_to_tp arm, follower executing PP0's schedule). 1. THE FOURTH SITE OF THE SAME COUNT ARITHMETIC IS GATED. scheduler.py `_get_new_batch_prefill_raw`: the `_maybe_yield_parked_carrier` gate (`get_num_allocatable_reqs(running_bs) <= 0 and chunked_req is None and not enable_priority_preemption`) ran ungated on a follower executing a FORWARDED schedule, and it is an ACTUATOR: it retracts a parked decode carrier (`_retract_decode_and_requeue`) on this rank's own seat count, a rank-local state change the peers do not make (RAENGE-NIE-UNEINS, construction half). Now `_count_veto and ...`, the same `rank_local_count_veto_applies(self._pp_scheduled_extents())` the sgl-project#1153 fix put on the three pre-loop count gates, the sgl-project#823 count arm and its batch_full_break. PP0 and every non-PP boot: unchanged expression. 2. STOP LINE COMPLETENESS. `get_num_allocatable_reqs` is min(pp_max_micro_batch_size, admission_limiter.current) - max(0, running_bs - parked); the STOP line printed local/limiter/running_bs/ parked but not the first min() term (=2 under the flip override on weg1b2, the adversarial reviewer's likeliest trigger). pp_admission_ congruence.FORWARDED_SCHEDULE_STOP_FORMAT gains `pp_max_mb={pp_max_mb}` after `limiter=`; `forwarded_schedule_stop_message` takes the kwarg; the scheduler builder probes `get_server_args().pp_max_micro_batch_size` (guarded, n/a on an unreadable value). The prefix 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' is byte-identical (the Boot-3 acceptance greps it). Final format: 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP rank={rank} slot={slot} told=[{told}] reached=[{reached}] census={census} local={local} limiter={limiter} pp_max_mb={pp_max_mb} running_bs={running_bs} parked={parked} r2t_avail={r2t_avail} headroom={headroom} group_limit={group_limit} batch_full_setter={batch_full_setter} batch_full_at_loop_entry={batch_full_at_loop_entry}: {refusal}' 3. THE LOAD-BEARING MEMBERSHIP LINE IS PINNED (reviewer mutant MC survived). With the count veto off, the post-loop `if missing:` is the only thing turning a follower's physical inability (add_one_req NO_TOKEN -> a told rid not reached) into a STOP. New T5 source pin: the membership block between `scheduled_extents = self._pp_scheduled_ extents()` and the `extra` check contains `if missing:` verbatim, no `_count_veto`, and the raise naming `missing rid(s)=`. Source pin, not a driven pass: nothing in this tree drives the ~700-line loop (same precedent as T2). No source change for this item. 4. reached=[] WART. `_pp_admission_reached_rids` was reset at pass entry and recorded only after the loop. New `Scheduler._pp_record_reached_ rids(can_run_list)`; called at the loop's in-loop `except PPScheduleRefused` (before `schedule_refusal = exc; break`) and at the existing post-loop site, so the STOP line names what the loop actually reached at the moment of a mid-loop raise. Note for the record: on this tree the add_one_req raise was already carried to the post-loop record (break -> alloc_group_end -> record -> raise); the only path that escapes both records is the pre-loop `adder.add_chunked_req` -> `_add_scheduled_req` raise (schedule_policy.py :1628), where can_run_list is genuinely empty, so reached=[] is truthful there. Matched check (error class: a count site left ungated on a follower): grep -n "_count_veto" python/sglang/srt/managers/scheduler.py lists the helper import (:227), the derivation (:10281), the three pre-loop gates (:10350 batch_full_or_empty_queue, :10363 min_free_slots_delay, :10413 no_allocatable_reqs), the parked-carrier yield (:10382), the sgl-project#823 count arm (:11076) and its batch_full_break (:11090). Tests (hermetic, CUDA_VISIBLE_DEVICES=""): test_pp_forwarded_refusal_stop_1153.py: red-first on the unedited source 7 failed / 15 passed (T1 pp_max_mb x3 incl. format string, T1 in-loop reached x2, T2 fourth-site pin, T4 follower-never-yields); after the edits 22 passed. T4 drives the REAL `_get_new_batch_prefill_raw` from its first line to the yield site on a SimpleNamespace stand-in (PP0 with 0 seats: yields once, then declines at no_allocatable_reqs_gate; follower on a non-empty told map: never yields, runs past both gates into a sentinel at policy.calc_priority). Both files: 46 passed (22 + test_pp_refused_pass_keeps_continuation_971.py 24). Mutants (each applied and reverted by string replace, sha1-verified): M1 drop the yield gate -> 2 red (T2 pin, T4 follower); M2 `if missing and _count_veto:` -> 1 red (T5); M3 drop the in-loop record -> 1 red (T1 source pin). ruff check parity with parent (pre-existing E402/E731/F811), no new findings; ruff format --diff hunk count unchanged vs parent (15/3) and the test file formatted (0 hunks). Evidence tier: DESK-PROVEN. Boot-3 acceptance unchanged: grep -F 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' beside 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank', zero 'ROW-DELIVER BATCH NULLED ... pass_voided=True' lines; the STOP line now carries pp_max_mb=.

…follower speaks before it can be stuck boot_855_weg1b5_cd5bb69607_0903_115008, rid 0c34259f, 11:55:13-11:55:30. All three ranks issued the same re-admission prefetch (gen 4, keys=13224, byte-identical FETCH CAP lines). PP0's completed in ~3 s, printed completed_synced=12288, and ADMITted prefix_lens=12288. PP1's and PP2's had not terminated. PP1 took the forwarded row into execute_scheduled_prefix and raised the designed group STOP 14 s later. The number PP0 admitted on was rank-local by construction: the MIN reduce behind "completed_synced" runs only under tp_world_size > 1, and this boot is --tp-size 1 --pp-size 3 -- attn_reduce_world=1 on 307/307 HICACHE-ROUND lines of the whole log. E1 managers/pp_prefetch_completion.py (new, pure, no I/O, no clock) plus a per-rid completion carrier on the sgl-project#791 ring lap, modelled exactly on sgl-project#968's parked-continuation table (stamp on the way up, absorb at PP0). No new all_reduce, no new message, no new blocking point on the admission path -- the PP0-authority order records a collective there as fatal. PP0 admits only when the group floor covers what it wants to schedule; a peer that is silent or still running contributes NO NUMBER (INCOMPLETE, never a zero), otherwise the rid is DEFERRED by name while other work proceeds. When the same length-priced bound the follower's own wait uses expires, PP0 CLAMPS to the group floor through the existing sgl-project#1059 note_observed_coverage channel instead of over-telling. want <= 0 (no store span) admits unconditionally, so a boot with no storage hit takes exactly the pre-sgl-project#1175 path. Kill switch SGLANG_PP_GROUP_COMPLETION=0 restores the old admission for an A/B. completed_synced now prints synced=yes|no and attn_reduce_world=N beside itself, so the field can no longer claim a sync it did not do. E2 pp_admission_congruence.execute_scheduled_prefix now DRIVES the writer it used to suspend: after each poll it calls check_prefetch_progress when the tree reports that path collective-free (world 1 = every all_reduce skipped by construction), and the expiry text carries prefetch_driven_in_loop=<bool> so an expiry can never be read as "the bytes were not there" when nothing drove the sole writer. Bounds unchanged. The sgl-project#1157 reaper's comment now states where it structurally cannot fire (it reports terminations; a prefetch that never terminates produces no line, so 0 REAPED is not evidence that nothing was reaped). E3 Every follower prints one bounded "sgl-project#1175 PREFIX-EXEC UNDER-COVERAGE" line at ENTRY, before any bound can expire, naming rid/local/scheduled/ deficit/bound. PP2 printed nothing at all about this rid and its silence was indistinguishable from health. The line also states honestly that a rank which never RECEIVES a decision row never calls this function and therefore still emits nothing -- that case is what E1+E2 remove. E4 (a) the healthy local == scheduled path no longer returns silently: one bounded, counted "sgl-project#968 PREFIX-EXEC no-op" line, so "0 materialised lines on the whole boot" reads as "never needed" rather than "never reached" (INDIKATOR-GESETZ). (b) the sgl-project#939 census line now names its population (population=retract_closure_only) -- SEAM_READMIT_ATTR is stamped only by the retract closure, so the two 13225-token queue-occupant re-admissions of this boot produced not one census line. The field is placed BEFORE fence_proceeds so sgl-project#1068's tail pin still holds. EVIDENCE (DESK-PROVEN; no boot, no GPU) red-first, parent cd5bb69 + only the three new test files: 16 failed, 3 passed, 1 collection error (pp_prefetch_completion absent) fixed tree: 36 passed (25 managers + 11 mem_cache) mutants: 12/12 killed, >= 3 per decision (E1 x4, E2 x3, E3 x3, E4 x2) matched checks re-run on the fixed tree: test_pp_admission_congruence_791, test_968_starvation_umbau, test_968_deletion_falsifiers, test_producer_phase_census_631, test_double_prefill_census_fence_1068, test_producer_phase_census_wiring_1061 -> 114 passed, 1 failed (test_the_module_imports_no_torch_at_all -- identical failure on the parent, "import torch" count 1 on both trees: pre-existing) bounded suite once, -n 8, hermetic, managers + touched mem_cache files: parent 262 failed / 5435 passed / 12 errors fixed 286 failed / 5472 passed / 12 errors name-level comm: 35 "regressions" and 11 "cures", ALL of them inside test_unified_radix_cache_unittest.py; 0 outside it, 0 involving the new files. That module run SOLO and SERIAL is 642 passed / 860 skipped / 0 failed on BOTH trees -- the delta is the known -n 8 crowding family (sgl-project#862/sgl-project#899), not this diff. ruff --select=F401,F821,UP037: 2 errors on both trees, the same two pre-existing ones; black/isort per file identical on both trees for all five touched production files, all four new files clean.

…ves, presence is the sum, and a follower never splits the group silently Closes the three blocking findings of the be3ec17 review. B1 (LAW WEAKENING). be3ec17 derived the demand from req._prefetch_span_tokens, a number stamped once at registration (scheduler.py:5355) that nothing in the tree ever clears. Req.truncate_prefix_to (schedule_batch.py:2357, called from scheduler.py:11495 and :10975 under the sgl-project#791/sgl-project#930 PP-told rule) empties prefix_indices AND host_hit_length and leaves that span standing, and the premise path re-reads the record non-destructively on every pass (phase_purity.py:1631). Worked breaking input: stamp 80009, registration match 79000 -> span 1008; the prefetch delivers 1008; PP0 then tells told=0 and the device-resident prefix is gone; the next pass read demand=min(80009,1008)=1008 against presence 1008 and called it a "hit", licensing a P=0 re-prefill of 79001 tokens on a stamp -- the exact kein-doppel-prefill (sgl-project#939) violation this witness exists to prevent. The parent RAISED on that record; the reviewed commit did not. FIX (the preferred remedy, upstream-minimal): the demand is max(0, stamp - resident) with resident = len(req.prefix_indices) + req.host_hit_length read AT WITNESS TIME -- the same expression the registration site computes. A truncation restores the full demand by construction: no new state, no clearer, no lifecycle to keep in step. phase_purity no longer reads _prefetch_span_tokens at all (AST-pinned); the field keeps its one legitimate consumer in scheduler.py. B2 (UNCAUGHT MUTANT). presence = matched + loaded is the single arithmetic the whole sgl-project#1176 fix turns on, and no case separated it from max(matched, loaded): every one had matched==0, or loaded==0, or a split whose larger half still landed inside the allowance. Shipped code was correct; the regression pin was missing. Added the separating input (stamp 20000, matched 9000, loaded 9000, allowance 4096: sum -> shortfall 2000 = hit, max -> shortfall 11000 = raise) on the property, on the witness, on the admission site, and on the duck-typed getattr default. B3 (A LOUD STOP TRADED FOR A SILENT RANK DIVERGENCE). be3ec17 made a follower stop raising -- correct under sgl-project#968/#969Z -- but left the returned state outside the tuple seam_transport_premise_holds accepts, so the follower still WITHHELD the seam premise. That boolean gates the whole prefill-batch build (phase_purity.py:1009 -> scheduler.py:8926 new_batch = None) and store_witness reads RANK-LOCAL records: under --tp-size 1 --pp-size 3 the packed MIN all_reduce (unified_radix_cache.py:3879-3907) is never taken. Measured input from weg1b6: stamp 6008, PP0 matched=5966/loaded=42 -> hit -> premise True; PP1 reaped at matched=100/loaded=0 -> contradiction -> premise False. Mismatched collectives -- a silent split, which raenge-nie-uneins forbids more strongly than the loud stop it replaced. FIX = option (ii) of the three offered: the follower COUNTS the candidate (taking PP0's standing verdict, #969Z) and REPORTS the contradiction on the EXISTING follower -> PP0 completion carrier (sgl-project#1175), where PP0 raises once, loudly, naming the peer. Option (i) -- making the whole premise PP0-anchored -- was not built: there is no PP0 -> follower channel for this verdict at that point in the pass, so it would have needed a new message or a collective on the admission path, and a collective there is a recorded fatal under the PP0-authority order. Option (iii) (restore the loud follower raise) was not built either: it is the very trade the review rejects, and it kills the follower while its peers admit. pp_prefetch_completion gains a CONTRADICTION sentinel (never min()'d into a floor, same rule as PENDING) plus the pure helper peers_reporting_contradiction, read in _admit_under_group_completion BEFORE the `want <= 0` early-out -- the dangerous combination is exactly "PP0 fetched nothing while a peer measured a contradiction". Evidence. Red-first on the parent be3ec17 in a scratch worktree with ONLY the two test files copied in (source untouched, git status shows two test paths): 16 failed / 52 passed -- 10 on B1 (classes G/K + the message pins in C and test_1157) and 6 on B3 (class M). B2 is green on the parent by construction (shipped code correct) and is proven by mutant M3/M4 instead. Fixed tree: 68 passed / 3 subtests. Neighbour suites (869c, w30, 968, 861j, 1173, 1068, 1060) 198 passed with 2 failures in test_tp_decode_formation_861j that reproduce identically on the parent worktree (pre-existing, not this diff). ruff parity per touched file against the parent: identical (0 findings each). Hermetic import smoke of all four edited modules passes.

…ck for all three exits ROOT, measured on boot weg2sn5s @ fd92440 under a FLEET-SHAPE load (shared ~16k anchored prefix + growing tails), all three D ranks: 22:27:51 #924D station=alloc_cow rid=664901103e55 mamba_slot=[7] site=finalize_match_result 22:27:51 WEG2 X-GATE rid=664901103e55 uncached=17236 X=8742 verdict=W31 22:27:51 W50 Weg2TpPrefillExceeded rid=664901103e55 uncached=17236 X=8742 on_idle [mamba] total=30, available=4, evictable=25, leaked_mamba_pages={7} mamba_leak_owners=[slot=7 slot_used=True last_event=ALLOC@seq6158 releaser=none-recorded] A prefix match drew a COW resume slot SPECULATIVELY (`MambaComponent.finalize_match_result`, stamping `mamba_slot_acquired_this_admission`). The X gate then refused the request by name and `_weg2_answer_x_refusals` returned it to the front WITHOUT the slot. That request has exactly ONE station line for its whole life: it never reaches `alloc` or `cache_finished_req`, so the refusal exit is the ONLY owner of that give-back. Zero `sgl-project#991` lines in the entire log confirms none ran. WHY IT READ AS INTERMITTENT, and why four clean 41k prompts proved nothing: BOTH conditions must hold -- `uncached > X` (long prompts only) AND a prefix match carrying mamba state (so the COW acquire happens at all). Every earlier repro put a unique nonce at the FRONT of the prompt to defeat prefix caching, which satisfies the first condition and structurally excludes the second. The fleet's real shape -- one shared system+tools prefix, growing tails -- leaked within two passes. THE CLASS: THREE admission-refusal exits, the give-back open-coded in two and absent from the third. * `scheduler.py` admission revert (sgl-project#991) -- FORK-OWN * `schedule_policy.py` PPScheduleRefused (sgl-project#791) -- FORK-OWN * `scheduler.py` `_weg2_answer_x_refusals` (sgl-project#1290/law 4, W31/W50) -- FORK-OWN All three are fork-own Weg-2 admission plumbing; the COW acquire they must compensate is upstream (`e0b692600f`, PR sgl-project#27118). That is the upstream-minimal statistic again: the defect sits in fork-own compensation around an upstream mechanism, not in the mechanism. FIX: lift the two open-coded bodies into ONE `release_admission_acquired_mamba_slot` (`mem_cache/common.py`) and call it from all three exits with identical guards (`mamba_slot_acquired_this_admission` and not `req.session`) -- a third copy would have been the second-bookkeeping answer. It also clears the match's other per-admission carry-overs (`mamba_cow_src_index`, `mamba_needs_clear`, `mamba_loadback_anchor_adopted`) so a refused request cannot carry a resume anchor into its next admission, and emits `#924D station=give_back site=<exit>` so the exit is visible in the trail. The wider partition on that exit needs nothing further: the request never allocated KV rows or a req_pool row (no `station=alloc`), the match's `inc_lock_ref` pairs are closed in-block at `mamba_component.py:488/:499`, and the exit already calls `release_aborted_request` / `terminate_prefetch`. Tests: `test_mamba_admission_giveback_q0.py`, 9 cases asserting the FULL partition after every station -- `expected == free U tree U live` AND all three pairwise intersections empty, i.e. the orphan direction AND the aliasing direction that the earlier suite could not see. RED-first against fd92440: 5/5 red including the specimen simulated on the real allocator (slot orphaned after the W50 exit); green on the fix. 3 mutants killed with an unmutated control: acquired-guard removed (frees batch-owned slots, sgl-project#1051's direction), session-guard removed, give-back made a no-op (the pre-fix exit). ruff: 127 findings across the three touched files, identical on the baseline tree.

fix echo + logprog on list prompts

f1c50be

Ying1123 merged commit 8d908a9 into main Jul 29, 2024

Ying1123 deleted the fix-multi-logprob branch July 29, 2024 00:09

Ying1123 mentioned this pull request Jul 29, 2024

[Bug] Prompt logprobs support #782

Closed

3 tasks

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Fix echo + lobprob for OpenAI API when the prompt is a list (sgl-proj…

f4b1a8c

…ect#791)

yuejiat0416 mentioned this pull request Sep 10, 2026

[NPU] Support GLM-5.2 DSpark loading #38860

Open

5 tasks

Navigation Menu

Uh oh!

Fix echo + lobprob for OpenAI API when the prompt is a list - #791

Fix echo + lobprob for OpenAI API when the prompt is a list#791
Ying1123 merged 1 commit into
mainfrom
fix-multi-logprob

Ying1123 commented Jul 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Ying1123 commented Jul 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant