docs: update README by zhyncs · Pull Request #788 · sgl-project/sglang

zhyncs · 2024-07-28T12:24:17Z

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

* Update test_utils.py * Update test_utils.py * Update test_ascend_debug_tensor_input_file.py * Update run_suite.py * Update run_suite.py * Change lint job to run on linux-amd64-cpu-1 * Update lint.yml * Refactor lint workflow for improved configuration * Update daily-build-test-npu-innersource.yml * Fix syntax error in merged PR benchmark workflow * Update pr-test-npu-debug.yml * Clean up blank lines in activation.py Removed unnecessary blank lines in activation.py. * Fix indentation in attention layer methods * Update io_struct.py * Update dbrx.py * Update phi.py * Update test_utils.py * Clean up unused imports in test_vlm_utils.py Removed unused imports from the test file. * Add blank line before main execution block * Add model path to TestLing class * Update test_ascend_llm_models_phi_4_multimodal.py * Update test_ascend_llm_models_smollm_1_7b.py * Update test_ascend_sleep_on_idle.py * Update test_ascend_llm_models_Qwen3_32B.py * Update test_serving_chat.py * Update run_suite.py * Update test_server_args.py * Update daily-build-test-npu-innersource.yml * Update pr-test-npu-debug.yml * Update activation.py * Update torch_native_backend.py * Update io_struct.py * Update phi.py * Update test_utils.py * Update test_ascend_llm_models_Stablelm_2_1_6b.py * Update test_ascend_llm_models_ling.py * Update test_ascend_llm_models_smollm_1_7b.py * Update merged-pr-benchmark.yml * Update run_suite.py * Update daily-build-test-npu-innersource.yml * Create pr-test-npu-debug-1.yml * Update run_suite.py * Update pr-test-npu-debug-1.yml * Update pr-test-npu-debug-1.yml * Create test_ascend_hicache_mha.py * Create test_ascend_sampling_backend.py * Update run_suite.py * Update pr-test-npu-debug-1.yml * Update run_suite.py * Update and rename test_ascend_deepep.py to test_ascend_deepep_mtp.py * Update test_ascend_deepep_mtp.py * Update test_ascend_deepep_mtp.py * Update test_ascend_mla_w8a8int8.py * Update test_ascend_tp4_bf16.py * Update daily-build-test-npu-innersource.yml * Create daily-build-test-npu-opensource.yml * Update daily-build-test-npu-opensource.yml * Update test_ascend_deepep_mtp.py

update kimi and 35b testcases

…ion, not before the exchange A boot reached serving, one admissible queued request was never served, and the process died. The wedge detector reported "1 queued, 0 running, NO first token for 340-370s and no prefill chunk either" and named its own gap in the same line -- "no phase-policy corroboration seen -- the wedge class is broader than that path". It was right: this is not an admission defect at all. EVIDENCE OF RECORD. py-spy on the specimen (WEDGE_788_specimen.log:732/915/1083) catches all three ranks in a circular wait: PP0, PP1 gloo waitSend _pp_commit_comm_work (scheduler_pp_mixin.py:2465) from _pp_forward_and_process_input_requests:1007 PP2 gloo waitRecv _pp_recv_typed_dict:2709 from _pp_recv_proxy_tensors:2803 The request chain was flushed at the TOP of the pass, before the rank had posted anything else it owed its peers that iteration -- the proxy tensor-dict send and the output-ring send both come later in _event_loop_pp_body. A downstream whose progress needs one of those was therefore waiting on a rank that was itself waiting on that downstream. The chain recv is posted only at the top of a pass (:420) and nowhere else in the loop body, so a rank parked in the proxy receive cannot drain the message its upstream is blocked flushing. THE FIX IS ONE ADDITION. _pp_forward_and_process_input_requests is byte-identical to before: it still commits send_req_work, then posts the async forward, then runs process_input_requests. What is new is _pp_commit_pending_req_work, called once per iteration from _event_loop_pp_body after the proxy isend and the output commit and BEFORE the phase-flip round hook. By the time that loop returns to the top-of-pass commit, the handle has already been waited on and cleared, so the old call site is a no-op there -- and the two disaggregation loops (:618, :765), which do not call the new method, keep their previous behaviour exactly. Deleting rather than moving the old call site is deliberate on both counts: it keeps the sgl-project#633 ordering contract literally intact (commit, then forward, then process, with send_req_work holding this pass's handle on return -- pinned by test_scheduler_pp_request_order_633), and it avoids a second staging slot. An earlier shape of this fix did stage the send separately; it broke that contract and left the two disagg loops overwriting a live P2PWork handle every pass. Precedent: a7ff250 "[sgl-project#753] Flush the output sends after the exchange, not before it" made the same move for the OUTPUT channel. This is that fix for the REQUEST channel, ungated -- the hazard is not gapped-specific. NEGATIVE FINDING, RECORDED RATHER THAN BURIED. A deterministic 3-process gloo reproduction of the specimen deadlock was attempted and NOT achieved under a faithful model of the loop. Two candidate mechanisms were tested and refuted: - Eager-send asymmetry. Hypothesis: an empty request list puts one 8-byte tensor on the wire and completes eagerly, so only a large first request can block. MEASURED FALSE: a 2-process probe with a 2.0s-delayed matching recv blocked for the full delay at every payload size from 8 B to 1 MiB in this build. No eager completion at any size. - Schedule asymmetry. A linear chain+proxy relay that flushes the old handle before dispatching the new one is deadlock-free by induction (each rank's flush needs only its immediate downstream one pass behind), and dozens of real 3-process runs across depths and offset combinations all completed. So the reduced two-channel model cannot close a cycle. The production graph has an edge that model lacks: the output ring closes last-rank -> rank 0 (_pp_send_recv_and_preprocess_output_tensors:3038, last-rank branch :3012-3014, via next_first_rank_mb_id at :416/:608/:754). That, and real GPU compute skew, are the two named unconfirmed candidates for the missing ingredient. Neither is built into the test, and the gap is stated in its docstring rather than papered over. The mechanism's evidence of record therefore remains the py-spy trace above plus the sgl-project#753 precedent, and the survival boot is the integration proof. RECOVERY RUNG. The detector gains one bounded action: after a wedge stays continuously alarming past a threshold well above the report threshold (default 3x, env SGLANG_ADMISSION_WEDGE_RECOVERY_SECONDS; non-positive values fall back to the default rather than firing every poll), it makes ONE forced-admission attempt per episode through the existing corridor_admission actuator and logs it loudly either way. Its own docstring is explicit that it relieves VRAM pressure at the admission site and will NOT move a comms deadlock like this one, and that calling an actuator from the watchdog thread is a new cross-thread shape over CUDA allocator state. TESTS test_pp_chain_flush_deadlock_788.py (new; 3 real gloo processes, CPU only) - load-bearing: runtime ordering pin. The SHIPPED _pp_commit_pending_req_work is wrapped and the observed sequence asserted to be proxy_send -> chain_flush -> round_hook for every pass and every non-last rank, so the flush cannot drift back past the collective and cost it the "last blocking op of the iteration" property its own comment relies on. - can-fail: the same run with the flush relocated to the top records a different sequence, so the pin discriminates placement. - the honest negative case described above, in place of the deleted hang-repro. A test that is green or red for the wrong reason is worse than no test. test_pp_flip_leftover_proxy_757.py: harness-only repair, no assertion touched. It was silently DEAD on this branch -- its _GlooWire predated rank_in_group/world_size and the src positional recv_typed_tensor_dict now passes, so it errored 3/3 before reaching an assertion. Now 3 passed and a working regression check again. Verified green with this diff: test_pp_chain_flush_deadlock_788 and test_scheduler_pp_request_order_633 (8 passed), test_phase_policy (90 passed), test_pp_flip_leftover_proxy_757 (3 passed). PRE-EXISTING FAILURES, MEASURED NOT ASSUMED. The managers slice reads 43F/2707P here against 39F/2705P at a pristine detached ff5651a worktree. Every delta is accounted for: -3 (the 757 repair above now passes), +5 stub drift and +1 ordering-contract break, both introduced by the earlier staging-slot shape and both gone with it, +1 test_pp_drain_completeness_787 which is INTENTIONALLY RED here -- that file's fix lives on fix/787-drain-completeness and it flips green at the merge, which makes it a free integration can-fail. The residual 39 are the same interface-drift family (rank_in_group, _pp_gapped_wire) plus one stale constant, all present at base with no part of this diff. DEBTS sgl-project#789 the two disaggregation PP loops keep the top-of-pass flush and therefore still carry this hazard sgl-project#757-harness-drift repaired here; the same drift still blinds test_pp_proxy_stamp_631 and test_pp_slot_last_batch_631 sgl-project#753-stale-constant 10.923 != 11.923 in the gapped entry protocol test sgl-project#766-pointer-stale the register points sgl-project#766 at ARM_defaultfull7.log, which is a 26-line host-ledger snapshot with no Bar1CollectiveAborted and no proxy/drain lines

…roject#788 line Both defects killed proof boots on the full-feature shape. sgl-project#788 (7530d96) is the PP request-chain flush ordering; sgl-project#787 is the disarm-time drain's one-shot-snapshot race. They are independent and land here together so the survival boot exercises both. test_pp_drain_completeness_787's primary case was intentionally RED on this line before the merge -- the test arrived first and its fix lives on the merged branch -- so it flipping GREEN here is a free integration can-fail.

…ving hot paths An isolation boot wedged for 25+ minutes with 120 ADMISSION-WEDGE markers. py-spy (two dumps 25 min apart, byte-identical, so hard-stuck) put PP0's MainThread INSIDE logging.emit while PP1 and PP2 starved in pp_chain_receiver.recv for a chain send PP0 never reached. ROOT CAUSE. PP0 admitted the first radix-carrying request and reached the sgl-project#767 instrument in HybridReqToTokenPool.alloc, which logged req.mamba_pool_idx -- a 1-element CUDA tensor -- as a %s argument. The format call runs Tensor.__repr__ -> _tensor_str -> a D2H copy -> a stream synchronize, inside logging.emit, on the admission path. The device was occupied by a spinning ncclDevKernel_SendRecv, so the sync never returned. The instrument produced ZERO output on that boot: the record died mid-format, which is why the log looks like the branch never ran. THE TRAP, NAMED SO IT IS NOT RE-INTRODUCED. Every obvious way to print the value synchronizes: %s/str()/repr(), .item(), .cpu(), .tolist(), float(), int(), and f-string interpolation. A fix that swaps one for another only relocates the sync. sync_free_tensor_repr returns host-resident metadata instead -- shape, dtype, device, and id() to correlate one tensor handle across lines -- and passes non-tensors through unchanged, so a value that is sometimes a tensor and sometimes a plain int is safe either way. SWEPT THE FAMILY (sgl-project#695: expensive work inside logging arguments on serving paths), fixing six more sites of the identical shape: mamba_component.py sgl-project#767-TRACE prefix-match model_runner.py sgl-project#767-TRACE cow_and_clear SKIP and body, per extend forward pass -- hotter than the original site mamba_radix_cache.py cache_finished / cache_unfinished / match dflash_solo_pool.py:_reclaim THE ONE WITH NO GATE The dflash site matters most: every other hit sits behind an opt-in debug flag, that one runs unconditionally on decode-time draft-slot allocation under real load. It computed self._slot_epoch[victims].max().item() purely for a diagnostic string. Victims are drawn from the ascending argsort's low end, so their epoch is bounded above by self._epoch, which is already a host int -- and the message text is REWORDED to match what is now reported ("at round X ... untouched since before this round") rather than quietly printing a different number under the old wording. LISTED, DELIBERATELY NOT FIXED, with reasons rather than silence: phase_flip_output_trace.trace_round and phase_flip_resident_carry's cutover falsifier (both bounded on purpose, the second says so in its own comment), ngram_corpus.debug_result (reachable only from a __main__ demo), and dspark_planner._log_verify_lens_decision -- mechanically the same shape and genuinely hot when armed, but its whole purpose is the exact per-request values, so an identity stand-in would gut the tool rather than trim it. It wants rate-limiting, not this treatment, and is left as a named follow-up. TEST. test_admission_log_no_device_sync_790.py, hermetic and CPU-only. A tripwire monkeypatches Tensor.__repr__/__str__/item/tolist/cpu/__float__/ __int__ to raise, and the test drives the REAL HybridReqToTokenPool.alloc through the sgl-project#767 branch rather than a re-implementation. Red-first was demonstrated by reverting just the one call-site argument: the tripwire fires through alloc -> logger.warning -> emit -> format -> getMessage -> msg % args, reproducing the incident's exact path. Green after: 5 passed. A can-fail case restores the identity formatting to prove the trap still fires from inside the real branch. Verified with the sgl-project#788 and sgl-project#787 suites alongside: 12 passed. Pre-existing and NOT caused by this change: 4 failures in test_mamba_anchor_seams_747.py (AttributeError: 'MambaComponent' object has no attribute 'cache' in _raw_token_pos), confirmed identical with the change reverted. Ruff findings in mamba_radix_cache.py and model_runner.py outside these hunks are pre-existing lint debt. This is the probable cure for the isolation boot's wedge: the log sync was the linchpin edge of that cycle. It does not touch sgl-project#789 -- one relay, two transports, no shared readiness contract -- which remains open debt.

…oor scope readable Two env-gated / one-shot diagnostics, both OFF or free by default, added to turn a mechanism proof into captured evidence on ONE instrumented boot. WHAT IS UNPROVEN. A TP=1/PP=3 boot deadlocks on the first radix-carrying request, measured twice. py-spy --locals showed the last rank holding a ScheduleBatch and blocking in _pp_recv_proxy_tensors for its slot, while both upstream ranks carried cur_batch=None and reported themselves idle -- so the proxy it waits for is never produced. The mb_ids were the CORRECT -1 stagger, so this is not a slot desync; the ranks diverge on BATCH PRESENCE. The mechanism is understood: PP ranks are N independent schedulers agreeing only by determinism (requests are chain-forwarded unconditionally, but each rank re-derives admission locally, and the proxy send is gated on that rank's own batch), and the #616g uniformity floors that would keep them aligned are scoped to tp_cpu_group -- which has ONE member on every rank of a TP=1/PP=3 boot, so all three floors switch off. What is missing is a captured value showing the ranks actually disagreeing. These diagnostics produce it, or falsify the theory honestly. 1. Scheduler._trace_pp_admission_verdict, called from get_new_batch_prefill where the verdict is known, behind SGLANG_PP_ADMISSION_TRACE (default False). Prints ADMIT/DECLINE, request count, up to four rids, per-request prefix lengths, available and evictable size, queue length, running batch size and the chunked flag. Truncated on purpose: this is a divergence signal, not a batch dump, and a log flood has cost a feature here a self-kill before. 2. A one-shot line at the #616g early return naming the tp_cpu_group world size together with pp_size and tp_size, and stating that the evict, host and mamba floors are off. The comment at that return reads "One rank: nothing to diverge from" -- true for TP, false for PP -- so the condition is now readable from the boot log instead of inferred from source. HOST-SIDE VALUES ONLY, and deliberately so. sgl-project#790 was a diagnostic passing a CUDA tensor as a logging argument: the formatting forced a D2H copy and a stream synchronize inside logging.emit and wedged the scheduler for 25 minutes. Prefix length is therefore taken with len(), which reads shape and does not synchronize; there is no .item(), .cpu(), .tolist() or float() on this path. The trace is wrapped so that a failing instrument degrades to one warning instead of killing the scheduler it is measuring. Default path is unchanged when the variable is unset. Verified: the module imports, the method resolves with its annotations, the env default reads False, and a source scan of the compiled method reports no sync-forcing call.

…n flush, not after it The wiring landed in f31fd5e was not bootable. On the instrumented boot it deadlocked on the first request with ZERO detector markers and zero GPU utilisation. py-spy, taken while the ranks were still alive: PP0 _pp_commit_pending_req_work (the request-chain flush) PP1, PP2 _pp_recv_admission_decision (waiting for the decision) The loop body ran recv-decision at :599, chain flush at :680, send-decision at :701. So the downstream ranks blocked on a message PP0 could only send at :701, PP0 blocked at :680 on those same ranks reaching the top of their next pass, and neither could move. A closed ring, silent: the ADMISSION-WEDGE detector never fired because this shape is not what it watches for, and the cards read 0% while three processes sat in gloo. THE CAUSE WAS THE COMMENT'S OWN REASONING. The send was placed after the flush because it "does not gate on any of them" and it seemed tidier to keep every per-iteration outbound flush in one place. Tidiness is not an ordering argument. sgl-project#788's own commit message states the rule this violated: a rank must satisfy everything a peer can be blocked on BEFORE it blocks on that peer. The admission decision is exactly such a thing, so it must be on the wire before the flush, and the flush is last precisely because it is the act that waits on a peer. Order is now recv(:599) -> send(:704, :729) -> flush(:745), and both comments say why rather than restating what. WHY THE TESTS DID NOT CATCH IT, recorded so the gap is not repeated: the integration test drives the decision path but not the real per-iteration ordering of the shipped loop against live peers, so a send placed after a blocking flush still passes. The suite went 190 green before this fix and 190 green after it -- unchanged, because it cannot see this class. Only the boot could, and only because zero GPU utilisation was noticed while the probe was supposedly running. Verified: 190 passed over ten files, unchanged from the pre-fix baseline.

…the send The proxy readiness gate refused to enter the blocking receive until the upstream's CHAN_DICT `sent` count exceeded this rank's `consumed` count. `sent` is published strictly after the post, so it says nothing while the upstream is still inside the send call. That window is empty for an ordinary isend -- but not for the first point-to-point op on a torch NCCL process group, which creates the 2-rank communicator lazily, and that creation is a rendezvous: the isend does not return until the peer enters the matching receive. Every boot therefore died on its first real prefill: PP0 cannot bump `sent` until the isend returns the isend cannot return until PP1 enters the receive PP1 will not enter the receive until `sent` bumps with the gate itself as one arc of the cycle. Boots instr7 and instr8 (2026-08-21) produced this identically; py-spy caught PP0 in isend -> send_tensor_dict -> _pp_send_dict_to_next_stage 33 s into a send it would never leave, while both downstream ranks raised "sgl-project#789 PROXY READINESS TIMEOUT ... posted 1681, consumed 1681". The message was not missing. It was being posted to the rank that refused to collect it. The gate's docstring asserted a stuck sender was "covered identically" to a sender that scheduled nothing; that assertion was the defect. Add a second counter, `attempted`, published on the line BEFORE the send call, and read only by the gate that would otherwise raise. "The upstream has irrevocably entered a send for me" is as positive a presence signal as "the upstream posted", and during a rendezvous it is the only one that exists. The module's ordering rule is unchanged and still governs `sent`; the new counter is not the phantom-message hazard in reverse, because the decision to send is already taken and unconditional at the bump, so the receiver waits on transfer and rendezvous time rather than on peer scheduling. Drain loops keep reading `sent`: they may take off the wire only what is provably already on it. Cost is one extra /dev/shm publish per dict message, measured at 13.6 us. Tests, all on this branch, measured not assumed: test_pp_proxy_readiness_rendezvous_789.py, 5 passed -- three live spawned processes, real gloo, the shipped functions, and a wire whose send blocks in a real dist.recv until the downstream enters recv, i.e. the NCCL lazy-init property in gloo primitives. The wire supplies a hazard rather than a guarantee, closing the direction of the trap where a test transport is more careful than production. Can-fail: neutering only PhaseFlipCounters.attempted, in the child, reproduces sgl-project#789 on both downstream ranks -- the metal specimen. test/registered/unit/managers full directory: 47 failed, 2753 passed. HEAD (96df16d) measured with the identical command: 47 failed, 2748 passed. The failing set is identical in both directions; the 5 added passes are the new file. The five holder/stub repairs in existing tests are interface drift only -- no assertion touched. Boot instr9, PP=3 on this rig with the reference instrumented configuration: health reached, 15 min 12 s uptime, one request served, the 8-request burst 8/8 in 3.1 s with per-request spec_accept_length mean 2.38 (sgl-project#779 gate PASS), 6 further back-to-back bursts under sustained load, 48 more requests, 0 failures, still serving afterwards. Zero sgl-project#789 raises, zero wedge markers, zero scheduler exceptions, and sgl-project#788 admission byte-identical on all three ranks including the 7-request batch. The first surviving boot of this series. Corridor is NOT passed on that boot and is not addressed here: NVML free sat flat at 5229/6612/5475 MiB per card under load against the 819-1229 MiB band, i.e. 4.4-5.0 GiB per card unclaimed against grants of 31800/18800/19800 MiB. That is a planner capacity item, and it is the first boot in this series on which the number means anything, because no earlier one ever served a request.

…p by a congruent count SGLANG_PP_ADMISSION_TRACE=1 wrote 146023 trace lines per rank on boot instr10, and on instr11 it produced a 5.9 GB log in about three hours of an IDLE server. The overwhelming majority were vacuous: sgl-project#788 PP-ADMISSION verdict=DECLINE n_reqs=0 rids= prefix_lens= ... queue=0 running=0 chunked=0 Nothing queued, nothing running, nothing admitted: no information, at a volume that buries the informative lines and fills the disk. The trace exists to prove RANK CONGRUENCE -- verdict_790.sh diffs the payloads across PP0/PP1/PP2 and requires them byte-identical -- so the suppression predicate must not be able to make two ranks disagree. pp_admission_verdict_is_vacuous therefore reads ONLY congruent payload fields (n_reqs, queue, running, chunked) and never wall-clock time, per-rank counters, log volume, sampling, or avail/evictable. The cadence is a COUNT, not a duration, for the same reason: a time-based limiter would fire at different passes on different ranks and manufacture a phantom divergence in the very evidence the instrument produces. Silence must not be ambiguous, so the instrument never simply goes quiet: every 1024 suppressed passes, and as a flush immediately before the next informative verdict, it emits sgl-project#788 PP-ADMISSION suppressed=N vacuous verdicts (...) since the last emitted line The roll-up deliberately does not spell "verdict=", so verdict_790.sh's payload grep never groups it, while the per-rank count grep still sees it -- counts stay congruent and comparable. Effect: ~146023 lines/rank becomes ~143; instr11's 5.9 GB / 3 h idle becomes roughly 6 MB. TESTS. test/registered/unit/managers/test_pp_admission_trace_vacuous_788.py without the fix -> 5 failed / 2 passed (the two passers are the over-suppression guard rails, which already held) with the fix -> 16 passed, 3 subtests passed, together with the sibling test_pp_admission_prefix_indices_tensor_796.py Can-fail proof for the property that matters: making the emission decision consult a rank-local value (available_size() > 100000) turns test_the_predicate_is_pure_in_the_congruent_fields red, with the two ranks' emission patterns diverging. ruff and codespell clean on the touched range.

… it with a void Boot instr11 wedged 4m18s into a flip drive: GPU util 0/0/0, log growth 846 B in 8 s, eight requests in flight and never served. py-spy on all three ranks gave a three-way ring: PP0 _event_loop_pp_body:779 -> ... -> _pp_recv_dict_from_prev_stage:3865 -> BLOCKING recv PP1 _event_loop_pp_body:603 -> pp_chain_receiver.recv:329 PP2 _event_loop_pp_body:603 -> pp_chain_receiver._advance:218 ROOT. A sgl-project#791 downstream retraction desynchronises the output ring, because the ring's two gates read two DIFFERENT ranks' mbs. reconcile_pp_admission_decision (pp_admission_congruence.py:462-483) amends the decision for every remaining DOWNSTREAM rank, and PP0 is upstream of every rank that can retract. The boot's last four admission passes, three passes after a tp_to_pp cutover, show it exactly: PP0 sgl-project#788 verdict=ADMIT n_reqs=1 rids=2f5e25a1... prefix_lens=512 PP1 sgl-project#791 unhonourable prefix on rank 1: rid=2f5e25a1... told=512 local=0 PP1 sgl-project#788 verdict=DECLINE n_reqs=0 PP2 sgl-project#788 verdict=DECLINE n_reqs=0 PP0 kept the microbatch and launched it; the downstreams had nothing for it. Last rank's send gate (:4101) reads its own empty slot and sends nothing; PP0's receive gate (:4306) reads its own occupied slot and blocks (:4315). _pp_output_exchange_due (:328) made the two the same EXPRESSION (sgl-project#753) but cannot make them the same FACT. PP0 never reaches the top of its next pass, so the pass-N+1 chain send is never posted and both downstreams block. The idle tail is load-bearing: a skipped send mid-burst is only a lag, since the pair is FIFO. It becomes a DEFICIT exactly when the pipeline goes quiet behind the retraction, which is what the log shows -- DECLINE, DECLINE, silence. sgl-project#796 IS NOT THE CAUSE, and this retires that hypothesis. The wraparound it removed is an ADMISSION_DECISION_KIND message, and recv_typed_tensor_dict (pp_typed_channel.py:136-145) returns only on expected_kind, stashing everything else. No wraparound could ever have released an expected_kind="output" receive, before or after sgl-project#796. Pinned by test_wraparound_kind_cannot_satisfy_an_output_receive. Restoring it would have put one unmatched message per pass back on the channel and unwedged nothing. FIX, following sgl-project#791's own law (decide on rank 0, carry the decision) rather than a timeout. _PP_OUTPUT_EXPECTED_KEY (:136) rides on the admission decision that already travels 0->1->... ->last in the SAME pass; PP0 publishes _pp_output_exchange_due(self.mbs[mb_id]), the identical expression on the identical object its own _do_recv will apply, and middle ranks forward it verbatim. The last rank (:4128) sends a void (_pp_void_output_payload, :4148) only for a slot PP0's own published verdict obliges it to receive -- so this is not the bounded-recv corpse in reverse. PP0 absorbs it (:4315, _pp_absorb_void_output, :4171), empties the slot so the loop's None guard holds, releases each request through the existing idempotent _release_dynamic_chunk_probe and re-queues it, and feeds record_return_trip from the chain-reconciled decision riding back inside the void. That last part also repairs collateral damage from sgl-project#796: record_return_trip (pp_admission_congruence.py:276) was fed ONLY by the wraparound, so since sgl-project#796 _learned_floor was never populated and prefix_len_for never clamped -- sgl-project#630's termination argument had no feeder at all. It does again, without an unmatched message. Absent key => False => no void => byte-identical to today, so stand-ins and pp_size<=1 are unchanged. TESTS. test/registered/unit/managers/test_pp_output_ring_retraction_wedge_791b.py, three live spawned processes, real gloo, the shipped functions: test_retraction_wedges_the_ring_without_the_fix (can-fail: neuters ONLY the fix IN THE CHILD and reproduces the specimen -- 3/3 ranks stuck, downstreams ahead of the rank they block on, the inverted stagger) test_ring_survives_the_retraction_with_the_fix (12/12 passes, 1 void absorbed on PP0, 1 request re-queued, 0 voids on PP1/PP2) test_void_payload_is_only_sent_when_the_first_rank_expects_one test_wraparound_kind_cannot_satisfy_an_output_receive -> 4 passed, three consecutive runs (40.95 / 39.57 / 38.44 s) Neighbours (795, 796) -> 13 passed. Wider mixin set -> 19 failed / 176 passed, IDENTICAL at HEAD (patch removed, re-run, re-applied); all 22 are _RingWire/_Group stand-ins missing is_first_rank/is_last_rank, i.e. sgl-project#796's own gate. codespell clean; the one ruff F841 at scheduler_pp_mixin.py:1395 is pre-existing and untouched. KNOWN GAP, filed rather than folded in: a PARTIAL retraction (one rid of several) keeps the ring matched, so PP0 processes a real output for a batch containing requests the downstream never ran -- a membership divergence with no wedge and no error. Same root asymmetry, different defect.

Boot instr14 logged ~18 MB/min under burst load, over the 100 MB/10min alarm threshold. A census of a 5 MB tail (27340 lines) found two emitters at ~95%: 13059 "sgl-project#788 PP-ADMISSION verdict=DECLINE" and 13068 "PARKED-DECODE carriers N parked". Both fire on EVERY scheduler pass. d59537c only collapses VACUOUS verdicts (n_reqs=0 queue=0 running=0 chunked=0). Under load queue>0 and running>0, so every DECLINE counts as informative and prints once per pass. THE OBVIOUS FIX DOES NOT WORK, and the data says so. Consecutive lines are almost never identical -- both emitters are PERIODIC, not constant: carriers 4 parked (+4 -2) of 4 resident carriers 2 parked (+2 -4) of 2 resident carriers 2 parked (+2 -2) of 2 resident <- period 3 verdict=DECLINE ... queue=1 running=2 verdict=DECLINE ... queue=1 running=4 <- period 2 Replaying the real tail through a strict run-length collapser recovers NOTHING: 13581 admission lines -> 9073 kept + 4508 roll-ups = 13581, unchanged. Run lengths are 1. So the collapse is a BOUNDED CYCLE DETECTOR over periods 1..8, of which run-length is merely the p=1 case. MEASURED on the full 51 MB / 9m59s instr14 log, replayed through the shipped CycleCollapse: sgl-project#788 verdict lines 141513 (20.4 MB) -> 1524 kept + 195 roll-ups PARKED-DECODE 140184 (25.4 MB) -> 225 kept + 195 roll-ups total 51212468 B -> 5850023 B = 11.4%, 8.8x peak minute 13.03 -> 1.24 MB/min; 10-min window 48.8 -> 5.6 MB CONGRUENCE, the property d59537c exists to protect. The sgl-project#788 collapse key is (verdict, n_reqs, rids, prefix_lens, queue, running, chunked). avail/evictable are deliberately EXCLUDED: they are the rank's own pool accounting and legitimately differ per rank, so keying on them would make two ranks collapse differently and manufacture a phantom divergence in the evidence verdict_790.sh diffs. Stated at the predicate and in the module header. PARKED-DECODE carriers is RANK-LOCAL by construction (slot_pool is this rank's mamba allocator, running_bs this rank's running_batch) and no gate diffs it across ranks, so it is collapsed on its own terms and documented as unusable for congruence. Both roll-ups dodge the gates' greps: the sgl-project#788 one never spells "verdict=" so verdict_790.sh step 3 ignores it while the per-rank count grep still sees it; the parked one never spells "PARKED-DECODE carriers" so prove_park_677.sh's count is not inflated by the reduction it reports. last_receipt is still set unconditionally -- only the log is throttled -- and park/readmit/evacuate stay unconditional as edge receipts. A chunked-prefill walk (prefix_lens 0, 512, 1024, ...) is pinned as never collapsed: it never repeats, and it is the actual congruence evidence. TESTS. test/registered/unit/managers/test_trace_cycle_collapse_788.py, single process, pure CPU, no torch. before wiring -> 10 failed / 9 passed ("500 identical DECLINEs produced 500 lines", "600 cycling receipts produced 600 lines") with the fix -> 114 passed / 5 subtests across this file and the 788, 796, 677, 731_744, 744, 659 neighbours behavioural neuter (CYCLE_COLLAPSE_MAX_PERIOD 8 -> 0, API intact) -> 18 failed / 1 passed, 0 AttributeError can-fail for congruence: keying the gate on the rank-local available_size() turns test_two_ranks_fed_identical_payloads_emit_identically red ruff check and format clean, isort clean, codespell rc=0. scheduler.py's format --check failure is pre-existing at HEAD and this hunk adds no diff.

…ction Boots instr15, instr16 and instr17 all died on the same raise, and it was not a leftover: ValueError: sgl-project#631 PP proxy/batch mismatch: received hidden_states with 126 row(s) for a 1 batch of 22 token(s) (instr17, 07:12:49 PP1) 126 = 22 + 104. PP0's batch is PP1's batch PLUS THE ONE REQUEST PP1 RETRACTED. Same pass, same slot, same epoch; the stamp was correct in every element, which is exactly why PROXY LEFTOVER REFUSED stayed 0 on all three boots. Nothing was stale, so no stamp discriminator -- per-slot generation, receiver-derived seq, rolling pass id -- could ever have caught it. Two earlier hypotheses are retired by this arithmetic: a within-epoch stale slot (there is no stale message) and a cutover landing between receive and use (the traceback shows a FRESH _event_loop_pp_body:960 after the re-dispatch, and pp_proxy_tensors is a loop local no re-dispatch carries across). MECHANISM, from boot_instr17.log:62997-63056, all 07:12:49: PP0 sgl-project#788 verdict=ADMIT n_reqs=2 rids=51a294650b...,5e744c29f8... prefix_lens=0,16896 PP1 sgl-project#791 unhonourable prefix on rank 1: rid=5e744c29f8... told=16896 local=0 PP1 sgl-project#788 verdict=ADMIT n_reqs=1 rids=51a294650b... prefix_lens=0 PP1 ValueError: 126 row(s) for a 1 batch of 22 token(s) reconcile_pp_admission_decision drops the unhonourable rid from `effective` (pp_admission_congruence.py:472-483), the admission loop omits it from THIS rank's batch (scheduler.py:6974-6991), self.mbs[mb_id] is the narrowed batch (scheduler_pp_mixin.py:806) -- while the upstream has already sent its decision and launched its own wider batch (:876, :947). A batch in flight cannot be amended. The flip is the TRIGGER, not the cause: it cold-starts the downstream radix cache, so PP0 offers a prefix PP1 has nothing for. #791b already fixed the OUTPUT-RING consequence of this same retraction (scheduler_pp_mixin.py:4317-4319); the proxy-width consequence was never covered. THE DISCRIMINATOR IS NOT ON THE WIRE, and it does not need to be: "did I retract anything from this pass's decision?" The receiver PERFORMED the retraction, at the top of the same pass, strictly before the proxy receive, and #791b already records the amendment per slot in _pp_admission_amended_by_slot. Nothing new crosses the wire and the receiver predicts nothing a sender wrote. The test is `retracted_by_rank == self`, not `retracted`, so an entry an earlier rank retracted is correctly ignored -- pinned as a regression case. WHY THIS BEATS THE SHIPPED TRIPWIRE WHERE IT MATTERS: chunked prefill caps every chunk at the same size, so two ranks running different request sets routinely present EQUAL widths. model_runner.py:4182 is blind to that -- silent wrong output, not a shape error. Pinned as test_a_same_width_divergence_is_still_refused. STATED PLAINLY, THIS DOES NOT YET PREVENT THE DEATH. It converts a shape error raised thirty layers deep into a boundary refusal that names rid, told= and local=. Prevention is a separate design call, deliberately not taken unilaterally here: PP0 must stop offering a prefix the downstream cannot honour, i.e. PPAdmissionCongruenceGuard.prefix_len_for needs the feedback sgl-project#796 deleted with the wraparound (scheduler_pp_mixin.py:3568-3594). The channel that survives sgl-project#796's "no send no peer must take" law is the OUTPUT message PP0 already receives from the last rank -- piggyback the retraction floor as a per-hop key, the _PP_OUTPUT_EXPECTED_KEY precedent. That still costs one wasted PP0 forward the first time, and voiding the retracting rank's pass desyncs PP0's own request state, so both halves must be decided together, on metal. ALSO FOUND, NOT FIXED HERE: two rank-local decode-retraction paths (scheduler.py:7517/:7489 and :7211-7312) diverge the same way and sgl-project#791 never touches them. TESTS. test_pp_proxy_retracted_pass_mispair_791c.py -- three live spawned processes, real gloo, shipped functions, real 126-row and 22-row tensors: test_a_retracted_pass_is_mispaired_without_the_retraction_test (can-fail: blinds ONLY entries_retracted_by_rank's return value in the CHILD, through scheduler_pp_mixin's own module globals, so every API, the guard body, the reconciliation and the amendment recording all still run -> 126 rows delivered for the 22-token batch = THE SPECIMEN, and the child's `effective` still shows the narrowed set, proving the retraction happened) test_a_retracted_pass_is_refused_by_the_receive_guard test_a_retraction_by_another_rank_does_not_refuse test_a_same_width_divergence_is_still_refused test_an_unretracted_pass_is_delivered_unchanged -> 5 passed; with the 631 and 795 neighbours 29 passed Neighbour set 631/795/791/791-wiring/791b/757 re-measured AT HEAD: 44 passed / 1 failed; after: 49 passed / 1 failed, the same pre-existing _RingWire.is_last_rank drift from sgl-project#796. ruff, ruff-format and codespell finding sets byte-identical to HEAD. The 631 test's residual pin is corrected: it had misdirected two investigations into hunting a stale message that never existed.

…ery rank extend = min(fill - prefix, cap). Three of the four quantities are now settled by measurement: cap -- group-uniform. sgl-project#610 subtracts each rank's surplus over the group minimum, and all three ranks read 12977. It also never bound: the binding term was chunk(4096) against extends of 253/254/ 277/278. prefix -- equalised by the sgl-project#791 truncation, both branches, code-read. extend -- differs. That is the observed 254 against 301. So the fill differs, and `_refresh_fill_ids` defines it as `origin_input_ids + output_ids` with a rank-identical prompt. `output_ids` is the single rank-local component of the whole equation. And `req.output_ids.append` lives in the RESULT path (batch_result_processor.py:276), which a VOIDED pass never reaches. These boots are void-dominated. That is a complete chain -- void pass, output falls behind, fill falls behind, extend differs, the sgl-project#631 guard fires -- and every link of it is code-read rather than argued. It is not yet MEASURED. This prints `fill_lens` and `out_lens` per rid on the existing `sgl-project#788 PP-ADMISSION verdict` line, which already carries `rids` and `prefix_lens` and already fires on every rank. It is the only carrier that allows a cross-rank comparison FOR THE SAME REQUEST. No new emission: the same `logger.info`, more fields -- the event perturbs this path (measured four times), the bytes do not (measured twice, and this is the third test). THE PREDICTION IS DIRECTIONAL, not just "a difference": the rank with the smaller `out_lens` must also show the smaller `fill_lens` and the smaller extend. A difference in the wrong direction refutes the chain rather than confirming it. If `out_lens` agrees across ranks the chain is sound but not what happened here, and the remaining writer is `pp_carried_fill_tail` (schedule_batch.py:1300-1305), which is also rank-local. I will not compute 301-254=47 into anything, whichever way the numbers land. Name the carrier, then measure the difference -- this window produced four numbers-fit traps and I walked into one. Evidence: desk. py_compile; the format verified by rendering it through logging rather than by counting placeholders. Belegstufe: DESK-BEWIESEN.

Three carriers were checked for an enrichment and all three fail: no rid-precise log exists after `req.output_ids.append`, `sgl-project#968 MINT` sits in the parked-continuation fact ring rather than the result path, and `sgl-project#788` fires at ADMISSION where `out_len` is structurally 0 (measured, boot 38). A new emission is therefore the only option, not a convenience -- and it is a STATE reading, not a timing signal, the same distinction that carried `published_fundable_floor`. WHAT IT MEASURES. extend = min(fill - prefix, cap). Three terms are settled: cap is group-uniform (sgl-project#610 pins each rank to the group minimum, measured identical at 12977) and never bound (4096 against extends of 253-278); prefix is equalised by the sgl-project#791 truncation, both branches. Fill is defined as `origin_input_ids + output_ids` with a rank-identical prompt, so `output_ids` is the single rank-local component of the whole equation -- and this append, in the RESULT path a voided pass never reaches, is the only place it grows. THREE CONSTRAINTS, each from a mistake made in this window: DENOMINATOR ALWAYS. seen/emitted/unreadable ride on every line whatever they are. A counter that only moves on confirmations is what made me quote a 12:1 neutrality basis that was really 10:5. FIXED CADENCE, NOT A HIT FILTER. "Emit only when interesting" cannot tell a zero from a never-evaluated -- the same shape that made boot 27 unreadable. Every 40th append, whatever it says. SENTINEL THAT CANNOT COLLIDE. `len(output_ids)` can legitimately be 0, so 0 and None are both unusable for "not read"; -1 is the sentinel and the message says so. `floor=unset` already saved one reading this way. THE DEATH FORM IS NOT A SIGNAL HERE, in either direction. Cumulative over 15 no-flip boots of this config: WIDTH=10, IDENTITY=5. At a one-in-three base rate a single flip is noise, and a single non-flip proves nothing. The only question this boot answers is what `out_len` says for the same rid on two ranks. Evidence: desk. py_compile; the format verified by rendering it through logging rather than by counting placeholders. Belegstufe: DESK-BEWIESEN.

… idiom Boot 43 reported `seen=0` on all three ranks with 27 prefill batches. The probe's own message says what that means -- "seen=0 means prepare_for_extend never ran, which is not the same thing" -- so it was readable, but the reason was mine. `len(getattr(_r, "prefix_indices", ()) or ())` asks `bool(tensor)`, which torch refuses for a multi-element tensor. Every call raised into the probe's `except` and nothing was ever counted. THE REPO DOCUMENTS THIS EXACT MISTAKE, AND I READ THE COMMENT IN THIS SESSION BEFORE MAKING IT. scheduler.py:8300, on the sgl-project#788 emitter: "sgl-project#796: NO `or []` HERE. `prefix_indices` is a tensor, and `x or []` asks `bool(x)` ... the effect was that EVERY admitting pass lost its trace line to the except below while the idle DECLINE passes logged fine -- boot instr6 showed all three ranks reporting a bare RuntimeError at the exact pass the first real request arrived ... The docstring above already said len() is the right spelling; the `or []` slipped in anyway." Same idiom, same failure mode, same silent `except`, one session apart -- and the earlier one cost a boot window too. Vigilance is not what stops this; the comment was read and the bug was written anyway. Fixed with the only correct spelling: an explicit None test and `len()`. AND THE PROBE NOW HAS A CAN-FAIL PROOF, which the first version never did: two synthetic requests, one holding the invariant (prefix 8192 against start 8192 -> BREAK 0) and one breaking it (prefix 7938 against start 8192 -> BREAK 254, and `len_input` correspondingly 508 instead of 254). The counter registers exactly one break. A probe that cannot be shown to fire on the condition it looks for is the same class as a counter without a denominator, and this one was shipped without it. Evidence: desk. py_compile; the isolation test that reproduced the raise before the fix and passes after it; the two-case can-fail proof. Belegstufe: DESK-BEWIESEN.

…ONGRUENT ZUGELASSEN -- meine "nie angekommen"-Erklaerung ist FALSCH S1-Durchgang, Schritt 1. Die rid-genaue Timeline aus dem Strandungs-Log kippt meine eigene Diagnose, und zwar in die praezisere Richtung. == DIE TIMELINE (logindex, boot_855_fix1027_..._112900, voll ingested) == Nach 11:57:30 nennt das Log GENAU EINEN rid: 2fe8dab34ab1. admission n=18 11:56:14 .. 11:58:05 flip_event n=100 11:56:17 .. 12:02:21 Die 18 Admission-Zeilen sind SECHS Runden zu je drei Raengen, und JEDE ist "sgl-project#788 PP-ADMISSION verdict=ADMIT n_reqs=1 rids=2fe8dab34ab14d" auf Rang 0 UND 1 UND 2. Die Raenge waren sich EINIG. Letzte Zulassung 11:58:05; letzter Prefill 11:58:07; danach nichts mehr. Und der Flip-Apparat nennt denselben rid noch bis 12:02:21 in FLIP EXTENT PROBE auf allen drei Raengen -- der Request ist also die ganze Strandung ueber RESIDENT, nicht verschwunden. == SELBSTKORREKTUR (fuenfte, und sie stand im CODE) == Mein Commit 70d85ed schloss: "DER REQUEST IST OBERHALB DER PHASE-POLICY VERLORENGEGANGEN", und ich hatte diese Erklaerung in die sgl-project#1028-Log-Zeile geschrieben ("its request never reached this scheduler"). Das ist FALSIFIZIERT: er hat den Scheduler erreicht, wurde sechsmal von allen drei Raengen zugelassen und blieb danach resident. Die Zeile haette den naechsten Leser genau so fehlgeleitet, wie mich "decoding in tp" fehlgeleitet hat -- also derselbe Fehler, den ich am selben Tag viermal katalogisiert habe, von mir selbst begangen und in den Baum committet. Text korrigiert auf "admitted and then stopped progressing somewhere DOWNSTREAM of admission". Der Grund, dass ich es fand: die rid-Timeline, nicht erneutes Nachdenken. == FOLGE: DER VERLUSTPUNKT LIEGT NICHT AUF S0 UND NICHT AUF S1 == Intake (S0) war bereits ausgeschlossen. Admission (S1) ist hiermit ebenfalls ausgeschlossen -- sie hat kongruent ADMIT gesprochen. Der Defekt liegt ZWISCHEN Zulassung/Prefill und dem laufenden Batch, also auf der MERGE-Bahn (S2). Damit ist die Station benannt, an der weitergesucht wird. == EIN KANDIDAT, AUSDRUECKLICH NOCH NICHT ALS URSACHE BEHAUPTET == Am Strandungspunkt feuert auf allen drei Raengen "SELF-MERGE REFUSED: last_batch is running_batch (bs=1)" (scheduler.py:7427-7457). Formal ist das genau die vom Gesetz verbotene Form: ein Guard, der einen Zustand erkennt, den sein eigener Kommentar "the resident set is corrupted" nennt, und dann REFUSAL-UND-WEITER macht statt die Gruppe zu stoppen -- und der Kommentar sagt selbst "a detector that only declines to act cannot stop a doubling -- the instance still died". Nach Upstream-Minimal ist ein Defekt in einer Kompensationsschicht ein LOESCH-KANDIDATEN-Befund; die eigentliche Frage ist, warum `last_batch is running_batch` ueberhaupt gilt (Aliasing). ABER, INDIKATOR-GESETZ: der Marker feuert 166.173-mal auf diesem Boot, also auch waehrend der ~26 Minuten, in denen alles funktionierte. Er ist damit NICHT hinreichend fuer die Strandung und wird hier als KO-OKKURRENZ gefuehrt, nicht als Ursache. Wer ihn ohne diese Zahl zitiert, berichtet eine Korrelation als Wurzel. == NAECHSTER SCHRITT (nicht mehr in diesem Zug) == S2/Merge-Bahn mit derselben Disziplin: Zeilen per emittiertem Literal am Pin relokalisieren, per trapsafe zaehlen, und die Frage stellen, die die sgl-project#1031-MERGE-PATH-PROBE (scheduler.py:7462ff) bereits woertlich formuliert -- "`#running-req: 0` heisst entweder der Prefill hat NIE gemerged, oder er merged und sgl-project#856-no-carry hat ihn vor der Decode-Runde retracted; nichts Gemessenes trennt die beiden". Genau diese Trennung ist jetzt die Aufgabe, und die Probe dafuer liegt schon im Baum. BELEG-STUFE: BOOT-BEWIESEN (rid-Timeline aus dem voll ingesteten Log). Kandidat SELF-MERGE: BEOBACHTET, ausdruecklich nicht kausal belegt.

… it without admission BOOT-8 ROOT, MEASURED. The extent was derived inside `build_pp_admission_decision`, reached only from `_get_new_batch_prefill_raw`'s tail -- after an admission loop with eight `continue` branches and an `if len(can_run_list) == 0: return` above it. On the passes where a host hit was live, the holder was skipped at scheduler.py's `prefetch_pending` branch before the adder ever saw it, the list came back empty, the method returned, and NO decision was built: 09:13:52 PP0/PP1/PP2 sgl-project#788 PP-ADMISSION verdict=DECLINE n_reqs=0 queue=1 reason=loop_skips(prefetch_pending=1(first=f7f997c0...)) 09:13:54 PP0/PP1/PP2 sgl-project#968 LOAD-BACK DEFERRED rid=f7f997c0... (kv=4618) ... holds no PP0 extent for it yet The chain fed itself: hit present -> prefetch pending -> skipped -> empty list -> no decision -> no extent -> deferral. `sgl-project#1040 EXTENT STATE-ALIGN` read 0 in the whole log while its sibling emitter logged 9 times, which is what localised this. TWO HALVES, both mechanism-agnostic on purpose. 1. CHOICE MOVES TO THE WRITER. `stamp_state_aligned_extent` runs at the two sites that write `Req.host_hit_length` -- `Req.init_next_round_input` and `match_prefix_for_req` -- both unpacking one `match_prefix` result. `Req.__init__` only zeroes the field and `truncate_prefix_to` is dead (0 callers, 0 name reads), so a request CANNOT hold a hit without passing one of them. Every `can_run_list` filler (`add_one_req`, `add_chunked_req`, `add_one_req_ignore_eos`, the dllm pair, `_add_scheduled_req`) must match before it is executable, so the match dominates all six. The dominator argument is made over the WRITERS deliberately: the call graph cannot carry it -- `call_path add_chunked_req -> match_prefix_for_req` walks past 156 unresolved edges and finds nothing, which is a bounded negative and proves nothing. The row builder now READS the stamped field on both branches and derives nothing. 2. PUBLICATION DECOUPLES FROM ADMISSION. `_publish_pp_decision_1041` is the one publisher and BOTH exits of `_get_new_batch_prefill_raw` go through it, including the empty-list return that boot 8 died on. Everything the loop SAW and did not admit rides as an `admitted=False` FACT CARRIER. Collected at ONE site at the top of the loop, before any skip -- appending inside each skip branch would be the per-path retrofit this slice exists to avoid, and would have missed the second, still-unproven decline mechanism exactly as the first was missed. CARRIER SAFETY IS STRUCTURAL, NOT CAREFUL. `forwarded_schedule` already filters `e.admitted and not e.retracted`, so a carrier can never enter `_pp_scheduled_extents` and therefore never reaches the sgl-project#791 membership compare (scheduler.py:10508/:10522) -- the PPScheduleRefused-storm direction is closed at the source. `reconcile_pp_admission_decision` passes it through verbatim; `forwarded_last_chunk`/`forwarded_fill_carry` key on `fill_len`, which it does not carry; `apply_pp_load_back_row` stamps it by rid, which is its whole purpose, and an unheld rid is already a counted no-op there. STANDING INSTRUMENT. `sgl-project#1041 EXTENT POPULATION seen/published/delta` per pass. `delta != 0` means a request held an extent and no entry carried it, i.e. a bypass. The three legitimate differences are structural and produce no delta: no host hit stamps no extent, `pp_size<=1` never reaches the publisher, and an unheld rid is the receiver's counter. A future bypass is now loud instead of a silent zero that costs a boot to read. CHECK (speed mode: one matched check). devtools/check_1041_factcarrier.py, hermetic, CUDA-free. Named failure class: non-admitted entries reaching readers that iterate the decision, dangerous direction a PPScheduleRefused storm. The three pinned directions: (1) carrier passes without refusal and is absent from forwarded_schedule / last_chunk / effective while still delivering its extent through reconcile and apply; (2) a real membership violation STILL refuses -- an admitted rid the rank does not hold is still nameable as MISSING, pinned so the gate cannot go quiet; (3) unknown rid raises nothing and stamps nothing. 20/20 PASS. sgl-project#1040 check re-run green (no regression), py_compile + import smoke, ruff clean on the edited congruence module. cell_1039: adds `sgl-project#1041 EXTENT POPULATION` delta==0 and PPScheduleRefused==0 as acceptance criteria, both voiding the PASS branch.

…; one timeout source; the store witness replaces the seam stamp ROOT (boot weg1b3 @ 6980c75, rid 679e4568, 84,027 tokens, cutovers 3+4): _prefetch_timeout_check_linear_func charged base + len(hash_value) x per_page and hash_value is empty until the serial prefetch thread's store probe has run, so the every-pass drain reaped the LONG re-admission at the bare base (effective 1.0 s: the sgl-project#968/sgl-project#1065 re-pricing 2.0/1.0 was a pair of dead locals overwritten by the hybrid parse's 1/0.25) with completed=0. The store had the pages; nobody asked it in time. phase_purity then counted the retract stamp as 'restored' and licensed a P=0 TP prefill (6 x 4096 recomputed). FIX F1a unified_radix_cache.py: _prefetch_priced_pages = len(hash_value) or len(token_ids)//page_size; _prefetch_timeout_budget_s = base + pages x per_page; the linear check compares against it (hiradix sibling same form under its own cfg.max; hi_mamba unreachable, registry.py:106-107). The unified budget is deliberately UNCAPPED (no PrefetchTimeoutConfig.max): an unprobed 1M-token op prices ~1000 s and admission waits on it through the PP0 _prefetch_done_for hold. Named, not hidden. F1b one timeout source: hybrid_cache_controller.parse_storage_backend_ extra_config pops its defaults from PrefetchTimeoutConfig (2.0 s + 1.0 s per KiToken); dead 2.0/1.0 locals and the 3.0/0.25 re-attach literals deleted; 'sgl-project#1157 PREFETCH TIMEOUT base= per_ki_token= source=' printed once per attach. F1c the reap is a line: 'sgl-project#1157 PREFETCH REAPED req= probed= requested_pages= hit_pages= completed= elapsed= budget=' per termination that is not a local completion. PrefetchOperation.probed_hit_tokens is stamped by the prefetch thread after its MIN; the record the admission loop pops (prefetch_loaded_tokens_by_reqid) is a PrefetchOutcome(int) annotated with hit_tokens/probed, written at completion, reap and revoke. F1d the seam premise is a measured witness: store_witness() reads the tree's own prefetch state (pending / hit / bounded / cold / unprobed); the `cache_protected_len>0 or cached_prompt_tokens_at_retract>0` stamp arm is DELETED. StoreWitnessContradiction is raised from the premise and from both admission arms and re-raised through the three fail-open probes; it leaves get_next_batch_to_run uncaught and is the group STOP (run_scheduler_process -> SIGQUIT), never a P=0 admission. REVIEW FIX (operator decisions 2026-09-03): B1 PrefetchOutcome now round-trips pickle/copy.deepcopy: positional __new__(loaded, hit_tokens=None, probed=False) + __reduce__. The keyword-only form survived dumps but not loads and rode req.storage_hit_length -> cached_tokens_storage -> details['storage'] -> send_pyobj -> detokenizer death on the first partial store hit. Both admission writes store int(loaded_tokens). Int subclass kept over a dataclass: smaller diff, every existing int reader untouched. B2 beside a stamp the witness MEASURES the stamped span: 'hit' only if loaded > 0 and stamp - hit_tokens <= one HiCache chunk; allowance = tree._prefetch_chunk_tokens (chunked_prefill_size, the one-chunk bound the sgl-project#1068 L2 truncation line already prices; fallback prefetch_threshold when the tree has no term). A revoke (loaded=0, the 40-token chat header answering every probe) or a probed shortfall beyond the allowance beside a stamp is a contradiction. Cold requests (stamp 0) keep today's reading. B3 raise text renamed to 'sgl-project#1157 STORE WITNESS CONTRADICTION ...' and the class to StoreWitnessContradiction: 'SEAM RESTORE' tripped the sgl-project#1068 seam-copy zombie gate (test_968_seam_copy_deleted_1068 red on R1, green now). Functions renamed store_witness / assert_store_witness_at_admission. N1 rank-uniform reap annotation: probed (MIN) and hit_tokens (MIN) ride two extra slots of the EXISTING packed all_reduce in check_prefetch_progress (the one that reduces min_completed_tokens); the record and the REAPED line are derived from the reduced vector only, never the thread-timing stamp. No new collective. N2 phase_policy.py hold reason now names the store witness instead of 'premise verified on the retract credit'. POPULATION CHANGES NAMED: (a) 'bounded' state: a stamped request shorter than prefetch_threshold counts as restored (no read can exist; recompute bounded below the threshold; the /health_generate shape). (b) A stamped request with no record, no ongoing prefetch and a prompt >= threshold (rate-limited, storage off, deferral refused) is now 'cold' -> premise refused -> exemption closed, where the stamp used to open it. (c) A mixed queue still opens on the witnessed request and a cold sibling rides at P=0 (pre-existing); the boot-4 count 'sgl-project#1157 PREFETCH REAPED probed=False completed=0' = 0 for retract re-admissions is the metal proof no LONG rides at P=0. (d) PP followers (pp_rank>0) run assert_store_witness_at_admission too: a follower raising is a process death via SIGQUIT (a STOP), not a verdict; it is not gated to pp_rank==0 here. RANK UNIFORMITY of the witness inputs: ongoing_prefetch removed only after the group MIN / under the drain barrier / at the cutover sweep; prefetch_loaded_tokens_by_reqid written from the reduced vector (N1) or on the revoke path whose enqueue the prefetch thread's MIN precedes; prefetch_deferred set from a rank-local counter but cleared/retried under the group gate (sgl-project#1068 slice 3 fix). TESTS (hermetic, CUDA_VISIBLE_DEVICES="" PYTHONPATH=<tree>/python) test/registered/unit/mem_cache/test_1157_reaper_prices_requested_span.py (20) test/registered/unit/managers/test_1157_seam_witness.py (25) Red measured before the fix on the R1 build: pickle.loads/deepcopy TypeError; _witness_from_outcome(stamp=80009, PrefetchOutcome(0,hit=40,probed)) == 'hit'; (4096,hit=4096) beside 80009 == 'hit'; no _reap_annotation_from_packed; no StoreWitnessContradiction. On the parent e63fd08 both files fail at collection (PrefetchOutcome / StoreWitnessContradiction absent). Inverted tests carry withdrawal docstrings: 861j test_a_stamp_without_a_store_read_no_longer_opens_the_gate, W30 test_a_stamp_alone_no_longer_opens_the_exemption, 968 test_premise_with_stamps_and_host_content_no_longer_holds_alone. Mutants killed (5/5): M1a keyword-only __new__ restored -> 2F; M1b int() dropped at the TP admission site -> 1F; M2 shortfall term dropped -> 2F; M3 reap annotation read from the local stamp -> 1F; M4 'SEAM RESTORE' token restored -> 3F (incl. the sgl-project#1068 zombie gate). SUITE (bounded selection, identical on both trees: test/registered/unit/ mem_cache + distributed/test_prefetch_progress_symmetry_580.py + managers test_*phase_purity*/test_*phase_policy*/test_*seam* minus test_phase_flip_spec_seam_631.py + test_968_starvation_umbau.py + test_tp_decode_formation_861j.py; pytest -q -p no:cacheprovider -rf, single process; the two new files EXCLUDED from the parent run, appended on the fixed run): parent e63fd08 (scratch worktree): 125 failed / 3446 passed / 927 skipped (174 s) fixed (this tree): 125 failed / 3492 passed / 927 skipped (172 s) per-test FAILED-id diff (comm): 96 FAILED ids on each tree, IDENTICAL sets (comm -13 and comm -23 both empty): 0 new failures, 0 newly passing; +46 passed = the 45 new tests plus one (not attributable from -q output; FAILED, ERROR and skipped sets identical) pre-existing failing files (parent, top): test_phase_policy.py 35, test_pool_invariant_live_double_912b.py 12, distributed/test_prefetch_progress_symmetry_580.py 8, test_seam_abandon_cap_631.py 7, test_mamba_double_free_924.py 5. Ruff parity measured per touched file (fixed / parent): cache_controller 14/14, phase_policy 0/0, phase_purity 0/0, scheduler 102/102, hicache_storage 0/0, hiradix 1/1, hybrid_cache_controller 0/0, unified_radix_cache 1/1, 968 0/0, W30 5/5, 861j 9/9; new files 0/0; totals 132 = 132. Not claimed clean. EVIDENCE TIER: DESK-PROVEN. Boot 4 proof lines: 'sgl-project#1157 PREFETCH TIMEOUT base=2.00s per_ki_token=1.00s source=PrefetchTimeoutConfig' once per rank; 'sgl-project#1157 PREFETCH REAPED ... probed=False' = 0 lines for retract re-admissions; 'HiCache prefetch success req=<LONG> completed_local=N>0' at the first pp_to_tp cutover; 'sgl-project#788 ADMIT ... prefix_lens>0' for the LONG; 'sgl-project#1157 STORE WITNESS CONTRADICTION' genuine count 0 (trapsafe_count.py --marker 'sgl-project#1157 STORE WITNESS CONTRADICTION' --genuine-regex 'RuntimeError: sgl-project#1157 STORE WITNESS CONTRADICTION|StoreWitnessContradiction: sgl-project#1157'); a nonzero is a group STOP naming rid/stamped/probed_hit/loaded/ allowance/shortfall.

…llot digest mismatch is a group STOP; the cold-build opener honours the cap Boot weg1b3 @ 6980c75 (log /spinning/evidence-665-f1/boot_855_weg1b3_6980c75eac_0902_234752.log): a /health_generate probe reached all three ranks at 23:54:18; PP0 was busy and dropped it in its dispatch loop, PP1/PP2 took the sgl-project#631 row-authority branch of the same loop and enqueued it (queue=6 vs 7, +1 through five seams). The #791b ballot saw the disagreement on every TP pass from 23:56:17 (18 mismatch lines, cadence 1..32, 0 restored) and fell back to the rank-local prefetch verdict, until at 23:59:54 the verdicts split: PP0/PP2 admitted 8f31846b and entered the forward, PP1 declined and sat in the request broadcast. The opener-side cold build window (x40, uncapped) held the hung forward until the 300 s watchdog. F2a ONE DISPOSAL, AT THE ORIGIN, BEFORE RELAY (upstream-minimal). request_receiver.recv_requests: on the request origin only, health-check requests meet the existing idle gate BEFORE the policy hook, the TP broadcast and the PP chain forward (busy -> ipc answered from the origin + dropped; idle -> kept and relayed like any request). The rank-conditional block in scheduler.process_input_requests (upstream idle gate + the sgl-project#631 follower-enqueue special case, 8da4ec5) is deleted: every rank dispatches the identical list and waiting_queue stays replicated. Drop line: 'sgl-project#1158 HEALTH-CHECK dropped at origin before broadcast rid=%s busy queue=%d running=%d' (the gate's own reading). The PP chain forward (_pp_forward_and_process_input_requests) sends the list recv_requests RETURNED, and the DP-attention work/control split runs inside _broadcast_reqs_across_ranks after the disposal, so no second path can deliver a probe the origin dropped. F2b A BALLOT DIGEST MISMATCH IS A STOP (raenge-nie-uneins). prefetch_ballot.unpack_prefetch_ballot raises PrefetchBallotDigestMismatch (RuntimeError) on min != max: '#791b PREFETCH-BALLOT DIGEST MISMATCH STOP rank=%d digest=%d group_min=%d group_max=%d queue_len=%d head=[%s]'. The (x, -x) pair rides a MIN all_reduce, so every rank of the reduce holds the same min and max and raises on the SAME pass. The scheduler's void ballot, the fallback to the rank-local verdict, the streak counters and the geometric cadence line are deleted from _update_uniform_pool_budget; a missing ballot slice in the TP loop is its own '#791b PREFETCH-BALLOT LAYOUT STOP'. None stays only for the single-rank / PP-loop callers of prefetch_done_under_ballot. Group stop = the existing run_scheduler_process except -> SIGQUIT -> kill_process_tree (_update_uniform_pool_budget is called from get_next_batch_to_run with no except in between); no new collective. advance_mismatch_streak stays for the sgl-project#823 head-congruence COUNT/ORDER degradation counters (its one remaining caller, _note_tp_head_degradation) and says so. F2c THE OPENER HONOURS THE CAP TOO. jit_cold_build.capped_cold_build_deadline(base, cap) = min(base * mult, base + cap) is the ONE formula for both readers: barlink_liveness. wait_timeout_s (host seconds) and jit_cold_build.resolve_timeout_cycles (device cycles, cap expressed at the documented 2 GHz nominal clock). cap = barlink_build_window.build_cap_s(), the same SGLANG_BARLINK_BUILD_WINDOW_ CAP_S the peers honour (60 s on this rig); cap 0 = no extension on both. F2d SIBLINGS. flush_cache on rank-local is_fully_idle: judged and left, documented in place -- with the one disposal at the origin the queues are replicated again and the predicate reads the same on every rank. tp_head_congruence ORDER arm: the 'except Exception -> SOURCE_RANK_LOCAL' degradation in _apply_uniform_head_order was one block and is now the named 'sgl-project#823 HEAD-ORDER APPLY STOP' (re-raised from the cause); it propagates through _get_new_batch_prefill_raw on the same path as the sgl-project#1153/sgl-project#1157 stops. _drain_prefetch_progress's replicated-queue premise holds again by construction (no edit). Tests (hermetic, CUDA_VISIBLE_DEVICES="" PYTHONPATH=<tree>/python): NEW test/registered/unit/managers/test_1158_one_disposal_and_stop.py (2-rank fake of recv_requests over a fake broadcast wire; AST proof that is_health_check_generate_req is consulted at exactly one disposal site and pp_rank no longer appears in process_input_requests; digest-mismatch raise with both digests / none on agreement; head-order STOP) and test/registered/unit/distributed/test_1158_opener_honours_cap.py (window open, mult 40, base b: host b+cap, device b+cap cycles, non-binding cap untouched, cap 0 = no extension, one formula). INVERTED with withdrawal docstrings: test_pp_prefetch_ballot_791b.py test_a_divergent_queue_head_voids_the_ballot_loudly -> ..._stops_every_rank; test_prefetch_ballot_divergence_823.py TheLatchIsGoneAndTheRecoveryEdgeExists (5 probes turned around) and TheFallBackBehaviourIsUnchanged (None = no ballot taken). Red-first on the parent 8a7dd38 (tests run from this tree against the parent's code, never copied in): one_disposal 16 failed / 4 passed, opener 4 failed / 1 passed, 791b 1 failed / 8 passed, 823 4 failed / 16 passed; the survivors pin unchanged behaviour (wrong-width None, agreement verdict, ballot-less local verdict, disabled gate, non-binding cap). Fixed tree: 54 passed. Mutants (each restored, all red): keep the probe when busy 2F; skip the disposal at the origin 3F; raise only when group_min == 0 4F; host reader uncapped 2F; device reader uncapped 3F; except degrades to SOURCE_RANK_LOCAL 2F. Bounded suite test/registered/unit/managers (+ the new distributed file on the fixed side only), pytest -q -rf -n 4 --dist loadscope: parent 8a7dd38 (scratch worktree, new files excluded): 246 failed / 4648 passed / 18 skipped / 8 errors (280.69 s) fixed: 246 failed / 4673 passed / 18 skipped / 8 errors (274.84 s) comm over the FAILED ids: 242 unique FAILED ids on each side, 0 only-on-parent, 0 only-on-fixed (-rf lists FAILED only; the 8 errors are collection errors on both sides, not listed by id) Harness stand-in: test_collective_family_siblings_610.py BudgetHarness gains `ps = SimpleNamespace(tp_rank=...)` because the reduce now reads `self.ps.tp_rank` for the STOP line (the file's own drift guard caught it, eighth drift of that harness); 11 passed. ruff 0.15.1 on the touched files: parent 103 errors / fixed 103 errors (parity; the two new files: 0). Evidence tier: DESK-PROVEN. Boot 4 proof lines: 'sgl-project#1158 HEALTH-CHECK dropped at origin' on PP0 only; every 'sgl-project#788 PP-ADMISSION ... queue=' triple equal across ranks after each health probe; grep -c '#969C READMIT-PREFETCH.*rid= HEALTH' = 0; 'Ballot void for this pass' = 0; every 'PHASE-FLIP SEAM RE-ADMISSION' triple identical; every '#1033c CUTOVER FORWARD WARMUP begin' matched by a done or a named abort within base+60 s; 'watchdog timeout' = 0. This commit audits and completes the partial staged work of a previous sgl-project#1158 builder (r2_partial_0903.patch); the drop line now prints the gate's own reading, and the opener test collects on a tree without the fix.

…igure, C2 upstream scope Two text-only repairs handed over by the round-3 code reviewer. No behaviour change, no test re-run. The commit also carries the rest of the in-progress sgl-project#1189 slot-publish change (the test file is new, so its C1 edit cannot be committed on its own). C1 -- a boot-8 figure already retracted in production comment prose was still shipped verbatim by the test that a boot-9 agent reads when it goes red. Re-measured against /spinning/evidence-665-f1/boot_855_weg1b8_e9d1a719ac_0904_064622.log: grep -c 'running_bs=7768' -> 0 max of the running_bs= series -> 7771 grep -c ' running=7768' -> 1, at :458441, on a 'sgl-project#788 PP-ADMISSION verdict=DECLINE' line, and the max of that series grep -c 'distinct rid' -> 0 grep -c 'PP-ADMISSION verdict=ADMIT' -> 183 grep -c 'PP-ADMISSION verdict=DECLINE' -> 27127 grep -c '#969N ADMIT' -> 150 distinct rids on '#969N ADMIT' -> 16 distinct rids on 'PP-ADMISSION verdict=ADMIT'-> 19 (the 16 are a strict subset) Edits in test/registered/unit/managers/test_1189_slot_publish_lap.py at :39-46 (module docstring), :447-451 (lap-(A) assertion message) and :1036-1042 (the sgl-project#1189 OUTCOME failure message): 'running_bs 7768' becomes 'running=7768' with the :458441 anchor and the instrument named, matching the form the production comment already uses; 'with at most 3 distinct rids' is DELETED, no query reproduces it; the 183/27127 pair now names its instrument, because '#969N ADMIT' counts 150 for the same word. C2 -- the lap-(A) comment claimed a scope its own helper docstring refutes. scheduler_pp_mixin.py:5238-5240 closed with "This is COVERAGE FOR A STATE THE FORK'S OWN PURITY FEATURE INVENTED, not a break with an upstream invariant." The second clause is false. The new else also fires on the purity-INDEPENDENT idle lap: scheduler.py's 'ret = None' there is the outer else of 'if not running_batch.is_empty() and not running_batch.is_prefill_only:', not the _decode_blocked arm above it. Upstream reaches that identical lap (main:scheduler.py:2737 carries the same '# Run decode (skip for prefill-only batches)' arm, ret = None at :2742, and zero occurrences of phase_purity or phase_decode_blocked_here in that file) and preserves the previous last_mbs[slot] on it -- guard/publish/no-else three times over, at main scheduler_pp_mixin.py :148/:155, :320/:326 and :507/:514. So the fork now clears an entry upstream keeps. Edit at scheduler_pp_mixin.py:5233-5277: the true, one-step-verifiable first clause ("STRICT PHASE PURITY CREATED THIS LAP ON PURPOSE", :5225) stays untouched; only the false clause is replaced, by what _pp_record_slot_last_batch.__doc__ already says -- the wider scope is deliberate, toward the non-PP loops' unconditional 'self.last_batch = batch'. Anchors are written as grep recipes, since both files are under active edit. Checks: python -m py_compile clean on both files. ruff reports 3 findings in scheduler_pp_mixin.py (E402 :268, F841 carries_flip_arm, E741 'l'), all three present unchanged at HEAD on the same symbols; zero findings in the test file. No pytest run: both edits are text-only.

docs: update README

3ebaa6f

zhyncs merged commit 628e1fa into sgl-project:main Jul 28, 2024

zhyncs deleted the doc branch July 28, 2024 12:24

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

docs: update README (sgl-project#788)

70a41f2

cherryblo added a commit to cherryblo/sglang-project that referenced this pull request Jul 2, 2026

Merge pull request sgl-project#788 from Sugar920/testcases

f9096aa

update kimi and 35b testcases

Navigation Menu

Uh oh!

docs: update README - #788

docs: update README#788
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:doc

zhyncs commented Jul 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

zhyncs commented Jul 28, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant