Fix #857 by kaifronsdal · Pull Request #858 · sgl-project/sglang

kaifronsdal · 2024-08-01T00:29:04Z

Motivation

Modification

Error gets thrown when using select without stream mode.

Added check in _execute_select in lang/interpreter.py

if self.stream_var_event:
    self.stream_var_event[name].set()

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

Add comprehensive documentation for TurboQuant integration into SGLang: - Paper summary with full mathematical specification (algorithms, theorems, codebook formulas) - SGLang architecture mapping with verified file paths and integration points - Qwen3.5 architecture notes (hybrid DeltaNet/attention, model params, implications) - Component references mapping QJL, PolarQuant, and MLX-VLM PR sgl-project#858 to paper concepts - Implementation roadmap with phased plan and design decisions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add TurboQuant KV cache quantization as an optional feature, enabled via --kv-cache-quantization turboquant flag. No changes to default behavior. Core library (new files): - codebook.py: Max-Lloyd codebook computation for Beta distribution - rotation.py: Haar-uniform rotation matrices, Gaussian projection matrices - quant_ops.py: MSE quantize/dequantize (Algorithm 1), Prod quantize/dequantize (Algorithm 2) with bit-packing for arbitrary bit-widths - config.py: TurboQuantConfig (QuantizationConfig subclass) - kv_cache_method.py: TurboQuantKVCacheMethod for RadixAttention SGLang integration: - turboquant_pool.py: TurboQuantTokenToKVPool with compact quantized storage and shared dequant buffers for existing attention backends - Register "turboquant" in BASE_QUANTIZATION_METHODS - Pass quant_config to RadixAttention in Qwen3.5 model - Add --kv-cache-quantization, --turboquant-bits, --turboquant-seed CLI args - Hook pool creation in model_runner_kv_cache_mixin.py Design: Keys use TurboQuant_prod (b-1 bits MSE + 1-bit QJL) for unbiased inner products. Values use TurboQuant_mse (b bits MSE) for optimal reconstruction. Dequant-then-FlashAttention strategy (Phase 1). Reference: arXiv:2504.19874, confirmed against MLX-VLM PR sgl-project#858 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…t; handshake liveness #861f -- THE ROOT FIX. W37-E deadlocked on the #861e formulation: seven retracted-with-output requests counted as decode work, so the demand term stayed silent for requests that were not decoding at all. They sat in the waiting queue needing a prefill pass TP may not run. GPU 0 % for 198 s, flips frozen at 9. A retracted-unfinished request IS PREFILL WORK waiting for the pp layout. Counting it as decode work was the category error that let both sides veto while neither served. decode_work_bs() now reports genuinely resident decoding ONLY; the d4 anti-chop protection moves to bundle_is_mid_flight(), gated on COMPLETED DECODE STEPS (MIN_DECODE_STEPS_PER_PHASE = 8) -- not wall seconds (d4 produced one token per flip cycle while every seconds guard was happy) and not "requests that exist somewhere" (which is what deadlocked W37-E). RED/GREEN proven behaviourally against 02bd706 in a TEMP WORKTREE, because `git stash` is banned here (shared stack across worktrees): 02bd706: decode_work_bs=7, demand=0 -> NO EXIT fix: decode_work_bs=0, mid_flight=False -> EXIT EXISTS THE EXISTENCE STAMP. `cached_prompt_tokens_at_retract` answers an ECONOMICS question and credits the ENTIRE prompt to any request with >=1 output token, so every backlog counter read 0 for the seven wedged requests. `Req.needs_prefill_pass` is stamped at reset_for_retract and answers the different question; _admissible_prefill_tokens counts the full prompt for a stamped request instead of subtracting a credit against a prefix tree the seam already dropped. #861g -- SERVABILITY INVARIANT. Two deadlocks in one night from independently correct vetoes (sgl-project#858: strict purity x sgl-project#856 no-carry, 150 flips and ZERO decode batches; W37-E as above). Two is a class, and each cost a GPU window. managers/servability_matrix.py proves per (request state x phase) that at least one path to service survives with ALL gates evaluated together. It INDEPENDENTLY rediscovers the W37-E deadlock and blames the same three terms; after the fix the deadlocked set is empty, exactly one cell closed and none opened. Carries the sgl-project#858 pair as a second red fixture. #861h -- HANDSHAKE LIVENESS. `_await_handshake` had a deadline and no liveness check, so a dead child cost the full timeout: py-spy showed the test at 0 % CPU while its child was already <defunct>. Now 250 ms slices with is_alive() checked every slice, immediate raise carrying the child's exit code, deadline as backstop. Measured: the same test now raises in ~15 s. SELF-CAUGHT: the first cut used `.poll()`; the object is a multiprocessing.Process, so it would have AttributeError'd on the very failure path it was written for. Corrected to match execute_script's idiom -- one liveness idiom per file. AND THE DEATH ITSELF, diagnosed hermetically: the child raises "No accelerator (CUDA, XPU, HPU, NPU, MUSA, MPS) or platform plugin is available" from ServerArgs.__post_init__ -> get_device() under CVD="". The whole scripted_runtime suite requires cards. Per the CUDA test discipline it now DECLARES that in its own conftest and skips (42 skipped) rather than reporting failures that say nothing about the tree. The skip consults the same get_device() the child calls, so condition and failure cannot drift apart. Two of my own #861e pins encoded the falsified design and are corrected with the reason recorded. The Cut-2 ast check is now FUNCTION-AWARE: a coherent accessor is exactly the place that may read the raw field.

…o rules, one deleted premise `build_flip_quiescence_fn` carried two rules whose justifications both died with sgl-project#856 (2026-08-24, "the flip carries no KV"), and they pulled in opposite directions -- one let a flip through that must now be refused, the other refused one for a reason that no longer exists. Fixing either alone would leave the predicate half-governed by a deleted mechanism. SITE 1 -- the between-chunks allowance LET THROUGH what must now be refused. Written 2026-08-09 (sgl-project#631 defect O), justified verbatim in its own docstring as "exactly the state the carry moves". sgl-project#856 deleted the carry: the seam retracts residents and drops the tree, so that state is freed, not moved. The predicate was never revisited. A 6019-token prompt needs two chunks, the flip commits between them, and the re-admission restarts at prefix_lens=0 -- measured on W38-B ("prefill still chunked (allocated=4096, needs=6045)") and visible in W37-H arm A as 51 flips, 132 pp prefills, 57 tp prefills, ZERO decode rounds, zero completions. The block is added under STRICT ONLY, and that condition is the design rather than a hedge. Blocking every incomplete chunk re-creates defect O, where a flip armed FOR a prefill could not land until that prefill had finished (the 32768-token prefill that ran in the slow layout and paid two cutovers for nothing, 2026-08-09 04:23:35-54Z). Under strict batching the flip is armed for the DECODE after the drain, so waiting IS drain-and-flip. Landed in `chunk_blocks_quiescence`, NOT at the `ready_fn` call site: the helper's own docstring says its two callers must never disagree, and they drifted apart once already (2026-08-09 20:31:38Z). Both callers now pass the term -- `ready_fn` via `purity_of` behind a try/except (a purity read may never break a flip), the park site via the already-cached `self._phase_purity` (no new import, and no second purity read on the hot path). SITE 2 -- the orphan gate REFUSED for a mechanism that no longer exists. It blocked on requests "not yet merged into the resident set THE CARRY HARVESTS" (sgl-project#631 defect L). There is no harvest, and de4f541 gave `_live_reqs` the identical population (running_mbs, last_mbs, running_batch, last_batch). REMOVED, not narrowed: narrowing a gate whose entire purpose has been absorbed elsewhere leaves a third stale premise behind. RESIDUAL, WRITTEN DOWN RATHER THAN LEFT IMPLICIT (own posten). Post-sgl-project#856 there is no carry in EITHER mode, so a mid-chunk flip discards the prefill in non-strict too. We decline to block there because an unconditional block re-creates defect O. THE CORRECT NON-STRICT ANSWER IS UNKNOWN AND IS FILED, NOT SOLVED. `if strict` must not be read as evidence that the non-strict path was analysed and found sound -- it was not analysed. Three claims outlived their justifications on the night this was written; this one says what it rests on. TESTS test_quiescence_no_carry_858.py, 8 passed. Each mutation verified to kill exactly its own test and nothing else: drop the strict block -> test_strict_blocks_an_incomplete_... FAILS make the block unconditional -> test_non_strict_still_allows_... FAILS restore the orphan gate -> test_..._does_not_consult_the_carry_ FAILS Two assertions pass BEFORE and after (a completed prefill still permits the flip; non-strict unchanged) -- without them a block-everything predicate reads green, which is how three red-first files were nearly mis-reported this night. `test_live_reqs_still_covers_that_population` guards the removal's own premise: nothing else asserts that enumeration, so it is the belt rather than belt-and-braces, and it reddens if `_live_reqs` ever stops reading last_mbs/last_batch. SUITES -- all four arms hermetic (CUDA_VISIBLE_DEVICES=""), verified by `nvidia-smi --query-compute-apps` reading 0 during the runs, and run SEPARATELY per sgl-project#749 (the managers/ + distributed/ combination is order-dependent on ~50 tests and cannot gate anything). Pre-existing reds measured against the branch tip 322f331, not inherited. managers/ change 15 failed / 4334 passed / 18 skipped / 356 subtests base 15 failed / 4326 passed / 18 skipped / 356 subtests identical failure sets BY NAME, set-difference empty both ways; passed delta +8 = exactly the new tests, nothing else moved. The base arm reproduces R7's independent measurement of the same tip (4326/15) exactly. distributed/ change 95 failed / 3017 passed / 12 skipped / 1170 subtests base 95 failed / 3017 passed / 12 skipped / 1170 subtests identical BY NAME across all 95 -- 92 line-start FAILED plus 3 parametrized `FAILED(dcp_size=2/3/4)` subtest reports that a `^FAILED` pattern misses; both arms agree on all of them. No formatter was run on any file: this tree's black is older than whatever formatted it, and isort has moved a deliberately-late `# noqa: E402` import here before. The change is hand-written to the surrounding style. NOT PROVEN HERE: that sgl-project#857's acceptance follows. This removes the mechanism that made the livelock inevitable; whether a full A-B-A cycle now completes with COMPLETIONS>0 is a metal question and is not claimed.

…plied it direction-blind sgl-project#858 traded the livelock for a deadlock, and the metal proved it in 12 minutes (boot_w40_857strict_0825_1931): 15 flips all with live_slots=0, 4 completions all inside ONE TP dwell after the last flip, none across a cycle, then 258 ADMISSION-WEDGE reports, 11 queued / 0 running, no first token for 535 s. 225 OF 228 QUIESCENCE HOLDS WERE tp_to_pp -- the direction sgl-project#858's premise does not cover. THE DEFECT IS IN MY OWN JUSTIFICATION, quoted from phase_flip_runtime.py: "Under STRICT batching the flip is not armed for the pending prefill: it is armed for the DECODE that follows the drain, so waiting for prefill to finish IS drain-and-flip rather than a stall." True for pp_to_tp. FALSE for tp_to_pp, which is armed FOR the prefill. Waiting there waits for work that strict forbids in the layout holding while we wait. A direction-specific argument, implemented direction-blind. AND IT HAD NO EXIT, which is why it presented as a wedge and not a slow flip. `validate_purity_policy_pair` already makes exactly this argument for PP -- a phase that "may not decode and cannot admit prefill ... has NO exit except the bounded window" -- and PP got `--phase-policy-pp-window-s`. TP got only `--phase-policy-tp-decode-floor-s`, a MINIMUM dwell that cannot end a hold. THE ASYMMETRY IS THE MISSING EXIT. FIX, three parts: 1. `prefill_runnable_in_current_layout(direction, purity)` -- the armed direction names the layout that currently holds; tp_to_pp consults `prefill_allowed_in_tp()`. `chunk_blocks_quiescence` gains `prefill_runnable_here` and blocks only when the work it waits for can actually progress. The predicate now asks "can what I am waiting for run right now", not "is waiting usually fine". Both callers pass it, resolved from the same local import block so they cannot drift. 2. `validate_tp_exit_pair` (phase_purity.py), mirroring the PP guard and wired beside it at boot. Refuses the checkable deadlocking triple: strict purity + strict drain mode + no bounded TP residency. Parse time, where refusing is free -- the same reason the PP guard exists instead of a runtime recovery. 3. Unit tests PARAMETRISED OVER BOTH DIRECTIONS. This is the check that would have caught sgl-project#858 at desk: a direction-blind predicate satisfies every single-direction test, and sgl-project#858's eight all passed. The decisive case asks ONE incomplete prefill in BOTH directions and requires the answers to differ under strict. TESTS test_quiescence_no_carry_858.py, 8 -> 17 passed. Mutation restoring the exact sgl-project#858 defect (dropping `prefill_runnable_here` from the condition) kills ONLY test_strict_does_not_block_tp_to_pp. Green-before-and-after controls kept, plus new ones: `off` purity is not refused by the boot guard, and a declared decode-stall SLO satisfies it. FILED, NOT FIXED HERE: the ADMISSION-WEDGE recovery gate fired 69 times with 69 x exit 'headroom-sufficient' -- a MEMORY-HEADROOM actuator answering a QUIESCENCE-HOLD wedge. It measured its own quantity truthfully; free headroom really was sufficient. INDIKATOR-GESETZ: it was never shown able to report the state it gates. Giving that path a wedge-cause discriminator is a different mechanism and would widen this commit past its evidence. SUITES -- all four arms hermetic (CUDA_VISIBLE_DEVICES=""), run SEPARATELY per sgl-project#749, measured against the branch tip c2e69c2, not inherited: managers/ change 15 failed / 4343 passed base 15 failed / 4334 passed identical failure sets BY NAME, set-difference empty both ways passed delta +9 = exactly the 8 -> 17 test growth distributed/ change 95 failed / 3017 passed base 95 failed / 3017 passed identical BY NAME, set-difference empty both ways EXTRACTION DISCIPLINE, because tonight produced three ways to misread these files and the third corrupts the comparison's own input: (1) ANSI escapes before the anchor -> `^FAILED` matched 0 of 15 (2) subtest failures with their own prefix -> `SUBFAILED(dcp_size=N)`, 3 of the 95, invisible to `^FAILED` and matched by a bare `FAILED` grep only because "SUBFAILED" contains it (3) UCX teardown warnings printed AFTER pytest's summary -> `tail -1` on a 2 MB file returns library noise, so the SUMMARY side reads empty and a count gate silently passes So: strip escapes, extract the summary BY PATTERN never by position, match `^(FAILED|SUBFAILED|ERROR)`, and gate on extracted-name-count == summary-count before any set-difference is allowed to mean anything. All four arms passed that gate: 15=15, 15=15, 95=95, 95=95. No formatter run on any file. NOT PROVEN HERE: that sgl-project#857's acceptance follows. sgl-project#858 removed the mechanism that made the livelock inevitable; this removes the deadlock that replaced it. Whether a full A-B-A cycle completes with COMPLETIONS>0 is the metal question.

… it had to prevent DETERMINISTIC, 2/2, AT EXACTLY 7 BATCHES. Pin dc4895e, boots boot_943bx_dc4895e1dc_0828_000240.log and _001113.log, no CUDA error involved: scheduler.py:9286 in _get_new_batch_prefill_raw assert self.chunked_req is None AssertionError immediately after `sgl-project#798 PP-ADMISSION pass voided on slot N` and `#797d own pass voided on slot N`. THE ASYMMETRY. `_event_loop_pp_body` runs two voids and calls the second "AND THE MIRROR OF IT". They were not mirrored. sgl-project#797 (this rank retracted) sets `_pp_admission_pass_voided` in `_pp_void_retracted_pass`, BEFORE `get_next_batch_to_run`, so scheduler.py's guard refuses the pass and it builds nothing -- which is what that guard's own comment demands: "The retraction voids the pass, so the pass must build nothing at all." sgl-project#798 (this rank's UPSTREAM did not launch) set the same flag AFTER the call, so its pass ran the whole of `_get_new_batch_prefill_raw` -- advancing `chunked_req`, possibly adopting a fresh `new_chunked_req`, taking lock refs, emptying the waiting queue -- and was then unwound retroactively by `_pp_void_own_batch`. THE UNWIND CANNOT BE COMPLETED, so this is a guard and not a bigger unwind. `_get_new_batch_prefill_raw` reaches `_retract_decode_and_requeue` (the sgl-project#679 relief ladder, the #888b seat yield), which sends `AbortReq` to the tokenizer over `ipc_channels` at scheduler.py:9626-9632. A message already delivered to another process is not scheduler state and no handler can put it back. Every other unrestored item is a matter of writing more restore code; this one is not. THE CONDITION WAS KNOWABLE EARLY ALL ALONG, which is what makes the guard possible. `_pp_upstream_launched_incoming` is written by `_pp_recv_admission_decision` (scheduler_pp_mixin.py:5482), whose own docstring records the placement -- "Positioned in `_event_loop_pp_body` strictly BEFORE `get_next_batch_to_run`" -- and the call site agrees (:1959 receive, :2073 plan). One shared predicate, `pp_upstream_void_pending`, is now read at both moments, so the refusal and the forward cannot drift apart. The void is still FORWARDED from where sgl-project#798 always forwarded it. CARRIED, NOT DROPPED. Once the guard empties the slot, `self.mbs[mb_id]` is None on exactly the passes the sgl-project#798 site used to find non-empty, so two instruments would have gone quiet unnoticed: * the sgl-project#801-spin livelock streak, which raises at 512. The guard now records `_pp_upstream_void_withheld_work` (this rank HELD work and was refused anyway) so the streak counts what it always counted. It is a pre-plan superset of "derived a batch" -- the safe direction: it can make a genuine streak visible sooner, never hide one, and an idle rank still clears it. * `_pp_idle_void_suppress_log`, whose contract is that it "can never outlive this pass". Its consume sat AFTER the empty-slot early return, so it would have leaked a True into the next, unrelated sgl-project#797 void and silenced a record nothing asked to silence. Read and cleared at the top now. has_chunked_req DELETED, not left to mislead again. `add_one_req` carried it as a parameter its body never read; it only ever forwarded to `add_one_req_ignore_eos`, and upstream removed that last consumer in 8cc7726 ("Super tiny remove unused argument"). Reading it as the guard against a second chunked request cost a boot window. What actually holds that invariant is budget arithmetic, and that is now recorded at the assert -- including its measured reachability: witnesses under /spinning/evidence-665-f1/witness_951/ drive the real PrefillAdder into three states where `add_chunked_req` returns the request while leaving `rem_chunk_tokens` positive, and a mid-pass replenishment of `rem_total_tokens` then lets the loop mint a second chunked request. witness_941_d2 is the negative control and does not reproduce without it. That general case is a SEPARATE posten -- it needs no PP void and its danger direction (wedging a request mid-prefill, the sgl-project#858 shape) needs its own analysis. The assert stays: it is the honest watcher. TESTS. test_pp_upstream_void_before_formation_951.py, 11 tests, red-first: 3 fail on dc4895e and pass here. The five over-fire arms (first rank, pp_size<=1, gapped wire, healthy upstream, healthy chunked continuation) are green in BOTH states, which is what makes the red mean something -- a guard that over-fires voids every pass on PP0 and serves nothing at all. Two mutants on the carried instruments, each killed by exactly its own test: consume-after-return revives the suppress leak, emptiness-only revives the streak reset. Desk gate scripts/gate_tier2_partitioned.py, frozen both sides on this tree: BEFORE 4757 passed / 2 failed / 18 skipped, AFTER 4768 passed / 2 failed / 18 skipped. Delta +11 is exactly this commit's new tests. Identical failure set both sides (test_collective_family_siblings_610.py, 2 genuine, pre-existing at the pin); count check 2 == 2. The lane shift (wide -15, narrow -30, serial +56) is the gate's own sha256 rule demoting the four touched test modules to the serial lane. ruff parity exact per file (105/105, 3/3, 21/21, 1/1) with no finding on an added line; black run only on files that were clean at the pin. NOT BOOTED. Boot half is /spinning/gpu-arb/TICKET_951_WINDOW.md.

…arithmetic not a check WHAT DIED. window-955-boot's second boot (pin 27bcb48, boot_943bx_27bcb4884f_0828_025835.log) took `assert self.chunked_req is None` in `_get_new_batch_prefill_raw` on ALL THREE ranks at 03:03:29 -- three seconds after `PHASE-FLIP DONE pp_to_tp (epoch 1) in 9709.5 ms`, the first clean cutover this family has produced, with 9 phase=tp batches already run. The line immediately before the crash on every rank: `PHASE-FLIP armed (tp_to_pp) but NOT QUIESCENT: a chunked prefill is incomplete` -- a continuation was demonstrably resident when a second one was minted. NOT A sgl-project#951 REGRESSION, and sgl-project#951 said so itself: its comment at the assert records that the invariant is held by ARITHMETIC (a surviving continuation has normally spent all of rem_chunk_tokens, so the fresh branch computes trunc_len <= 0), that this is BREAKABLE with witnesses, and that sgl-project#951 closes only the PP instance -- "It does NOT close the general case, which needs its own posten and its own danger-direction analysis." AND ITS GREEN WAS VACUOUS. window-951 read 0/0 on this line while every batch it saw was phase=pp. Boot 2 is the first evidence the TP side is reachable at all, which is why the new suite proves reachability before asserting anything. WHY THE EXISTING CLEAR DOES NOT COVER IT. phase_flip_runtime.py:1853-1856 clears `scheduler.chunked_req` at the seam only for a request whose id() is in the RETRACTED target set. A continuation that survives the flip un-retracted -- the designed behaviour, see `chunk_blocks_quiescence` -- is never in that set. A cutover also resizes the pool, which is precisely the mid-pass replenishment witness_941_a needs to mint the second continuation. DANGER DIRECTION, and it decides where the fix goes. The two ways to restore the invariant are not symmetric: * refuse the FRESH chunked admission -- nothing of it is committed, no KV is held, no chunk has run. It waits one pass, via the requeue-for-free the admission loop already relies on. Nothing is lost, so no double prefill. It cannot starve: the resident continuation is consuming chunks, and when it finishes the fresh one is admitted. * drop the RESIDENT continuation (clear chunked_req at the cutover) -- that re-prefills a request mid-flight. That is the double prefill the standing law forbids outright, and the sgl-project#858 wedge shape besides. The resident continuation is never the one to give way. FIX. `PrefillAdder.chunked_req_outstanding`, stamped by the scheduler at the one point where residency is settled -- right after the add_chunked_req branch, which matters because the sgl-project#906 seam refusal KEEPS the continuation without calling the adder at all, so residency cannot be inferred from the adder's own calls. The two fresh-request mint sites in `add_one_req` return AddReqResult.OTHER while it is set. No new mechanism: `_add_scheduled_req`'s `carried_chunk` flag is the precedent and already names this assert while refusing for the same reason. Default False, so any adder built outside the scheduler behaves exactly as before. THE ASSERT STAYS. It is the honest watcher and has now named its own reachability twice; nothing here weakens it. TESTS. test_second_chunked_req_959.py, 7 tests, driving the REAL PrefillAdder through add_chunked_req / preempt_to_schedule / add_one_req -- adapted from /spinning/evidence-665-f1/witness_951/witness_941_a.py rather than inventing a second harness. Includes the reachability proof (the continuation really does survive with chunk budget left), a CANFAIL arm showing the unguarded adder reproduces the crash precondition (both a resident and a freshly minted continuation), that the resident one keeps its committed geometry byte for byte (kein Doppel-Prefill), and that the guard does NOT fire when nothing is resident, so the fix costs no throughput. Desk gate, /spinning/htsglang-gpu/.venv, CVD="": BEFORE (pin) serial 878 passed / 2 failed AFTER (sgl-project#958+sgl-project#959) serial 895 passed / 2 failed; wide 3701; narrow 202 +17 = exactly the 10 tests of sgl-project#958 plus these 7. Failure set IDENTICAL, both the known pre-existing test_collective_family_siblings_610.py pair. Tally gate OK on all three lanes. 88/88 across the 946/951/955/958/959 suites. No boot. Metal proof belongs to the window ticket.

…resident at all Boot 14 (cf16281) died on PP1 after 39 s: `assert self.chunked_req is None`, scheduler.py:9784 in `_get_new_batch_prefill_raw`. Third recorded fundstelle of this family after :9286 (sgl-project#951) and :9367 (sgl-project#959). ROOT, and it corrects my own attribution in the boot-14 register entry. The invariant behind that assert is held "by ARITHMETIC, not by a check" (scheduler.py's own comment). sgl-project#959 therefore gave the two `add_one_req*` mint sites an explicit `chunked_req_outstanding` check, and SKIPPED the third -- `PrefillAdder._add_scheduled_req`, the forwarded-schedule execution path -- reasoning at schedule_policy.py that it "already has its own (`carried_chunk`)". That is the guard-comment-names-the-hazard trap. `carried_chunk` answers "is THIS request the resident continuation". The invariant needs "is there a resident continuation AT ALL". It covers a request being re-announced; it does not cover a DIFFERENT named request becoming a second continuation while the first is resident. Two of three sites guarded, one not. Boot 14 is that gap on metal: the resident continuation survived `add_chunked_req`, so `chunked_req_outstanding` was True and both sibling sites correctly refused -- and this site minted anyway, on another rid the same forwarded schedule named. sgl-project#994 EXPOSED THIS, IT DID NOT CREATE IT. Boot 13 never reached the line because the sgl-project#791 geometry refusal killed every pass before a batch was built; sgl-project#994 removed that refusal. My register entry attributed the death to sgl-project#994's effect on `rem_chunk_tokens` arithmetic. That was wrong: both sibling sites were already guarded and did refuse. The arithmetic was not the hole; the missing third guard was. Withdrawn here rather than left standing. FIX: the sibling guard, at the site that lacked it. WHY A PASS REFUSAL AND NOT A REQUEST SKIP -- the danger direction, which is the whole question here. On a forwarded schedule this rank may NOT drop a named request: the upstream's hidden states for it are already on the wire, which is what this same method already raises `PPScheduleRefused` for a few lines above. Running the chunk WITHOUT announcing it is worse -- the continuation would be untracked and re-prefilled next pass, the double prefill the standing law forbids. So the disposal is the one this path already owns: refuse the PASS by name, let sgl-project#791/sgl-project#797 void and re-derive. It cannot starve: the resident continuation is consuming chunks, and when it finishes `chunked_req` is None and the schedule is executable. This is the same direction sgl-project#959 chose ("the resident continuation is never the one to give way; the fresh admission is") -- not the sgl-project#858 mid-prefill wedge, which would be clearing `scheduler.chunked_req` instead. FUTURE CHECK: `grep -c 'if self.chunked_req_outstanding:'` over schedule_policy.py is now 3 and equals the number of `new_chunked_req` writers. A fourth writer that skips it is visible as an inequality rather than as a boot death. Execution proof (speed mode, one instrument per link): `note_second_continuation_refused(req, "_add_scheduled_req")` counts it and the refusal text names itself, so boot 15 measures whether the guard fires at all -- absence of the assert alone would be green-by-absence. Evidence: desk. py_compile; local-import dominance and guard-before-raise- before-mint verified by source order in the loaded module; guarded-site count 3 == mint-site count 3. Belegstufe: DESK-BEWIESEN.

…n, die es nicht gibt == DER BEFUND (gemessen, nicht vermutet) == Boot_855_704bgroup2: 57 Prefill-Batches, NULL Decode-Batches, eine 6-Token-Probe -- und 42 Flips. Die Gruende standen im Log, es hatte sie nur nie jemand gezaehlt (`PHASE-POLICY arming`, scheduler.py:13627 -- die Zeile existierte bereits, ich musste kein Instrument bauen): 21x pp_to_tp: "idle Ns >= Ns, returning to the decode resting layout" 21x tp_to_pp: "pending prefill N tok > 0 (purity: prefill cannot run in tp, nothing decoding)" 21 Rundtrips auf einer LEEREN Kiste. Die zwei Regeln zertifizierten einander: diese hier flippte fuer nichts nach TP, die Prefill-Arbeit der Health-Probe konnte dort nicht laufen, die tp-waertige Regel flippte sofort zurueck. Bei gemessenen 8,07 s je Rundtrip (4,016 + 4,050 s, RECONCILED `flips`-Tabelle) sind das ~169 s Naht fuer nichts. == DIE WURZEL == `idle` ist an dieser Stelle der STARKE Leere-Term (`decode_work_bs() == 0 AND not work_exists()`) -- es ist also BEWIESEN, dass weder ein residenter Decode-Bundle noch ein Prefill irgendwo geschuldet ist. Die Regel flippte trotzdem, um sich in der Ruhe-Schicht zu POSITIONIEREN. Diese Praemisse ist fuer eine KALTE Ankunft verkehrt herum, und eine andere bekommt eine leere Kiste nicht: ein neuer Request braucht ZUERST einen Prefill-Pass, und Prefill laeuft unter strict purity nicht in TP. In TP zu ruhen heisst, die naechste Ankunft zahlt tp_to_pp zum Prefillen UND pp_to_tp zum Dekodieren. In PP zu ruhen ist fuer genau diese Ankunft strikt besser. KEINE Dwell-Erhoehung: ein laengerer Dwell macht die Schleife langsamer, nicht abwesend, und bei NULL Arbeit ist kein Timer lang genug, um den Handel positiv zu machen (sgl-project#819-Preisfrage, die ein Timer per Konstruktion nicht beantwortet). == FEHLERKLASSEN-PASSENDER CHECK (Direktaufruf von decide(), 3 Faelle) == Klasse a) Format-Spec auf einem Conditional im f-String: beide Zweige ausgefuehrt -> "idle 0.0s" / "idle 11.5s". AST + Import gruen. Klasse b) falscher Zweig / Verhungern: CASE1 idle+leer in PP -> direction=None, Reason traegt sgl-project#1011 PASS CASE2 Prefill geschuldet -> "prefilling in pp (50000 tok pending)" PASS (nimmt den idle-Zweig NICHT) CASE3 Decode-Arbeit da -> direction=pp_to_tp via DRAINED PASS <- das ist der sgl-project#858/sgl-project#1006-Verhungerungsbeweis: mit echter Arbeit feuert der Flip weiterhin. Belegt, nicht argumentiert. == ZWEISEITIGE ABNAHME AUF METALL (boot_855_1011idle) == (i) LEERLAUF, 11 min 39 s, Deadman aktiv, 7 health_generate-Proben bedient: Flips 0 | PHASE-POLICY armings 0 | sgl-project#1011-Refusals 77 | health 200 Vorher auf vergleichbarem Leerlauf: 42 Flips. JETZT NULL. (ii) LAST, conc=4 x 200 tok: 4/4 fertig, 0 Fehler, TTFT min 0,30 / med 0,31 / MAX 0,31 s -- keine Verhungerung Completion 11,38-11,85 s Flips: GENAU 1, armiert via DRAINED "N req decoding" -- der Flip feuert, wenn echte Arbeit wartet. == WAS DIESER SCHNITT NICHT TUT -- und meine eigene frueher zu grosse Behauptung == Ich hatte A als groessten Hebel gegen die 68-%-Flip-Steuer benannt. GEMESSEN FALSCH, und die Korrektur gehoert hierher: conc=4, 600 tok: 88,64 tok/s (vorher 90,47) -- unveraendert in der Streuung bs1, 600 tok: 25,09 tok/s (vorher 25,37) -- unveraendert Unter DAUERLAST sind die Flips nachfrage-getrieben (bs1 zahlt weiter ~2 Flips je Request: 12 Flips auf 6 Requests), und genau das ist die 68-%-Steuer. Der Leerlauf-Befund bleibt gueltig und der Schnitt beseitigt reine Verschwendung ohne Preis -- aber er kauft die Last-Steuer NICHT zurueck. BELEG-STUFE: BOOT-BEWIESEN fuer beide Abnahmehaelften (Zahlen und n oben). Der Drain-and-Flip-Kontrakt (sgl-project#925) ist unberuehrt; nur der Leerlauf-Zweig mit `rest_phase == PHASE_TP` aendert sein Verdikt.

…les that could hold it Two corrections, both read off this message's own output on boot_855_1076filebacked (15:32:40): "rank 1 has held slot 0 occupied (rids=NONE) for 90.0s". WRONG SLOT. The throttle arm is taken when ANY slot is occupied (`any(b is not None for b in self.mbs)`), but the message reported `self.mbs[mb_id]` -- whichever slot the loop happened to be on. Those are different slots, so `rids=NONE` read as "no request is stuck" when it meant "wrong key". Every occupied slot is now listed with its own rids, and the loop's slot is reported separately so the two can never be conflated again. INVISIBLE THROTTLES. `_unresolved_rounds`, `_terminator_spent` and `_offer_streak` each have exactly ONE clear site: the `elif entry.admitted:` arm of `record_return_trip`, which runs only when the ring TURNS, while their increments are deliberately lap-free ("the INCREMENT must survive a broken ring, the CLEAR only has to work when the ring turns"). A stopped ring therefore cannot clear the states that throttle the next attempt -- gate and clearer in one cycle, the sgl-project#955 / sgl-project#888-D2 / sgl-project#858 / sgl-project#748 family, and a self-sustaining wedge rather than a slow peer. That hypothesis could NOT be tested against the 1076 wedge, and the reason is the finding: none of the three is logged anywhere in the tree. Grepping them returned zero because they have no emitter -- absence of an INSTRUMENT, not absence of the state, and quoting that zero as an exclusion would have been the benign-zero class this strand has paid for four times today. They are now printed at the one moment the question is live, with the reading rule beside them: non-zero on a stopped ring = the clearer sits behind the gate; all zero = the throttles are innocent and the stall is upstream of admission. Desk: matched check drives the real method with an occupied slot 1 while the loop sits on slot 0 -- it names slot1, keeps "loop was on slot 0" separate, and prints all three throttle values.

Fix select w/o streaming bug

48e85c7

Ying1123 merged commit 0c0c813 into sgl-project:main Aug 1, 2024

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Fix sgl-project#857 (sgl-project#858)

0829cce

McZyWu added a commit to McZyWu/sgl-sglang that referenced this pull request Jul 2, 2026

add ci permission (sgl-project#858)

77425e1

Navigation Menu

Uh oh!

Fix #857 - #858

Fix #857#858
Ying1123 merged 1 commit into
sgl-project:mainfrom
kaifronsdal:main

kaifronsdal commented Aug 1, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

kaifronsdal commented Aug 1, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants