Use min new token ratio at start by hnyls2002 · Pull Request #701 · sgl-project/sglang

hnyls2002 · 2024-07-23T10:24:42Z

No description provided.

…le, falsifier PRIO. Deeper root beneath the sgl-project#698 wedge. Desk slices only; the wiring that changes serving behaviour is deliberately NOT in this commit. Root cause, one sentence: chunked prefill bounds the COMPUTE per step, not the KV COMMITMENT, and admission was reading the compute bound as if it were a memory decision. schedule_policy.py:1389-1407 charges the budget trunc_len -- one 512-token chunk -- while admitting a request whose real commitment is its entire remaining length. A 327,680-token request is therefore admitted on a 512-token affordability check, a 640x under-charge. The non-chunked branch directly above charges req.extend_range.length, the real figure; only the chunked path substitutes the chunk for the commitment. That explains the specimen without needing a second actor, which the specimen requires: ONE request (new-seq 1, new-token 512, cached 0) drove usage 0.95 -> 1.00 with no retract, abort or finish. Its own prefix locks as it grows, and a locked chain cannot be evicted to fund its own growth -- which is why the sgl-project#698 relief correctly reports "freed 0". sgl-project#698 made the failure legible; it could not fix it. Design decisions (DESIGN_701_chunked_admission.md): (a) Fund the full remaining length at admission; spill/retract of the request's own prefix comes later and is NOT a prerequisite. Funding is a correctness fix that cannot regress into a wrong answer, and it does not foreclose spill: a future spill capability simply raises the fundable total and the SAME rule then admits more. Stated honestly, this makes near-capacity requests slower -- some that are admitted today will defer. Trading throughput for not-deadlocking is the correct trade, and "freed 0" is the evidence that today's behaviour is a stall rather than a throughput win. (b) Head-of-line: refuse LOUDLY when the remaining length exceeds total pool capacity (can never fit at any future time), defer when it exceeds free + unlocked-evictable, admit otherwise. All three derived from pool arithmetic. There is deliberately no "90 percent of pool" style constant. (c) sgl-project#631 defect O expressed as one counting truth, effective_running_bs, so the ladder / delayer / idle-flip consumers converge on it instead of each re-deriving that a resident-but-batchless request means idle. planner/chunked_admission.py carries no rig threshold, no tuned fraction and no hardware or model name, per the binding generality clause. It has NO chunk-size parameter at all: the chunk is exactly what the old code substituted for the commitment, so the substitution is made unrepresentable, and a test asserts that passing chunk_tokens raises. Falsifier as specified by the ticket: a request whose remaining length exceeds free + unlocked-evictable must be refused or deferred, never admitted. Plus the specimen as a regression, and a generality test pinning that a request at 99 percent of a large pool admits while one at 101 percent of a small pool refuses, under the same function with no constant between them. Tests: 9, red first. Planner suite 2620 passed, 2 failed -- both the pre-existing test_rejected_evidence_pins failures verified earlier against clean integration/r2. ruff clean. NOT INCLUDED, and requires F4-r4 coordination before any deploy: wiring the rule into schedule_policy.py's chunked branch, which changes admission on the serving line.

…h reserve is the blocker Review-gate item 2. Every term from config or instruments, nothing fitted. KV term is byte-exact from config, not fitted: kv_cache_dtype fp8_e4m3 (1 byte) x num_key_value_heads 4 x head_dim 256 = 1024 B per token per ATTENTION layer for K, 2048 B for K+V. The boot log's per-rank K sizes at 436,766 tokens match on all three ranks (2.92 / 2.08 / 1.67 GiB predicted and logged). This falsifies the 4096 B bf16 constant in Slot-3's doc directly. Mamba residency derived from the allocation sites (memory_pool.py:583-608, :655-665, :693-705) with max_mamba_cache_size 12 (13 slots), max_running_requests 4 (5 spec slots), speculative_num_draft_tokens 4, temporal_state_shape (48,128,128), conv_dim 10240, win 3, bf16: temporal_state 13 x 48x128x128 x 2B = 19.50 MiB/GDN-layer conv_state 13 x 10240x3 x 2B = 0.762 intermediate_ssm_state_cache 5 x 4 x 48x128x128x2B = 30.00 intermediate_conv_window_cache 5 x 10240x6 x 2B = 0.586 total = 50.85 MiB/GDN-layer The gate's 19.5 MB/GDN-layer is the temporal component ALONE. The full residency is 50.85, dominated by the SPECULATIVE intermediate cache at 30.00, so a model charging only 19.5 under-charges GDN layers by 2.6x. Forward validation against the live [28,20,16] boot chain (avail-after-weights minus avail-after-pool): predicted 6.874 / 4.910 / 3.928 GiB against measured 6.91 / 4.95 / 3.97 -- under 1.1 percent on every rank. The residual is nearly CONSTANT at ~40 MiB rather than scaling with layers or attention, which is the evidence that the per-layer terms are complete: a missing per-layer term would have shown a slope. CORRECTION to my own blocker claim. I previously named the +-7 percent unbooted-floor uncertainty as what blocks bootable predictions. Wrong in emphasis. Retro-prediction runs the equation backwards and needs everything the sizer sets aside before the pool; every term is now config-derived except the GRAPH RESERVE, which the chain puts at 6,372 / 2,650 / 2,377 MiB per rank after subtracting the measured floors. Next to that the floor uncertainty is second-order. The fix is not to model the graph reserve independently but to have the pool solve consume the sizer's own reserve terms, as it must already consume the sgl-project#676 floor. Structural finding encoded (gate 3.5): no cut keeping rank2 = layers 48-63 can beat the incumbent pool. Rank2 is byte-identical across both boots (weights 10.40 GiB in each log) and binds at 436,766, a hard min-rule ceiling. Pool gains require shrinking rank2's ATTENTION count or Part B decoupling, not rank0/rank1 rebalancing. Also on record: the incumbent's binder is PP2 per the boot log (PP1 cap 463,406, PP2 436,766). My rev5 calibration assigned it to rank1 and solved a free constant from that assumption; this derivation needs no such fit and supersedes it. Docs only. Items 1 (canonical pp_cut convergence) and 3 (sgl-project#701 slice-3 rework) not started.

…take the four gate corrections The review gate adjudicated the attention-only divisor in this branch's favour -- white-box at the allocator, byte-exact against the boot log's K sizes -- and found four defects of mine. This lands the convergence and all four. ## Convergence: one canonical solver Merged c5afff7 (Slot-2 rev5), which adopts the attention-only rule and adds the per-layout arming floor and mamba terms. My duplicate FamilyPoolModel / stage_family_capacities / family_phase_pool are DELETED; rev5's PhasePoolModel is the only pool model. Slot-2 leads the solver; this branch owns layout_ladder, ladder_controller and the arena model, which now consume rev5 instead of reimplementing it. Two additions to pp_cut.py that do not duplicate rev5: - kv_mib_per_token_per_attn_layer_from_config() + kv_dtype_width_bytes() - decoupled_phase_pool(), the part B projection, rebased on PhasePoolModel ## E1: the KV cell was FITTED, and was wrong by 2x DESIGN_704 read the cell as 4096 B (bf16). The shipped config is fp8_e4m3 and the cell is 2048 B for K+V, 1024 B for K. Fitting it against an observed pool is what produced both the dtype error and the bogus "0.83 of observed" fudge -- they cancelled into something that looked calibrated. The cell is now consumed from config (2 x kv_heads x head_dim x dtype_width). Unknown dtypes raise rather than default, because a wrong default is a silent 2x on every pool number; 'auto' raises too, since it names no width. ## E2/E3: the binding rank, and the retraction I claimed the incumbent binds on rank1. The boot log says PP2 binds at 436,766. The functional form was right; my free-bytes vector was not, and that produced the retraction: [33,15,16] leaves rank2 holding layers 48-63 -- byte-identical to the incumbent's rank2, measured cap 436,766. Under the min-rule no cut keeping that rank2 can exceed it, so my claimed 457,604 over-predicted an UNCHANGED rank's measured capacity by 4.8%. Expected actual ~387k, about -11%. The arm is withdrawn. [32,16,16] had already failed its gate on metal at 416,796, the same error class. The "discriminating experiment" justification was void: the divisor was settled white-box, so the window would have bought nothing. What replaces it is stronger than the arm I lost: NO cut that keeps rank2 = layers 48-63 can beat the incumbent pool. Pool-positive rungs must shrink rank2's attention count, or wait for part B. That is structural, not a number needing re-measurement. ## E4: GDN residency was under-charged 2.6x, and it falsified a claim The full per-GDN-layer figure is 50.85 MiB, not 19.5: temporal_state 19.5 + speculative intermediate_ssm_state_cache 30.0 (5 spec slots x 4 draft tokens) + conv_state 0.762 + intermediate_conv_window 0.586. This falsified a structural claim of mine. §3.7 said two rungs sharing an attention profile price EXACTLY the same, so the deeper strictly dominates. That held only because the 30 MiB speculative term was missing. Corrected, such rungs differ by their GDN residency (~3,250 tokens at 8 attention layers): a weak real trade, not a domination. test_under_an_arena_equal_attention_profiles _have_equal_pool failed the moment the constant landed and is replaced by test_under_an_arena_same_attention_profile_differs_only_by_gdn_residency, which asserts the gap is explicable by GDN residency and nothing else. ## Part B defects D4/D5/D6 - D4: the byte-identity gate as written could never pass. "decoupled vs coupled" is A-vs-B -- an LSE merge sums in a different float order than monolithic attention -- so it would fail forever on correct code and be waived. Respecified: (1) decoupled-vs-decoupled determinism, byte-identical across runs and boots, CPU-sampled inputs; (2) agreement with the coupled reference within a tolerance fixed before the run. Gate 1 is never waived. - D5: "TP identity wins" was unconditional and cannot be. The measured TP vector puts ~43.8% of KV rows on rank0, the rank with the least free bytes at exactly the deep rungs part B exists to unlock, so prize 1 and prize 3 can contradict. TP identity now carries a per-rung feasibility bound, and the rung yields before the vector does. - D6: committing rung changes only at a fully-quiescent boundary can starve. 640 chunks per max-length prompt with overlapping admissions means the boundary may never arrive -- and ascent is needed precisely while chunked prefills are active. That is the ladder's own sgl-project#701-shaped wedge. Paired with an admission hold plus bounded drain and an urgency-derived deadline; asymmetric, since only a safety-seeking ascent may hold admission. ## Model discipline LadderInputs now REQUIRES an arming_floor_for(counts) provider and refuses construction without one -- a constant floor is the E3 error. Rev5's own docstring records the known gap: the sgl-project#676 solver derives the floor from a measured seam draw, so an unbooted cut has no solved floor, leaving ~±500 MiB (~±32,000 tokens, ~7%) of uncertainty on every unbooted rung. No rung's predicted pool is a boot gate on its own; the retro-prediction gate against all four measured points (434,878 / 435,822 / 436,766 / 416,796) comes first, and Slot-2 owns it. The withdrawn headline table is deleted rather than patched. Re-deriving those numbers before the retro gate passes would repeat the mistake. ## Test results 63 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean. - test_kv_cell_from_config_704.py (7, replaces test_pp_cut_family_pool_704.py): reproduces the logged K sizes 2.92/2.08/1.67 GB from config alone with zero free parameters; pins that a bf16 reading misses the log by exactly 2x; refuses unknown dtypes and 'auto'; keeps the 0.47-bytes-per-element dimensional record that settled the dispute. - test_layout_ladder_704.py (25) and test_ladder_controller_704.py (12): ported to rev5's API. test_descend_only_happens_at_low_fill now states its thresholds RELATIVE to the solved ladder -- its absolute token constants silently encoded the KV dtype and became meaningless when the cell was corrected. - rev5's test_pp_cut_phase_pool_702.py + test_pp_cut_prefill_speed_702.py (19): green across the merge. The rig fixtures are labelled STRUCTURAL, NOT CALIBRATED PREDICTORS: they pin monotonicity, hysteresis and arena residency, which are invariant to the free vector. No pool value in them is bootable.

…the real holes The review gate BLOCKED slice-3 as premised. I verified both of its central claims in-code before reworking, and both hold: * schedule_policy.py:1464 already gates the FULL lifetime (total_tokens >= self.rem_total_tokens -> NO_TOKEN). There is no missing full-length gate at first admission, so my "admitted on a 512-token affordability check, a 640x under-charge" story is FALSE and is retracted in the design doc, the module docstring and both test files. * The site I cited, :1389-1407, is the ignore_eos branch, reachable only with ignore_eos AND tree_cache.disable. The serving line runs radix ON, so the specimen went through the MAIN branch at :1569-1610. I cited a path the specimen could not have taken. Defect 1 (cross-pass reservation) -- the actual deadlock channel, now closed. PrefillAdder is rebuilt each pass and reserves only remaining DECODE, and only for requests present in running_batch.reqs -- which a resident-but-batchless chunked request need not be (sgl-project#631 defect O biting the accounting itself). A live chunked request's remaining PREFILL was therefore represented nowhere in later passes, so later admissions spent its committed future and the deadlock returned with two actors. ChunkedCommitmentLedger carries the commitment across passes (commit / spend per chunk / release on finish-abort-retract, keyed by request id rather than batch membership), and effective_rem_total_tokens is what a later pass must spend against. Double-commit and overspend are refused rather than silently absorbed. Defect 2 (paper-evictable overcount) -- the likely specimen mechanism. rem_total_tokens counts full_evictable_size() while the allocator recovers only mamba-recoverable bytes, stated in-code at :734-737: the gate passes on paper and relief later frees 0, and the chain need not be locked, only mamba-coupled. PoolState.evictable_unlocked_tokens is REMOVED, not renamed, so a stale caller fails loudly; the field is now recoverable_evictable_tokens and callers must pass min(full_evictable, mamba_recoverable). Defect 3 (forever-defer wedge). refuse now fires at the ACHIEVABLE ceiling (capacity minus permanent reserves), not raw capacity. The band between the two deferred on every pass forever, and since a non-CONTINUE verdict breaks the FCFS loop the whole queue wedged behind it with no usage-1.00 tell -- the same syndrome this ticket exists to fix. Defer now carries aging telemetry so a wedge is observable. Defect 5 (defer x flip). deferred_head_blocks_idle_flip: a deferred head is pending work, not idleness, so an idle/flip detector cannot park an instance with a blocked queue. Defect 4 (sites) and the falsifier gap. The gate's three integration falsifiers now run against a REAL PrefillAdder with only the pool and tree as doubles, in test/registered/unit/managers/test_chunked_commitment_701.py, so the cross-pass tests fail against scheduler behaviour rather than against a module's absence -- which the gate correctly said the original nine could not do. Tests: 24 total (13 new integration + 11 reworked arithmetic), red first. managers + planner suites: 4730 passed, 2 failed -- both the pre-existing test_rejected_evidence_pins failures verified earlier against clean integration/r2; managers is at zero. ruff clean. Still NOT wired into schedule_policy.py: that is the behaviour change and needs F4-r4 coordination. The sibling inventory it must cover is :1569-1610 (main), :1388-1407 (ignore_eos), :1230-1276 (chunk continuation, commitment-blind) and :1124-1141 (dLLM analog).

…ady for the boot Derives nothing. Reads the four instrumented terms and refuses what it cannot read. That discipline is the session's most expensive lesson: three external re-derivations of the sizer's arithmetic missed the measured boot by +20, -3.8 and -12 percent, because the reserve tracks per-rank CUDA-graph capture and config cannot see it. RankInstruments carries one PP stage as emitted numbers: budget, the three budget posts, rest, mamba allocated, available_bytes, cell_size, tokens, the per-layout arming floor and the layer split. verify_sizing_chain asserts the sizer's own identity, rest = budget - sum(posts). It reconciles on all three live stages within 0.9 MiB against a 3 MiB tolerance set by the posts' three-decimal-GiB rounding. This is the one link validatable without a new boot, and it pins that the emission is COMPLETE: an unemitted post would surface here as a residue. recover_reserve_mib recovers the reserve as rest - available_bytes rather than modelling it: 8,848 / 3,818 / 5,164 MiB on the live boot, a 2.3x spread. Two traps encoded as behaviour rather than comments: * mamba_charge_mib returns the ALLOCATION, not the budget post. The post under-charges by a constant 0.852 on every rank, so a solve fed from posts carries ~150 MiB/rank of systematic optimism -- which was the original plan for this field before the discrepancy was resolved. * the arming floor is NOT subtracted separately. It is held back after the profiler, so it already sits inside the recovered reserve; charging it again would understate every pool by ~2 GiB. A test pins that the floor fits inside the reserve and that the prediction still lands on the boot. predict_tokens_for_cut takes no default reserve and raises when it is missing, because the reserve does not transfer between layouts. An unbooted cut is an extrapolation and the call site has to say so. world_pool_tokens applies the PP min-rule and reports the binding stage. Retro-prediction status: the incumbent reproduces from instruments alone (436,766, chain reconciled). The remaining three boots close the four-boot gate the moment a boot carrying 2a6305d emits available_bytes directly -- until then that field is pinned from tokens x cell_size, which the instrument will confirm or refute rather than being assumed correct. Tests: 10, red first. Planner suite 2643 passed, 2 failed -- the pre-existing test_rejected_evidence_pins pair. ruff clean. Prior queue item for the record: the sgl-project#701 slice-3 rework landed earlier as 845ac92 (five gate holes, 24 tests, cross-pass falsifiers against a real PrefillAdder).

…gl-project#701 ## The gate, respecified so it can actually pass DESIGN_704 originally demanded byte-identity "decoupled vs coupled". That is A-vs-B, not A-vs-A: an LSE merge sums partials in a different floating-point order than monolithic attention, so bit-exact agreement is not a property correct code has. As written the gate would fail forever on a correct implementation, and the predictable outcome is that someone waives it -- worse than having no gate. planner/lse_merge_gate.py harnesses both halves: GATE 1, DETERMINISM, byte-identical A-vs-A, never waivable. The same inputs merged twice must be bit-identical. This catches the most likely silent defect in a distributed merge -- folding partials in ARRIVAL order rather than rank order, which yields a different rounding every run and is invisible in any single run. No tolerance is applied, because a tolerance would hide exactly what the gate is for. GATE 2, AGREEMENT with the coupled reference within a tolerance fixed BEFORE the run. The tolerances are arguments with no defaults on purpose: a tolerance chosen after seeing the numbers is not a gate. merge_partials() is the CONTRACT the GPU path must satisfy, written to match layers/dcp/comm.py:228-262 (cp_lse_ag_out_ar_mha_uneven): all-gather every rank's LSE, reduce with one logsumexp over the stacked axis, so merge order is RANK order fixed by the collective and never arrival order. Inputs are sampled on CPU by construction, not by convention: torch.randn on-GPU is not architecture-identical across the 3080s and the 5090, and a harness that seeded on device would make gate 1 fail for a reason that has nothing to do with the merge. Deliberate asymmetry: given rank_order the merge reorders before reducing; WITHOUT it, list order IS taken as rank order. A caller that shuffles and stays silent has a bug, and the harness must not launder it into a plausible answer. ## The D6 admission hold shares a failure mode with a live bug Recorded as a hard precondition. The sgl-project#701/sgl-project#698 chunked-prefill admission deadlock is currently the dominant live defect -- #running-req: 0 on 90.6% of prefill rounds, zero completions -- and it is the SAME wedge D6 describes, already happening for another reason. An admission hold dropped into an admitter that is already starving would deepen the deadlock rather than bound a drain, and would then be indistinguishable from it in a log. So the hold ships GATED: it may fire only when admission is demonstrably live, and is disabled outright until sgl-project#701 lands. My mechanism and an existing bug share a failure mode, and the ordering between them is not optional. ## Instrumentation warnings carried into the design cache_hit_rate reports 0.0 despite real hits (separate filed bug) -- count hits from log lines and token counts instead; a gate written against that counter would pass or fail for reasons unrelated to what it measures. And acceptance of the "real cache hit across flip AND reboot" kind is unfalsifiable until sgl-project#701 is fixed, because the cache is STARVED rather than broken: such a test would fail for the wrong reason and must not consume a boot window. ## Test results 138 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean. test_lse_merge_gate_704b.py (9, new): bit-identity across repeated runs; inputs CPU-sampled and seed-reproducible; merge order is rank order not arrival order; a silent shuffle is caught rather than tolerated; assert_deterministic carries a CAN-FAIL proof against a deliberately flaky merge; one partial returns unchanged; the sharded merge reproduces a monolithic softmax to 1e-10 on float64; the agreement report gates on a pre-fixed tolerance; shape mismatches refused.

…locked chain The BOTH-BLOCKED relief reported freed==0 as one thing. It is two. evict_from_tree_cache returns 0 either because it ran and could not reach anything -- a pool held behind a frontier it cannot peel -- or because it was SKIPPED, since `avail >= num_tokens` already and there was nothing to do (the bare `return 0` at the tail of its standard-allocator branch). The live 21:44:37 specimen was the second kind and the log called it the first: asked the tree cache for 512 rows, it freed 0; 139507 rows now reachable. Eviction delivered NOTHING ... the pool is held by something the frontier cannot reach 139507 rows were available against 512 wanted. Nothing was stuck; eviction was never needed. The message would have sent the next reader hunting an in-flight chunked request's protected prefix that was not there -- the counter-vs-actuator mistake this routine exists to catch, committed by the instrument itself. The relief now measures availability on the same side of the call the actuator decides from, and says which state it is in. The benign branch additionally names where to look instead: KV is not the binding resource, so the state-slot bound (mamba/GDN slots) is the candidate -- a request needs a slot even when the pool is plentiful. Tests are BEHAVIOURAL, not source inspection: they drive the method with the actuator and availability stubbed, so an edit that collapses the branches fails. test_zero_with_sufficient_avail_is_reported_as_SKIPPED test_zero_with_insufficient_avail_still_escalates ...... can-fail: the pathological reading must survive; an always-SKIPPED fix fails here test_nonzero_freed_reports_the_remedy_ran test_the_two_zero_branches_differ_in_DIAGNOSIS_not_just_in_numbers test_availability_is_quoted_so_the_claim_is_checkable CAN-FAIL PROVEN BY MUTATION, and the mutation corrected the test suite itself: collapsing the branches first failed only 1 of 5, because the "distinguishable" test compared raw strings and the quoted availability differed (139507 vs 0) even though the diagnosis was identical -- it passed for the wrong reason. It now compares digit-stripped text, and the same mutant fails 2 of 5. Reverted and re-verified green after each run. -> 16 passed across the 701 + both-blocked-routing + post-evict-rows suites. ruff check + format clean.

…-layout divisor pp_max_micro_batch_size auto-computes as max_running_requests // pp_size. That is correct for classic PP: the stages run micro-batches of one batch, so each may hold only its share. It is wrong under the phase flip, because DECODE never runs in the PP layout -- it runs in the TP layout, which has no pipeline to divide by. The prefill layout's divisor was throttling the decode phase. On this deployment the effect is total: max_running_requests=4, pp_size=3, so the default was max(4 // 3, 1) = ONE. MEASURED BEFORE THE FIX, under a sustained depth-5 load for 90 s: #running-req over 1,813 prefill rounds: 0 -> 973, 1 -> 792, 2 -> 48 66 decode batches, 9 completions, 0 BOTH-BLOCKED events Read that carefully, because it corrects the framing this task started with: NOTHING WAS DEADLOCKED. Requests completed, the box flipped, eviction was never defeated. The scheduler was faithfully obeying a cap of one against a configured ceiling of four. The "90.6% of prefill rounds with #running-req: 0" symptom is under-parallelism, not a wedge -- and #running-req 0 on a PREFILL line is anyway normal when nothing is decoding, which is most rounds when only one request may decode at a time. The flip branch returns the full max_running_requests. It is not unbounded: get_num_allocatable_reqs still mins it against the admission limiter, the request-slot pool, and the mamba/GDN state headroom, so the state pool stays the real ceiling. This only stops a divisor from pre-empting all three. Extracted to a pure helper so the rule is testable without booting a scheduler. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): test_the_live_config_no_longer_caps_at_one ... 4/3/flip -> 4, was 1 test_classic_pp_is_unchanged ................. CAN-FAIL: the division must survive where it is correct ((4,3)->1, (12,3)->4, (8,2)->4, (7,2)->3). A fix that simply stopped dividing passes the first test and fails this. test_never_returns_below_one, test_pp_size_zero_does_not_divide_by_zero test_the_scheduler_uses_the_helper ........... wiring pin test_the_call_site_is_actually_reached_from_init -> 33 passed, 12 subtests, across the 701 + 698 + 677 scheduler suites. ruff clean in both changed regions; the file's other findings pre-exist. The wiring pin earned its place immediately: its first version asserted the call lived in __init__ and went RED, because the call site is init_model_worker. That is the same placement trap that broke init_parked_decode_set in sgl-project#677 -- pinning the wrong method is how a fix gets believed without ever running. The second pin asserts __init__ still calls that method, so the first cannot be satisfied by dead code. Not yet validated on metal: the next boot re-runs the depth-5 load and the claim is that decode concurrency rises above 2.

@1

…ot the bottleneck a764308 predicted that removing the pp_size divisor would raise decode concurrency above 2. Measured on metal at f2003bd, same depth-5 harness, same 90 s, and read on the SAME instrument in both arms (decode-batch lines): pre-fix (cap = max(4 // 3, 1) = 1): 54 rounds @1, 12 @2, 3 @4 -> max 4 post-fix (cap = 4): 30 rounds @1, 15 @2, 3 @3 -> max 3 Decode concurrency ALREADY reached 4 with the cap at 1, and reached 3 with the cap at 4. The prediction is refuted. Completions moved 9 -> 10, which is noise at this sample size, and BOTH-BLOCKED stayed at 0 in both arms. A MEASUREMENT ERROR OF MINE ALMOST HID THIS. The first pass compared #running-req on PREFILL lines before against DECODE lines after -- different instruments, and the mismatch flattered the change ("concurrency now reaches 3, up from 2"). The pre-fix decode distribution had to be recovered from the rotated log to make the arms comparable. Changing the instrument between arms is how a null result gets published as a win. THE CHANGE IS KEPT, but on correctness grounds only, and its claim is downgraded accordingly: dividing a TP-layout decode cap by the PREFILL layout's stage count is wrong regardless of whether it currently binds, and get_num_allocatable_reqs still mins against the limiter, the request-slot pool and the mamba/GDN headroom. It is no longer offered as a throughput fix. WHERE THE BOTTLENECK ACTUALLY IS, from the same run: 10 requests x ~19k prompt tokens in 90 s is ~2.1k tok/s of prefill, against 5,938 tok/s measured on this rig with a single long prefill. The workload is prefill-dominated (19k in, 300 out), so decode concurrency is close to irrelevant to its wall time. The next sgl-project#701 probe belongs on prefill throughput under concurrency, not on the admission cap.

The relief compares availability against `want` = chunked_prefill_size. That is an arbitrary chunk, not what the blocked work needs, so `avail >= 512` does not license the conclusion "KV is not the binding resource". The first version of this branch drew exactly that conclusion, and the live 22:22:33 specimen shows why it is unsafe: BOTH BLOCKED: 0 req resident, 97922 tok pending RELIEF: 19004 rows were already available against 512 wanted -> "KV is therefore NOT the binding resource here" 19004 rows against 97922 pending tokens. KV may well be exactly what is binding; the instrument had simply asked a question 190x smaller than the demand and generalised the answer. That is the same over-claim from a partial instrument that this routine exists to catch in others, committed by it twice now -- first by conflating the two zeros, now by trusting the wrong yardstick. The branch now reads `inp.pending_prefill_tokens` and withholds the strong claim when the real demand exceeds availability, while keeping it where it is warranted. Worth stating plainly: the relief's diagnosis ALSO contradicts the phase-policy line printed immediately above it, which asserts "the binding resource is KV, not the layout" without measuring. One of the two is wrong on every such specimen. Naming the disagreement is left to a follow-up; this commit only stops my half from asserting more than it measured. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): test_pending_above_avail_withholds_the_not_binding_claim ... the specimen test_pending_below_avail_still_makes_the_claim ............. CAN-FAIL: the strong claim must survive where warranted; an always-hedging fix fails it -> 7 passed in the file, 24 passed across the 701 + 698 suites. ruff clean.

…l attributes Follow-up: the binding needed code reading, not a live scheduler. Verified against /spinning/wt-678-deploy. WHAT EXISTS: scheduler.forward_ct monotone; += 1 at the TOP of run_batch (scheduler.py:6933), so it counts batch ATTEMPTS len(scheduler.waiting_queue) pending requests (:7538, :7725) len(scheduler.running_batch.reqs) running requests (:7539, :7726) load_inquirer._get_num_pending_tokens() pending tokens (load_inquirer.py:54-72, sums req.seqlen over the waiting queue plus the chunked remainder) WHAT DOES NOT EXIST, and it changes what the signal can promise: there is NO monotone completion, decode-step or committed-chunk counter. metrics_reporter.num_generated_tokens is reset every reporting interval (metrics_reporter.py:849, :1080), so it cannot carry a delta across a window. CORRECTION TO MY OWN CLAIM. I told the coordinator "forward_ct alone is NOT sufficient". That was overstated. forward_ct IS sufficient for the 16:23 wedge: no batch forms, so it never increments, and the defect was purely the is_active gate. Where it is genuinely insufficient is the RETRY-LOOP shape -- a batch that re-runs without committing anything advances ATTEMPTS while nothing progresses, which is exactly the sgl-project#701 self-deadlock silhouette. Separating those needs a committed-chunk counter, and that is a one-line instrumentation ask rather than a redesign. The binding is therefore honest about its reach: it fills the attempt signal from a real monotone attribute and leaves completions/decode_steps at ZERO rather than inventing motion it cannot observe. A test states that limit explicitly rather than hiding it. SchedulerBinding holds attribute paths as DATA, not lambdas, so a binding can be asserted against a synthetic object hermetically and a rename in the tree surfaces as a refusal. A stale binding raises: a binder that quietly returns a frozen counter reads as a PERMANENT wedge, the worst failure available to a wedge detector. 21 tests (5 new), hermetic, ruff + codespell clean. Live can-fail on a real wedge remains a window item.

…-chunk, the failure is per-total A chunked request's own committed prefix fills the pool its own next chunk must allocate from, and it cannot evict itself to make room. Analysis, hermetic falsifier, and priced options. NO BUILD -- the recommendation touches admission policy and is held for GO. ## (a) Reachability, at file:line in the deploy tree 1. Each committed chunk LOCKS its prefix. radix_cache.py:546 -- cache_unfinished_req ends with inc_lock_ref(new_last_node), moving the prefix from evictable to PROTECTED. 2. The admission budget EXCLUDES protected space. schedule_policy.py:809-825 -- rem_total_tokens is available_size() + evictable_size(). Nothing protected counts. 3. So every chunk shrinks the budget its own successor is checked against. The request eats its own runway, one locked chunk at a time. 4. Eviction cannot recover it. radix_cache.py:569-575 -- evict walks evictable_leaves only, and the request's own prefix is locked. THE REQUEST CANNOT EVICT ITSELF TO FUND ITSELF. THE TREE ALREADY KNOWS. schedule_policy.py:993-995 names it: "the prefill input must transiently fit the device. If not, this is the DEEP case (PS2) -> reject, today's wedge/wait behaviour", and the born-spill admission logs that it admits a request whose "full lifetime would wedge". What was missing is the arithmetic saying exactly when. THE CONDITION. With budget A at admission and chunk C, a request of length L commits chunks until A - k*C < C, so a single request deadlocks IFF L > A. The sharp edge: ADMISSION IS PER-CHUNK, THE FAILURE IS PER-TOTAL. Concurrently the condition is on the SUM -- several individually-admissible requests deadlock collectively, which no per-request check catches. It presents as a hang rather than a reject because the request gets MOST OF THE WAY first (the falsifier shows >= pool - C committed before stalling). ## (b) Falsifier Hermetic, no GPU. Pins: a fitting request completes; L > pool self-deadlocks after near-complete progress; the threshold is the POOL not the chunk (a bigger chunk changes granularity, not outcome); and two individually-admissible requests deadlock COLLECTIVELY. ## (c) Options, priced A admission gate (refuse unless full L fits, and RESERVE it) stops wedge: yes | serves L>pool: NO | needs: nothing cost: refuses long prompts; reservation held for the lifetime, so long-prompt concurrency drops sharply B self-evictable prefix to host (sgl-project#703 composes) stops wedge: yes | serves L>pool: YES | needs: the host tier cost: host traffic; USELESS WITHOUT HOST BUDGET -- the falsifier shows it still deadlocking at zero budget, so sgl-project#703 is a real dependency C preempt/retract another request does not help the single-request case: the only holder IS the victim A subtlety the model makes explicit: the gate must RESERVE the full length, not merely check it. Checking without reserving lets two requests both pass and then collide -- the very shape it exists to stop. ## Recommendation: A first as a safety property, then B as a capability The reasoning is about failure SHAPE, not throughput. A wedge is the worst available outcome because it is SILENT -- sgl-project#699 established that /health reports 200 through it and the watchdog is disarmed by the very condition that defines it. A refusal is loud, diagnosable and attributable. Converting a silent wedge into a noisy refusal is a strict improvement EVEN WHEN THE REFUSAL IS UNWELCOME, and option A does that with no new subsystem. A alone is not a sufficient end state: it makes prompts longer than the device pool permanently unservable, and on this rig a 327,680-token context against a 436,278-token pool leaves no room for a second concurrent long prompt. B follows because it is the only option that keeps SERVING the request, and it is exactly what sgl-project#703's host tier is for. They are complementary, not alternatives. 269 tests green, hermetic, ruff + codespell clean. Traffic hold honoured.

…ch had already refuted ## The retraction My sgl-project#701 section concluded "admission is per-chunk, the failure is per-total". That premise is FALSE. schedule_policy.py:1464 gates the FULL lifetime at first admission: if total_tokens >= self.rem_total_tokens: ... return AddReqResult.NO_TOKEN So a single request longer than the pool is REFUSED, not admitted-then-wedged, and my falsifier's headline case modelled a defect the shipped code prevents. Worse: the retraction already existed ON THIS BRANCH. Commit 845ac92 ("[sgl-project#701] Slice-3 rework: retract the mechanism story, close the real holes") had verified :1464, retracted exactly this story in its design doc, module docstring and both test files, and closed the real defect. I re-derived the refuted version without reading it. I searched the deploy tree for the mechanism but never checked whether sgl-project#701 was already owned and answered. ## What the real defect is, per the work that already exists 1. Paper-evictable funds an admission the evictor cannot honour: schedule_policy.py:734-737 -- rem_total_tokens includes full_evictable_size() while the allocator recovers only MAMBA-recoverable bytes. Gate passes; later relief frees 0. 2. The missing reservation is CROSS-PASS. PrefillAdder is rebuilt each pass and reserves only remaining DECODE, so a resident chunked request's remaining PREFILL is represented nowhere later. Later admissions spend its committed future, and the deadlock needs TWO ACTORS -- which is why a single-request analysis could never find it. ## Duplicates deleted planner/chunked_admission.py already provides ChunkedCommitmentLedger, decide_chunked_admission, effective_rem_total_tokens, defer-age tracking and idle-flip blocking, with tests in planner/ AND managers/ -- the latter running against the REAL PrefillAdder, which is strictly better evidence than the toy model I wrote. So I deleted my duplicate managers/chunked_admission.py and my planner/chunked_deadlock.py rather than keep a second, weaker account. ## The one genuinely additive piece, kept A monotone committed_chunks counter on the EXISTING ledger's spend() -- the single commit path. It lets the sgl-project#699 detector separate a retry loop (attempts advancing, nothing committing) from real progress, which forward_ct cannot do because it counts ATTEMPTS (scheduler.py:6933). Not rewound by release(): a progress counter that goes backwards reads as a restart to any watcher. progress_liveness gains an `attempts` field and retry_loop_detection, binding forward_ct to attempts and the ledger's committed_chunks to the commit signal. When the ledger is absent the commit signal stays at ZERO rather than borrowing the attempt count -- an invented commit would make a retry loop look like progress, which is the failure being hunted. ## sgl-project#702 noise floor (window 0b6c7db) A-vs-A spread is 14.1%, clean single-stream prefill ~1,820 tok/s. Rungs predicting less than +14.1% are NOT FINDINGS and are now flagged below_noise_floor: [28,17,19] (+12.0%) is disqualified outright. The recommended picks are unaffected -- [42,11,11] (+81.8%) and [44,10,10] (+100%) clear it widely. Also noted: 1,820 tok/s single-stream against 3,307 implied by the pipelined stage times is consistent with one stream not filling a 3-stage pipeline, so single-stream figures must not be substituted for pipelined ones. 237 tests green, hermetic, ruff + codespell clean.

…s the ledger the host had ROOT. evict_from_tree_cache gates the eviction on uniform_avail_for_evict(...) < num_tokens. That reads tree_cache.uniform_avail_floor, published ONCE per iteration (scheduler.py:4142-4144) as the group MIN of available_size(). Allocations made later in the same iteration were never charged against it, so late in an iteration the number is stale-OPTIMISTIC: with floor >= num_tokens the eviction is SKIPPED entirely, the alloc then fails against the live pool, and the raise reports a tree full of evictable tokens nothing ever asked for. Not a new class -- the HOST sibling already had this fixed. Its own comment states the reasoning: "a stale floor over-admits ... charging admissions against the floor removes the staleness without a second collective". The DEVICE sibling never got the ledger. This adds it, mirroring sgl-project#645 exactly: uniform_admitted_since_floor, charged on the success path of alloc_token_slots, reset by the scheduler in the same call that publishes the next floor so it never outlives the number it corrects. Sufficient, by sgl-project#645's argument: live availability is at least avail_at_publish - admitted, and avail_at_publish >= floor, so a request clearing floor - admitted fits the real pool. Rank-uniform by construction: num_tokens comes from the replicated batch, so every rank charges the same amount at the same allocation and the predicate stays identical across ranks -- the #616g invariant this must not break, pinned by a test. fundable_extend_tokens reads the same predicate, so admission inherits the correction for free. OVERLAP VERDICT vs sgl-project#701(a), checked in code rather than assumed: NOT one defect, on the available evidence. * evictable and protected are DISJOINT by construction -- inc_lock_ref moves tokens out of evictable_size_ into protected_size_ (radix_cache.py:605-606) and dec_lock_ref moves them back (:622-623). So a reported evictable count never includes protected-prefix pages. * The only specimen present in any accessible log is 66039 (available=273 + full_evictable=65766, 512 requested). Its "Full LRU list evictable size: 65766" matches full_evictable_size EXACTLY. That sanity check is an independent traversal of the eviction list, and divergence is precisely what it exists to detect -- so this specimen's evictable was genuinely reachable, refuting the paper-evictable hypothesis for it. It is also the specimen sgl-project#681 already diagnosed (mamba tombstone leaf) and paid in MambaRadixCache. evict_full. * The "167k evictable" specimen is NOT in any log I can reach, so its decomposition into mamba-recoverable vs paper-only cannot be done and is not inferred. If F4-r4's 1f594e7 instrument catches a recurrence, its skipped-vs-ran line decides it directly. So the staleness defect is proven STRUCTURALLY (the host/device asymmetry) and fixed with a can-fail falsifier; it is not claimed as the cause of a specimen whose instrument output does not exist. One self-inflicted defect caught by the suite: getattr(tree_cache, "uniform_admitted_since_floor", 0) yields a Mock on an unconfigured double, and int(Mock()) is 1, not 0 -- silently shaving a token off the floor and breaking test_a_published_floor_is_returned (499 != 500). Guarded by an isinstance check, with the reason recorded. Third appearance of the sgl-project#624 stub-drift class. The host sibling carries the identical latent exposure and is left untouched here to keep the blast radius small; worth a follow-up. Tests: 7, red first, including the can-fail proof that an uncharged stale floor really does skip. managers 2113 passed / 0 failed. mem_cache is 944 failed / 772 passed both before and after this change -- a large PRE-EXISTING red suite, verified by patch round-trip on the clean tree (945/2870 baseline across managers+mem_cache without the new file). ruff clean on all three files, compared against HEAD.

…t the single chokepoint ## What is wired ChunkedCommitmentLedger is now visible to PrefillAdder, flag-gated DEFAULT ON (chunked_admission_enabled). Off restores the pre-sgl-project#701 arithmetic byte-for-byte, so the flag is the A/B arm for a reviewed window. THE LEDGER IS OWNED BY THE SCHEDULER, NEVER BY THE ADDER. A PrefillAdder is rebuilt every pass, so anything it held itself would forget a resident chunked request's outstanding prefill exactly when the next pass needs it -- which IS defect (b). It is passed in and never constructed there. The subtraction lands at the single chokepoint, rem_total_tokens, via effective_rem_total_tokens(budget, ledger). That covers ALL FOUR sibling sites at once (:1569-1610, :1388-1407, :1230-1276, :1124-1141) because every one of them reads that property rather than recomputing the budget. Four hunks, 29 insertions; :734-737 deliberately UNTOUCHED so the merge train with Slot-2's branch stays clean. ## Falsifier: the two-actor deadlock through the REAL PrefillAdder A resident request commits 80,000 tokens; a FRESH adder (the rebuild) then reports 100,000 unwired and 20,000 wired. Spending returns the budget incrementally, release returns it fully, and a missing ledger is not an error. Red on unwired, green on wired, same adder both ways. ## Defect (a) is NOT wired, deliberately An earlier draft of this commit priced rem_total_tokens against "actually recoverable" rather than full_evictable_size(). THAT HYPOTHESIS IS REFUTED (sgl-project#694, Slot-2 704240c): evictable and protected are DISJOINT BY CONSTRUCTION -- inc_lock_ref moves tokens out of evictable_size_ into protected_size_ (radix_cache.py:605-606) -- and the readable specimen's LRU evictable size matched an independent traversal exactly. The real root was a STALE FLOOR (uniform_avail_for_evict published once per iteration, never charged by later allocations), fixed on Slot-2's branch. The change and its tests are reverted rather than left passing against a mock of a distinction the code does not make. A note in the test file records why, so the idea is not re-derived a third time. Also removed on the way: the harness mocked `mamba_recoverable_size`, which exists NOWHERE in mem_cache/ -- only mamba_evictable_size does. A test built on that mock would pass against itself and fail against the real tree. ## Flip integration: SPEC ONLY, for F4-r4 phase_flip_runtime and the idle-flip integration are his boundary and are not touched. DESIGN_704 carries the contract: (1) ask deferred_head_blocks_idle_flip before arming, since a deferred head is pending work not idleness; (2) count resident chunked requests via effective_running_bs (sgl-project#631 defect O -- a resident-but-batchless request appears in no running_batch.reqs); (3) outstanding_tokens() is non-zero exactly while some prefill is unfinished, and arming across that applies the seam move to KV still being written -- drain or refuse, his call since he owns the window economics. All three are read-only questions against an object the scheduler already holds. ## Third-site sweep Per the standing instruction: no further site found where rem_total_tokens is funded by a quantity the relief path cannot deliver. Had one appeared it would be reported with file:line, not fixed inline. Tests green, hermetic. Lint on the touched test file is unchanged from HEAD (3 pre-existing findings, none introduced); schedule_policy.py deliberately NOT run through ruff format, which reformats unrelated legacy code and would have doubled the diff.

…faster A permanently-red suite hides every new regression (sgl-project#380/sgl-project#585 test-honesty class). Hermetic mem_cache stood at 944 failed / 777 passed / 707 skipped in 248 s. It now runs 777 passed / 1651 skipped / 0 failed in 32 s. CLASSIFICATION, and it is simpler than the count suggested: (a) GPU-required under CVD="" ...... 944 of 944 (100%) (b) stub-drift / sgl-project#624-class ........ 0 (c) genuinely broken code paths .... 0 (d) obsolete tests ................. 0 Every one of the 944 shared a single root: get_device() raising "No accelerator (CUDA, XPU, HPU, NPU, MUSA, MPS) or platform plugin is available". Red meant "no GPU here", never "broken". Verified by taking the non-GPU remainder of the FAILED lines, which is EMPTY. Two secondary findings, both of which made this worse than it had to be: * CustomTestCase._callTestMethod wraps every test in utils.common.retry, which re-raises a bare Exception("retry() exceed maximum number of retries.") WITHOUT chaining the cause (no `from e`). The real one-line reason was discarded, so 841 of 944 reported an opaque retry message. That is why a single environmental cause looked like a suite-wide catastrophe. * the same wrapper retried a DETERMINISTIC environment failure several times each, which was most of the runtime -- hence 248 s -> 32 s. FIX. retry() honours exactly one exception: SkipTest is re-raised immediately and never retried. So a directory conftest converts the environmental error into a skip AT ITS SOURCE, which fixes the masking and the retry storm together and needs no edit to any test. It patches get_device on both the defining module and the package namespace, because the tests bind the name at import time and conftest is imported first. Installed ONLY when there is genuinely no accelerator, so a GPU run is unchanged and any real failure stays red; it fires only for tests that actually call get_device, so the 777 that already pass keep running -- the passing count is identical before and after, which is the check that no green test was silenced into a skip. One test needed an individual marker: test_memory_allocated calls torch.cuda.memory_allocated() directly and never goes through get_device. Marked with skipUnless and the reason, rather than widening the patch to cover torch.cuda. PHANTOM MOCK removed (mine, from the sgl-project#701 harness): tc.mamba_recoverable_size was mocked and NO SUCH ACCESSOR EXISTS -- the real one is mamba_evictable_size (mamba_radix_cache.py:1177, unified_radix_cache.py:3189). Nothing read it, so it never produced a false green, but a mock of a non-existent API teaches the next reader an interface the tree does not have, which is the same sgl-project#380 shape. Both it and its unused `recoverable` parameter are gone, with a comment naming the real accessor. mem_cache + managers together: 2883 passed, 0 failed (baseline 945 failed / 2870 passed). ruff clean.

…cket Two docs, both prep: no branch pointer on a real line moves here. MERGE_TRAIN_2_FOLLOWUP.md -- the follow-up train, measured against the head train 1 projects (integration/r2 + 621 + 699 + 673-lockstep + 4c84637 + 67572ce + 677 = 6bab764c33), not against r2. Order and trial-merge result: reconcile/cluster-b-seam-model clean; fix/673-teardown-stack ONE conflict, docs/dev/MERGE_TRAIN_2026-08-17.md, doc only, no source conflict anywhere in the train; fix/728-max-bytes-uniform clean. Assembled head c5a4c1bca8. Ancestry finding, which moves the kernel rebuild: 4512136 and a8b068b are ancestors of reconcile/cluster-b-seam-model (YES) and of nothing else in the train -- not of 6bab764c33, 887a6d4 or a66f5e2. So the separately listed "sgl-project#441 kernel commits" step is a NO-OP and should be struck, the sgl-kernel rebuild is triggered by step 1, and train 1 is unaffected (it merges 67572ce, which carries neither). The rebuild step is named with the runbook's wheel-pin discipline (arch list 86;120, MAX_JOBS=4, nvcc from the venv cu13 toolkit) and its acceptance is boot-gated, not desk-gated. Baselines on c5a4c1bca8, hermetic (CUDA_VISIBLE_DEVICES=""): mem_cache 1086 passed / 0 failed (1651 skipped); managers 19 failed / 2367 passed; planner 8 failed / 2842 passed (all test_webui/chess, missing optional dep); distributed 27 failed / 2764 passed vs 21 on train 1; test_scheduler_teardown_673.py 10 passed. The +6 on distributed is attributed rather than asserted. On the three implicated files: 887a6d4 9 failed, a66f5e2 9, feat/704 9, fix/602 9, fix/701-ledger-wiring 15, reconcile 15, train-2 head 15. The extra failures are pre-existing on fix/701-ledger-wiring and inherited unchanged; no merge in this train creates one. Reported as "21 pre-existing + 6 inherited from sgl-project#701", not as green. DESIGN_706_BOOT.md -- the boot-side open items are closed as a decision, and section 5 is a run-card F4-r4 can execute: preconditions, verbatim flag set, ordered steps, pass/fail by greppable log string, abort conditions. Boot WITHOUT the second host pool. The phase_flip_host_pools builder stays unwritten deliberately: it does not fit (5.37 GB remainder, both pools need ~27 GB on rank 0's ratio) and the cross-phase path does not need it, since sgl-project#706 made the disk tier geometry-neutral. One code-verified correction to this doc's own earlier framing: with --phase-flip-rebind-hicache OFF, rebind_for_cutover returns None on the flag check BEFORE phase_pools_for is called, so the recommended boot logs no sgl-project#719 line AT ALL -- the previous text said the unarmed rebind refuses at every cutover. Armed without a second pool it does refuse, exactly once per cutover, logged-never-raised at ERROR level. Both expectations are stated, because an expected ERROR line is what stops a boot for no reason. Acceptance counts the prefill log line (#cached-token > 0), never cache_hit_rate, which reports 0.0 despite real hits.

…cket Two docs, both prep: no branch pointer on a real line moves here. MERGE_TRAIN_2_FOLLOWUP.md -- the follow-up train, measured against the head train 1 projects (integration/r2 + 621 + 699 + 673-lockstep + 4c84637 + 67572ce + 677 = 6bab764c33), not against r2. Order and trial-merge result: reconcile/cluster-b-seam-model clean; fix/673-teardown-stack ONE conflict, docs/dev/MERGE_TRAIN_2026-08-17.md, doc only, no source conflict anywhere in the train; fix/728-max-bytes-uniform clean. Assembled head c5a4c1bca8. Ancestry finding, which moves the kernel rebuild: 4512136 and a8b068b are ancestors of reconcile/cluster-b-seam-model (YES) and of nothing else in the train -- not of 6bab764c33, 887a6d4 or a66f5e2. So the separately listed "sgl-project#441 kernel commits" step is a NO-OP and should be struck, the sgl-kernel rebuild is triggered by step 1, and train 1 is unaffected (it merges 67572ce, which carries neither). The rebuild step is named with the runbook's wheel-pin discipline (arch list 86;120, MAX_JOBS=4, nvcc from the venv cu13 toolkit) and its acceptance is boot-gated, not desk-gated. Baselines on c5a4c1bca8, hermetic (CUDA_VISIBLE_DEVICES=""): mem_cache 1086 passed / 0 failed (1651 skipped); managers 19 failed / 2367 passed; planner 8 failed / 2842 passed (all test_webui/chess, missing optional dep); distributed 27 failed / 2764 passed vs 21 on train 1; test_scheduler_teardown_673.py 10 passed. The +6 on distributed is attributed rather than asserted. On the three implicated files: 887a6d4 9 failed, a66f5e2 9, feat/704 9, fix/602 9, fix/701-ledger-wiring 15, reconcile 15, train-2 head 15. The extra failures are pre-existing on fix/701-ledger-wiring and inherited unchanged; no merge in this train creates one. Reported as "21 pre-existing + 6 inherited from sgl-project#701", not as green. DESIGN_706_BOOT.md -- the boot-side open items are closed as a decision, and section 5 is a run-card F4-r4 can execute: preconditions, verbatim flag set, ordered steps, pass/fail by greppable log string, abort conditions. Boot WITHOUT the second host pool. The phase_flip_host_pools builder stays unwritten deliberately: it does not fit (5.37 GB remainder, both pools need ~27 GB on rank 0's ratio) and the cross-phase path does not need it, since sgl-project#706 made the disk tier geometry-neutral. One code-verified correction to this doc's own earlier framing: with --phase-flip-rebind-hicache OFF, rebind_for_cutover returns None on the flag check BEFORE phase_pools_for is called, so the recommended boot logs no sgl-project#719 line AT ALL -- the previous text said the unarmed rebind refuses at every cutover. Armed without a second pool it does refuse, exactly once per cutover, logged-never-raised at ERROR level. Both expectations are stated, because an expected ERROR line is what stops a boot for no reason. Acceptance counts the prefill log line (#cached-token > 0), never cache_hit_rate, which reports 0.0 despite real hits. (cherry picked from commit 60414e9)

…-project#701's commitment ledger `_budget_state_stub` had drifted behind `PrefillAdder.rem_total_tokens` again: sgl-project#701 appended the cross-pass COMMITMENT LEDGER to the tail of that property (schedule_policy.py:883-885) and the stub carried neither name, so both admission cases died with AttributeError: 'PrefillAdder' object has no attribute 'chunked_admission_enabled' This is the sixth drift of this harness after #616g, sgl-project#639, #639b, #791b and sgl-project#794, and the sixth time `TheAdderStubTracksTheBudgetPredicate` named it instead of letting an AttributeError surface inside an unrelated admission assertion. The guard keeps earning its place. BOTH NAMES ARE BOUND, not just the one the failure reported. `chunked_admission_enabled` was the reported miss; `commitment_ledger` was the next one queued behind it. Binding only the reported name is exactly what reshipped this incident at #639b, whose own note says so. THE PAIR EXERCISES THE SHIPPED CHOKEPOINT RATHER THAN STEPPING AROUND IT. `effective_rem_total_tokens` returns its input unchanged when the ledger is None (planner/chunked_admission.py:233-235), so enabling the flag runs the real tail and provably cannot move the budget this harness is about. Setting the flag False would ALSO make the guard green while quietly taking that tail out of the tested path -- the weaker of the two bindings, and the one this file's own history argues against. TESTS (hermetic, CVD="", CPU only) test_collective_family_siblings_610.py 11 passed (was 3 failed / 8 passed). The can-fail is already resident: `test_the_guard_can_fail` plants a field that does not exist and asserts the guard reports exactly it. Battery over the 7 files that carried every failure, under /spinning/htsglang-gpu/.venv (datasets 5.0.0, full collection): base 500be7e 22 failed / 35 passed before this 15 failed / 46 passed after this 12 failed / 49 passed -3, exactly the tests repaired here. This branch still adds none. Found while opening sgl-project#823: the enforcer there has to add slots to the same packed reduce this harness models, so an already-drifted guard would have been the thing standing between that change and a silent index shift. Repaired first, deliberately. No boot was run. This is desk work.

W9 wiring, part 1 of 2: the group now COMPUTES the uniform decision every TP-loop iteration. Consuming it in batch formation is part 2. PLACEMENT. The block sits between the mamba pair and the ballot, at the same seam the sgl-project#794 corridor width uses and for the same reason its comment gives: everything above is indexed from the HEAD and the ballot is indexed from the TAIL (`len(vals) - (PREFETCH_BALLOT_SLOTS + 2)`), so an insertion here leaves both readings intact. Both new blocks capture an explicit index BEFORE appending and are read back by it; nothing is read by a negative offset, which is the mistake #639b's note records -- "the `t[-2]`/`t[-1]` the host floor used to read would have silently started reading MAMBA availability". The full payload order is written out in the comment so the next person does not have to reconstruct it. THE PULL-FORWARD, and it is not a new idea in this function. The sort key `num_matched_prefix_tokens` is populated by `calc_priority`, which runs LATER in the pass, so at reduce time it is zero or last pass's value and reducing it would make the group agree on a stale number. #791b already solved this exact shape here for the prefetch verdicts: pull the RANK-LOCAL computation forward to the reduce site -- no collective, once per TP-loop iteration -- and memoise it for the batch formation to consume. `_local_head_prefix_matches` does the same, bounded to the canonical head, against a tree `calc_priority` was about to walk in full anyway. THE COUNT VOTE GETS ITS OWN SLOT rather than being derived from the availability floor, because `get_num_allocatable_reqs` is bounded by `admission_limiter.current` (:6526-6529) -- rank-local floating state the availability reduce does not capture, so a count derived from the uniform avail would still diverge. Neither vote can break the reduce: both are wrapped, and a rank that cannot price contributes the absent/unpriced sentinel, which can only delay admissions or leave the local limit untouched. Never a collapse to bs=0. THE DRIFT GUARD CAUGHT ME, WHICH IS THE POINT OF HAVING REPAIRED IT FIRST. Widening the payload made `TheHarnessTracksTheProductionSurface` fail with BudgetHarness has drifted behind Scheduler._update_uniform_pool_budget: ['_local_admit_limit', '_local_head_prefix_matches'] -- the seventh drift of that harness after #616g, sgl-project#639, #639b, #791b, sgl-project#794 and sgl-project#701, and the FIRST caught in the same change that caused it rather than a quarter later. Then it named `get_num_allocatable_reqs` behind them, the transitive member, exactly the cascade the file warns about. Both votes are BOUND from Scheduler so the harness keeps modelling the real contract, and both ride neutrally there: the harness's waiting_queue is empty, so the head vote is an empty canonical set of absent sentinels. For `get_num_allocatable_reqs` the guard's own message offers a stand-in as the alternative to binding, and a stand-in is right: the shipped method is bounded by the admission limiter and carries the sgl-project#677 parking branch, so binding it would oblige this harness to model a limiter and a phase policy to answer a question it is not asking. Constant and EQUAL on both ranks deliberately -- a divergent count is test_tp_head_congruence_823's subject, and making it diverge here would put a second unrelated variable into the budget cases. TESTS (hermetic, CVD="", CPU only, real gloo where the suite uses it) 55 passed across test_collective_family_siblings_610 (the harness that models this reduce), test_pp_prefetch_ballot_791b (the tail-indexed ballot, i.e. the reading most at risk from a widening), test_prefetch_ballot _divergence_823 and test_tp_head_congruence_823. That the ballot suite still passes IS the evidence that the tail indexing survived the insertion. STILL PART 2, so W9 stays preflight_pass N: `calc_priority` and the candidate loop do not yet consume `_uniform_head_match_lens` / `_uniform_admit_limit`. The decision is computed and published; nothing acts on it yet, so this commit changes no batch. No boot was run. This is desk work.

…eciding who DESK BUILD of TICKET_943_REISSUE_REACHABILITY.md. sgl-project#937 refuses to publish a prefetch whose binding generation went stale across a cutover -- correctly: the sgl-project#943 bisection put the garbage fix at exactly that commit, and every pin that publishes stale spans returns 1/7 coherence. What the refusal leaves behind is a request owed its prefix, and the only correct way to return it is a FRESH fetch under the binding that is current now. THE SHAPE, and every piece of it is chosen to avoid inventing a second version of something that exists: * At the refusal, the req_id is recorded as owed -- a NAME and a COUNT, never the operation, the span or the indices. Keeping any of those is what would tempt the re-stamp that `StaleStampRewrite` (a882e64) already refuses. * `take_agreed_reissue` picks ONE request the whole group agrees on, using the shape `drain_retired_prefetch` already proved: MIN over `[d, -d]` yields the group min and max in one pass, and only `min == max != 0` is agreement. * The candidate set is the INTERSECTION of "owed" and "present in this rank's waiting queue". Voting on "owed" alone could agree on a request some rank cannot act on, and that rank would then sit out the collective its peers entered -- the failure, not a smaller version of it. * The re-issue itself runs through the ordinary `Scheduler._prefetch_kvcache`, so it inherits the existing participation vote, the rank-local eligibility handling and the symmetric-mode branch instead of reproducing them. * The count is REPORTED, never gated. `_MAX_PREFETCH_REISSUES` already carried that rule in its own docstring; a cap would be a rank-local predicate deciding collective participation. I WROTE THE sgl-project#580 FAILURE INTO THE CODE MEANT TO PREVENT IT, and it is recorded at the site rather than quietly corrected. The first draft of the scheduler block read `if self.tree_cache._reissue_pending:` before calling the gate -- a rank-local predicate in front of a collective, so a rank with nothing owed would skip the all_reduce its peers had already entered. The comment on the reap two lines up states the rule verbatim for the same reason. The call is now unconditional and the vote answers 0 for an empty candidate set precisely so it needs no guard. WHY THE LIVE MEASUREMENT WAS NOT ACCEPTED AS THE PROOF. Boot a810ef6 measured the refusal verdict rank-uniform (DIVERGES 0, AGREES 3, over 111 cutovers and 48 refusals). That is one boot on one rig at TP=3. Building a collective on it would make the uniformity load-bearing and checked nowhere, so the sgl-project#580 direction is held by a TEST that INJECTS the divergence the boot never showed. TESTS, hermetic (CUDA_VISIBLE_DEVICES=""), /spinning/htsglang-gpu/.venv. GLOO FALSIFIER, three spawned processes, the real gate and the real `_all_reduce_attn_groups` in each child, both arms bounded by a deadline so a hang is a REPORTED timeout: * split verdict (ranks disagree WHICH request): guarded -> no rank takes, no rank enters the follow-on vote, all three finish. Ungated -> rank 0 takes req-A while ranks 1 and 2 take req-B and all three act, i.e. the ranks re-register DIFFERENT requests. That is the assertion, measured: `distinct taken == {req-A, req-B}`. The first version of this arm was a three-way OR that any outcome satisfied; it was sharpened after checking what the mutant actually did. * lonely verdict (only rank 0 owes): ungated -> one rank enters the collective alone, the literal wedge. Guarded -> nobody enters. ANCHOR SURVIVES A CUTOVER, the coverage the bisection exposed as missing -- no test pinned what happens to a prefix across a cutover at all. Pinned as BOTH halves at once, because pinning only the first is satisfied by the very defect sgl-project#937 removed: the prefix is RECOVERABLE (an agreeing round hands the request back to be re-fetched) and the old span is UNRECOVERABLE (the stale operation cannot be re-stamped; the retained state is a count, asserted). RED-FIRST against the pre-#943b tree: 7 failed, 5 passed; extracted FAILED-name count 7 == summary "7 failed". THE DRIFT GUARD CAUGHT ME, which is the system working. The frozen A/B went 45 -> 47 failures: `test_collective_family_siblings_610` noticed that the drain grew a member (`_prefetch_kvcache`) its harness did not carry. That guard's own comments count the previous times it paid for itself (#616g, sgl-project#639, #639b, #791b, sgl-project#794, sgl-project#701, sgl-project#823); this is the next. Fixed by giving the harness a STAND-IN -- it exercises the budget reduce, not the prefetch path, and with `enable_hicache_storage` False the shipped drain returns {} before reaching the re-issue -- and the stand-in raises if it is ever actually called, so a moved early-return cannot leave these cases silently pinning nothing. FAMILY, FROZEN A/B over the same 66 files, base vs this tree, after the fix: base 45 failed, 1472 passed, 865 skipped (extracted names 45) built 45 failed, 1472 passed, 865 skipped (extracted names 45) diff of the FAILED-name sets: IDENTICAL. New + touched files together: 31 passed. ruff: scheduler 103 findings before and after (none introduced), everything else clean. black clean. NOT PROVEN, AND IT IS THE WHOLE POINT OF THE NEXT WINDOW: that this actually returns the prefix on metal. The acceptance is encoded in devtools/bisect_869b_anchors.sh as a CONJUNCTION -- cached>0 on a repeated prompt AND 7/7 coherence, never one alone -- and verified to FAIL both known states: the pre-sgl-project#937 pin (cached=0, 1/7) and the current tree (cached=0, 7/7). It can only pass in the state no pin in f1a3391..dd0e3bc has reached.

hnyls2002 added 2 commits July 23, 2024 10:22

use min ratio at start

a3894e2

init schedule conserv -> 1

fef6a29

hnyls2002 merged commit 2686844 into main Jul 23, 2024

hnyls2002 deleted the fix-new-token-ratio branch July 23, 2024 18:52

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Use min new token ratio at start (sgl-project#701)

86f04fa

Navigation Menu

Uh oh!

Use min new token ratio at start - #701

Use min new token ratio at start#701
hnyls2002 merged 2 commits into
mainfrom
fix-new-token-ratio

hnyls2002 commented Jul 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

hnyls2002 commented Jul 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant