Fix hf config loading by ispobock · Pull Request #702 · sgl-project/sglang

ispobock · 2024-07-23T15:32:30Z

Motivation

When testing with chatglm3-6b, this line will fail because chatglm's config uses num_layers instead of num_hidden_layers.

Modification

Ref vllm, for registered models, use extended config class to load.

ispobock · 2024-07-23T15:35:49Z

Tested chatglm3-6b:
hf:

prefill logits tensor([1.2461, 1.2451, 9.1641,  ..., 1.2441, 1.2451, 1.2451], device='cuda:0', dtype=torch.float16)
prefill logits tensor([2.9043, 2.9043, 6.4297,  ..., 2.9004, 2.9043, 2.9023], device='cuda:0', dtype=torch.float16)
prefill logits tensor([-2.7617, -2.7637,  4.6641,  ..., -2.7637, -2.7617, -2.7617], device='cuda:0', dtype=torch.float16)

sglang:

prefill logits (final) tensor([[ 1.2510,  1.2500,  9.1562,  ...,  1.2480,  1.2490,  1.2490],
        [ 2.9023,  2.9023,  6.4336,  ...,  2.9023,  2.9023,  2.9023],
        [-2.7656, -2.7656,  4.6562,  ..., -2.7656, -2.7637, -2.7656]],
       device='cuda:0', dtype=torch.float16)

… percent at L=16 Desk halves of sgl-project#700. Paused mid-ticket for the sgl-project#702 priority insert; part 3 (the identity arm) is specified but no arm has been run. (1) Wiring contradiction resolved: the KERNEL HEADER is stale, not the wiring. fused_recurrent_linear_replayssm.py:10-12 claims the feature is "NOT yet wired into the memory pool / radix cache / scheduler / backend dispatch". It is wired in all four: ring allocation and documented layout (memory_pool.py:469-477, 611-633), cursor reset on slot alloc (:1530), copy_from with a stated invariant and dst reset (:941-968), flush/clear (:908), write_pos carried in the offload/restore blob (:1017, :1049, :1075), the radix donate capped to the last flush boundary (mamba_radix_cache.py:599-608), its unified-cache mirror (mamba_component.py:409-417), backend plus CUDA-graph static buffers (hybrid_linear_attn_backend.py, 54 refs), and dispatch (gdn_backend.py:497-537). So sgl-project#700 does NOT close as refused-until-fixed: the seam hazard the ticket anticipated -- a donate or copy mid-window against an unflushed ring -- is addressed head-on and documented at both sites. No correctness hole found. Two findings remain. (a) The copy_from invariant is enforced only under debug_memory_pool, off by default, so in production it rests on caller discipline in a docstring; a fourth caller would violate it silently. That is the sgl-project#624 drift class -- hardening candidate, not a bug today. (b) _build_tree's docstring says the pools are built "on CPU", but construction reaches utils/common.py:1322 and raises "No accelerator ... is available" under CUDA_VISIBLE_DEVICES="". A second comment that does not match its code. Execution status, stated plainly: the ticket asked to settle this BY EXECUTION and that is blocked hermetically by (b) -- the pre-existing TestFlushResetsMambaPool fails identically, so this is not introduced here. The determination above is therefore made by reading. The executable proof is committed and waiting for a GPU window: TestReplaySsmRadixSeam, four tests covering ring allocation, dst-cursor reset, the can-fail proof that the documented invariant trips under the debug guard, and one that pins (a) as a fact by showing an unflushed source is copied silently with the guard off. THESE TESTS HAVE NEVER RUN AND ARE NOT REPORTED GREEN. (2) The net count left open in sgl-project#325: total(L) = S + R + ((L-1)/2)R + S/L with S = 3,145,728 B and ring append R = 32,960 B/step (d 24,576 + k 8,192 + g 192; note replayssm_k is per K-head, 16, not per V-head). At the default L=16 the saving is 40.4 percent of chain bytes, a 1.68x reduction -- below the header's "roughly halved" because the reconstruction re-read is priced at zero there. The optimum is L* = sqrt(2S/R) = 13.8, so the shipped default sits within 0.1 percentage points of optimal and the curve is flat from L=8 to L=24. L=1 comes out worse than baseline at -0.5 percent, the sanity result a correct count must reproduce. Against sgl-project#325's >10 percent gate this clears by 4x versus 0.99 percent for the best fusion. (3) Identity arm specified, not run: CPU-sampled inputs, A-vs-A floor first, probe kept under the ~109-token GDN prefill non-determinism ceiling, GDN scalar-gate only, and the identity result gates whether the perf arm is worth running at all. Docs plus one window-gated test class. No enable-flip, no deploy.

…tor co-solved The sgl-project#602 capacity solve withdrew the cut for the POOL objective. The question actually asked -- more prefill on the 5090 -- is a time objective and was never solved for. This adds it, and then corrects its own first framing. Timing model, calibrated from the measured 49.2/154.8/116.4 ms at [28,20,16] (1.757/7.740/7.275 ms per layer). Two objectives are reported side by side because they disagree: SERIAL sum(layers_r * ms_r) and PIPELINED max over stages. Backtest: the model reproduces the calibration point exactly, and the hand-arithmetic anchor holds -- [42,12,10] gives 239.43 ms against 320.4, i.e. 1.338x, matching the 1.34x expected by hand. Result table under the rank0 cap (x_serial / x_pipelined vs incumbent): [28,20,16] 1.000 / 1.000 incumbent [42,12,10] 1.338 / 1.667 the anchor [42,11,11] 1.341 / 1.818 DOMINATES the anchor on BOTH axes [42, 1,21] 1.367 / 1.013 serial-optimal and useless once pipelined The serial optimum is a degenerate cut that collapses pipelined throughput, so reporting only one objective would have recommended it. [42,11,11] is the candidate to arm. Rank0 cap recomputed rather than quoted: at 724.3 MiB/layer against a 31,800 MiB budget, 42 layers leave 1,379 MiB (corridor OK), 43 leave 655 MiB (CORRIDOR VIOLATED), 44 overflow by 69 MiB. So DESIGN_691's "about 42" is confirmed, and the binding constraint at 42 is the CORRIDOR, not raw weight overflow -- 43 fits the card and still must be refused. Co-solve correction (user, restating sgl-project#320/sgl-project#492): revision 1 priced a "pool cost" per candidate while holding the KV token vector PINNED. That is single-family optimization and sgl-project#485 forbids it. Layers moved to rank0 free exactly their weight bytes on the rank they left, and the uneven-DCP / rank-kv-ratio machinery relocates the displaced KV share onto those bytes. Rank0's cap therefore bounds rank0's SHARE of the token split, not the world pool. The conservation is exact, not approximate, and the tests prove it rather than assert it: total VRAM is fixed and the same 64 layers of weights exist wherever they sit, so total free bytes are invariant; under DCP token-sharding one token costs kv_per_token_per_layer * total_layers wherever it lands. World pool is therefore identical across every feasible cut, and the only residual is second-order -- seam/staging in both directions plus TP-phase redistribution -- which is ITEMIZED and whose sum equals the delta exactly (no fudge term). Scope note owed to the earlier verdict: "29,19,16 is 6.3 percent worse" held only under a pinned vector. Co-solved, that comparison does not stand as a capacity argument. Calibration limit stated in the model docstring and pinned by a test: one measured cut gives one point per rank and cannot separate per-layer slope from fixed per-stage cost. fixed_ms defaults to zero, the OPTIMISTIC end, so every speedup here is an upper bound. A second measured cut per rank would pin it. Tests: 15, red first. Planner regression 41 passed across the 702 and 602 suites. ruff clean. No deploy, no arm.

…d my co-solve was wrong Arm B [42,11,11] was armed on my solver's advice and OOM'd on rank0 at KV pool commit, twice. The defect was mine and it was a phase confusion. Revision 2 computed the world pool as the SUM of per-rank token capacities with a free-proportional vector, and reported "world pool conservation is exact". That is the TP-phase rule: under tensor parallelism the model is width-sharded, every rank holds a slice of all 64 layers, per-token cost is uniform, tokens are sharded across ranks, and the vector CAN relieve a tight rank. Under PP prefill the pool is LAYER-sharded. Every rank stores KV for ALL tokens for ITS OWN layers, so pool = min_i(free_i / (layers_i * cost)) and the token vector does not enter at all. F4-r4 proved the second half on metal: rank0's vector share was cut 4.3x between the two failed attempts and rank0's memory moved by ZERO -- identical reserved 30.60 GiB both times. So the conservation result was an artifact of applying the wrong phase's rule, and prefill speed and pool capacity are in direct opposition along the cut axis. This module already contained stage_kv_capacities, the min-rule machinery; the co-solve did not use it. Deleted rather than deprecated: cosolve_prefill_cut, WorldMemory, CoSolvedCut and the tests that asserted sum-rule conservation. Leaving a callable solver that encodes falsified physics is worse than leaving none, and green tests asserting the wrong rule are worse still. Added, census-calibrated to F4-r4's table (reproduced to ~0.02%): stage_pp_capacities / pp_phase_pool -- MIN rule, takes NO vector argument and raises TypeError naming the metal proof if one is passed, so the TP rule cannot be reapplied to the PP phase at a call site; tp_phase_pool -- SUM rule, separate column, independent of the PP cut and vector-relievable; solve_prefill_cut_tradeoff -- both speedups AND both pool columns per candidate, with a min_pool_ratio filter, because an unfiltered speed ranking recommends exactly the cuts that cannot hold the context. Backtest / can-fail proof: the solver now predicts the boot that failed. [42,11,11] -> 202,572 model tokens, 0.53x the incumbent, live-equivalent 229,287 -- matching F4-r4's independently scaled "near 229k". Sanity anchor [32,16,16] -> 419,666 against his 419,734. Weight term corrected to the census 450.7 MiB/layer (374.2 full-attention / 476.2 linear) from the 724.3 revision 2 used, +61%. A test pins why that was the wrong headline: at 724.3 arm B looks like a WEIGHT OVERFLOW and never reaches a pool number, while at 450.7 the weights fit comfortably and the arm still fails on the min-rule. Same failure, two diagnoses. New finding neither revision had: the pool is not monotone in the cut. The BINDING RANK SWITCHES -- at the incumbent rank1 binds because it carries 20 layers, so moving layers OFF rank1 RAISES the pool. The incumbent [28,20,16] is therefore not pool-optimal. The pool peaks at [30,18,16] (462,231, +20%) and only collapses past the crossover. Real trade, cuts that keep at least the incumbent pool: [32,16,16] x_ser 1.081 x_pipe 1.250 pool 1.09x live-equiv 475,012 [33,15,16] x_ser 1.103 x_pipe 1.330 pool 1.02x live-equiv 443,738 versus what revision 2 recommended: [42,11,11] x_ser 1.341 x_pipe 1.818 pool 0.53x live-equiv 229,287 [46,9,9] x_ser 1.484 x_pipe 1.915 pool 0.37x live-equiv 160,912 Tests: 19, red first, including the arm-B backtest. Planner suite 2593 passed, 2 failed -- both the pre-existing test_rejected_evidence_pins failures verified earlier against clean integration/r2. ruff clean. No arm, no deploy.

…3 us threshold Desk half of sgl-project#705, the sgl-project#485 phase matrix applied to DECODE as sgl-project#702 applied it to prefill. No GPU touched. Verdict: the desk-net is positive if and only if a blocking TP all-reduce costs more than 14.3 microseconds on this rig. Everything else is priced; the whole question collapses to that one measurable threshold, which is the only number this desk could not source. Two premise corrections, one each way. (a) "20 GB of GDN weights" is ~4x too high for the part that actually moves. The MoE block is 77.7 percent of all weights and does NOT move -- it stays sharded on every layer. Only the GDN attention module relocates: 110.5 MiB per layer, 5,304 MiB total. (b) "75 percent of depth becomes sync-free" is FALSE, and this cuts against the proposal. Every layer is MoE (qwen3_next.py:580, is_layer_sparse = True unconditionally) and the MoE block issues its own all-reduce whenever moe_tp_size > 1 (moe/fused_moe_triton/layer.py:2047, :2062). No layer becomes sync-free at any depth. The removal is 48 of 128 collectives per round, 37.5 percent, not 75 percent of depth. (c) Cutting FOR the proposal: the 48 removed all-reduces are the BLOCKING kind (RowParallelLinear calls tensor_model_parallel_all_reduce synchronously, linear.py:2340), while the surviving MoE all-reduce is DEFERRED and joined in the next layer's prepare_attn (communicator.py:888-891, sgl-project#597). So the removed collectives are worth more per unit than a count comparison suggests. Bandwidth cost, priced: +0.668 ms/round for GDN attention weights (sharded 1,768 MiB each binds on the 3080s at 2.439 ms; solo 5,304 MiB on the 5090 is 3.107 ms) plus +0.018 ms for state, so +0.686 ms/round total -- far below the ticket's ~+5 ms envelope. The reason is that the sharded case is bound by the SLOWEST rank, so trading three slow readers for one fast one costs much less than the aggregate-bandwidth framing implies. Break-even is 0.686 ms / 48 = 14.3 us per collective, a low bar for a PHB round trip on a 10 KB payload. Capacity ledger: rank0 +4,688 MiB (GDN weights +3,536, GDN states +1,152 at 12 mamba slots), each 3080 frees 2,344 MiB, world net EXACTLY ZERO. Here that conservation is correct rather than the sgl-project#702 error: decode really is the TP phase, the pool really is the sum, and the DCP vector really can relieve a tight rank. sgl-project#702's mistake was applying this rule to PP prefill, where the pool is layer-sharded and takes the min. Residual tension named: capacity wants rank0's token share lower, bandwidth wants 2.4:1:1. Recommendation is to hold 2.4:1:1 and absorb the 4,688 MiB out of rank0's KV share, since the world pool is conserved and the 3080s were just handed 2,344 MiB each. Ranks 1/2 become zero-shard for the GDN family per sgl-project#115, an intermediate rung rather than all-or-nothing, as sgl-project#324 per-(rank,family) ratios allow. Ledger gap stated rather than smoothed: the census on disk is the PP layout, so world-neutrality holds as an accounting identity but the per-rank FIT of +4,688 MiB on rank0 is unverified. One conditional A/B arm defined, gated on measuring the break-even first, with the wait column as the acceptance signal and a falsifier on rank0 compute rising by ~0.69 ms and no more. Must not be co-armed with a sgl-project#702 prefill-cut change -- one phase at a time or neither result is attributable. Docs only. No arm, no deploy.

… arena ladder, and that reprices it The actuator survey came back and it changes the design, so the design changes rather than the finding being reinterpreted. ## R1: there is no runtime cross-rank weight mover, and the absence is deliberate regime_stages.py:100 defines REACH_NO_WEIGHT_MOVER -- "no runtime actuator moves weights, switching arms needs a restart (sgl-project#354/sgl-project#357)". StagePlan.reachable (:156) excludes it, so such a stage cannot be selected, and regime_act.py:121-130 wires exactly kv/vram/phase. Corroborated at uneven_perf.py:7744-7751. So sgl-project#363's "weight mover" is a REFUSAL CODE, not an actuator -- the opposite of the counter-versus-actuator failure, and honest. It is not something to wire to. What exists is weights_arena.py + phase_flip_boot.py:361: a fixed-address VRAM arena refilled by a host->device memcpy from a boot-baked pinned image, checksum-verified, with a rollback arm, and containing no dist.* call at all. On this no-P2P rig that is the better primitive anyway -- a rank-to-rank transfer would stage through host memory regardless -- and the fixed address keeps captured CUDA graphs valid across a rung change. ## The consequence, which is not favourable The arena is sized for the DEEPEST rung and resident at EVERY rung: a rank does not get its weight bytes back when the ladder sits shallow. solve_arena_ladder() models this, and two things fall out. First, the shallow rungs get poorer. Over a rank0 span of [31,38] the top rung drops from 518,433 to 390,700 live-equivalent -- below the incumbent's observed 434,878. The ladder costs ~10% pool at the top to buy up to 1.231x pipelined prefill at the bottom. Standing alone that is a marginal trade, and this commit says so in the design rather than shipping the flattering number. Second, and structurally: with free memory pinned by the arena, a rung's pool depends only on its ATTENTION count. Two rungs sharing an attention profile price identically and the faster one strictly dominates. Under an arena the ladder's real axis is the attention-count vector, not the raw layer cut. That invalidates my own earlier Slice 1 pair. [33,15,16] <-> [35,14,15] was chosen because it moves zero attention layers, which is exactly what makes it useless AS A LADDER PAIR. It is retained only to prove the mover mechanically; a pair that actually trades must cross an attention boundary, so Slice 1b uses [35,14,15] <-> [38,13,13]. The alternative mover -- reallocate per rung, restoring the full per-rung pool at the cost of a CUDA graph recapture -- is named and left open. Both solvers are in the branch; Slice 2 measures the recapture cost and closes the fork. ## Part B repriced by its own survey The cross-rank LSE merge is real, N-way and rank-order deterministic (layers/dcp/comm.py:228-262, all_gather places rank i at index i), which RETIRES the merge-order risk. But two blockers are confirmed: DCP groups are subsets of one PP stage by construction (parallel_state.py:3142-3159) and under PP prefill no DCP group is instantiated at all (dcp_group_guard.py:1-42); and the KV pool is DIMENSIONED by weight ownership -- model_runner_kv_cache_mixin.py:2466-2470 filters attention layer ids to the rank's own [start_layer, end_layer) before the pool is built, so a rank has no row-space for a layer it does not own. Part B is construction, not assembly; slice 3 splits into 3a/3b/3c at 4-7 windows, with the pool allocation rework on the critical path. layers/dcp/test_weightless_kv_math.py is confirmed test-only: it imports no torch and reimplements the LSE formula locally. It is not evidence of a wired path. ## Test results 60 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean on all new files, codespell clean. - test_ladder_controller_704.py (10, new): oscillation falsifiers on adversarial fill traces. The anti-flap test carries a CAN-FAIL PROOF: with the hysteresis collapsed to a bare threshold the controller emits 400 moves under a 1-token jitter; with it intact, zero. An earlier version of that test probed an INACTIVE threshold and passed for both, i.e. it was vacuous -- fixed by exposing active_descend_threshold() and asserting a fixed point. - test_layout_ladder_704.py (25, +5): arena sizing, the equal-attention-profile domination property, the deeper-ladder-costs-pool trade, and an unaffordable arena refused with its shortfall. - test_pp_cut_family_pool_704.py (6) and the two sgl-project#702 suites (19): unchanged. The controller is a DECISION FUNCTION and says so in its docstring: it moves no bytes and nothing is wired to it yet. Stated because a decision function mistaken for an actuator is the exact confusion this survey was run to avoid.

…ut arming floor consumed Three deliverables from the solver conflict, plus a retraction of my own rev4 validation claim. (1) ADJUDICATED KV-SCALING RULE -- Slot-3 is right, verified against the allocator, not taken on report. Token-scaling KV lives ONLY on full-attention layers: HybridLinearKVPool is documented "KV cache with separate pools for full and linear attention layers" (memory_pool.py:3606), full_kv_pool takes layer_num=self.full_layer_nums (:3688), and full_layer_nums = len(full_attention_layer_ids) (:3637). Linear/GDN layers hold per-SEQUENCE MambaPool slots, which is a residency subtraction and never a divisor. My rev3 divided by the rank's TOTAL layer count and was wrong. RETRACTION of my own rev4 test. I called two live boots a "metal discriminator". They are not, on two counts. First, this checkpoint places attention uniformly every 4th layer, so any multiple-of-4 cut has total/attn == 4 on every rank and the two rules agree exactly -- and BOTH available points are multiple-of-4. Second, I had solved two free constants from those two points, so reproducing them was circular. The rule is adopted on the allocator source. A test now pins the non-discrimination so nobody repeats the claim. (2) PER-LAYOUT SOLVED ARMING FLOOR. F4-r4's prime suspect is confirmed and quantified rather than asserted: rank0's floor rises 1728 -> 2255 MiB between the two layouts, and at 8 attention layers and 2048 B/token that is 33,728 tokens against a 39,919-token over-prediction -- 84 percent of the +9.6 percent common mode, leaving ~1.5 percent residual. arming_floor_mib is now a REQUIRED per-rank field with no default, so a constant cannot be passed by accident, and the docstring points callers at arming_floor_target_bytes (sgl-project#676). Known gap recorded in the docstring rather than smoothed: that solver derives the floor from a MEASURED seam draw, so an unbooted layout has no solved floor. Predicting one needs a draw-versus-layout model that does not exist. Every row for an unbooted cut therefore carries its proxy's uncertainty, about +-500 MiB = +-32,000 tokens = ~7 percent. (3) BACKTEST of both metal points: incumbent 436,767 against live 436,766, and [32,16,16] 416,794 against live 416,796, with the binding rank switching r1 -> r0 across them. [42,11,11] is treated as an INEQUALITY, not a point, because it OOM'd and never solved a pool; the model puts it far below the pin and no constant is fitted to it. Calibration honesty guard added: two metal points give two equations for three ranks, so rank2's free constant is BOUNDED, not identified -- a test perturbs it and shows neither reproduced point moves. (4) RE-EMITTED CUT TABLE, and it changes the recommendation. [33,15,16], the cut queued for boot, is predicted at 387,949 -- 11.2 percent BELOW the incumbent, i.e. it would fail the same pool gate that [32,16,16] failed at -4.4 percent. Better candidates on both axes: [30,18,16] at 539,003 (+23.4 percent pool, 1.111x pipelined) and [31,17,16] at 509,301 (+16.6 percent pool, 1.176x pipelined). The rev3-vs-rev5 gap at [33,15,16] is +3.12 percent, so that cut does discriminate the rules -- but only by about twice the residual and half the floor uncertainty, which is worth knowing before spending a boot on it. Tests: 12, red first. Planner suite 2622 passed, 2 failed -- both the pre-existing test_rejected_evidence_pins failures verified earlier against clean integration/r2. ruff clean. Canonical-implementation convergence with Slot-3's parallel pp_cut is NOT done here and needs the review gate; this commit makes the rule and its calibration limits explicit so the merge has one adjudicated form to converge on.

…tor co-solved The sgl-project#602 capacity solve withdrew the cut for the POOL objective. The question actually asked -- more prefill on the 5090 -- is a time objective and was never solved for. This adds it, and then corrects its own first framing. Timing model, calibrated from the measured 49.2/154.8/116.4 ms at [28,20,16] (1.757/7.740/7.275 ms per layer). Two objectives are reported side by side because they disagree: SERIAL sum(layers_r * ms_r) and PIPELINED max over stages. Backtest: the model reproduces the calibration point exactly, and the hand-arithmetic anchor holds -- [42,12,10] gives 239.43 ms against 320.4, i.e. 1.338x, matching the 1.34x expected by hand. Result table under the rank0 cap (x_serial / x_pipelined vs incumbent): [28,20,16] 1.000 / 1.000 incumbent [42,12,10] 1.338 / 1.667 the anchor [42,11,11] 1.341 / 1.818 DOMINATES the anchor on BOTH axes [42, 1,21] 1.367 / 1.013 serial-optimal and useless once pipelined The serial optimum is a degenerate cut that collapses pipelined throughput, so reporting only one objective would have recommended it. [42,11,11] is the candidate to arm. Rank0 cap recomputed rather than quoted: at 724.3 MiB/layer against a 31,800 MiB budget, 42 layers leave 1,379 MiB (corridor OK), 43 leave 655 MiB (CORRIDOR VIOLATED), 44 overflow by 69 MiB. So DESIGN_691's "about 42" is confirmed, and the binding constraint at 42 is the CORRIDOR, not raw weight overflow -- 43 fits the card and still must be refused. Co-solve correction (user, restating sgl-project#320/sgl-project#492): revision 1 priced a "pool cost" per candidate while holding the KV token vector PINNED. That is single-family optimization and sgl-project#485 forbids it. Layers moved to rank0 free exactly their weight bytes on the rank they left, and the uneven-DCP / rank-kv-ratio machinery relocates the displaced KV share onto those bytes. Rank0's cap therefore bounds rank0's SHARE of the token split, not the world pool. The conservation is exact, not approximate, and the tests prove it rather than assert it: total VRAM is fixed and the same 64 layers of weights exist wherever they sit, so total free bytes are invariant; under DCP token-sharding one token costs kv_per_token_per_layer * total_layers wherever it lands. World pool is therefore identical across every feasible cut, and the only residual is second-order -- seam/staging in both directions plus TP-phase redistribution -- which is ITEMIZED and whose sum equals the delta exactly (no fudge term). Scope note owed to the earlier verdict: "29,19,16 is 6.3 percent worse" held only under a pinned vector. Co-solved, that comparison does not stand as a capacity argument. Calibration limit stated in the model docstring and pinned by a test: one measured cut gives one point per rank and cannot separate per-layer slope from fixed per-stage cost. fixed_ms defaults to zero, the OPTIMISTIC end, so every speedup here is an upper bound. A second measured cut per rank would pin it. Tests: 15, red first. Planner regression 41 passed across the 702 and 602 suites. ruff clean. No deploy, no arm.

…icit solver output Two independent calibrations converged on the same blind spot from the same structural cause, so it is now a first-class output rather than a footnote each of us rediscovers. Pool side (sgl-project#702 rev5): two metal points give two equations for three ranks. Rank1 binds at [28,20,16] and rank0 at [32,16,16]; rank2 binds at neither, so its free constant is bounded, not identified. Timing side (planner/timing_calibration.py): the slice-1a pair [28,20,16] <-> [29,19,16] is three different calibrators -- rank0 weak, rank1 strong, rank2 carrying ZERO information because its layer count does not move. Every candidate cut so far keeps rank2 at 16 layers, and the [33,15,16] discriminator does too. A window spent on it cannot identify rank2 on either axis. calibration_coverage() makes that visible BEFORE the window is spent. The key distinction the type encodes: a rank whose layer count does not move is UNIDENTIFIED, not weakly levered. Its regressor column is identically zero, so no sample size recovers it, and chunks_for_target() returns infinity rather than a large finite number that would invite someone to buy more chunks. The sample-demand formula is the two-sample standard error on a difference of boot means, N = (sqrt(2) * SD / (target * |dn| * ms_per_layer))^2, which reproduces the timing analysis independently: 583 chunks for rank0 and 30.0 for rank1 against its 584 and 31, at SD=3 ms and a 10 percent target. Agreement from two directions on numbers neither fitted to the other. publishable_intercept() gates an intercept before it reaches the solver: refused without a standard error, refused at N=0, and refused when |value| < 2 SE. Refusing beats emitting fixed_ms=0, because downstream a zero reads as a MEASUREMENT that the fixed cost is absent when in fact the pair could not see it. This is the same refusal the timing solver already makes, expressed as a shared gate so the two solvers cannot disagree about what counts as measured. suggest_rank_moving_arm() proposes arms in BOTH directions when a rank's term is load-bearing, since one direction confounds the term with any monotone drift. Tests: 9, red first. ruff clean. Nothing rig-specific: coverage is a property of the two layer vectors, sample demand a property of the lever and the noise.

…ive requirements FOUR-BOOT GATE VERDICT: pinned available_bytes CONFIRMED at the binder, REFUTED at the non-binders, and the refutation is fully explained. PP0 pinned 6,261,477,376 metal 8,526,565,376 -26.57% PP1 pinned 4,472,483,840 metal 4,741,928,960 -5.68% PP2 pinned 3,577,987,072 metal 3,575,365,632 +0.07% <- binder I had pinned available_bytes = pool x cell for EVERY stage, which assumes every stage is exactly saturated. Under the min-rule only the BINDER is; the others carry slack. So the pin was determinate only at PP2 and is confirmed there to 0.07 percent. Same pattern in the reserve: mine 5,163.8 MiB vs metal 5,166.3 at PP2 (-0.05 percent), but +7.2 and +32.3 percent at PP1/PP0. The method is validated where it was decidable and the error elsewhere is the min-rule, not a modelling defect. World pool 436,446, binding rank PP2, both as the boot reports. BONUS SETTLED: cell_size is 14336 / 10240 / 8192 = exactly 7 / 5 / 4 attention layers x 2048 B. The sgl-project#702 attention-divisor now rests on the running system's ABSOLUTE values, not on a ratio, so the multiple-of-4 degeneracy that made [28,20,16] and [32,16,16] non-discriminating no longer matters. CORRECTION to my 0.852 claim, on F4-r4's challenge, which was right. The budget post is "mamba state pool + speculative intermediate state + prefill activation reserve" while the allocated line sums conv + ssm + intermediate_ssm + intermediate_conv. Those are not the same quantity, so "the mamba post under-charges mamba by 14.8 percent" is WITHDRAWN. What survives is stronger in kind: the post covers a nominal SUPERSET of the allocated line and is nevertheless SMALLER by 0.155 / 0.111 / 0.089 GiB, which cannot be legitimate. The gap is real and at least that size; which sub-term carries it needs the post's three components emitted separately -- the same missing-instrument fix a third time. The solver charges the ALLOCATION either way, because that is what the allocator took. Reserve figures corrected throughout to metal: 6,688 / 3,561 / 5,166 MiB, a 1.88x spread (my earlier 8,848 / 3,818 / 5,164 back-out was the non-binder artifact above). ENTRY POINT reshaped to Slot-3's five requirements: 1. attn_counts DERIVED from layer_families via PoolContext, never passed -- an inconsistent pair is now unrepresentable, and it is the sgl-project#702 divisor. 2. arming_floor_for required, no default (E3), and still NOT subtracted: it lives inside the recovered reserve and charging it twice would understate every rung by ~2 GiB. 3. binding stage and per-stage caps RETURNED, so the controller does not recompute them in a second code path. 4. PoolContext is built once by the caller; each solve is pure arithmetic with no NVML, census or config parsing, so an enumeration over thousands of cuts is cheap. 5. KV cell from kv_mib_per_token_per_attn_layer_from_config, converged VERBATIM from feat/704-prefill-ladder into pp_cut together with kv_dtype_width_bytes and its width table, so there is one implementation rather than two that must agree. It yields exactly 2048 B, matching metal. Optional explain=True returns per-stage weights/mamba/floor/free-for-KV terms. Tests: 21 across the two suites, red first. Planner suite 2654 passed, 2 failed -- the pre-existing test_rejected_evidence_pins pair. ruff clean. Slot-3 was not reachable by name from this session; providers accept callable or mapping so his call sites can be matched without a breaking change.

…ith the pool price stated The user's question from 2026-08-16, unowned: more layers on the 5090, what does it cost? The capacity solver answers "what cut holds the most context", a different objective, so it never covered this. planner/prefill_frontier.py solves it and charges THREE prices per candidate, because quoting only the first is how a cut gets recommended that cannot serve. ## Two decisions fall out, and they are independent 1. COUPLED OR DECOUPLED -- and coupled cannot buy depth at any price. Today the pool collapses: [33,15,16] costs 17% of context for 1.33x, [42,11,11] costs 85% for 1.82x. Nothing past ~[34,15,15] is purchasable. Under decoupling the pool is EXACTLY cut-independent at 514,034, i.e. +17.7% over the incumbent's observed 436,766, because total weight bytes and total GDN state are invariant under a re-cut -- only their distribution moves. So the pool price of depth is not merely affordable under decoupling, it is NEGATIVE. That inverts the usual framing: decoupling is not a cost centre bought for speed, it is what makes the speed free of context loss. 2. WITH OR WITHOUT THE PIPELINING LEVER -- and this one has a trap. Net speedup without cross-chunk pipelining IS NOT MONOTONE in depth. It peaks at [42,11,11] (1.660x) then FALLS: 1.630x at [43,10,11], 1.569x at [44,10,10]. Past the peak the collective overhead grows faster than the compute gain, so a deeper cut is ACTIVELY WORSE, not merely diminishing. A frontier reporting only compute speedup would recommend exactly those cuts, which is why they carry needs_pipelining. ## The pick today, decoupled, no lever : [42,11,11] 1.660x net at +17.7% pool with the lever built : [44,10,10] 2.000x net at +17.7% pool coupled, if decoupling slips: [34,15,15] 1.333x for -24%, or [33,15,16] for 1.330x at -17%; beyond that the context loss is not a trade, it is a failure DO NOT pick [28,17,19] : decoupled it is net 0.980x, SLOWER than the incumbent -- at incumbent depth the collective buys nothing and still costs its overhead. Decoupling does not pay for itself until ~[29,17,18]. The lever is worth the last 0.34x and nothing else, which is the honest way to rank it against other work. ## Discipline No rig constant appears in the solver; this rig's figures are calibration data in the test. available_bytes_for and attn_counts_for are injected because they depend on the boot's own instruments, which no arithmetic here can invent. Every speedup is an UPPER BOUND until slice 1a-i lands the timing intercept (fixed_ms=0 is the optimistic end). Pool figures extrapolate the four-boot gate's metal available_bytes across cuts and carry the unquantified reserve term pending Slot-2's instrument boot -- the solver reports measured=False for exactly that reason. Overhead uses measured links with the authoritative card mapping. 201 tests green, hermetic (CUDA_VISIBLE_DEVICES=""), ruff + codespell clean. test_prefill_frontier_702.py (9): the decoupled pool exactly cut-independent; the coupled pool collapsing with depth; net-without-pipelining NOT monotone with an interior peak; the deepest cut winning only with the lever; cuts past the peak flagged; the shallowest cuts failing to pay for themselves; self-labelling; a foreign profile with different depth/stages/links/geometry; malformed inputs refused.

…losed form: coupled is far kinder than I said The instrument boot landed and sgl-project#707 closed, so the frontier now prices pool with Slot-2's closed form instead of my extrapolation. ## His formula, verified from the capture allowed_tokens = id_space + (free_at_measure - arming_floor - margin) / cell holdback_frac = 1 - allowed_tokens / (profiled_bytes / cell) Reproduces the reported 45.143 / 44.074 / 60.258 % to 0.000 pp on all three ranks. adjusted == available_bytes confirmed to the capture's 0.1 MiB print precision, and cell == attn_layers x 2048 EXACTLY (14336/10240/8192 against 7/5/4) -- a third independent confirmation of the sgl-project#702 divisor, now from a live sizer rather than allocator source or a K-size back-computation. It is a CAP, not a subtraction, so the resting free column still holds the arming floor; nothing is set aside twice. PP pays because a flip must ARM out of that column; the TP pass measures 0.000% because there is no flip to arm. ## Why the binder holds back most, and it is not waste PP2's bracket (free_at_measure - arming_floor - margin) is 0.0 MiB -- it sits exactly AT its arming floor -- against PP0's 2164.8 and PP1's 260.2. So its allowed tokens collapse to id_space, i.e. it SETS the pool. It simultaneously has the smallest cell (4 attention layers), hence the largest raw capacity and therefore the largest holdback FRACTION. Binding and holding back most co-occur by construction. Reading PP2's 60.3% as waste is exactly backwards. ## Three changes to the frontier, one a correction to me 1. COUPLED IS FAR KINDER THAN I REPORTED. My extrapolation put [33,15,16] at -17% and [42,11,11] at -85%. The truth is -5% and -56%, because allowed_tokens is floored at id_space, which does not shrink with the cut. So [33,15,16] buys 1.330x for FIVE PERCENT of context, TODAY, with neither decoupling nor the lever -- the cheapest real win on the whole frontier, and my earlier table hid it. 2. THE SEAM CAP BOUNDS THE DEPTH. Past n0=51 a rank's free column no longer clears its arming floor, so the layout cannot ARM a flip. Those cuts are REFUSED by the provider and never reach the frontier -- absent, not priced as a tiny pool. An extrapolation cannot produce that boundary; a closed form can. 3. BOTH OPTIMA ARE INTERIOR, for two different reasons. Without the lever, overhead outgrows the compute gain past [42,11,11] (1.660x). WITH it, the raw compute speedup itself peaks at [44,10,10] (2.000x) and then FALLS -- piling layers onto the fast card eventually makes a TAIL stage the bottleneck. "More layers on the 5090" has a limit that is not about memory at all. Both headline picks survive unchanged: [42,11,11] at 1.660x today, [44,10,10] at 2.000x with the lever, and needs_pipelining stays flagged on [43,10,11] and [44,10,10] and deeper. ## Provenance is now per layout FrontierPoint.pool_measured replaces the single frontier-level flag: the booted incumbent is MEASURED and reproduces its own binder; every other row is extrapolated across ONLY the free_at_measure shift, charged per FAMILY (374.2 MiB per attention layer, 476.2 per linear, 51.20 per GDN layer). One global flag would let the measured row launder the extrapolated ones. A physical detail the per-family split exposes: converting a linear layer into an attention one at constant layer count FREES 153.2 MiB (476.2 - 374.2 weights, plus 51.20 of GDN state that attention layers do not carry) -- but enlarges the cell, so fewer tokens per byte. The two effects pull opposite ways, which is why the frontier prices cell and bracket separately. ## Test results 212 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff + codespell clean. test_seam_holdback_707.py (8, new): the closed form reproducing every reported holdback; the cell identity; the binder-holds-back-most structural test; the booted layout round-tripping to its own metal available_bytes; the per-family shift; a layout that cannot arm refused by name; a zero-attention stage refused. test_prefill_frontier_702.py (12): coupled declining but buying early steps cheaply, both optima interior, the seam-cap depth bound, per-layout provenance.

…ch had already refuted ## The retraction My sgl-project#701 section concluded "admission is per-chunk, the failure is per-total". That premise is FALSE. schedule_policy.py:1464 gates the FULL lifetime at first admission: if total_tokens >= self.rem_total_tokens: ... return AddReqResult.NO_TOKEN So a single request longer than the pool is REFUSED, not admitted-then-wedged, and my falsifier's headline case modelled a defect the shipped code prevents. Worse: the retraction already existed ON THIS BRANCH. Commit 845ac92 ("[sgl-project#701] Slice-3 rework: retract the mechanism story, close the real holes") had verified :1464, retracted exactly this story in its design doc, module docstring and both test files, and closed the real defect. I re-derived the refuted version without reading it. I searched the deploy tree for the mechanism but never checked whether sgl-project#701 was already owned and answered. ## What the real defect is, per the work that already exists 1. Paper-evictable funds an admission the evictor cannot honour: schedule_policy.py:734-737 -- rem_total_tokens includes full_evictable_size() while the allocator recovers only MAMBA-recoverable bytes. Gate passes; later relief frees 0. 2. The missing reservation is CROSS-PASS. PrefillAdder is rebuilt each pass and reserves only remaining DECODE, so a resident chunked request's remaining PREFILL is represented nowhere later. Later admissions spend its committed future, and the deadlock needs TWO ACTORS -- which is why a single-request analysis could never find it. ## Duplicates deleted planner/chunked_admission.py already provides ChunkedCommitmentLedger, decide_chunked_admission, effective_rem_total_tokens, defer-age tracking and idle-flip blocking, with tests in planner/ AND managers/ -- the latter running against the REAL PrefillAdder, which is strictly better evidence than the toy model I wrote. So I deleted my duplicate managers/chunked_admission.py and my planner/chunked_deadlock.py rather than keep a second, weaker account. ## The one genuinely additive piece, kept A monotone committed_chunks counter on the EXISTING ledger's spend() -- the single commit path. It lets the sgl-project#699 detector separate a retry loop (attempts advancing, nothing committing) from real progress, which forward_ct cannot do because it counts ATTEMPTS (scheduler.py:6933). Not rewound by release(): a progress counter that goes backwards reads as a restart to any watcher. progress_liveness gains an `attempts` field and retry_loop_detection, binding forward_ct to attempts and the ledger's committed_chunks to the commit signal. When the ledger is absent the commit signal stays at ZERO rather than borrowing the attempt count -- an invented commit would make a retry loop look like progress, which is the failure being hunted. ## sgl-project#702 noise floor (window 0b6c7db) A-vs-A spread is 14.1%, clean single-stream prefill ~1,820 tok/s. Rungs predicting less than +14.1% are NOT FINDINGS and are now flagged below_noise_floor: [28,17,19] (+12.0%) is disqualified outright. The recommended picks are unaffected -- [42,11,11] (+81.8%) and [44,10,10] (+100%) clear it widely. Also noted: 1,820 tok/s single-stream against 3,307 implied by the pipelined stage times is consistent with one stream not filling a 3-stage pipeline, so single-stream figures must not be substituted for pipelined ones. 237 tests green, hermetic, ruff + codespell clean.

… the sizer must NOT claim it VERDICT: no dead reservation found. The gap is the arming floor (deliberate) plus min-rule residue (structural), so this is the labeled accounting table the brief asks for in that case, not a forced fix. Per-rank, from the sgl-project#704/sgl-project#707 instrument chain rather than modelled: rank budget weights mamba rest holdback allowed used slack PP0 31800 16064.5 916.5 14819.5 6690.1 8129.5 5964.7 2164.8 PP1 18800 10061.8 654.3 8083.4 3562.7 4520.7 4260.5 260.2 PP2 19800 10699.8 523.3 8576.3 5167.9 3408.4 3408.4 0.0 Total slack 2,425 MiB, and the shape is the finding: it is almost all on PP0, and the BINDER PP2 has exactly zero. HOLDBACK is deliberate, established in sgl-project#707: the pool is capped so the resting free column still holds the arming floor. Sizing below it is the boot that holds the corridor and never flips (sgl-project#656 E/G). Not claimable. SLACK is structural and still not claimable BY THE SIZER. The pool is one global token count, so a non-binding rank cannot spend its extra allowance without the binder moving -- PP0's 2,164.8 MiB is not booked-and-untouched, it is unreachable at this cut. A sizer that claimed it would hand the pool tokens PP2 cannot back, and PP2 is the rank that OOMs. That is the sgl-project#593-family direction the brief warns about, so I wrote NO red-first test for it: the falsifier would have to assert a wrong answer. WHERE THE CAPACITY ACTUALLY IS: the slack is a property of the CUT, and that is already solved on another strand. sgl-project#702 rev5 found the incumbent [28,20,16] is not pool-optimal -- the binding rank switches, so moving layers OFF the binder raises the pool, and [30,18,16] gives +20 percent AND 1.11x pipelined prefill. Claim it by rebalancing the cut, not by relaxing a reserve. WHAT I DO NOT CLAIM: the ticket's 2.0 / 5.7 / 3.7 GiB per card is a DIFFERENT operating point from the instrumented boot, and the instruments that would attribute it per-term only exist on boots carrying 2a6305d / 5f3e61f / f55c1a8. Mapping this table onto those three numbers is the cross-boot arithmetic this strand has already retracted twice, so I did not. Ask for the next boot to capture the three instrument lines at that operating point and the table re-runs against it directly -- a window item. One thing that CAN be said without it: against the measured arming floors (1728 / 1825 / 2467 MiB) the per-card free exceeds the ~1024 MiB corridor target BY CONSTRUCTION, because on a flip-enabled boot the arming floor is the binding level, not the corridor. Reading the gap against 1024 overstates it on every rank. Docs only. No sizer change, deliberately.

…er a conservative floor THE MEASUREMENT IS THE POINT AND THE GATE IS SECONDARY -- that way round, on purpose, and it is a change from the shape I was asked for. Three external-kill crashes now, all flip-proximate: 03:22:17 death, flip 03:21:05 -> rank0 lost ~7 s after the flip 03:31:17 death, flip 03:31:06 -> 11 s 03:38:53 death, flip 03:38:07 -> 46 s Two of the three killed RANK0 ALONE (rank0 contributes 2 log lines while the peers contribute 60-134 each -- the absent rank is the one that died), and rank0 is PP0, the 35 G RSS process, the largest on the box. The third took the whole tree symmetrically. A 21:24 kill is ledger-confirmed as the kernel OOM killer. BUT STEADY-STATE HOST HEADROOM IS ~38 G. So a floor small enough to be safe would never have fired on any of these, and a floor large enough to fire would defer constantly. I have NO measured projected host transient for the flip, and I will not invent one to make a gate look decisive -- that is the hand-fitted input mistake I already made once on sgl-project#702. So the guard LOGS host availability at every flip, which nobody currently records, and defers only under a deliberately conservative floor: * availability collapses at flips -> transient candidate confirmed, and the floor can then be set FROM DATA rather than from a guess; * availability stays flat while kills continue -> the flip is exonerated and lane RSS spikes gain. The discriminator works whichever way it points, which is the property that justified shipping ahead of attribution. FLOOR = 4 GiB, justified from this box, not chosen round: ~10% of the measured 38 G steady headroom -- enough to absorb the lane RSS jitter sharing this container (pytest/git spikes run under 2 G) without deferring on noise. Deliberately NOT the 10 G PINNED_HOST_RESERVE_BYTES: that reserve protects PERMANENT pins; a staging transient is by definition returned. BOUNDED DEFER, 3 then escalate-and-proceed. A permanent hold is WORSE than the hazard: the flip is how this instance alternates prefill and decode, so holding forever converts a POSSIBLE process kill into a CERTAIN half-service outage -- and the kill is recoverable by a restore while the outage is not self-clearing. The escalation says PROCEEDING WITH EYES OPEN and prints every term. UNREADABLE HOST RAM STANDS THE GUARD DOWN. pinned_host_memory_bytes returns (None, None) when there is no honest number, and refusing a flip on a fabricated figure is worse than not checking -- the refusal is the thing with a service cost. sgl-project#696 INTERACTION, NAMED NOT ABSORBED: the defer reason is the shared constant DEFERRED-HOST-RAM, so SLO / ARM-UNFUNDED accounting sees a host-RAM defer as its own fact. A flip that did not arm because the HOST was tight is not the same event as one that could not fund its VRAM seam, and merging them would hide exactly the signal sgl-project#721 is collecting. It is a named defer reason, not a new policy: the defer rides the existing unanimous too_small ballot. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): healthy allows and does NOT name a defer ..... CAN-FAIL: a guard that deferred on healthy state would pass every firing test and be useless low headroom defers; boundary is exactly transient+floor unreadable stands the guard DOWN defer is bounded, then ESCALATED and proceeds every term quoted; defer reason is the shared constant; degenerate inputs -> 8 passed + 6 subtests; 82 passed + 19 subtests with flip-runtime and 703. ruff clean in both changed regions. CAN-FAIL PROVEN BY MUTATION: a never-defer mutant fails 5 of 8; reverted green. The guard is wrapped so it can never break a flip: on any exception the flip proceeds UNGUARDED and says so, rather than being refused on an unknown.

…or that exists The rung machinery already existed and was NOT rebuilt: layout_ladder.py already solved rungs as a Pareto frontier, indexed them by occupancy (Rung.admit_up_to_tokens) and derived hysteresis bands (descend_below_tokens / ascend_above_tokens). What was missing was the PRICE of a step, and the price was wrong by an order of magnitude. _solve_transitions charged a step as moved_layers x weight_mib_per_layer over the gating link. That prices a cross-rank weight mover, which does not exist and whose absence is explicit (regime_stages.py:100, REACH_NO_WEIGHT_MOVER). The actuator that DOES exist is PhaseFlipStacks.refill (phase_flip_boot.py:361, arena_refill :539, dst.copy_(payload) :576): a CONTIGUOUS host->device memcpy of a whole boot-baked arena image. The bytes on the wire are the same whether one layer moves or six. With sgl-project#690-rev2's 9614.9 MiB per rank over the measured links (13/13/6.4 GB/s, authoritative mapping) and nothing crossing a rank boundary -- so the refills run CONCURRENTLY and the slowest card sets the step -- a rung change costs 1575.3 ms: 10.7x to 21.3x the moved-layer estimate, and 38-79% of a whole phase flip (sgl-project#690's fixed 2.0-4.2 s). The band consequence is real but small (every ascend trigger drops ~30k tokens, under 1% of pool): the mispricing mattered for the DECISION, not for the guard. The consequence that changes how a ladder must be driven: the switch cost is CONSTANT IN THE DISTANCE TRAVELLED. Crossing twelve rungs one at a time costs 12 x 1.575 = 18.9 s; crossing them in one jump costs 1.575 s. A constant-cost actuator inverts the usual intuition -- rungs are choices of destination, not stations to stop at. Per-step value is also wildly unequal: on the draining leg at arena depth 40, one step buys 23% for 1.575 s (payback 8.4 s) and the next buys 1.8% for the same 1.575 s (payback 87.9 s). Arena depth is the master knob and a real trade, since the arena is sized for the deepest reachable rung and resident at every rung: deepest rank0 | rungs | roomiest pool | fastest | end-to-end payback 36 | 2 | 421,894 | 1.0071 | 224.7 s (dead) 38 | 5 | 418,848 | 1.1538 | 11.8 s 40 | 8 | 364,413 | 1.2532 | 7.8 s At depth 36 the ladder is economically dead; at 40 it is live but the roomiest rung has lost 13.6% of its pool purely for holding the option. A defect this found in my OWN new API: the first cut of solve_fill_ladder required ASCENDING fill levels. Rising fill only ever moves to roomier, SLOWER rungs, so the interface could express nothing but forced retreats and reported an infinite payback on every one -- the discretionary step, the one the function exists to price, was unreachable through it. Monotone in either direction now, and both legs are pinned by test. The two kinds of step are distinguished explicitly: ASCEND is MANDATORY (its alternative is not "stay fast" but "stop admitting", so infinite payback is correct and must not be read as "never do this"), DESCEND is DISCRETIONARY and is the only step sgl-project#677 decides. sgl-project#677 integration: a cut jump and a phase flip are both an arena refill, so they spend the SAME budget and a rank cannot do both at once -- the controller ranks them rather than running two policies. A discretionary cut jump is admitted only when it is a jump not a step, its payback fits sgl-project#677's backlog-derived window, it beats a flip for the same stall (a flip changes regime; a cut jump buys a prefill factor within one), and the hysteresis band already permits it -- economics are a veto on top of the guard, never a replacement. Mandatory ascends bypass all four. HONEST LABELLING: no cut is recommended and nothing is added to the user's morning list, which carries sgl-project#702's cut alone. Every pool and speed number is solver output on the STRUCTURAL free-bytes fixture the review gate found was fitted against the incumbent, and inherits the +-500 MiB unbooted arming-floor uncertainty (~7% at 8 attention layers). The cost side rests on measured inputs, but 1575 ms is an arithmetic prediction from measured bandwidth, not a measured switch: the ladder has never performed a rung change on metal. 9614.9 MiB is taken from sgl-project#690-rev2 and assumed uniform per rank. Test results (hermetic, CUDA_VISIBLE_DEVICES="", interpreter /spinning/htsglang-gpu/.venv/bin/python3): test/registered/unit/planner/test_fill_ladder_704a.py (new) 20 passed test/registered/unit/planner/test_layout_ladder_704.py 25 passed test/registered/unit/planner full suite 2825 passed, 123 skipped, 157 subtests passed, 0 failed (2805 before this slice) can-fail by breakage, two independent neuters: summing the per-rank refills instead of taking the max (i.e. denying concurrency) turns 2 tests red reverting the bands to the moved-layer estimate turns test_the_real_cost_makes_the_BANDS_STRICTLY_MORE_CONSERVATIVE red restoring returns 20 passed Desk only. No GPU, no serving, deploy tree untouched.

…le, and what the phase matrix owes Docs only. No code changed, nothing built or wired. ## sgl-project#702 decision table (docs/DECISION_702_cut_table.md) Re-derived on the sgl-project#707 closed form with the calibrated fixture. Incumbent pool reads 436,275 here against the 436,766 in older notes; all percentages use 436,275 so they are internally consistent. cut attn pool speed noise(14.1%) gate [28,20,16] (7,5,4) 436,275 1.0000 -- is the gate [30,18,16] (7,5,4) 436,275 (+0.0%) 1.1111 BELOW IT PASSES [33,15,16] (8,4,4) 415,859 (-4.7%) 1.3299 above FAILS Switch cost 1575 ms from the incumbent either way (distance-independent, per #704a); payback 15.75 s and 6.35 s respectively. THE TRADE: the two candidates fail in OPPOSITE places, which is the decision. [30,18,16] passes the stage-1 "pool >= incumbent" gate and cannot be validated by measurement -- its 11.1% gain sits BELOW this rig's measured 14.1% A-vs-A noise floor, so no boot here can tell it from the incumbent. [33,15,16] clears the noise floor comfortably but needs a 4.7% capacity waiver. No ranking is offered; the pick is the user's. [30,18,16]'s pool is identical STRUCTURALLY, not coincidentally: it keeps the (7,5,4) attention split and leaves PP2 (layers 48-63) untouched, and PP2 is the binder. Per-rank capacity is [594,615/462,920/436,275] at the incumbent and [517,464/570,931/436,275] here -- PP2's column is unchanged, so moving two layers between PP0 and PP1 cannot move a pool PP2 sets. CORRECTION: I could not reproduce a "+20% pool" figure for [30,18,16]; its coupled pool is +0.0%. The nearest +17.7% on the list is the DECOUPLED pool, which is cut-INDEPENDENT (the #704b R6 sum-rule result: 513,875 for every cut including the incumbent) and therefore not attributable to any cut. SOLVER DEFECT FOUND, flagged not fixed: prefill_frontier.py:144-156 enumerates, per lead depth, ONLY the tail split that minimises pipelined time. At n0=30 that is [30,16,18] (130.95 ms), so [30,18,16] (139.32 ms) is never generated -- yet [30,16,18] holds 343,951 tokens (-21.2%) against [30,18,16]'s 436,275 (+0.0%). The frontier is Pareto-optimal in SPEED but not complete in POOL: a roomier, slightly slower cut at the same lead depth cannot appear on it. [30,18,16] strictly dominates the incumbent (equal pool, more speed) and was invisible for exactly that reason, which is why it never reached the sgl-project#702 block. The fix would be to keep the Pareto set over (pool, speed) per lead depth instead of the speed-argmin; other pool-preserving cuts may also be hidden. Validation-boot evidence is specified in falsification order, and includes the one that matters most: for [30,18,16] the prefill A/B is expected to be INCONCLUSIVE by construction -- a boot reporting a clean 1.11x has measured its own noise, and that result must be rejected rather than celebrated. Appended to the morning plan file as a dated block per its amendment convention; head-298 sha256 verified byte-identical before and after (81560e94b7628e6c8...). That file is not version-controlled, hence this copy. ## sgl-project#485 completion verdict (docs/VERDICT_485_phase_matrix.md) Directive quoted verbatim from /spinning/htsglang/CLAUDE.md:87-96 (user law 2026-08-03), which is NOT tracked in this worktree. The two facets under sgl-project#485 are the same ticket, not a number collision: the matrix doctrine plus its prefill-column solver, and the seam/memory certification gating it. VERDICT: PARTIALLY DELIVERED, do not close. DELIVERED prefill column for the attention-vs-GDN/linear boundary (c08f613, b675630, b3c9802, e645aa7), with a second axis on the attention row (sgl-project#492 d82a778, corrected by sgl-project#503 447249a). Its own feasibility facet (407b382) is RED: the threshold is not reachable at the certification pool. SUPERSEDED d937d5f refuses the decode-side family split on sound MEASURED reasoning -- against the honest proportional (uneven-TP) baseline the split is worth +0.090 ms/round, ~0.3% of a 30 ms bs=1 round. But this closes ONE CANDIDATE CELL (concentration), not the column. Named tension: the directive says single-family arms are DIAGNOSTIC only and must never be phrased as a phase-level verdict, and that commit is worded as one. The narrow reading is adopted. OPEN O1 no (family x phase) structure exists anywhere -- pp_cut.py solves prefill only, family_split.py returns by_family with NO phase argument or dimension; the "matrix" is prose over two single-phase solvers. O2 the decode solver is DEAD CODE: solve_family_placement has exactly two references, its definition (family_split.py:141) and its test, zero call sites. O3 vocab, experts, nonlinear kernels and per-quant-lane linear layers were never cut at all. O4 the law covers diffusion, SR/video, TTS/ASR and training tenants -- entirely untouched, and the largest open surface. O5 the prefill column is not arm-able while Facet B is red. NOT sgl-project#485 sgl-project#704/#704a/#704b are a PP-cut layout LADDER and its arming machinery. They consume the prefill-column solve as an input and must not be counted toward the matrix; counting them would inflate its delivery with work on a different axis. Status in one line: one of two columns, one of many families, one of several workloads -- and the table is still prose. Every load-bearing claim spot-verified against the tree rather than taken from the survey: directive text read at source, solve_family_placement call sites grepped, d937d5f's baselines read from the commit body, [30,18,16] priced directly rather than inferred from frontier membership. Sanity: 35 passed (test_prefill_frontier_702.py, test_fill_ladder_704a.py). Desk only, no GPU, deploy tree untouched.

…ision set nearly doubles THE DEFECT (self-flagged in 4c5e594): solve_prefill_frontier enumerated, per lead depth, ONLY the tail minimising pipelined time. At n0=30 that is [30,16,18] (130.95 ms, 343,951 tokens), so [30,18,16] (139.32 ms, 436,275 tokens) was never generated -- six percent of speed bought at twenty-seven percent of pool, decided inside an enumeration and never surfaced. [30,18,16] holds the incumbent's own pool while running 1.11x faster, so it STRICTLY DOMINATES the layout in production, and the solver whose purpose is to find such cuts could not emit it. It reached the sgl-project#702 table only because I priced it by hand. THE FIX: per lead depth, keep the Pareto set over (pipelined time, pool). Sort by time ascending, keep strict pool improvements. The first element is the speed-argmin, so every pre-sgl-project#723 row still appears unmoved; the rest are additions, each roomier than everything faster. needs_pipelining became a property of the DEPTH, evaluated on that depth's fastest tail and shared by its siblings. Per-tail evaluation would flag a roomier, slower sibling as needing a lever it is not reaching for -- that cut trades speed for pool, it does not buy depth. Using the depth's fastest tail also reproduces the pre-sgl-project#723 flags exactly, because that tail used to be the only candidate at its depth. RESULT: 25 -> 41 candidates, 16 previously invisible. [31,17,16] STRICTLY BETTER than the cut this ticket was opened for: same pool as the incumbent (436,275, +0.0%), 1.1765 prefill, and at 17.7% its gain is ABOVE the rig's 14.1% A-vs-A noise floor -- so unlike [30,18,16] a boot could actually confirm it. Payback 10.50 s vs 15.75 s. Cuts that BEAT the incumbent pool, all previously invisible: [30,19,15] +15.5% @ 1.0526, [31,18,15] +9.8% @ 1.1111, [29,20,15] +6.1% @ 1.0000. The old frontier's roomiest row WAS the incumbent; it no longer is. FALSIFICATION FOUND, and it changes [33,15,16] on the user's morning list. [32,16,16] also surfaced, predicted pool-neutral (+0.0%) at 1.2500 -- and it is the layout that was BOOTED and FAILED its pool gate (STAGE 1 VERDICT 2026-08-16). available_bytes_for_cut shifts a bracket captured at the INCUMBENT by weight and GDN terms only (seam_holdback.py:145-147), so a change in the ARMING FLOOR is structurally invisible to the model. [32,16,16] measured rank0's floor rise 1728 -> 2255 MiB (+527) and is the only booted layout with the (8,4,4) split. Applying that measured delta: [32,16,16] +0.0% -> -4.7% (consistent with its observed gate failure) [32,17,15] +3.1% -> -4.7% [33,15,16] -4.7% -> -12.4% So [33,15,16]'s advertised -4.7% is OPTIMISTIC; against the only measured arming floor for its own attention split it is -12.4%. The +-500 MiB caveat is not abstract -- it was measured at +527 MiB and has already failed one boot. A tidy rule I asserted in a first test draft was FALSE and the test caught it: "pool-neutral cuts keep the (7,5,4) split". [32,16,16] is predicted pool-neutral WITH the changed split. What separates the safe family is the attention split being unchanged, not the predicted pool. The test now pins both families so the booted counter-example keeps its row. Test results (hermetic, CUDA_VISIBLE_DEVICES="", interpreter /spinning/htsglang-gpu/.venv/bin/python3): test_frontier_pool_pareto_723.py (new) 12 passed red-first before the fix: 3 failed on the missing cut, and the must-not-move pins were already green at that point test_prefill_frontier_702.py 15 passed, unchanged test/registered/unit/planner full suite 2837 passed, 123 skipped, 157 subtests, 0 failed (2825 before) can-fail by breakage: reverting the enumeration to the speed-argmin (one `break`) turns 6 tests red while every pre-sgl-project#723 pin stays green, which is the property that matters -- the fix adds rows without moving one. Decision set appended to docs/DECISION_702_cut_table.md and to the morning plan file as a dated block; plan head-298 sha256 verified byte-identical before and after (81560e94b7628e6c8...). No recommendation added: the set is now complete rather than filtered, and the pick remains the user's. Desk only, no GPU, deploy tree untouched.

…ferable vs lazy-fundable The floor was measured once and frozen, never designed, and that is visible in the source rather than a matter of opinion: arming_floor = corridor band floor + seam_entry_reserve, where the first half is a stated policy with a stated tolerance (1024 - 20% = 819 MiB) and the second is one scalar whose own docstring calls it "the shipped allowance". Per-rank it decomposes exactly: rank 0 (5090) 1728 MiB = 819 + 909 seam draw rank 1 (3080 x4) 1825 MiB = 819 + 1006 rank 2 (3080 x8) 2467 MiB = 819 + 1648 Rank 2 holds 81% more than rank 0 with no recorded reason. The defect is not the size of the number, it is that it is ONE number: a monolithic holdback cannot be traded, because trading requires knowing which part buys what. STAGING, AND WHAT ACTUALLY FORCES IT. Under barlink a cross-card transfer is one PCIe crossing, so a host bounce (two) is strictly worse and is not offered there. But the reshard stages in VRAM on both sides regardless of transport, and rollback semantics are NOT why: _dist_exchange allocates a uint8 receive buffer per peer and sends a gathered contiguous buffer (kv_reshard.py:939-995), because the wire format is a flat byte stream per peer while the destination rows are scattered ids. The obstacle to BAR1 landing in-place is the SCATTER, not rollback and not write ordering -- so the cross-card staging term goes to ~zero only under a layout co-design that makes a peer's destination extent contiguous. That is the largest structural reduction available here. RESTORE, NEVER REBUILD adopted as a named invariant: the flip may pay copy time, never build time. Consequence: capture-moment workspace is a BOOT-time component, not a per-flip floor term (both layouts captured once, graph state parked to host and restored per flip, 40-85 ms band, sgl-project#464 coalescing pending). The weights refill already complies. TIME PRICE of host-bouncing, from the H2D rates I measured in sgl-project#690 rather than nameplate: 125 / 199 / 181 ms for the whole seam draw on ranks 0/1/2, i.e. 4-6% of a ~3.1 s flip to return 909-1648 MiB per card permanently. Stated as a FLOOR on the cost, not a wall-clock delta: the refill is already 41-52% of the flip and shares the same link, so bounces queue behind it -- worst on rank 1's x4 card, where reslotting remains the cheapest single intervention. NAMED AS UNMEASURED: the seam draw cannot currently be split into components; nothing records it per component. Until one instrument attributes the peak instant to (send buffers, receive buffers, graph state, allocator transient), any per-component trade is arithmetic on an undivided number -- the same error class as pricing a flip on intention rather than completion. Also records the two honest limits on the rebuilt evict rung (sgl-project#717): it now delivers less than it prices by design, so repricing must key on the DELIVERED amount; and the ~413 MiB it leaves on the table is recoverable only by evicting deeper than the cap. Desk analysis, no code change. Feeds sgl-project#702 repricing and sgl-project#677 economics.

…aim: the arena tails are measured and recorded The sgl-project#702 repricing under the cold-spill doctrine found the number I said did not exist. NOTE_677 section 8 stated "no 1456 MiB figure appears anywhere in the records or source". WRONG: managers/phase_flip_seam_reserve.py, record_path docstring, records the measured per-rank arena tails -- "1436 MiB on rank2 against 466 MiB on rank1 and 0 on rank0". 1436 rather than 1456, but plainly the figure the earlier candidate meant. I missed it by grepping the docs tree and the ticket number instead of the module that owns the quantity. The sgl-project#685 ticket attribution was still wrong (it is an UnboundLocalError); the NUMBER was real. It also settles, without a boot, what NOTE_677 left as a candidate. Against the standing pool reduction per rank -- 704 / 801 / 1443 MiB, i.e. arming_floor_subtrahend_bytes = floor - max(corridor law, already_reserved) -- the measured tail is 99.5% of rank 2's, 58% of rank 1's, and 0% of rank 0's. Rank 2's floor excess IS its arena tail. The repricing itself lands in the evidence tree (not a git repo): /spinning/evidence-665-f1/NOTE_702_CUT_TABLE.md sections D1-D6, referenced from an append-only block in PLAN_PERF_PIPELINE_2026-08-16.md (head-298 sha256 verified byte-identical, 81560e94b7628e6c). Headline: crediting the standing reduction back, [31,17,16] recovers to 530,381 -- EXACTLY [31,18,15]'s spill-funded figure, because once floors stop binding both bind PP0 on the same bytes. The pool advantage that made [31,18,15] interesting was an artefact of VRAM-resident floors, as the user said; with it gone [31,17,16] wins outright on speed and is the only one of the two above the 14.1% noise floor. Build NOT undertaken and NOT small: nothing credits ON-DEMAND capacity against the standing floor, and crediting the evict rung's PRICED capacity would be pricing on intention rather than completion -- sgl-project#717's defect one layer up, with an OOM at arm time as its failure mode. Ticket filed at D6 with one live _record_seam_peak reading as prerequisite. Desk only, no boot, no GPU, no serving contact.

…unification sgl-project#612: declare the decoupled_kv construction site in RUNTIME_COMMUNICATOR_GROUPS. Declaration completeness only -- the contract checks construction SITES, and initialize_decoupled_kv_group has no production caller yet, which the entry says rather than implies. No behaviour change. sgl-project#584: file both constants the newly-merged ratchet caught, with differentiated verdicts rather than a baseline bump. planner/boot_instruments.py:: _CHAIN_TOLERANCE_MIB is NOT A DEMAND DECISION and the call site was read to say so: it is the rounding slack for verifying the budget-chain identity, reserving nothing. managers/phase_flip_seam_reserve.py::DEFAULT_ARMING_MARGIN_MIB is NEEDS AUDIT: it genuinely gates a VRAM decision, its own comment argues it is an instrument tolerance, and that call belongs to the sgl-project#662-F4 owner. TICKET_702_unify_revisions.md files the decision that stopped fix/602-fill-side: WorldMemory canonical (boot 2 validated it), PhasePoolModel's attention-layer divisor and sgl-project#723 frontier completeness ported as an EXTENSION, never a parallel model. The four mechanical resolutions from that aborted merge are recorded so the next attempt does not re-derive them.

The retracted-provenance gate refuses a vector whose LINEAGE is bad. This is the other half: a vector admitted as a SEED made a claim about the future -- "I am a pre-boot estimate and the measured per-rank capacity supersedes me in-process" -- and a boot where that never happened is now refused rather than warned about. That case is not hypothetical. boot_798_0822_0629 reached three install-capable sizing sites at dcp_size=3 with allow_install=True and role='seed', declined every one on a predicate about the worker's LABEL instead of its pool, and served the seed [29, 19, 16] as if it were a decision. Nothing failed. The advisory printed and was ignored, which is the defect class this task is about: a path that is logged and never enforced. A warning here would be the same thing again, so this raises. Three signals, because refusing on fewer would refuse correct boots: armed -- a seed was actually resolved (a pin asserts its value and arms nothing) calibration -- a site that could REALLY have superseded it was reached (dcp_size > 1 AND allow_install), so "never superseded" means declined and not "never had the chance" superseded -- a verdict was reached Refuse iff armed AND calibration AND NOT superseded. Two design points that each prevent a false refusal. The latch is disarmed on the VERDICT, not on the write: the install site writes nothing when the measured optimum equals the active vector, so disarming on the write would refuse a boot whose seed happened to be right. And the latch is process-global, so a genuine draft-pool worker declining its own install does not trip it -- its target runs in the same process and its verdict disarms the claim for both. Placed at scheduler.py:1448, after build_phase_flip_tp_stack rather than after init_memory_pools, because under the flip it is the TP stack that carries the DCP layout; the earlier position would refuse every flip boot. CAN-FAIL, EXECUTED, both directions, driving the real sizing method in the boot-0629 configuration (is_draft_worker=True, is_draft_pool_worker=False): predicate `not self.is_draft_worker` -> installs [29, 19, 16] -> FIRED predicate `not self.is_draft_pool_worker` -> installs [29, 17, 18] -> silent So the gate would have refused the boot that shipped and admits the one that is fixed. Neutering assert_seed_superseded to return unconditionally turns both falsifiers red; verified separately. 15 new tests; 84 green across the six sgl-project#797 suites. ruff clean on the new files and unchanged in count on the edited ones (utils.py 1, scheduler.py 99, mixin 0, all pre-existing); the two codespell hits in utils.py are pre-existing and unchanged in count. WHAT sgl-project#797 IS WORTH, corrected. The commit that landed the install fix quoted this runtime's own projection, "max_total_num_tokens 1213952 -> ~1335296". That number is not deliverable and should not be cited. Measured on boot _798_0822_0646: installing [29, 17, 18] in place of [29, 19, 16] moved the world pool by ZERO tokens, 990656 either way. The phase-flip boot caps every TP rank at the PP id space (448910 tokens, phase_flip_boot.py:206), clipping rank 0 from its profiled 620560 and making rank 0 the binding rank under both vectors, since both give it ratio 29. The threshold is cap0 > 29 * min(360392//17, 375560//18) = 605056, about 35 % above today's operating point; lifting it is the PP cut's problem (sgl-project#723/sgl-project#702), where cut and vector have to be solved together. The DEFECT remains real on any cut without that ceiling. This commit also annotates the install log line, whose "-> ~N" is scored on uncapped capacity and is what made a non-deliverable figure quotable, and drops the "roughly 10 % of the KV pool" line from the suite docstring.

fix hf config loading

14b0601

merrymercy merged commit 824a77d into sgl-project:main Jul 23, 2024

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Fix hf config loading (sgl-project#702)

2fdc588

Navigation Menu

Uh oh!

Fix hf config loading - #702

Fix hf config loading#702
merrymercy merged 1 commit into
sgl-project:mainfrom
ispobock:fix_model_config

ispobock commented Jul 23, 2024

Uh oh!

ispobock commented Jul 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ispobock commented Jul 23, 2024

Motivation

Modification

Uh oh!

ispobock commented Jul 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants