Reduce hardcoded logic of kernel usage by wisclmy0611 · Pull Request #707 · sgl-project/sglang

wisclmy0611 · 2024-07-23T22:02:34Z

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

…sgl-project#707 checkpoint (AEON-FP8-MTP) + unmodified club-3090 bench.sh, bs=1 wall-based: narrative 84.26 (+22.8% vs 68.6), code 111.04 (+24.9% vs 88.9), CV 3.5-3.7%, samples coherent; fair k=3 arm NOT measured (budget), ready as run_arm.sh fair (~6 min); wedge root was NEITHER AEON nor bar1: rank-0 reserve too small -> SIGKILL in capture -> survivors spin at 100% SM zero PCIe (the observed picture); second finding: high-accept ladder [1..5] does NOT boot at standard reserve under KV 7,3,3 + solo draft (needs ~+700 MiB on draft rank, adaptive_graph_memory.py:205 comment overclaims)

…22.0% vs vLLM sgl-project#707), code 108.25 (+21.8%), fair-vs-voll within joint spread => the +22-25% lead is essentially ALL engine/placement/transport at equal k, adaptive-k share small-to-undetectable; sgl-project#311 FALSIFIED with reversed sign — fp8 draft is SLOWER (+4 to +12% ms/Verify vs bf16; bs=1 drafter is latency- not bandwidth-bound, per-step dequant adds cost; accept holds), bs=1 verdict hardens to -34 to -50%; side proofs: draft-quant flag verified not to reach target, extra reserve requirement belongs to high-accept ladder not AEON; harness gap: battery_release_locks is process-local, orphans locks from new shells

…r reserve demand DERIVED from its own posts (per built rung: flashinfer float workspace + captured tokens at sgl-project#68 coefficient, mode-aware offload/resident reduction) instead of guessed — charged to the solo-draft GPU only; one helper feeds installed reserve, pinned advisory and sgl-project#265 fundability so they cannot drift; sgl-project#707 geometry derives GPU0 4160->5344 (clears the measured 4554 want); explicit reserve never inflated silently, pinned/boot errors now carry concrete ladder-aware suggestions; docstring overclaim fixed; 23 new tests, 121/121 uneven_tp_args, 2345 sweep failure set unchanged

…laySSM gate built sgl-project#707 CLOSED. The holdback is not a reserve in the intuitive sense and never was, which is why four attempts to model it as one failed. It is a CAP: allowed_tokens = id_space + (free_at_measure - arming_floor - margin)/cell adjusted_bytes = allowed_tokens * cell holdback_frac = 1 - allowed_tokens / (profiled_bytes / cell) from phase_flip_seam_reserve.floor_allowed_tokens -- "largest id space whose RESTING FREE COLUMN still holds the arming floor". An equality, solved in closed form, reached through the one funnel every sizing path crosses. Verified on all three ranks of the instrumented boot: model 45.141 / 44.072 / 60.256 percent against reported 45.143 / 44.074 / 60.258. Agreement 0.002 pp, the residual being MiB rounding in the logged budget. id_space 435,334 on all three. The three questions answered. (1) The percentage is a CONSEQUENCE of capping the pool so the resting free column still holds the floor, not a parameter anything computes. (2) The PP pass pays because it is the layout a flip departs from and must rest above the floor; the TP-stack pass has no flip to arm from and spends its whole budget -- holdback 0.000 percent on all three ranks confirms. (3) The BINDER holds back most because both extremes meet on PP2: smallest cell (4 attn -> 8192 B) gives it the LARGEST raw token capacity of the three (1,097,728), while its resting free column sits 7.4 MiB above its arming floor -- essentially on it -- pinning allowed at barely more than id_space. It binds for the same reason it holds back most, which is why the two always co-occurred and looked like coincidence. Retired: all fitted candidates, and the [33,15,16] discriminator boot, which now has no calibration value. The rank2 blind spot is preserved explicitly because it still governs TIMING calibration, which this result does not touch. Handed to Slot-3: rung pools can go from extrapolated to EXACT without booting each rung. cell is config-derived, id_space/free_at_measure/margin come from the current seam record, arming_floor from sgl-project#676, and the layout shift is arithmetic over two established constants -- free_at_measure(cut) = free_at_measure(booted) - delta_weights (374.2 full-attention / 476.2 linear per layer) - delta_mamba (51.20 MiB/GDN-layer). What stays extrapolated is only that shift, which is a far smaller claim than carrying another layout's capture behaviour and should be labelled as such rather than as measured. sgl-project#700, both halves built. The wiring contradiction is RESOLVED in-code: the kernel header's "NOT yet wired into the memory pool / radix cache / scheduler / backend dispatch" was stale and false in all four places it named, and the header now carries the file:line evidence for each plus the note that reading it as "inert" cost sgl-project#325 revision 1 a wrong conclusion. The byte-identity half is planner/replayssm_identity.py plus scripts/replayssm/identity_probe.py: A-vs-A floor before A-vs-B, probe capped under the ~109-token GDN determinism ceiling, CPU-sampled inputs only, GDN scalar gate only, and same-tokens-with-nonzero-delta classified as NOT byte-identical rather than rounded up. An unrun measurement is a refusal, never a default-on. The probe refuses before touching a GPU -- verified by running it with 256 tokens. --enable-linear-replayssm stays OFF: wiring resolved, fidelity unmeasured. Tests: 11 for the identity gate, red first. Planner suite 2665 passed / 2 failed -- the pre-existing test_rejected_evidence_pins pair. ruff clean.

…losed form: coupled is far kinder than I said The instrument boot landed and sgl-project#707 closed, so the frontier now prices pool with Slot-2's closed form instead of my extrapolation. ## His formula, verified from the capture allowed_tokens = id_space + (free_at_measure - arming_floor - margin) / cell holdback_frac = 1 - allowed_tokens / (profiled_bytes / cell) Reproduces the reported 45.143 / 44.074 / 60.258 % to 0.000 pp on all three ranks. adjusted == available_bytes confirmed to the capture's 0.1 MiB print precision, and cell == attn_layers x 2048 EXACTLY (14336/10240/8192 against 7/5/4) -- a third independent confirmation of the sgl-project#702 divisor, now from a live sizer rather than allocator source or a K-size back-computation. It is a CAP, not a subtraction, so the resting free column still holds the arming floor; nothing is set aside twice. PP pays because a flip must ARM out of that column; the TP pass measures 0.000% because there is no flip to arm. ## Why the binder holds back most, and it is not waste PP2's bracket (free_at_measure - arming_floor - margin) is 0.0 MiB -- it sits exactly AT its arming floor -- against PP0's 2164.8 and PP1's 260.2. So its allowed tokens collapse to id_space, i.e. it SETS the pool. It simultaneously has the smallest cell (4 attention layers), hence the largest raw capacity and therefore the largest holdback FRACTION. Binding and holding back most co-occur by construction. Reading PP2's 60.3% as waste is exactly backwards. ## Three changes to the frontier, one a correction to me 1. COUPLED IS FAR KINDER THAN I REPORTED. My extrapolation put [33,15,16] at -17% and [42,11,11] at -85%. The truth is -5% and -56%, because allowed_tokens is floored at id_space, which does not shrink with the cut. So [33,15,16] buys 1.330x for FIVE PERCENT of context, TODAY, with neither decoupling nor the lever -- the cheapest real win on the whole frontier, and my earlier table hid it. 2. THE SEAM CAP BOUNDS THE DEPTH. Past n0=51 a rank's free column no longer clears its arming floor, so the layout cannot ARM a flip. Those cuts are REFUSED by the provider and never reach the frontier -- absent, not priced as a tiny pool. An extrapolation cannot produce that boundary; a closed form can. 3. BOTH OPTIMA ARE INTERIOR, for two different reasons. Without the lever, overhead outgrows the compute gain past [42,11,11] (1.660x). WITH it, the raw compute speedup itself peaks at [44,10,10] (2.000x) and then FALLS -- piling layers onto the fast card eventually makes a TAIL stage the bottleneck. "More layers on the 5090" has a limit that is not about memory at all. Both headline picks survive unchanged: [42,11,11] at 1.660x today, [44,10,10] at 2.000x with the lever, and needs_pipelining stays flagged on [43,10,11] and [44,10,10] and deeper. ## Provenance is now per layout FrontierPoint.pool_measured replaces the single frontier-level flag: the booted incumbent is MEASURED and reproduces its own binder; every other row is extrapolated across ONLY the free_at_measure shift, charged per FAMILY (374.2 MiB per attention layer, 476.2 per linear, 51.20 per GDN layer). One global flag would let the measured row launder the extrapolated ones. A physical detail the per-family split exposes: converting a linear layer into an attention one at constant layer count FREES 153.2 MiB (476.2 - 374.2 weights, plus 51.20 of GDN state that attention layers do not carry) -- but enlarges the cell, so fewer tokens per byte. The two effects pull opposite ways, which is why the frontier prices cell and bracket separately. ## Test results 212 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff + codespell clean. test_seam_holdback_707.py (8, new): the closed form reproducing every reported holdback; the cell identity; the binder-holds-back-most structural test; the booted layout round-tripping to its own metal available_bytes; the per-family shift; a layout that cannot arm refused by name; a zero-attention stage refused. test_prefill_frontier_702.py (12): coupled declining but buying early steps cheaply, both optima interior, the seam-cap depth bound, per-layout provenance.

…t and load A static window is wrong in both directions -- too short at high load (the backlog never clears) and too long at low load (decodes wait behind an empty prefill window). planner/phase_window.py solves it. ## The amortization argument Over a cycle C = T_p + T_d + flips*F the work arriving in C must clear in C, so T_p = rho_p*C, T_d = rho_d*C and: stability floor C >= flips*F / (1 - rho) latency ceiling C <= (budget - F) / rho_d The floor is where the backlog stops growing however the windows are split; the ceiling is the wait a request meets arriving just after the prefill window shuts. Flip overhead is flips*F/C, which FALLS as the cycle lengthens, so throughput always wants a longer window and THE ECONOMIC CHOICE IS THE LARGEST ADMISSIBLE CYCLE, floored by stability. Not a midpoint, and not a constant. ## The sharpest result, and it reprices sgl-project#690 The two constraints move in OPPOSITE directions with F: the floor rises as flips*F while the ceiling falls as -F/rho_d. A dearer flip does not merely add an overhead line -- IT CLOSES THE FEASIBLE BAND FROM BOTH ENDS, and past some F the band shuts entirely: no window length works, at any split. On this rig with a 10 s TTFT budget: F=2.0 rho=0.30 -> cycle 53.3 s, overhead 7.5% F=2.0 rho=0.50 -> cycle 32.0 s, overhead 12.5% F=3.0 rho=0.50 -> cycle 28.0 s, overhead 21.4% F=4.2 rho=0.50 -> cycle 23.2 s, overhead 36.2% any F, rho=0.80 -> REFUSED, floor above ceiling So at rho=0.8 this rig is already refused at every measured flip cost. Halving the flip cost does not halve an overhead; it REOPENS CONFIGURATIONS THAT ARE CURRENTLY IMPOSSIBLE, which is a far stronger argument for sgl-project#690 than "2-4 s is slow". ## Two refusals, because a policy that quietly does the impossible is worse - THE SEAM MUST BE ABLE TO ARM. If the layout's free column no longer clears its arming floor there is no flip to schedule at any window length. This composes directly with sgl-project#707's closed form and the n0 <= 51 depth bound it implies, and is checked BEFORE any arithmetic. - THE DECODE WINDOW MUST BE WORTH ENTERING. Batch formation (sgl-project#689) collapses toward size 1 below a queue threshold, so flipping early buys a fraction of the decode rate for a full flip cost. That is a floor on the cycle, C >= q/(lambda*rho_p), and AT LIGHT LOAD IT BINDS INSTEAD OF STABILITY -- exactly the regime where a static window over-flips. ## Discipline Every quantity injected; the rig's figures are calibration data in the test. A foreign profile (flip 0.05 s, rho 0.8, 2 s budget, 50 arrivals/s) pins the generality and lands at under 2% overhead, showing the policy is about the RATIO of flip cost to cycle rather than about this rig. 225 tests green, hermetic (CUDA_VISIBLE_DEVICES=""), ruff + codespell clean. test_phase_window_677.py (13): the window as a function of flip cost rather than a constant; the floor diverging toward saturation and refusing rho>=1 as a capacity problem; the band closing from both ends; a dear flip under load admitting NO window; the economic choice being the ceiling; overhead falling with cycle length; halving the flip cost worth more than an overhead line; the batch floor and the case where it binds instead of stability; the seam refusal; a pure-prefill regime with no ceiling; a foreign profile; malformed inputs.

… the sizer must NOT claim it VERDICT: no dead reservation found. The gap is the arming floor (deliberate) plus min-rule residue (structural), so this is the labeled accounting table the brief asks for in that case, not a forced fix. Per-rank, from the sgl-project#704/sgl-project#707 instrument chain rather than modelled: rank budget weights mamba rest holdback allowed used slack PP0 31800 16064.5 916.5 14819.5 6690.1 8129.5 5964.7 2164.8 PP1 18800 10061.8 654.3 8083.4 3562.7 4520.7 4260.5 260.2 PP2 19800 10699.8 523.3 8576.3 5167.9 3408.4 3408.4 0.0 Total slack 2,425 MiB, and the shape is the finding: it is almost all on PP0, and the BINDER PP2 has exactly zero. HOLDBACK is deliberate, established in sgl-project#707: the pool is capped so the resting free column still holds the arming floor. Sizing below it is the boot that holds the corridor and never flips (sgl-project#656 E/G). Not claimable. SLACK is structural and still not claimable BY THE SIZER. The pool is one global token count, so a non-binding rank cannot spend its extra allowance without the binder moving -- PP0's 2,164.8 MiB is not booked-and-untouched, it is unreachable at this cut. A sizer that claimed it would hand the pool tokens PP2 cannot back, and PP2 is the rank that OOMs. That is the sgl-project#593-family direction the brief warns about, so I wrote NO red-first test for it: the falsifier would have to assert a wrong answer. WHERE THE CAPACITY ACTUALLY IS: the slack is a property of the CUT, and that is already solved on another strand. sgl-project#702 rev5 found the incumbent [28,20,16] is not pool-optimal -- the binding rank switches, so moving layers OFF the binder raises the pool, and [30,18,16] gives +20 percent AND 1.11x pipelined prefill. Claim it by rebalancing the cut, not by relaxing a reserve. WHAT I DO NOT CLAIM: the ticket's 2.0 / 5.7 / 3.7 GiB per card is a DIFFERENT operating point from the instrumented boot, and the instruments that would attribute it per-term only exist on boots carrying 2a6305d / 5f3e61f / f55c1a8. Mapping this table onto those three numbers is the cross-boot arithmetic this strand has already retracted twice, so I did not. Ask for the next boot to capture the three instrument lines at that operating point and the table re-runs against it directly -- a window item. One thing that CAN be said without it: against the measured arming floors (1728 / 1825 / 2467 MiB) the per-card free exceeds the ~1024 MiB corridor target BY CONSTRUCTION, because on a flip-enabled boot the arming floor is the binding level, not the corridor. Reading the gap against 1024 overstates it on every rank. Docs only. No sizer change, deliberately.

…le, and what the phase matrix owes Docs only. No code changed, nothing built or wired. ## sgl-project#702 decision table (docs/DECISION_702_cut_table.md) Re-derived on the sgl-project#707 closed form with the calibrated fixture. Incumbent pool reads 436,275 here against the 436,766 in older notes; all percentages use 436,275 so they are internally consistent. cut attn pool speed noise(14.1%) gate [28,20,16] (7,5,4) 436,275 1.0000 -- is the gate [30,18,16] (7,5,4) 436,275 (+0.0%) 1.1111 BELOW IT PASSES [33,15,16] (8,4,4) 415,859 (-4.7%) 1.3299 above FAILS Switch cost 1575 ms from the incumbent either way (distance-independent, per #704a); payback 15.75 s and 6.35 s respectively. THE TRADE: the two candidates fail in OPPOSITE places, which is the decision. [30,18,16] passes the stage-1 "pool >= incumbent" gate and cannot be validated by measurement -- its 11.1% gain sits BELOW this rig's measured 14.1% A-vs-A noise floor, so no boot here can tell it from the incumbent. [33,15,16] clears the noise floor comfortably but needs a 4.7% capacity waiver. No ranking is offered; the pick is the user's. [30,18,16]'s pool is identical STRUCTURALLY, not coincidentally: it keeps the (7,5,4) attention split and leaves PP2 (layers 48-63) untouched, and PP2 is the binder. Per-rank capacity is [594,615/462,920/436,275] at the incumbent and [517,464/570,931/436,275] here -- PP2's column is unchanged, so moving two layers between PP0 and PP1 cannot move a pool PP2 sets. CORRECTION: I could not reproduce a "+20% pool" figure for [30,18,16]; its coupled pool is +0.0%. The nearest +17.7% on the list is the DECOUPLED pool, which is cut-INDEPENDENT (the #704b R6 sum-rule result: 513,875 for every cut including the incumbent) and therefore not attributable to any cut. SOLVER DEFECT FOUND, flagged not fixed: prefill_frontier.py:144-156 enumerates, per lead depth, ONLY the tail split that minimises pipelined time. At n0=30 that is [30,16,18] (130.95 ms), so [30,18,16] (139.32 ms) is never generated -- yet [30,16,18] holds 343,951 tokens (-21.2%) against [30,18,16]'s 436,275 (+0.0%). The frontier is Pareto-optimal in SPEED but not complete in POOL: a roomier, slightly slower cut at the same lead depth cannot appear on it. [30,18,16] strictly dominates the incumbent (equal pool, more speed) and was invisible for exactly that reason, which is why it never reached the sgl-project#702 block. The fix would be to keep the Pareto set over (pool, speed) per lead depth instead of the speed-argmin; other pool-preserving cuts may also be hidden. Validation-boot evidence is specified in falsification order, and includes the one that matters most: for [30,18,16] the prefill A/B is expected to be INCONCLUSIVE by construction -- a boot reporting a clean 1.11x has measured its own noise, and that result must be rejected rather than celebrated. Appended to the morning plan file as a dated block per its amendment convention; head-298 sha256 verified byte-identical before and after (81560e94b7628e6c8...). That file is not version-controlled, hence this copy. ## sgl-project#485 completion verdict (docs/VERDICT_485_phase_matrix.md) Directive quoted verbatim from /spinning/htsglang/CLAUDE.md:87-96 (user law 2026-08-03), which is NOT tracked in this worktree. The two facets under sgl-project#485 are the same ticket, not a number collision: the matrix doctrine plus its prefill-column solver, and the seam/memory certification gating it. VERDICT: PARTIALLY DELIVERED, do not close. DELIVERED prefill column for the attention-vs-GDN/linear boundary (c08f613, b675630, b3c9802, e645aa7), with a second axis on the attention row (sgl-project#492 d82a778, corrected by sgl-project#503 447249a). Its own feasibility facet (407b382) is RED: the threshold is not reachable at the certification pool. SUPERSEDED d937d5f refuses the decode-side family split on sound MEASURED reasoning -- against the honest proportional (uneven-TP) baseline the split is worth +0.090 ms/round, ~0.3% of a 30 ms bs=1 round. But this closes ONE CANDIDATE CELL (concentration), not the column. Named tension: the directive says single-family arms are DIAGNOSTIC only and must never be phrased as a phase-level verdict, and that commit is worded as one. The narrow reading is adopted. OPEN O1 no (family x phase) structure exists anywhere -- pp_cut.py solves prefill only, family_split.py returns by_family with NO phase argument or dimension; the "matrix" is prose over two single-phase solvers. O2 the decode solver is DEAD CODE: solve_family_placement has exactly two references, its definition (family_split.py:141) and its test, zero call sites. O3 vocab, experts, nonlinear kernels and per-quant-lane linear layers were never cut at all. O4 the law covers diffusion, SR/video, TTS/ASR and training tenants -- entirely untouched, and the largest open surface. O5 the prefill column is not arm-able while Facet B is red. NOT sgl-project#485 sgl-project#704/#704a/#704b are a PP-cut layout LADDER and its arming machinery. They consume the prefill-column solve as an input and must not be counted toward the matrix; counting them would inflate its delivery with work on a different axis. Status in one line: one of two columns, one of many families, one of several workloads -- and the table is still prose. Every load-bearing claim spot-verified against the tree rather than taken from the survey: directive text read at source, solve_family_placement call sites grepped, d937d5f's baselines read from the commit body, [30,18,16] priced directly rather than inferred from frontier membership. Sanity: 35 passed (test_prefill_frontier_702.py, test_fill_ladder_704a.py). Desk only, no GPU, deploy tree untouched.

…here the PP bytes go sgl-project#1011 -- CHECKED THE LIVE LOG FIRST, as the order required, and the honest answer is that most of it needs no rebuild. Across three boots (restore, final, wb): holds in tp with pending prefill > 0 ............ 0 holds citing tp-decode-floor-s or pp-window-s ... 0 holds citing min dwell ......................... 1345 / 1248 / 1 holds citing IDLE-LOCKED "< break-even N tok" ... 1342 / 1234 / 0 The exits are ALREADY work-gated: the policy never holds in TP while prefill work waits, and the two clocks never appear as a hold reason at all. The only reasons that fire are min-dwell (the sanctioned last thrash guard) and the PRICED economy ("0 tok < break-even 27410 tok"), which is exactly the sgl-project#677/sgl-project#819 form the order wants anti-thrash to have. pp_window_s is dead code on this config by construction: phase_policy.py:3840 gates it on `cap <= 0`, and cap > 0 whenever decode_stall_slo_s > 0 (ours is 180). CHANGED, the one place that did actuate against the drain law: the decode stall cap (phase_policy.py:3828). It was the only thing permitted to cut a drain short, and its own message printed "N tok prefill still pending" while doing it. It now DEGRADES TO A DETECTOR whenever pending_prefill_tokens > 0 -- a named WARNING with wait time and backlog -- and actuates only on an empty backlog. THE ONE HONEST CONSEQUENCE, stated and not softened: under a continuous prefill stream the backlog never reaches zero, so carried decodes wait UNBOUNDEDLY. That is the operator's decision; the warning is the whole mechanism that keeps it visible. BOOT-PROVEN: boot up, coherent (Paris / 143-67=76), 39 cutovers under agent-shaped load, 0 detector fires and 0 actuations -- i.e. no regression. The actuation path was already latent (0 fires in the two prior boots), so this boot proves the change is harmless, NOT that the detector fires. Naming that limit rather than claiming a proof I do not have. sgl-project#1014 -- WHERE THE BYTES GO. Answered from the ledger's own per-rank posts, same boot, both phases (GiB): post PP0 pp -> tp PP1 pp -> tp PP2 pp -> tp weights + runtime state 13.572 -> 4.252 7.408 -> 2.719 8.342 -> 3.662 gapped corridor holdback 1.000 -> 1.000 1.000 -> 1.000 1.000 -> 1.000 mamba state pool 0.731 -> 0.731 0.426 -> 0.365 0.304 -> 0.365 speculative intermediate 0.877 -> 0.877 0.511 -> 0.438 0.365 -> 0.438 prefill activation reserve 1.000 -> (none) 1.000 -> (none) 1.000 -> (none) rest (= the KV pool) 13.875 ->24.195 8.014 ->13.837 8.324 ->13.870 COUNT-CHECK PASSES in BOTH phases: 31.055 / 18.359 / 19.335 GiB = 31800 / 18800 / 19800 MiB = rank_gpu_memory_mib exactly. No unattributed MiB. So the entire PP->TP KV gain decomposes into exactly two terms: weights booked differently 18.69 GiB (29.32 -> 10.63 summed over ranks) prefill activation reserve 3.00 GiB (1.0 per rank, PP only) total 21.69 GiB = the measured KV delta (30.21 -> 51.90) AND THE FIRST TERM DOES NOT SURVIVE INSPECTION. The PP-phase weight posts (13.572 / 7.408 / 8.342, sum 29.32) are right: a layer-wise split holds the whole 27.5 GiB checkpoint plus runtime state. The TP-phase posts (4.252 / 2.719 / 3.662, sum 10.63) are NOT: with the flip vector 32,16,16 the expected shards are ~13.75 / 6.9 / 6.9. The TP weight post is low by ~3x. VERDICT per post, as asked: weights + runtime state ... MIS-BOOKED, not mobilisable. The TP phase's larger KV budget is substantially an accounting artifact, and the 1,274,048-token figure rests on it. Corroborating evidence from the same boot: "TP pool sized to the PP id space: 616670 tokens" and the earlier "max_total_tokens=616670 is larger than the profiled value 450402 -- use the profiled value" -- the inflated budget is largely NOT spent. prefill activation reserve NEEDED-IN-PP (prefill runs there; absent in TP by construction). 3.00 GiB, not free. gapped corridor holdback NEEDED, flip-functional, and per sgl-project#707 the floor is itself measured. Arming floor is 1037 MiB solver-derived (band floor 819 + seam entry 218 + arming margin 192). Shrink-by-measurement only. mamba / speculative NEEDED, and they are already near-identical across phases (<= 0.073 GiB delta). CONSEQUENCE FOR THE 1M CHALLENGE, computed rather than aspired: the world per-token cost is 32,768 B and is layout-invariant (ANALYSE_799 §5.2). 1M tokens therefore needs ~31,250 MiB of KV world-wide against the ~20,300 MiB the PP phase holds today -- about +10,950 MiB of REAL bytes. The ledger above shows no idle posten of that size: the only 18.69 GiB "difference" is a mis-booking in the other phase, not memory sitting unused in this one. So PP does not reach 1M by copying the TP budget. Whether it can reach it at all is a question for the post-#1009a solver with the FA split solved budget-proportionally instead of pinned 8/4/4 -- and the token axis alone is capped at 639,800 (+6.51%) at this cut, also from ANALYSE_799. I am not projecting a number I cannot source.

Ying1123 force-pushed the lmy-code-clean branch from 2ff87ff to fe27699 Compare July 23, 2024 22:03

remove hardcoded logic

313befb

Ying1123 force-pushed the lmy-code-clean branch from fe27699 to 313befb Compare July 23, 2024 22:04

Ying1123 approved these changes Jul 23, 2024

View reviewed changes

update

610ca50

Ying1123 approved these changes Jul 23, 2024

View reviewed changes

Ying1123 merged commit a523a3c into main Jul 23, 2024

Ying1123 deleted the lmy-code-clean branch July 23, 2024 23:42

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Reduce hardcoded logic of kernel usage (sgl-project#707)

d1363a9

Navigation Menu

Uh oh!

Reduce hardcoded logic of kernel usage - #707

Reduce hardcoded logic of kernel usage#707
Ying1123 merged 2 commits into
mainfrom
lmy-code-clean

wisclmy0611 commented Jul 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wisclmy0611 commented Jul 23, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants