docs: update README by zhyncs · Pull Request #819 · sgl-project/sglang

zhyncs · 2024-07-30T06:11:51Z

Motivation

To ensure the usability and stability of SGLang installation for users, it is recommended to install via PyPI. If users want to use install from source, then use the rel branch.

It is agreed thereafter that both PyPI and rel are stable versions tested, while the main branch is used for development iteration. cc @merrymercy @Ying1123 @hnyls2002

Modification

Briefly describe the changes made in this PR.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

… the allowlist is empty Operator decision on the fork the previous commit left open, and it turns on a fact from the code rather than on the wrapper's prose. The sgl-project#677 hold was licensed to veto "the plain timer/economics exit". That licence was never really about the arm being a timer; it rested on an UNSTATED ASSUMPTION -- that vetoing the timer leaves some other backstop armed. The legacy pp_window stopwatch destroys the assumption: it sits behind a `cap <= 0` guard, so it fires ONLY when the decode-starvation cap is absent, which makes it the LAST anti-pinning bound in every state where it fires. Vetoing the last bound is an unbounded hold. That is verbatim the condition test_sustained_backlog_still_leaves_pp_via_the_window exists to prevent ("PP returned 'holding in pp' on every call, without end") and the shape of the live wedge family this ticket started from. The assumption outranks the prose, so the stopwatch is an exit as well. THE RULE THAT FOLLOWS, written into the dataclass and pinned by a test rather than left in this message: An arm may carry hold_eligible=True only if, in EVERY state where that arm fires, a SECOND INDEPENDENT anti-starvation bound is armed. THE ALLOWLIST IS THEREFORE EMPTY, and empty is the honest state, not a loss. The seam stays: it is the socket for a future arm that really is backstopped, so such an arm is added by stating the claim instead of re-deriving this whole argument -- and with no member, no arm can be held at all. sgl-project#677's economics is not dead with it; it lives in the window-length machinery and in the threshold repricing (sgl-project#819), on the flip-DECISION side where a flip can be weighed before one is chosen, rather than as a veto on an exit the rules already decided. That wiring pointer is recorded in the code and in the COORD so nobody re-attaches it here. The three anti-starvation tests are left untouched. They are the guarantee carriers, and they now pass because the code agrees with them again. TESTS (hermetic, CUDA_VISIBLE_DEVICES="", PYTHONPATH at this worktree; no boot -- a flip-decision change is acceptable only under load, which is a window post): test/registered/unit/managers/test_hold_allowlist_817.py: 12 passed. New pins: the allowlist is empty and the admission condition is written where it binds (the dataclass a future author actually reads, not the history), the stopwatch is shown from the source to be the arm that fails the condition, and -- as behaviour rather than as a count -- no reachable PP_TO_TP arm can be held while the list has no member. ALL EIGHT wrapper-caused failures are now green. Suite diff over the 25 suites importing phase_policy, branch vs base 587e4c2: 14 named failures -> 6, and the diff contains ONLY fixed entries, no new failure. The remaining 6 are test_vacuous_decode_exit_730.py, pre-existing and unrelated to the wrapper (proven earlier by disabling the wrapper outright: they do not move). Mutants, all KILLED: base denylist wrapper restored -> 15 failed stopwatch back to hold_eligible=True -> 5 failed blocked exit put back in the allowlist -> red allowlist inverted back into a denylist -> 11 failed black, isort, ruff, codespell clean.

… floor from its own cap; solve the arming floor; name the break-even provenance Three ordered items on top of the funding authority. A) THE PEER VETO, AT ITS SOURCE (sgl-project#812 / kein-bindender-rang) The live defect is not _floor_rows, it is what happens to its output. In _floor_ppm, kv_backing_relief.py:167: if floor_rows >= current_rows: return _SHRINK_SCALE # 100% == "no shrink" Measured 06:32:05, all three ranks carrying the SAME floor 128549 -- correct, because under PP a request's tokens occupy KV on every stage, so the live set is genuinely replicated -- against caps that are unequal BY DESIGN: PP0 backed 212992 floor 128549 -> 60.4% of its own cap PP1 backed 124928 floor 128549 -> 102.9% <- the defect PP2 backed 133120 floor 128549 -> 96.6% PP1's 102.9% clamps to 100%, the group MAX takes it, explain_kv_target computes target = max(desire, max_floor), and NOBODY shrinks -- vetoing PP0's fully fundable 84443-row plan on the very rank that needed it. The sibling _shrink_ppm documents this exact trap for the ambition side ("on an uneven fleet the smallest pool's 'no change' is the smallest number in the group and silently wins"). The lesson was never applied to the floor side. Fix, local only -- the collective FORM is untouched, no new collective on the admission path (DESIGN_679's warning stands): _floor_rows now derives its result against this rank's own backed rows and clamps DOWN to them. An unbacked row cannot be reserved, so a floor above the cap was never a meaningful quantity. The condition is logged as an UNDER-BACKED RANK -- a defect about that rank's backing, whose answer is to grow it -- and floor_exceeds_local_cap() splits the two cases _floor_ppm used to collapse: floor == cap is a HEALTHY full pool, floor > cap is the defect. They returned the identical value before, which is how one rank's local problem became a group-wide freeze. HONEST SCOPE: the clamp makes the quantity truthful at the point it is computed and names the defect. It does NOT by itself unblock the veto, because a clamped floor still equals the cap and still returns the neutral element under a PROPORTIONAL agreement. Replacing that proportion is a change to what the group agrees on -- enforcement, and metal. Ticket in the COORD. B) DEFECT A: THE ARMING FLOOR IS THE FREE VARIABLE, THE BAND IS NOT band floor 819 + seam entry reserve 512 = 1331 arming floor + arming margin 192 = 1523 band ceiling 1229 <- 294 MiB short The corridor band (819-1229 free per card under load) is a hard user rule: below it is a breach, above it boot acceptance has FAILED. It does not move. The arming floor is DERIVED and the seam entry reserve is an allowance, so the floor is what must give. solve_arming_floor() reports the largest reserve that fits -- 218 MiB against the 512 shipped -- and refuses BY NAME otherwise. It does not auto-correct: cutting the reserve changes what the seam may spend while it runs, which needs metal. Also withdraws a piece of advice that named a forbidden state. The abandon message said the flip "is retried when occupancy drops"; clearing a 1331 MiB watermark means leaving the acceptance band from ABOVE, so no amount of waiting can end it -- and 18f measured exactly that (draining the load did not lift the lock). _arming_floor_advice() now retracts the retry advice whenever the watermark is unreachable. C) sgl-project#819: 7004 IS NOT A SEED, AND THE STALENESS IS UNEVEN 7004 is a literal nowhere; break_even_tokens computes N = C / (1/X - 1/P) and 7004 is what the shipped inputs produce (asserted in the suite). The three inputs do NOT share a provenance, which is the actual finding: C flip cost SELF-CORRECTING -- FlipCostEstimator.observe() is fed real cutover durations, seeded from DEFAULT_FLIP_COST_S=3.2 X TP prefill 1681.0, env-overridable, NEVER measured at runtime P PP prefill 7245.5, same shape One input self-corrects and two cannot. A rig whose prefill ladder differs from the sgl-project#631 mainrig silently solves N against another machine's hardware unless a human sets two env vars, and nothing tells them to. BreakEvenProvenance makes that visible instead of implicit. TEST RESULTS (hermetic, CUDA_VISIBLE_DEVICES="") test/srt/test_funding_authority_770.py 48 passed (was 40) test/srt/test_floor_local_cap_812.py 16 passed (new) test/srt/mutants_770.sh 11/11 KILLED (M9/M10/M11 new) test/srt/mutants_812.sh 4/4 KILLED ruff --select=F401,F821,UP037 3 findings on phase_flip_runtime.py, identical to base (2090, 2581, 7022->7137 shifted by inserted lines) -- ZERO new codespell clean on all new files TWO MUTANTS SURVIVED FIRST CONTACT AND BOTH WERE REAL. MA3 (clamp rounds UP past the cap) survived because the fixture used cap 124928 with page 64 -- exactly divisible, so round-up and round-down agree and the test could not tell them apart. Refixtured on 124900. MA5 (drop the min() guard in _floor_ppm) is an EQUIVALENT mutant, proven not patched: the early return means the expression is only reached with floor < current, where ceil(floor*1e6/current) <= 999999 < _SHRINK_SCALE. Verified numerically; the guard is kept as defence-in-depth and the mutant is recorded in the harness rather than run, because a harness listing an unkillable mutant reports a permanent false gap.

16g: funding_authority, solve_arming_floor at 218 instead of 512, the retraction fix to the abandon message, and sgl-project#819 provenance. Clean merge. Shares phase_flip_runtime.py with feat/822 and kv_backing_relief.py with the already-merged sgl-project#816; hunk regions checked disjoint before merging (sgl-project#770 at :6120-6972 vs sgl-project#822 at :3741-3798/:8029).

…gl-project#821, sgl-project#823, sgl-project#824, sgl-project#770/sgl-project#812, sgl-project#815; close the section 18 breach Merge-checklist duty for the five wave-2 stages on this branch (fix/801, fix/824, feat/770, feat/822, fix/815). Mechanism text is taken from the commits' own measured evidence, not restated from ticket titles. WHAT WAS ALREADY THERE, checked before writing rather than after. feat/822 catalogued ITSELF mid-train -- 21ff075 carries +49 lines of this file -- so sgl-project#822 already has a complete section 3 entry including the sgl-project#816 clamp firing-rate baseline. No second sgl-project#822 entry was written. sgl-project#801, sgl-project#815, sgl-project#821, sgl-project#823, sgl-project#824 and sgl-project#770 had zero prior mentions; sgl-project#812 was name-checked inside the KV-backing-relief bullet as a consequence of the cap, with no entry of its own. SECTION 12, five new families: Said-once (sgl-project#823) -- a fact that changes during a process, reported once at the moment it first became true, is a stale label afterwards. Both instances live in _update_uniform_pool_budget: the floors' coverage said three times at startup and never again through four cutovers in 55 s, and a divergent TP queue head logged as an event when it is a duration. The mutant that makes the case: disabling the recovery edge left all 14 tests green before the streak was extracted. Blind-instrument (sgl-project#821) -- the scheduler watchdog's activity predicate reads exactly the two values a rank parked in the dict receive freezes, so the instrument best placed to see the wedge is the one structurally unable to. Non-monotone-source (sgl-project#824) -- a monotone guarantee (the protected prefix) derived from a non-monotone tracker (mamba_last_track_seqlen). Landed as a measured assertion, cache_protected_len=16384 against page_aligned_len=8192. Invariant-for-funding (sgl-project#770, sgl-project#812) -- a group-derived per-rank floor asking PP1 for 102.9% of its own cap, and the withdrawal of a floor clamp that bought funding by breaking the levelling invariant (1 failed with, 12 passed without). Stub-drift (sgl-project#815) -- ten helpers drifted behind named commits, and the rule the repair follows: faithfully or not at all. This entry also RECORDS why seven cases in test_pp_flip_slot_hold_631.py are deliberately left red -- the faithful stub is a real rank identity, which drags the whole admission-receive path in with it. That is documentation of a known state, not a claim that it is fixed. Plus a NUMBER-COLLISION note, because the log is ambiguous without it: 9d13bf0 is subject-tagged [sgl-project#824] and names its test file _824.py but is canonically register-sgl-project#823 (floor-scope reporter, strand 16f, arriving on fix/801-admission-recv-liveness); the canonical sgl-project#824 is 658d0c1 on fix/824-chunked-prefill-protected-len. No rebase, so the titles stand and the mapping is written down instead. SECTION 7: sgl-project#801's dead-peer measurement, which REFUTES the premise under the sgl-project#816 survivor analysis -- an exited or SIGKILLed peer makes the admission receive raise after 1.02 s, while alive-never-sends and SIGSTOP both block unbounded, so a rank silent for 119.7 s was not waiting on a corpse. SECTION 2: the funding authority (sgl-project#770/sgl-project#584/sgl-project#819) -- one question, posts declared by name, the three priced losses (L1 the unreachable 2638 MiB, L2 a 3437-row ask against 8192-row granularity returning a silent zero at 65% of the shortfall, L3 claimed=0 against an accounted 107 MiB), and the arming floor solved rather than asserted (512 MiB reserve needs 1523 against a 1229 ceiling = structurally unreachable; 218 is the largest that fits). sgl-project#819's break-even carries its provenance because X and P are env-overridable defaults that are never runtime-measured. SECTION 18 -- THE BREACH, CLOSED. Rule 5 says a PR adding a reusable module adds its entry in the SAME PR. Three modules had landed without one: kv_row_ownership.py (sgl-project#822), funding_authority.py (sgl-project#770), uniform_floor_scope.py (sgl-project#823). prefetch_ballot.py (#791b) had the same gap from an EARLIER merge and is added here labelled as backfilled -- the provenance is stated rather than quietly absorbed, because the checker only tests citations that EXIST, so a missing entry costs nothing and is exactly the failure mode rule 5 cannot catch. Each entry says what is NOT wired, in those words. Both authorities are so far consulted only for the arming floor, the census and the retire/audit pair; the clamp firing-rate metric, format_violations, diagnose_floor_band, uniform_absolute_floor and slack_above_uniform_floor have no production caller at this commit. Recording an unwired export as wired is how a catalog becomes a liability. One trap recorded: funding_authority's docstring at :56 names a solve_funding function that does not exist -- the solve is can_fund at :378. TWO DEFECTS FOUND WHILE WRITING, both fixed here: 1. Section 18 cited `watchdog.py:88`, which resolves to python/sglang/srt/watchdog.py -- a file that does not exist in this tree. The intended target is turnkey/watchdog.py:88 (the retired generation probe), now qualified. This was a live red test, not a cosmetic issue. 2. The sgl-project#822 section 3 entry cited kv_backing_relief.py:504 for exposure_over_backing; :504 is a docstring line and the def is at :547. The entry landed already drifted. Corrected, with the drift noted in place per the header rule that the code wins. TESTS. test/registered/unit/docs/test_building_blocks_catalog_538.py, the mechanical section-18 pointer checker: 91 passed / 1 failed BEFORE (the watchdog.py:88 breakage), 101 passed / 0 failed AFTER. The ten new cases are the ten new citations, and every one resolves. The indicator is proven in both directions without needing an injected mutant: it was red on a real bad pointer of exactly the class it claims to detect, and is green now. codespell on the file: clean.

…, and can be wrong W24 residual defect (iii), and the ticket's own remedy is NOT what shipped. The 09:01:37 specimen carries TWO defects; "break the band when the decode bundle is not draining" is neither of them. THE ARITHMETIC THAT REFUSES THE TICKET'S REMEDY. At the measured decode_contention (sigma) = 1 the scheduler gives prefill absolute priority per iteration -- `_differential_flip_threshold` records the measurement verbatim: "an iteration with any prefill chunk pending runs THAT batch and never reaches the decode branch". So while prefill is pending in TP the decode bundle CANNOT shrink, by construction. "Bundle not draining" is therefore IMPLIED by the band's own operating condition, not evidence about it, and a band that broke on it would collapse to the plain break-even for every load with a request decoding -- silently deleting the sgl-project#665-F1 differential model. That is a policy rewrite wearing a bug fix. (A) THE DETECTOR READ A BAR THE POLICY NEVER APPLIED. The alarm printed `bar_tok=20057` and fired on `pending 22887 > 20057`, while the hold it indicted names its own bar in its own text: `> N=20057 but <= 30086`. 30086 is `effective_flip_threshold(cfg, running_bs=1)`, the differential bar, which at sigma = 1 is N0 x (1+2B)/(1+B) -- exactly 20057 x 1.5. The policy compared 22887 against 30086 and HELD, correctly. So W24's single LAYOUT-ECONOMY ANOMALY is a FALSE POSITIVE, and the "first metal catch of the detector" reads better as its first metal self-indictment. This is sgl-project#819's ONE READING rule -- "the bar the policy APPLIED and the bar the log REPORTS can never be two different numbers" -- holding inside phase_policy and breaking at the module boundary. It is the sgl-project#851 class root exactly ("the DECIDERS still read their own bookkeepers"), occurring inside an sgl-project#851-family detector, which is the same shape sgl-project#853(i) already found once this build. The applied bar is now passed from the one authority that computes it, and is REQUIRED rather than defaulted: a default is the mechanism by which a caller silently re-creates the divergence. The gate takes max(break-even, applied), which bounds the blast radius to one direction -- it can only RAISE the bar, so it can only remove false positives and can never invent an alarm. That also covers strict purity, where the threshold is 0 by construction. (B) THE BAND HAD NO FALSIFIER FOR ITS OWN PREMISE. It was the ONLY hold in `_decide_from_load` that could not be wrong. Every neighbour carries a bound: min dwell yields to `starved` (sgl-project#768), drain mode to the sgl-project#833 stall deadline, the idle lock to the idle dwell (sgl-project#748). The band says "prefilling it in tp beats the round trip" -- a claim priced at the TP prefill RATE, true only while the backlog is actually being prefilled here. The falsifier is taken on the axis the claim is made on: PREFILL PROGRESS. `pending_prefill_tokens` is "admitted but not yet computed", measured at the chunk fill boundary, so it drops every round a chunk is computed -- even mid-way through one long prompt. Frozen for a whole decode window means no chunk was computed for a whole decode window. The clock is sgl-project#677(a)'s existing `last_prefill_progress_at`; the window is `drain_stall_deadline_s`, the same quantity sgl-project#833 and the sgl-project#838 detector already use, so policy and detector cannot come to hold two different ideas of one decode window. Unstamped reads as no stall, so a caller that never observed cannot flip on it. THE TWO HALVES AGREE ON THE SPECIMEN INSTEAD OF DOUBLE-COUNTING IT. W24's pending oscillated 0 -> ~22.5k -> 0 on a ~5-min period, so prefill progress was live and this break would have stayed SILENT there -- consistent with (A), which says that hold was right. That is what makes them two separable defects rather than one defect described twice. NOT REBUILT (prior-art gate): the ticket pairs (iii) with "a completed flip should clear the staging backoff". Already implemented -- `note_flip_completed` pops last_abandon_at / arm_refusals / arm_hold_until / arm_degraded for the direction -- and already pinned by test_phase_policy_flip_reachability.py:: test_a_completion_clears_the_staging_rate_limit_outright. Cited, not duplicated. A band-break is paced by that limiter like any other arm, which is correct: `_decide_rules` applies it after `_decide_from_load`, and `_demand_outweighs_a_retry` still overrides it when the backlog outweighs the wait. WHAT THIS DOES NOT CLOSE: it does not move the flip. W24's stuck phase was FUNDING -- 153 arms refused after the policy had already said the load wanted the flip, 43 of 45 binding refusals reading cause=phantom_capacity, which is sgl-project#852's territory. (iii) must not be cited as a flip-stickiness fix. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_band_premise_853.py 15 passed test_layout_conformance_838.py (updated) unregressed test_flip_threshold_repricing_819.py unregressed test_phase_policy_flip_reachability.py unregressed test/registered/unit/managers/ 3833 passed, 18 skipped, 334 subtests (759 s) test/registered/unit/mem_cache/ 1710 passed, 1658 skipped, 361 subtests (127 s) ruff check + format clean Red-first, both halves and both directions. (A) was red as the specimen FIRING -- "a hold inside the policy's own band was alarmed on" -- with the can-fail direction proving a detector that merely went quiet dies (pending above the applied bar still fires, and a window-3-shaped span at 119x the bar is above BOTH bars and stays an anomaly). (B) was red as the frozen backlog holding forever, measured: 25 observations at a constant 25065 tok returned "too short for the round trip" every time, with the can-fail direction being that a progressing prefill, a sub-window stall, a sub-band backlog and an unexpired min dwell must all still HOLD.

…being ambiguous W25 ran a sustained TP-sticky prefill phase -- `Prefill batch phase=tp` live on all three ranks at 11:38:51, 16-24k prompts, no cutover after 11:33:57 -- and THE USER FOUND IT BY EYE while grep -c "LAYOUT-ECONOMY ANOMALY" boot_w25_0824_1125.log -> 0 for the whole boot. Fourth instance of the failure form sgl-project#838 exists to prevent, against a standing order that a human eye is never the instrument. THE DETECTOR IS EXONERATED, AND THAT IS SAID PLAINLY RATHER THAN PAPERED OVER. Its class-2 gate 5 requires `pending > the bar the policy applied`. The hold reasons across the sticky phase: 19x pending prefill <= N=18614, running it in tp (BELOW the bar) 6x > N=18614 but <= 27921 with 1 req decoding (inside the band) 25x staging rate limit / 18x drain mode / 7x min dwell Pending was 16-20k against a LIVE break-even of 18614: sgl-project#819 reprices N from the measured seam, the seam had risen to 8.50s against a 3.2s seed, and the bar moved 7004 -> 18614 with it. The policy's own arithmetic said TP was cheaper, so an alarm would have been precisely the over-eager mutant `test_mutant_overeager_detector_dies` exists to kill. NO ALARM IS ADDED HERE. The real defect behind the sticky phase is the SEAM COST that raised the bar -- W25 root R1, recorded in NOTE_851 and NOT fixed by this commit. WHAT IS ACTUALLY BROKEN IS THAT THE ZERO CANNOT BE READ. `c2 = 0` conflates: ran every round and correctly declined (what happened) never ran (the W24 shape, one ticket ago) not wired on this pin threw inside the try/except and was swallowed That is the sgl-project#851 defect class -- the silent multi-valued zero -- occurring INSIDE the detector, one commit after sgl-project#853(i) had to remove the identical shape from the exposure gate. It is why a person's eye ended up being the instrument, and it is the same remedy: the instrument announces what it decided. LAYOUT-ECONOMY CHECKED (sgl-project#838) the verdict ran and declined, with the declining gate and both bars in the line Counted always, logged on a 60 s heartbeat (DECLINE_REANNOUNCE_S, deliberately slower than the 10 s alarm cadence: an alarm is urgent, a decline is a liveness proof and must be cheap enough that nobody switches it off). At INFO, not DEBUG -- W25 ran at INFO and would not have seen a DEBUG line, which is the correction sgl-project#853(i) already had to make once on the exposure clamp. The counter field gains `c2ok`, so `c2=0, c2ok>0` reads healthy and `c2=0, c2ok=0` reads never-ran. `_should_say` takes the cadence as a PARAMETER rather than growing a second copy that could drift from the first. THE CAN-FAIL DIRECTION IS THE WHOLE RISK and is pinned: a fix that downgraded real anomalies into heartbeats would satisfy every liveness assertion in the file while destroying the detector. So an anomaly must still alarm, must count as an anomaly and never as a check, the two markers must be distinct strings, and an alarm line must not contain the CHECKED marker -- that last one because sgl-project#853(i) already caught a marker quoting another marker's text and restoring the ambiguity while passing everything else. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_economy_detector_liveness_854.py 14 passed test_layout_conformance_838.py (periodic field updated for c2ok) unregressed test_band_premise_853.py unregressed test/registered/unit/managers/ 3847 passed, 18 skipped, 334 subtests (737 s) ruff check + format clean Red-first: `note_economy_declined` did not exist, so every liveness assertion failed at AttributeError, and the periodic-field test failed as 'c1=1, c2=2, c2ok=0' != 'c1=1, c2=2' -- the field genuinely changed shape and the existing pin was updated deliberately rather than loosened. Also recorded in docs/dev/NOTE_851_build_caveats.md: W25's three new roots (R1 seam cost, R2 the gate short ~210 MiB beside kv-slack's 2432 MiB, R3 sgl-project#852 still over-promising a stable 88 MiB and saying so through its own discriminator). sgl-project#852 must not be cited as closed.

W25 went TP-sticky on 16-20k prompts and the user caught it by eye. The policy was right and the PRICE was wrong, and the price was wrong in a way this module had already written down and then not applied to itself. phase_policy.py:82 C = round-trip flip cost, seconds break_even_tokens "never repays the {flip_cost_s}s round trip" But `observe_flip_leg` fed ONE LEG per sample -- its own docstring even computes the round trip it is not feeding, "tp_to_pp 11490 + pp_to_tp 5681 = 17171 ms" -- and BOTH DIRECTIONS went into ONE EMA. THE TWO LEGS ARE NOT THE SAME QUANTITY. On the binding rank W25 measured tp_to_pp 10466-13181 ms against pp_to_tp 5078-6545 ms. An EMA fed both alternately converges to neither. REPRODUCED EXACTLY, which is what makes this measured rather than argued. Replaying PP0's eleven PHASE-FLIP DONE totals through one estimator at ALPHA=0.3 gives 5.0779 6.6944 6.2494 7.5450 7.2426 9.0241 8.2740 9.2457 8.4356 9.3990 8.5041 -- and the boot's own decision lines printed N=15853/18110/18464/18614 at exactly the samples pricing to 7.2426/8.2740/ 8.4356/8.5041. To the token, four times. The blend settled at 8.50 s: BELOW every tp_to_pp leg and ABOVE every pp_to_tp leg. The true round trip was 11.6 + 6.4 = 18.06 s. The bar also oscillated ~2000 tok with flip-direction parity (8.50 after a pp_to_tp, 9.40 after a tp_to_pp) -- an artifact of the blend that says nothing about cost. sgl-project#819's own closing sentence is the rule it broke one level up: "a component and its container are different quantities and an EMA fed both alternately converges to neither." Two directions are different quantities too. THE CORRECTION RAISES THE BAR (C 8.50 -> 18.06 s, N 18614 -> ~39500), so it makes TP-stickiness on 16-20k prompts MORE correct, not less. That is stated plainly rather than softened: the remedy for a bar that is too high is a cheaper seam, not a permanently under-priced one. dN/dC = 2188.8 tok/s says what each second of seam is worth once it is. WHAT SHIPPED. `RoundTripFlipCost` holds one `FlipCostEstimator` PER LEG and sums them. The leg estimator is REUSED, not rebuilt, so every property sgl-project#677 pinned on it holds per leg -- including that it tracks DOWN as readily as up, which is what makes a future seam fix actually lower the bar instead of latching high. The seed is split in half, so an uncalibrated instance values exactly the round-trip seed and the pre-sgl-project#856 path is unchanged. An undirected reading is treated as a whole round trip and split evenly (so callers that really measured one stay honest); a direction this class does not know is REFUSED rather than filed under a guess. PROVENANCE GAINED A THIRD WORD. `flip_cost_measured()` is a boolean over a quantity with three states and printed the middle one as "measured". `flip_cost_provenance()` returns seed / half-measured (<leg> only) / measured. Same class of fix as sgl-project#853(i) on the exposure gate and sgl-project#854 on the economy detector, one layer further in. AND THE CONSUMER IS RECONCILED, which is the half that is usually missed. The sgl-project#838 economy detector refuses to question a bar priced off the seed because "an assumption is not the policy's own claim". A HALF-measured round trip is still half assumption, so it is refused on the same ground: `flip_cost_fully_measured()` requires BOTH legs. The blast radius is one-directional -- the detector can only DECLINE more often, never alarm more often. DELIBERATE TEST CHANGES, not loosened ones: * `test_the_leg_total_becomes_the_price` now asserts the leg's own value AND the round trip (11.4901 + 3.2/2). Asserting 11.4901 for C would be asserting that one leg is the whole round trip -- the defect itself. * `_measure` floors at 2x MIN_ESTIMATE_S: each leg carries its own band, so a round trip cannot be cheaper than two leg-minimums. * `test_a_seeded_price_is_not_evidence` now also asserts "half-measured". TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_round_trip_price_856.py 19 passed test_flip_threshold_repricing_819.py unregressed test_flip_cost_calibration_677.py unregressed test_flip_threshold_honesty_777.py unregressed test_flip_cost_clamp_directions_677.py unregressed test_layout_conformance_838.py unregressed test_economy_detector_liveness_854.py unregressed ruff check + format clean on the changed source ALSO FOUND, NOT FIXED HERE (recorded in NOTE_856_seam_cost_ledger.md): `observe_flip_leg` is called only from the flip-COMPLETION branch, so a boot whose flips are all refused or abandoned prices off the seed for the whole session -- and sgl-project#777's staleness WARNING is gated on the same event, so such a boot gets neither a reprice nor a warning. W25 did not manifest it (33 completed cutovers). Same silent-zero shape, one more instance.

…here the PP bytes go sgl-project#1011 -- CHECKED THE LIVE LOG FIRST, as the order required, and the honest answer is that most of it needs no rebuild. Across three boots (restore, final, wb): holds in tp with pending prefill > 0 ............ 0 holds citing tp-decode-floor-s or pp-window-s ... 0 holds citing min dwell ......................... 1345 / 1248 / 1 holds citing IDLE-LOCKED "< break-even N tok" ... 1342 / 1234 / 0 The exits are ALREADY work-gated: the policy never holds in TP while prefill work waits, and the two clocks never appear as a hold reason at all. The only reasons that fire are min-dwell (the sanctioned last thrash guard) and the PRICED economy ("0 tok < break-even 27410 tok"), which is exactly the sgl-project#677/sgl-project#819 form the order wants anti-thrash to have. pp_window_s is dead code on this config by construction: phase_policy.py:3840 gates it on `cap <= 0`, and cap > 0 whenever decode_stall_slo_s > 0 (ours is 180). CHANGED, the one place that did actuate against the drain law: the decode stall cap (phase_policy.py:3828). It was the only thing permitted to cut a drain short, and its own message printed "N tok prefill still pending" while doing it. It now DEGRADES TO A DETECTOR whenever pending_prefill_tokens > 0 -- a named WARNING with wait time and backlog -- and actuates only on an empty backlog. THE ONE HONEST CONSEQUENCE, stated and not softened: under a continuous prefill stream the backlog never reaches zero, so carried decodes wait UNBOUNDEDLY. That is the operator's decision; the warning is the whole mechanism that keeps it visible. BOOT-PROVEN: boot up, coherent (Paris / 143-67=76), 39 cutovers under agent-shaped load, 0 detector fires and 0 actuations -- i.e. no regression. The actuation path was already latent (0 fires in the two prior boots), so this boot proves the change is harmless, NOT that the detector fires. Naming that limit rather than claiming a proof I do not have. sgl-project#1014 -- WHERE THE BYTES GO. Answered from the ledger's own per-rank posts, same boot, both phases (GiB): post PP0 pp -> tp PP1 pp -> tp PP2 pp -> tp weights + runtime state 13.572 -> 4.252 7.408 -> 2.719 8.342 -> 3.662 gapped corridor holdback 1.000 -> 1.000 1.000 -> 1.000 1.000 -> 1.000 mamba state pool 0.731 -> 0.731 0.426 -> 0.365 0.304 -> 0.365 speculative intermediate 0.877 -> 0.877 0.511 -> 0.438 0.365 -> 0.438 prefill activation reserve 1.000 -> (none) 1.000 -> (none) 1.000 -> (none) rest (= the KV pool) 13.875 ->24.195 8.014 ->13.837 8.324 ->13.870 COUNT-CHECK PASSES in BOTH phases: 31.055 / 18.359 / 19.335 GiB = 31800 / 18800 / 19800 MiB = rank_gpu_memory_mib exactly. No unattributed MiB. So the entire PP->TP KV gain decomposes into exactly two terms: weights booked differently 18.69 GiB (29.32 -> 10.63 summed over ranks) prefill activation reserve 3.00 GiB (1.0 per rank, PP only) total 21.69 GiB = the measured KV delta (30.21 -> 51.90) AND THE FIRST TERM DOES NOT SURVIVE INSPECTION. The PP-phase weight posts (13.572 / 7.408 / 8.342, sum 29.32) are right: a layer-wise split holds the whole 27.5 GiB checkpoint plus runtime state. The TP-phase posts (4.252 / 2.719 / 3.662, sum 10.63) are NOT: with the flip vector 32,16,16 the expected shards are ~13.75 / 6.9 / 6.9. The TP weight post is low by ~3x. VERDICT per post, as asked: weights + runtime state ... MIS-BOOKED, not mobilisable. The TP phase's larger KV budget is substantially an accounting artifact, and the 1,274,048-token figure rests on it. Corroborating evidence from the same boot: "TP pool sized to the PP id space: 616670 tokens" and the earlier "max_total_tokens=616670 is larger than the profiled value 450402 -- use the profiled value" -- the inflated budget is largely NOT spent. prefill activation reserve NEEDED-IN-PP (prefill runs there; absent in TP by construction). 3.00 GiB, not free. gapped corridor holdback NEEDED, flip-functional, and per sgl-project#707 the floor is itself measured. Arming floor is 1037 MiB solver-derived (band floor 819 + seam entry 218 + arming margin 192). Shrink-by-measurement only. mamba / speculative NEEDED, and they are already near-identical across phases (<= 0.073 GiB delta). CONSEQUENCE FOR THE 1M CHALLENGE, computed rather than aspired: the world per-token cost is 32,768 B and is layout-invariant (ANALYSE_799 §5.2). 1M tokens therefore needs ~31,250 MiB of KV world-wide against the ~20,300 MiB the PP phase holds today -- about +10,950 MiB of REAL bytes. The ledger above shows no idle posten of that size: the only 18.69 GiB "difference" is a mis-booking in the other phase, not memory sitting unused in this one. So PP does not reach 1M by copying the TP budget. Whether it can reach it at all is a question for the post-#1009a solver with the FA split solved budget-proportionally instead of pinned 8/4/4 -- and the token axis alone is capped at 639,800 (+6.51%) at this cut, also from ANALYSE_799. I am not projecting a number I cannot source.

…n, die es nicht gibt == DER BEFUND (gemessen, nicht vermutet) == Boot_855_704bgroup2: 57 Prefill-Batches, NULL Decode-Batches, eine 6-Token-Probe -- und 42 Flips. Die Gruende standen im Log, es hatte sie nur nie jemand gezaehlt (`PHASE-POLICY arming`, scheduler.py:13627 -- die Zeile existierte bereits, ich musste kein Instrument bauen): 21x pp_to_tp: "idle Ns >= Ns, returning to the decode resting layout" 21x tp_to_pp: "pending prefill N tok > 0 (purity: prefill cannot run in tp, nothing decoding)" 21 Rundtrips auf einer LEEREN Kiste. Die zwei Regeln zertifizierten einander: diese hier flippte fuer nichts nach TP, die Prefill-Arbeit der Health-Probe konnte dort nicht laufen, die tp-waertige Regel flippte sofort zurueck. Bei gemessenen 8,07 s je Rundtrip (4,016 + 4,050 s, RECONCILED `flips`-Tabelle) sind das ~169 s Naht fuer nichts. == DIE WURZEL == `idle` ist an dieser Stelle der STARKE Leere-Term (`decode_work_bs() == 0 AND not work_exists()`) -- es ist also BEWIESEN, dass weder ein residenter Decode-Bundle noch ein Prefill irgendwo geschuldet ist. Die Regel flippte trotzdem, um sich in der Ruhe-Schicht zu POSITIONIEREN. Diese Praemisse ist fuer eine KALTE Ankunft verkehrt herum, und eine andere bekommt eine leere Kiste nicht: ein neuer Request braucht ZUERST einen Prefill-Pass, und Prefill laeuft unter strict purity nicht in TP. In TP zu ruhen heisst, die naechste Ankunft zahlt tp_to_pp zum Prefillen UND pp_to_tp zum Dekodieren. In PP zu ruhen ist fuer genau diese Ankunft strikt besser. KEINE Dwell-Erhoehung: ein laengerer Dwell macht die Schleife langsamer, nicht abwesend, und bei NULL Arbeit ist kein Timer lang genug, um den Handel positiv zu machen (sgl-project#819-Preisfrage, die ein Timer per Konstruktion nicht beantwortet). == FEHLERKLASSEN-PASSENDER CHECK (Direktaufruf von decide(), 3 Faelle) == Klasse a) Format-Spec auf einem Conditional im f-String: beide Zweige ausgefuehrt -> "idle 0.0s" / "idle 11.5s". AST + Import gruen. Klasse b) falscher Zweig / Verhungern: CASE1 idle+leer in PP -> direction=None, Reason traegt sgl-project#1011 PASS CASE2 Prefill geschuldet -> "prefilling in pp (50000 tok pending)" PASS (nimmt den idle-Zweig NICHT) CASE3 Decode-Arbeit da -> direction=pp_to_tp via DRAINED PASS <- das ist der sgl-project#858/sgl-project#1006-Verhungerungsbeweis: mit echter Arbeit feuert der Flip weiterhin. Belegt, nicht argumentiert. == ZWEISEITIGE ABNAHME AUF METALL (boot_855_1011idle) == (i) LEERLAUF, 11 min 39 s, Deadman aktiv, 7 health_generate-Proben bedient: Flips 0 | PHASE-POLICY armings 0 | sgl-project#1011-Refusals 77 | health 200 Vorher auf vergleichbarem Leerlauf: 42 Flips. JETZT NULL. (ii) LAST, conc=4 x 200 tok: 4/4 fertig, 0 Fehler, TTFT min 0,30 / med 0,31 / MAX 0,31 s -- keine Verhungerung Completion 11,38-11,85 s Flips: GENAU 1, armiert via DRAINED "N req decoding" -- der Flip feuert, wenn echte Arbeit wartet. == WAS DIESER SCHNITT NICHT TUT -- und meine eigene frueher zu grosse Behauptung == Ich hatte A als groessten Hebel gegen die 68-%-Flip-Steuer benannt. GEMESSEN FALSCH, und die Korrektur gehoert hierher: conc=4, 600 tok: 88,64 tok/s (vorher 90,47) -- unveraendert in der Streuung bs1, 600 tok: 25,09 tok/s (vorher 25,37) -- unveraendert Unter DAUERLAST sind die Flips nachfrage-getrieben (bs1 zahlt weiter ~2 Flips je Request: 12 Flips auf 6 Requests), und genau das ist die 68-%-Steuer. Der Leerlauf-Befund bleibt gueltig und der Schnitt beseitigt reine Verschwendung ohne Preis -- aber er kauft die Last-Steuer NICHT zurueck. BELEG-STUFE: BOOT-BEWIESEN fuer beide Abnahmehaelften (Zahlen und n oben). Der Drain-and-Flip-Kontrakt (sgl-project#925) ist unberuehrt; nur der Leerlauf-Zweig mit `rest_phase == PHASE_TP` aendert sein Verdikt.

docs: update README

df7d1a8

zhyncs requested review from Ying1123, hnyls2002 and merrymercy July 30, 2024 06:11

zhyncs merged commit bece265 into sgl-project:main Jul 30, 2024

zhyncs deleted the doc branch July 30, 2024 06:17

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

docs: update README (sgl-project#819)

158a4ee

Navigation Menu

Uh oh!

docs: update README - #819

docs: update README#819
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:doc

zhyncs commented Jul 30, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

zhyncs commented Jul 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

zhyncs commented Jul 30, 2024 •

edited

Loading