Fix List input bug by yichuan-w · Pull Request #838 · sgl-project/sglang

yichuan-w · 2024-07-30T19:21:52Z

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Fix one small bug in openai/adapter

change line 364: if isinstance(prompt, str) or isinstance(prompt[0], str):

while the code in main will cause error when serving list input

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.

hnyls2002 · 2024-07-30T20:40:06Z

LGTM.

…g/round8 A presence withhold whose every clause is beyond the armed service turn's reach stops running the 60 s clock out, and the detector alarms that were logged but grepped by nobody (sgl-project#850 plus sgl-project#838's two) reach monitoring. Desk evidence on the merged branch: 3749 passed / 0 failed on the full managers suite with nothing concurrent, against 3726 passed / 1 failed (pp_admission_wraparound, pre-existing) on pristine integ/round8 -- delta +22 is exactly this branch's new tests. Metal UNPROVEN; booked as W22.

…, and can be wrong W24 residual defect (iii), and the ticket's own remedy is NOT what shipped. The 09:01:37 specimen carries TWO defects; "break the band when the decode bundle is not draining" is neither of them. THE ARITHMETIC THAT REFUSES THE TICKET'S REMEDY. At the measured decode_contention (sigma) = 1 the scheduler gives prefill absolute priority per iteration -- `_differential_flip_threshold` records the measurement verbatim: "an iteration with any prefill chunk pending runs THAT batch and never reaches the decode branch". So while prefill is pending in TP the decode bundle CANNOT shrink, by construction. "Bundle not draining" is therefore IMPLIED by the band's own operating condition, not evidence about it, and a band that broke on it would collapse to the plain break-even for every load with a request decoding -- silently deleting the sgl-project#665-F1 differential model. That is a policy rewrite wearing a bug fix. (A) THE DETECTOR READ A BAR THE POLICY NEVER APPLIED. The alarm printed `bar_tok=20057` and fired on `pending 22887 > 20057`, while the hold it indicted names its own bar in its own text: `> N=20057 but <= 30086`. 30086 is `effective_flip_threshold(cfg, running_bs=1)`, the differential bar, which at sigma = 1 is N0 x (1+2B)/(1+B) -- exactly 20057 x 1.5. The policy compared 22887 against 30086 and HELD, correctly. So W24's single LAYOUT-ECONOMY ANOMALY is a FALSE POSITIVE, and the "first metal catch of the detector" reads better as its first metal self-indictment. This is sgl-project#819's ONE READING rule -- "the bar the policy APPLIED and the bar the log REPORTS can never be two different numbers" -- holding inside phase_policy and breaking at the module boundary. It is the sgl-project#851 class root exactly ("the DECIDERS still read their own bookkeepers"), occurring inside an sgl-project#851-family detector, which is the same shape sgl-project#853(i) already found once this build. The applied bar is now passed from the one authority that computes it, and is REQUIRED rather than defaulted: a default is the mechanism by which a caller silently re-creates the divergence. The gate takes max(break-even, applied), which bounds the blast radius to one direction -- it can only RAISE the bar, so it can only remove false positives and can never invent an alarm. That also covers strict purity, where the threshold is 0 by construction. (B) THE BAND HAD NO FALSIFIER FOR ITS OWN PREMISE. It was the ONLY hold in `_decide_from_load` that could not be wrong. Every neighbour carries a bound: min dwell yields to `starved` (sgl-project#768), drain mode to the sgl-project#833 stall deadline, the idle lock to the idle dwell (sgl-project#748). The band says "prefilling it in tp beats the round trip" -- a claim priced at the TP prefill RATE, true only while the backlog is actually being prefilled here. The falsifier is taken on the axis the claim is made on: PREFILL PROGRESS. `pending_prefill_tokens` is "admitted but not yet computed", measured at the chunk fill boundary, so it drops every round a chunk is computed -- even mid-way through one long prompt. Frozen for a whole decode window means no chunk was computed for a whole decode window. The clock is sgl-project#677(a)'s existing `last_prefill_progress_at`; the window is `drain_stall_deadline_s`, the same quantity sgl-project#833 and the sgl-project#838 detector already use, so policy and detector cannot come to hold two different ideas of one decode window. Unstamped reads as no stall, so a caller that never observed cannot flip on it. THE TWO HALVES AGREE ON THE SPECIMEN INSTEAD OF DOUBLE-COUNTING IT. W24's pending oscillated 0 -> ~22.5k -> 0 on a ~5-min period, so prefill progress was live and this break would have stayed SILENT there -- consistent with (A), which says that hold was right. That is what makes them two separable defects rather than one defect described twice. NOT REBUILT (prior-art gate): the ticket pairs (iii) with "a completed flip should clear the staging backoff". Already implemented -- `note_flip_completed` pops last_abandon_at / arm_refusals / arm_hold_until / arm_degraded for the direction -- and already pinned by test_phase_policy_flip_reachability.py:: test_a_completion_clears_the_staging_rate_limit_outright. Cited, not duplicated. A band-break is paced by that limiter like any other arm, which is correct: `_decide_rules` applies it after `_decide_from_load`, and `_demand_outweighs_a_retry` still overrides it when the backlog outweighs the wait. WHAT THIS DOES NOT CLOSE: it does not move the flip. W24's stuck phase was FUNDING -- 153 arms refused after the policy had already said the load wanted the flip, 43 of 45 binding refusals reading cause=phantom_capacity, which is sgl-project#852's territory. (iii) must not be cited as a flip-stickiness fix. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_band_premise_853.py 15 passed test_layout_conformance_838.py (updated) unregressed test_flip_threshold_repricing_819.py unregressed test_phase_policy_flip_reachability.py unregressed test/registered/unit/managers/ 3833 passed, 18 skipped, 334 subtests (759 s) test/registered/unit/mem_cache/ 1710 passed, 1658 skipped, 361 subtests (127 s) ruff check + format clean Red-first, both halves and both directions. (A) was red as the specimen FIRING -- "a hold inside the policy's own band was alarmed on" -- with the can-fail direction proving a detector that merely went quiet dies (pending above the applied bar still fires, and a window-3-shaped span at 119x the bar is above BOTH bars and stays an anomaly). (B) was red as the frozen backlog holding forever, measured: 25 observations at a constant 25065 tok returned "too short for the round trip" every time, with the can-fail direction being that a progressing prefill, a sub-window stall, a sub-band backlog and an unexpired min dwell must all still HOLD.

…being ambiguous W25 ran a sustained TP-sticky prefill phase -- `Prefill batch phase=tp` live on all three ranks at 11:38:51, 16-24k prompts, no cutover after 11:33:57 -- and THE USER FOUND IT BY EYE while grep -c "LAYOUT-ECONOMY ANOMALY" boot_w25_0824_1125.log -> 0 for the whole boot. Fourth instance of the failure form sgl-project#838 exists to prevent, against a standing order that a human eye is never the instrument. THE DETECTOR IS EXONERATED, AND THAT IS SAID PLAINLY RATHER THAN PAPERED OVER. Its class-2 gate 5 requires `pending > the bar the policy applied`. The hold reasons across the sticky phase: 19x pending prefill <= N=18614, running it in tp (BELOW the bar) 6x > N=18614 but <= 27921 with 1 req decoding (inside the band) 25x staging rate limit / 18x drain mode / 7x min dwell Pending was 16-20k against a LIVE break-even of 18614: sgl-project#819 reprices N from the measured seam, the seam had risen to 8.50s against a 3.2s seed, and the bar moved 7004 -> 18614 with it. The policy's own arithmetic said TP was cheaper, so an alarm would have been precisely the over-eager mutant `test_mutant_overeager_detector_dies` exists to kill. NO ALARM IS ADDED HERE. The real defect behind the sticky phase is the SEAM COST that raised the bar -- W25 root R1, recorded in NOTE_851 and NOT fixed by this commit. WHAT IS ACTUALLY BROKEN IS THAT THE ZERO CANNOT BE READ. `c2 = 0` conflates: ran every round and correctly declined (what happened) never ran (the W24 shape, one ticket ago) not wired on this pin threw inside the try/except and was swallowed That is the sgl-project#851 defect class -- the silent multi-valued zero -- occurring INSIDE the detector, one commit after sgl-project#853(i) had to remove the identical shape from the exposure gate. It is why a person's eye ended up being the instrument, and it is the same remedy: the instrument announces what it decided. LAYOUT-ECONOMY CHECKED (sgl-project#838) the verdict ran and declined, with the declining gate and both bars in the line Counted always, logged on a 60 s heartbeat (DECLINE_REANNOUNCE_S, deliberately slower than the 10 s alarm cadence: an alarm is urgent, a decline is a liveness proof and must be cheap enough that nobody switches it off). At INFO, not DEBUG -- W25 ran at INFO and would not have seen a DEBUG line, which is the correction sgl-project#853(i) already had to make once on the exposure clamp. The counter field gains `c2ok`, so `c2=0, c2ok>0` reads healthy and `c2=0, c2ok=0` reads never-ran. `_should_say` takes the cadence as a PARAMETER rather than growing a second copy that could drift from the first. THE CAN-FAIL DIRECTION IS THE WHOLE RISK and is pinned: a fix that downgraded real anomalies into heartbeats would satisfy every liveness assertion in the file while destroying the detector. So an anomaly must still alarm, must count as an anomaly and never as a check, the two markers must be distinct strings, and an alarm line must not contain the CHECKED marker -- that last one because sgl-project#853(i) already caught a marker quoting another marker's text and restoring the ambiguity while passing everything else. TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_economy_detector_liveness_854.py 14 passed test_layout_conformance_838.py (periodic field updated for c2ok) unregressed test_band_premise_853.py unregressed test/registered/unit/managers/ 3847 passed, 18 skipped, 334 subtests (737 s) ruff check + format clean Red-first: `note_economy_declined` did not exist, so every liveness assertion failed at AttributeError, and the periodic-field test failed as 'c1=1, c2=2, c2ok=0' != 'c1=1, c2=2' -- the field genuinely changed shape and the existing pin was updated deliberately rather than loosened. Also recorded in docs/dev/NOTE_851_build_caveats.md: W25's three new roots (R1 seam cost, R2 the gate short ~210 MiB beside kv-slack's 2432 MiB, R3 sgl-project#852 still over-promising a stable 88 MiB and saying so through its own discriminator). sgl-project#852 must not be cited as closed.

W25 went TP-sticky on 16-20k prompts and the user caught it by eye. The policy was right and the PRICE was wrong, and the price was wrong in a way this module had already written down and then not applied to itself. phase_policy.py:82 C = round-trip flip cost, seconds break_even_tokens "never repays the {flip_cost_s}s round trip" But `observe_flip_leg` fed ONE LEG per sample -- its own docstring even computes the round trip it is not feeding, "tp_to_pp 11490 + pp_to_tp 5681 = 17171 ms" -- and BOTH DIRECTIONS went into ONE EMA. THE TWO LEGS ARE NOT THE SAME QUANTITY. On the binding rank W25 measured tp_to_pp 10466-13181 ms against pp_to_tp 5078-6545 ms. An EMA fed both alternately converges to neither. REPRODUCED EXACTLY, which is what makes this measured rather than argued. Replaying PP0's eleven PHASE-FLIP DONE totals through one estimator at ALPHA=0.3 gives 5.0779 6.6944 6.2494 7.5450 7.2426 9.0241 8.2740 9.2457 8.4356 9.3990 8.5041 -- and the boot's own decision lines printed N=15853/18110/18464/18614 at exactly the samples pricing to 7.2426/8.2740/ 8.4356/8.5041. To the token, four times. The blend settled at 8.50 s: BELOW every tp_to_pp leg and ABOVE every pp_to_tp leg. The true round trip was 11.6 + 6.4 = 18.06 s. The bar also oscillated ~2000 tok with flip-direction parity (8.50 after a pp_to_tp, 9.40 after a tp_to_pp) -- an artifact of the blend that says nothing about cost. sgl-project#819's own closing sentence is the rule it broke one level up: "a component and its container are different quantities and an EMA fed both alternately converges to neither." Two directions are different quantities too. THE CORRECTION RAISES THE BAR (C 8.50 -> 18.06 s, N 18614 -> ~39500), so it makes TP-stickiness on 16-20k prompts MORE correct, not less. That is stated plainly rather than softened: the remedy for a bar that is too high is a cheaper seam, not a permanently under-priced one. dN/dC = 2188.8 tok/s says what each second of seam is worth once it is. WHAT SHIPPED. `RoundTripFlipCost` holds one `FlipCostEstimator` PER LEG and sums them. The leg estimator is REUSED, not rebuilt, so every property sgl-project#677 pinned on it holds per leg -- including that it tracks DOWN as readily as up, which is what makes a future seam fix actually lower the bar instead of latching high. The seed is split in half, so an uncalibrated instance values exactly the round-trip seed and the pre-sgl-project#856 path is unchanged. An undirected reading is treated as a whole round trip and split evenly (so callers that really measured one stay honest); a direction this class does not know is REFUSED rather than filed under a guess. PROVENANCE GAINED A THIRD WORD. `flip_cost_measured()` is a boolean over a quantity with three states and printed the middle one as "measured". `flip_cost_provenance()` returns seed / half-measured (<leg> only) / measured. Same class of fix as sgl-project#853(i) on the exposure gate and sgl-project#854 on the economy detector, one layer further in. AND THE CONSUMER IS RECONCILED, which is the half that is usually missed. The sgl-project#838 economy detector refuses to question a bar priced off the seed because "an assumption is not the policy's own claim". A HALF-measured round trip is still half assumption, so it is refused on the same ground: `flip_cost_fully_measured()` requires BOTH legs. The blast radius is one-directional -- the detector can only DECLINE more often, never alarm more often. DELIBERATE TEST CHANGES, not loosened ones: * `test_the_leg_total_becomes_the_price` now asserts the leg's own value AND the round trip (11.4901 + 3.2/2). Asserting 11.4901 for C would be asserting that one leg is the whole round trip -- the defect itself. * `_measure` floors at 2x MIN_ESTIMATE_S: each leg carries its own band, so a round trip cannot be cheaper than two leg-minimums. * `test_a_seeded_price_is_not_evidence` now also asserts "half-measured". TESTS (hermetic, CUDA_VISIBLE_DEVICES=""): test_round_trip_price_856.py 19 passed test_flip_threshold_repricing_819.py unregressed test_flip_cost_calibration_677.py unregressed test_flip_threshold_honesty_777.py unregressed test_flip_cost_clamp_directions_677.py unregressed test_layout_conformance_838.py unregressed test_economy_detector_liveness_854.py unregressed ruff check + format clean on the changed source ALSO FOUND, NOT FIXED HERE (recorded in NOTE_856_seam_cost_ledger.md): `observe_flip_leg` is called only from the flip-COMPLETION branch, so a boot whose flips are all refused or abandoned prices off the seed for the whole session -- and sgl-project#777's staleness WARNING is gated on the same event, so such a boot gets neither a reprice nor a warning. W25 did not manifest it (33 completed cutovers). Same silent-zero shape, one more instance.

…chable, not merely outranked USER DIRECTIVE (2026-08-24): all pending prefill is collected and processed in the PP layout, then all decode in TP, then prefill again -- and NEVER any work in the wrong layout, "egal wie lang der flip dauert". Flip duration is recorded from here on, not traded against. THE BAND HAD TO STOP BEING CONSULTED, and that is the whole content of this slice. `drain_mode` already existed and its PP half was already strict (the DRAINED rule flips at `pending <= pp_exit_tokens`, default 0). The TP half was not: the break-even band sits ABOVE the drain exit in `_decide_from_load` and can `return _no(...)`, so a correct drain rule underneath is NOT sufficient -- the band holds TP while prefill waits, and prefill then runs in the decode layout, which is the one thing this mode forbids. `drain_mode_strict` gates the band off. One condition, not a new engine. DELIBERATELY NOT a third `--phase-flip-policy` value: that enum selects the manual-vs-auto ENGINE, not the arming rule, and it is a different axis from the `phase_purity` MODE_STRICT enum. Conflating any two of the three would make the next reader's mental model wrong in a way the tests would not catch. STRICT IMPLIES DRAIN, refused rather than half-applied: the exit strict relies on lives in the drain block, so strict-without-drain would gate the band away and then fall through to the very economics it exists to remove -- configured in appearance, neither mode in behaviour. THE ECONOMIC MODE IS UNTOUCHED for every other workload, and the tests say so in both directions: same load, same numbers, the band still holds TP without strict and cannot be the reason for anything with it. The N~=28,050 repricing from the previous slice stays DOCUMENTATION -- in this mode it is not the trigger, so it is not tuned further. TWO TEST TRAPS THIS FILE FELL INTO AND NOW NAMES, because both would have made it pass while proving nothing: * a config whose band cannot be ENTERED (threshold == N) passes whatever the code does. `TestTheBandIsOpenAtAll` pins threshold 3000 against N 1000. * comparing strict against economic-WITH-drain shows no difference, because the existing drain rule already flips a drained bundle on any backlog. The sub-N test compares against PURE economics (drain off). * a state with `bundle_at_phase_entry=0` arms the "decode phase ran EMPTY" rescue in EVERY mode, hiding the difference behind a rule neither owns. RECIPE (boot_w29.sh), from this session's own scorer finding: the W28 specimen had 153 prefill batches EXECUTING IN TP LAYOUT. That followed from `--phase-flip-purity prefill_in_tp`, so the recipe now boots `--phase-flip-purity strict --phase-policy-drain-mode-strict`. w29_score.py reads conformity from the scheduler's own `phase=pp/tp` batch markers -- independent of the sgl-project#838 detector on purpose, so the detector stays falsifiable rather than self-certifying -- and now also SCORES purity stand-downs: the `_relaxed` escape valve stays (correctness over starvation) but every stand-down is a departure from the target mode and must not pass silently. GATE (foreground, family-batched): phase policy suites 172 passed managers (chunked) 3193 passed; 1 failure + 1 collect error, both identical at HEAD server_args cli-metadata / ratchet / migration 18 passed, 19 subtests strict-batch suite 9 passed ruff F/E7/E9 clean; format clean.

…retracted for W30 ran the strict-batch acceptance and found two blockers behind the pool leak it had come to prove fixed. Neither is in the pool accounting, which was clean across 150 flips. FIX A -- `_live_reqs` enumerates `last_mbs`. Arm 2 died on all three ranks in 21 s: ResidentCarryError: PHASE-FLIP-CARRY 1 request(s) are reachable only through last_mbs/last_batch at the cutover: ['56fddcc3c0ef...'] Two functions disagreed about what "resident" means. `_live_reqs`, the authority the RETRACTION uses, read running_mbs / running_batch / last_batch / chunked_req. `orphan_resident_reqs`, the cutover guard, checks last_mbs AND last_batch. So `last_mbs` was a route the retraction could not see and the guard did check: a request freshly prefilled under event_loop_pp sits in last_mbs[slot], was never retracted, and the guard correctly refused. Fixed at the authority, exactly as the W27 fix was -- it simply stopped one route short. TWO sites needed it: enumerating without CLEARING would retract the request and leave the guard's reference intact. The guard is untouched and stays as strict as it was; its docstring rule holds ("a bug to raise, not a carry to widen"). FIX B -- the seam re-admission is FLIP TRANSPORT, and only that. Arm 1: 150 flips in 17 minutes, 129 prefill batches, ZERO decode batches, 28 of 28 client requests timing out at 600 s. The arm auditor called it 12 times: "armed pp_to_tp (... 1 req decoding ...), the cutover COMMITTED into the target layout, and it still built no batch in 8 rounds". Every link is a shipped decision. The policy arms pp_to_tp BECAUSE a request is ready to decode; the sgl-project#856 seam then retracts that very request (no-carry); re-admitting it in TP needs a read-through prefill; strict purity forbids prefill in TP absolutely (`Prefill batch phase=tp` count for the whole arm: 0); so TP builds nothing and the policy flips back. The arm's justification is destroyed by the arm's own execution. The re-admission recomputes NOTHING -- the tokens were prefilled in the PP window and their KV is in the canonical store from the sgl-project#703 fence. It is a cache restore: seam mechanics, the same category as the KV the flip moves. So it is exempt BY NAME rather than by a purity stand-down, which would let ordinary prefill into TP and be counted a violation by both the sgl-project#838 detector and w29_score.py -- making the acceptance unpassable by our own instruments and dishonest about the user's rule. THE MARKER IS SEAM-SPECIFIC ON PURPOSE. `is_retracted`/`retracted_stain` already existed and were readable at the gate, but `reset_for_retract` is reached from four paths: decode-OOM preemption, the PD prefill path and the PP void path set the identical booleans. Keying on them would exempt every preempted request's re-prefill, which is real work. So the cutover stamps its own `Req.seam_readmit_epoch`, spent on the one re-admission it licenses. The builder additionally keeps an exempt batch to stamped requests only, so a new arrival cannot ride along inside it. Rank-uniform: the stamp comes from a group-unanimous cutover and is read off the replicated waiting_queue. THE LIVELOCK SAFETY NET. The purity valve arms on flips that are guarded, abandoned or refused. W30's flips all COMMITTED, so the valve was blind to it and stood down 0 times through a ten-minute livelock. The signal it needed was already being written to a log with no consumer: ARM-VERDICT-WRONG is now booked as a streak and read as a fourth cause. Its rank-uniformity is the weakest of the four and says so -- it is bounded away, ~32 batchless rounds deep, past anything transient. w29_score.py now counts it as a HAZARD, so if this net fires the run fails loudly: it is a net under the acceptance, never a way to satisfy it. Verified both ways -- 13 hazards on the W30 arm-1 specimen it previously scored as 0, still 0 on the W29 specimen. FIX C -- an arm may not destroy its own justification. `PhasePolicyConfig.seam_readmit_available`, static boot config. The DRAINED branch refuses with a named reason when the target cannot re-admit what the cutover retracts. NOT gated on `inp.target_can_admit`, the tempting one-liner: at arm time the residents are not yet retracted and so not yet stamped, so that term is False for the very arm that would make it true, and gating on it deadlocks the other way. TESTS 15 new (test_seam_transport_exemption_w30.py) + 4 new (test_retracted_leaves_live_universe_856.py), red-first. CAN-FAIL, the dangerous direction: a genuine new request, an OOM-preempted request, and an empty queue must ALL still be blocked in TP; the round flag must not latch; exactly one site in python/sglang/ may write the stamp. CAN-FAIL, the guard direction: `assert_no_orphan_resident_reqs` must STILL raise without the consume -- if it stops, the guard was widened rather than satisfied. DRIFT-DETECTOR: every residency route the guard checks must appear in both the authority and the consume path. managers unit suite: 8 failed / 4035 passed, against 8 / 4016 before this slice and 9 / 4004 at e8bce97. Same 8 pre-existing failures throughout; +19 passes = the 19 tests added here. Targeted policy/purity/flip family: 283 passed. ruff clean, delta 0. Window record and specimens: /spinning/gpu-arb/W30-RESULT.md

W37-D, live: 258 'Prefill batch phase=tp' batches, every one #new-token 4096 / #cached-token 0, mean cached 0.0%. The seam-transport exemption permits prefill in the TP layout on the ground that 'the re-admission recomputes nothing -- it is a cache restore'. On metal it recomputed everything: #cached-token was 0 on ALL 1441 occurrences in the boot and storage_hit was 0. That is real prefill work in the decode layout, 258 times -- the user's strict-batch law broken by the exemption written to respect it. PRE-EXISTING, NOT F2-INTRODUCED, proven before patching: 'Prefill batch phase=tp' appears 6x in W37-B and 12x in W37-C, both before #861c was in the tree. W37-D shows 231+ because the instance finally works (222 flips vs 18), so F2 AMPLIFIED an existing defect rather than creating one. CLASS: a guard or exemption that ASSERTS a factual premise in prose and never verifies it at runtime. Same class as #861c/F1, where my own guard assumed two host pools have equal slot counts. Both silent for a whole window. Both fixed the same way: check the premise where it is relied upon. SIBLINGS: F1 (pool-size equality) fixed in #861c; this one found because the class was named but the SWEEP was scoped to pool sizing only -- a sweep-completeness failure I am recording as such rather than as a new class. FUTURE-CHECK: seam_transport_premise_holds() reads cache_protected_len at the gate; a test pins that prefill_blocked_here ANDs the two, so the check cannot be installed-and-unreachable (the W32 shape). WHY NARROWING IS SAFE NOW AND WAS NOT BEFORE: the exemption exists because W30 ping-ponged with zero decode batches, and the ROOT of that ping-pong was the blind counter #861c/F2 fixed. With the existence term in place a held TP prefill DEMANDS the flip to PP and PP admits it, so the hold has somewhere to go. The workaround is narrowed only because its cause is gone. sgl-project#838 GAP CLOSED: layout_conformance had terms for hold ECONOMY and routing provenance and none for the user's law itself, so it logged through all 258 violations and flagged zero. work_layout_verdict() adds the conformity term and keys on MEASURED bytes, not on the caller's intent: a batch claiming transport while recomputing tokens is a violation. Verified on five arms including the exact specimen (flagged), a genuine restore (not flagged), right-layout, non-strict, and decode-in-pp. W30 exemption tests updated deliberately: their doubles now carry a restored prefix by default, because that is what those tests were always about, and the cold case gets its own explicit tests rather than being a silent default. Tests: 785 passed on the touched suites (purity/conformance/838/856/policy/ seam/861/859), 28 in the W30 suite. ruff: zero delta.

The merge-regression pass runs ruff with the repo's OWN selection (.pre-commit-config.yaml: --select=F401,F821,UP037), not ruff's defaults -- the default run reports 575 findings on these files and measures a gate nobody enforces. Under the enforced selection: 0 F401, 0 F821, 22 UP037. Nineteen were already there at 0cd27d9, measured rather than assumed by running the same selection against the base blobs of the same files; three are new, all in phase_policy.py, all forward references introduced by sgl-project#889/sgl-project#893 (effective_pp_exit_term, stranded_decode_s, superseded_pp_bound_warning). Fixed all 22 rather than the 3. Both files carry `from __future__ import annotations`, so the quotes are inert and their removal is the autofix the hook itself applies; fixing only the new three would leave the file split between two conventions, and leaving all 22 hands the next toucher a pre-commit failure they did not cause. 91 passed across the suites over both files (sgl-project#889, sgl-project#893, sgl-project#896, the arm outcome suite, sgl-project#838 layout conformance) after the rewrite. Codespell, for the record, is NOT clean and is not made clean here: 12 hits on train-touched files against 11 at the same base. The single new one is "oder" in test_one_chunk_tp_prefill_887.py:8 -- inside a verbatim German user directive the test quotes as its premise, the same class as the "egal" quote that has sat at phase_policy.py since before this train. A quote is not a typo, and rewriting one to please a dictionary would falsify the record it exists to preserve.

…omote, and the silently-wrong pair ONE build closing the publish -> cached_tokens chain, plus the health-probe confound that made every measurement of it ambiguous. THE TWO BINDING TERMS (both desk-proven at the tree, both root, not symptom): 1. The promote `pending -> effective` existed at exactly ONE site, inside `_event_loop_pp_body`. The warm prefill on this rig runs boot-measured in the TP phase (the instance flags it itself: `sgl-project#838 work_in_wrong_layout class=prefill phase=tp`, 9 genuine violations), where that body never runs. PP0 published into `pending` and nobody ever promoted it: `sgl-project#968 LOAD-BACK PROMOTED` = 0/0/0 over three whole boots, and that is a REAL zero -- its cadence counts promotions, so the first would have printed. Fix: `promote_pp_load_back_pending()`, idempotent, called from both loop bodies AND from `_get_new_batch_prefill_raw` (the phase-neutral consumer), above every early return. 2. `init_pp_loop_state` nulled `_pp_load_back_pending` UNCONDITIONALLY while the line directly below guarded `_pp_load_back_effective` with `hasattr`. The unconditional null was deliberate ("only pending is a per-pass quantity") and is only fatal BECAUSE the promote was phase-bound. Fix 1 is the root; this is the guard that carries the fact across the cutover. REGRESSION IN e0bc960, FOUND AND FIXED: that commit moved the offer write to the load-back site but kept only the `_hit > 0` half, deleting the only deleter -- the old `= _hit if _hit > 0 else None`. The comment it removed states the lost property verbatim ("goes to None on its own"). The offer then outlived its own truth: stale-HIGH re-publish -> a peer told an extent its host tier no longer covers -> `EXTENT UNHONOURABLE` -> PPScheduleRefused, which voids the pass for the WHOLE GROUP. Deleter restored at the spend. STAGE A: the stamp population and the entry population were two different sets -- the stamp sits behind eight `continue` branches (`already_in_batch` is the sgl-project#946 shape: `add_chunked_req` appends the continuation to `can_run_list` BEFORE the loop, which then skips it), while the decision is built from `can_run_list`. Boot-measured in the gap: `loop_skips( prefetch_pending=1(first=bce57cf8...))`, 3x, the same 4618-token rid. The key is now stamped over `can_run_list` itself, so the two populations are identical by construction. This also closes the NEXT link, which was already visible: the publish filter needs `load_back_len AND load_back_key`, so an entry with a live offer and no key would still have published 0. SILENTLY-WRONG PAIR, mandatory in this build because fixes 1+2 are what arm the path -- once live it returns wrong tokens rather than crashing: * The adopted GDN state is the node's END state, at `_applied`; the S1 clamp cuts KV to `told` and cannot cut a transfer that already ran. The scan would resume at `told` from a state that has consumed `told.._applied-1`. No assert can fire; the KV geometry is self-consistent at `told`. And it is the EXPECTED relation, not a corner (a4c9a7a: "stale-low by construction", measured 1215 -> 1216 -> 2114). The whole load-back now falls instead; the state is NEVER trimmed, per the hazard the code already spells out. The slot is returned before the raise, because the exception bypasses the sgl-project#991 revert site. * Mamba slot starvation used to skip the restore while the KV prefix grew to `told` -- warm-booked prefix over a state never loaded. It now fails the whole load-back. KV extent and GDN state stand or fall together, one rule. HEALTH ISOLATION (sgl-project#942, standing user order): a /health_generate probe is a liveness instrument, not work. Upstream only dispatches one when the box is `is_fully_idle`, so at idle the probe IS the entire backlog, and under purity `tp_threshold` is 0 -- its single token decided the layout of the whole instance. Excluded from all FOUR backlog terms via `_pending_prefill_tokens(include_health=False)` and `_admissible_prefill_tokens(include_health=False)`, at the economy readers only. THE SPLIT IS AT THE CALL SITE, AND THAT IS THE WHOLE POINT. A blanket exclusion is the obvious form and is wrong in the silent direction: `tp_compute_fits_in_one_chunk` grants only while `0 < pending < chunk`, so subtracting the probe there too collapses the sgl-project#887 grant and the probe is served by nothing at all. Service readers keep the default; economy readers exclude. No new tag: upstream already stamps the rid at the /health_generate source, and its own predicate reads `.rid` via getattr, so it applies unchanged to Req, TokenizedGenerateReqInput and PPAdmissionEntry. MEASURABILITY (without these the next boot cannot read its own result): * `cached_tokens` was structurally blind for exactly the population the warm acceptance lands on: the booking sits under `if not req.retracted_stain` and the stain is an ever-flag, while the cutover design retracts and re-admits everything. Now booked per admission by assignment (never `+=`, so the double count the stain guards stays impossible) and the stain is kept, because two other consumers read it. * The publish probe counted PASSES: with fewer than 64 publishes the `%64` arm never fired and the whole budget went to the boot's cold head, leaving every warm opportunity unobserved. Now event-cadenced, and health is its own column rather than mixed into the denominator. * `sgl-project#783` named a term that was not the one firing (it printed two identical numbers); each of the three terms now names itself. * The `sgl-project#998` sample printed `items()[:3]` of a never-pruned dict, i.e. the OLDEST entries forever -- one genuine event echoed 35k+ times, which reads as a request that stayed resident all boot. Now `[-3:]`. Ordering guard at `truncate_prefix_to`: it zeroes `host_hit_length`, which the offer is captured from. Dormant only because sgl-project#1015 deleted the receiving end; whoever restores that wire must re-check this order first. Tests: test/srt/test_health_isolation_942.py, 7 tests against the production functions, all green; can-fail proven by mutation (exclusion -> no-op gives 3 failures, incl. "a health probe armed the tp-ward flip"). Cross-module exception identity for the new refusal verified as identical. AST checks on signature/call-site agreement (9 sites) and on control-flow ordering (35/35, 24 failures against pristine HEAD). ruff 108 = 108 vs HEAD, no new findings. Not proven on metal at the symptom -- that is the next boot.

…t waited for a chunk the gate refuses to build BOOT-PROVEN EDGE (boot_855_1050dev_0840f82601_0831_115825, the boot 20 livelock). The open question of the handover was WHY the broken chunk never completes before the 30 s abandon. It is not a race: inside the armed window 12:22:19-12:22:49 PP0 logged ZERO `Prefill batch` lines and 15 `Decode batch` lines. The chunk was never attempted, and the re-arm is in the SAME SECOND as the abandon (12:22:49), so no unarmed window existed in which it could be. N9's remaining candidate (abandon deadline vs chunk completion time under growing backlog) is therefore FALSIFIED. THE BINDING TERM, with file:line. `prefill_blocked_here` (phase_purity.py:1087-1095) grants the sgl-project#887 one-chunk exception only when BOTH `prefill_allowed_in_tp_now(spent)` AND `tp_compute_fits_in_one_chunk` hold -- deliberately, so the gate and the sgl-project#838 detector permit the same batches (REGISTER_OPEN_876 on sgl-project#887). `tp_compute_fits_in_one_chunk` is `0 < pending < chunked_prefill_size`. The last grant was 12:22:11 with new_tokens=3392 against a 4096 cap; the backlog then crossed the cap and never came back (12347 -> 48866 -> 101075 tok), so the gate refused on every later round, SILENTLY, at its final `return not _relaxed(...)`. The grant is spent AT the grant (`_spend_tp_compute_chunk`), so a refusal on the `fits` term leaves the budget untouched. The hold (`prefill_runnable_in_current_layout`, phase_flip_runtime.py:156) read ONLY that budget, kept answering "TP can still make progress", and `chunk_blocks_quiescence` kept the rank un-quiescent for ever: PP0 armed 29 / committed 4 / abandoned 48, `this rank ready=0` in all 72 abandon lines, backlog 4618 -> 101075 tok. CLASS: a hold predicate that predicts another gate's decision in a different currency. SIBLING SWEEP over the three consumers of the sgl-project#887 grant: the gate itself reads both terms; `maybe_arm_phase_policy` (sgl-project#942, scheduler.py:13611- 13632) reads both terms and says so in its own comment; the flip hold read one. The fix makes the third consistent with the two that were already right, by passing the gate's OWN side-effect-free probe rather than a second copy of it. FUTURE CHECK: a new reader of `tp_compute_budget_remaining` that makes a runnability claim must also read `tp_compute_fits_in_one_chunk`. AND THE OPERATOR'S CONDITION, which turns this from a deletion into a replacement. Once the hold correctly reports "this layout cannot finish the chunk", the cutover commits over the broken chunk. That is licensed by sgl-project#988 riding the prefix across (loss <= 1 chunk = the sgl-project#939 grant) ONLY while the COMPLETED pages are settled in the store. The old strict arm blocked on "incomplete"; it now blocks on the real hazard -- an unacked prefix -- read from the EXISTING sgl-project#792/sgl-project#703 fence (`ongoing_write_through`, the same counter sgl-project#1050's `reclaim_rows_for_drop` and `on_idle` gate on), never a second ledger. Unknown is refusal, never permission. The hold is bounded by the ack (milliseconds), with the 30 s park deadline still the outer backstop; no second clock. UNTOUCHED AND PINNED BY TEST: mid-admission (no pool row -> still blocks), non-strict (residual, still filed not solved), and pp_to_tp drain-and-flip (runnable HERE -> still blocks, which is what keeps all prefill in PP). The PP-microbatch-in-flight half has its own predicate and is not read here. DETECTION: `_why_not` returned one string for three different holds. It now names which term is holding, including the new unacked-prefix case. DESK CHECK (matched to the error class of this edit -- a control-flow inversion in a restructured predicate, which py_compile and import smoke are structurally blind to): devtools/check_1033_quiescence_edge.py, 24 cases, hermetic CUDA_VISIBLE_DEVICES="", all pass. Includes a RED-FIRST arm that evaluates the PRE-FIX expressions on the measured boot-20 state (budget 2, pending 48866, cap 4096) and asserts they produce the wedge, and the POST-FIX expressions on the same state and asserts they commit only against an acked prefix. Plus compile + import + an order proof that both probes are assigned before either read inside get_next_batch_to_run. NOT YET BOOT-PROVEN. This is DESK-PROVEN at the predicate; the metal proof is boot 21.

fix a bug in the main/ add some usage info

e36a6a4

hnyls2002 approved these changes Jul 30, 2024

View reviewed changes

hnyls2002 merged commit bb0501c into sgl-project:main Jul 30, 2024

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Fix List input bug (sgl-project#838)

7d611db

Navigation Menu

Uh oh!

Fix List input bug - #838

Fix List input bug#838
hnyls2002 merged 1 commit into
sgl-project:mainfrom
yichuan-w:smallbug_in_adpater

yichuan-w commented Jul 30, 2024 •

edited by github-actions Bot

Loading

Uh oh!

hnyls2002 commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

yichuan-w commented Jul 30, 2024 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modification

Checklist

CI States

Uh oh!

hnyls2002 commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yichuan-w commented Jul 30, 2024 •

edited by github-actions Bot

Loading