Adjust default mem fraction to avoid OOM by Ying1123 · Pull Request #823 · sgl-project/sglang

Ying1123 · 2024-07-30T08:30:36Z

No description provided.

zhyncs · 2024-07-30T08:33:25Z

May you merge the main branch?

zhyncs · 2024-07-30T08:33:48Z

We could test the latest PR E2E test.

## Motivation  ## Modifications  ## Accuracy Tests  ## Benchmarking and Profiling  ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).

…g said it once The #791b ballot ALREADY detects that the TP replicas' waiting-queue heads have parted -- it rides an all_reduce(MIN) that runs once per TP-loop iteration, so the detector and its carrier both exist. On a mismatch it latched _prefetch_ballot_mismatch_logged, logged ONCE PER PROCESS, and fell back to the rank-local verdict. Its own comment called a divergent queue head "a deeper breakage than a divergent prefetch verdict" and then recorded it exactly once. WHAT THAT COST, from specimen /spinning/evidence-816-18f/wedge_0823_055757 (boot 0516, PP=3 with --enable-phase-flip): 05:55:19 sgl-project#797 void storm begins on PP1 05:55:38 this warning fires -- one line per rank, the last the process ever emits about it. PP0 digest 887126098, PP1+PP2 1471852626. 05:55:53 void storm core: 7710 #797d/sgl-project#798 lines in seven seconds 05:56:13 PHASE-FLIP DONE pp_to_tp (epoch 5) 05:56:15 flip back ABANDONED: "live slot set divergence cannot be repaired this round: the group's union reaches row 240831 and the poorest rank has only 1208" 05:56:15 last real forward progress 05:56:18 the three ranks each build a DIFFERENT prefill batch (#new-seq 1 vs 3, #cached-token 0 vs 16384, #queue-req 6 vs 3) 05:57:57 py-spy: PP0+PP1 in the spec VERIFY arm (eagle_worker_v2.py:2246), PP2 in the EXTEND arm (:2151); all three GPUs pinned at 100%, stacks frozen across three samples 06:00:43 external SIGTERM -- the wedge never resolved itself That 05:55:38 line was the earliest evidence in the whole chain, and the latch made it a single event with no duration attached. "It healed after N passes" and "it never healed" are the same silence to a latched logger, and they call for opposite responses. THIS CHANGES NO DECISION. The fall-back to the rank-local prefetch verdict is untouched; prefetch_done_under_ballot is not modified and its behaviour is pinned by test, because "only logging" is a claim rather than a licence. What changes is that the divergence is counted, its onset is reported, a persisting one is re-reported on a bounded cadence, and the RECOVERY EDGE is reported with the length of the divergence that just ended. NOT UNBOUNDED LOGGING, and the same boot is the argument: 7710 void lines in seven seconds is what a per-iteration warning on a hot loop becomes. The cadence reports streaks 1, 2, 4, 8, ... to a cap and then every cap-many passes -- a handful of lines per minute for a divergence that never heals. THE DECISION IS A PURE FUNCTION, AND A SURVIVING MUTANT IS WHY. advance_mismatch_streak lives in prefetch_ballot.py rather than inline. With it inline the recovery edge could only be checked by grepping the shipped source, and a source grep cannot tell a live branch from `elif False:` -- measured: mutant M2 (recovery edge disabled) left all 14 tests GREEN. After extraction the same mutant kills two tests that drive the machine directly. Same lesson as the sgl-project#820 M5 survivor: a probe that measures presence is not measuring reachability. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): 20 passed, 7.7s. RED-FIRST on unchanged 587e4c2 in a scratch worktree: 10 of 14 red (before the state-machine class was added). The 4 green are the "nothing decides differently" pins plus the detector's own can-fail arm, which hold on base by construction. Noted as a limit, not a strength. Mutants, after extraction, each killed by the arm that owns it: M1 latch restored DEAD (1) M2 recovery edge disabled DEAD (2) -- SURVIVED before extraction M3 cadence logs every pass DEAD (4) M4 total resets across recovery DEAD (1) Regression: every registered unit test touching prefetch_ballot or the uniform reduce -- 40 failed / 65 passed on branch, 40 failed / 45 passed on unchanged base (the 20 difference is this commit's new file). Identical failures, all pre-existing holder drift. Zero regressions. black clean; ruff (F401,F821,E731) and codespell report the same pre-existing hits on base and branch, none in the changed region. isort wanted to renormalise this file's whole import block, which is pre-existing disorder and not this change; reverted, so the diff is the block and nothing else (0 import lines touched). Root cause of the wedge remains OPEN. This makes the earliest signal in the chain legible; it does not stop the divergence. Acting on the mismatch (refusing admission, or a uniform named abort off the already-agreed reduce) needs metal and belongs to the 18 lane.

…nges, said once ROOT OF THE 0516 WEDGE, named (COORD-strand16f-801-build.md B.10). All three rank-uniformity floors -- evict, host, mamba -- are published from ONE MIN-reduce over `tp_cpu_group` in `_update_uniform_pool_budget`. When that group holds one member the reduce is a no-op and every floor switches OFF. scheduler.py's own comment at that site already says what that costs and calls it "the measured cause of a pipeline deadlock": "With pp_size>1 the ranks that must agree are NOT in this reduce group." And #616g's docstring states the chain that follows from unfloored eviction: the radix trees stop being replicas, `match_prefix` returns a rank-dependent prefix, `prepare_for_extend` derives extend_num_tokens from it, and every per-layer TP all_reduce of that forward is entered with a rank-dependent token count. Specimen /spinning/evidence-816-18f/wedge_0823_055757 (boot 0516) carries the scope line exactly three times, once per rank, all reading `world=1 -> floors OFF ... pp_size=3 tp_size=1`. 19 s later the queue-head digests had parted (#791b); at 05:56:18 the three ranks built prefill batches with #cached-token 0 against 16384 -- rank-dependent prefix match, the exact fingerprint #616g predicts; at 05:57:57 two ranks sat in the spec verify arm and one in the extend arm, all GPUs at 100%. WHY A LATCH WAS THE WRONG SHAPE. Under --enable-phase-flip the scope is not a boot constant: phase_flip_runtime rebuilds the TP group per phase (`want_tp_size = n if tp_phase else 1`), so the floors are ON through TP decode and OFF through PP prefill, switching at every cutover. Boot 0516 completed FOUR cutovers in 55 s and reported its coverage ONCE, at the first PP iteration. "Off for the whole run" and "off for the prefill half of every cutover" are different situations with different fixes and read identically; coverage coming BACK was never reported at all. Now: report on transition, both directions, first observation always. The decision is `uniform_floor_scope.scope_transition`, a pure function, for the sgl-project#823 reason -- inline, a test could only grep this file for a branch, and a grep cannot tell a live branch from `elif False:`. ENFORCES NOTHING, deliberately. The reduce group is not widened and no admission or eviction decision changes. Closing the gap means changing what the group agrees on; that needs metal and is B.10's 18-lane ticket. A MODULE FUNCTION, NOT A Scheduler METHOD, and that was measured. The first cut was a method and broke 61 tests across six files at once: the uniformity-floor suites drive `_update_uniform_pool_budget` bound onto `_FakeScheduler` stand-ins that bind only what they name, so `self._report_...` raised AttributeError on every one. Same binding trap `_pp_wait_for_proxy_readiness`' alias comment documents. A module function reaching the holder via getattr needs no binding; a `getattr(self, ..., None)` guard at the call site was rejected as the silently-skip shape. ALSO FIXES A REGRESSION THIS BRANCH SHIPPED IN sgl-project#823. `BudgetHarness` in test_collective_family_siblings_610.py drifted behind the reduce, which now writes `_prefetch_ballot_mismatch_streak`/`_total`. sgl-project#823's regression scope did not include that file, so the drift shipped. It is the FOURTH time this harness has fallen behind the reduce (after #616g, sgl-project#639, #639b) and the fourth time the file's own drift guard is what named it -- that guard earned its line count again here, catching my change rather than a stranger's. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): 16 passed. RED-FIRST: the module is new, so on unchanged 587e4c2 the file is a collection error rather than per-test reds. Stated as the weak evidence it is; the mutants carry the weight here. Mutants, each killed by the arm that owns it: M1 latch: only ever report the first observation DEAD (3) M2 a singleton group counted as covering DEAD (8) M3 never quiet: report every iteration DEAD (4) M4 coverage coming back swallowed DEAD (4) M5 ON-side call site dropped DEAD (1) -- SURVIVED until an arm was added for the CALL SITES; every other test reads the reporter's own body and never its callers. Same shape as sgl-project#823's M2. Regression: 10 files covering every registered unit test that touches the uniform floors or the ballot -- 54 failed / 98 passed on branch against 54 failed / 62 passed on unchanged base (the +36 are this branch's two new files). comm -13 EMPTY. Zero regressions. black clean; ruff (F401,F821,E731) and codespell clean on the new files; 0 import lines touched in scheduler.py.

Four commits from 16f, each tested and pushed separately: [sgl-project#801] a dead PP peer is not the wedge (the transport already reports it), [sgl-project#821] the watchdog could not see the wedge it was best placed to diagnose, the ballot-divergence observability commit, and the uniformity-floor scope reporter. NUMBERING, recorded because the titles do not match the register: the head commit 9d13bf0 is titled '[sgl-project#824]' but canonically it is the register's sgl-project#823 artefact (the floor-scope reporter). Register sgl-project#824 is 18f's chunked-prefill protected-len crash fix, merged as its own stage below. No rebase -- the commit titles stand as written and the mapping is documented here, in the COORD and in the catalog. Clean merge. Shares test_collective_family_siblings_610.py with fix/815 (not yet merged): checked at hunk level, sgl-project#801 adds two class-level counters to BudgetHarness (:465) while sgl-project#815 fixes _budget_state_stub (:540/:557) -- different symbols, different drift instances, no collision.

18f's metal-proven crash fix; register sgl-project#824 (see the numbering note on the fix/801 stage merge -- 16f's '[sgl-project#824]'-titled commit is register sgl-project#823). Clean merge, production surface disjoint from the other wave-2 stages (mamba_ckpt_utils / mamba_radix_cache / mamba_component).

…-project#701's commitment ledger `_budget_state_stub` had drifted behind `PrefillAdder.rem_total_tokens` again: sgl-project#701 appended the cross-pass COMMITMENT LEDGER to the tail of that property (schedule_policy.py:883-885) and the stub carried neither name, so both admission cases died with AttributeError: 'PrefillAdder' object has no attribute 'chunked_admission_enabled' This is the sixth drift of this harness after #616g, sgl-project#639, #639b, #791b and sgl-project#794, and the sixth time `TheAdderStubTracksTheBudgetPredicate` named it instead of letting an AttributeError surface inside an unrelated admission assertion. The guard keeps earning its place. BOTH NAMES ARE BOUND, not just the one the failure reported. `chunked_admission_enabled` was the reported miss; `commitment_ledger` was the next one queued behind it. Binding only the reported name is exactly what reshipped this incident at #639b, whose own note says so. THE PAIR EXERCISES THE SHIPPED CHOKEPOINT RATHER THAN STEPPING AROUND IT. `effective_rem_total_tokens` returns its input unchanged when the ledger is None (planner/chunked_admission.py:233-235), so enabling the flag runs the real tail and provably cannot move the budget this harness is about. Setting the flag False would ALSO make the guard green while quietly taking that tail out of the tested path -- the weaker of the two bindings, and the one this file's own history argues against. TESTS (hermetic, CVD="", CPU only) test_collective_family_siblings_610.py 11 passed (was 3 failed / 8 passed). The can-fail is already resident: `test_the_guard_can_fail` plants a field that does not exist and asserts the guard reports exactly it. Battery over the 7 files that carried every failure, under /spinning/htsglang-gpu/.venv (datasets 5.0.0, full collection): base 500be7e 22 failed / 35 passed before this 15 failed / 46 passed after this 12 failed / 49 passed -3, exactly the tests repaired here. This branch still adds none. Found while opening sgl-project#823: the enforcer there has to add slots to the same packed reduce this harness models, so an already-drifted guard would have been the thing standing between that change and a silent index shift. Repaired first, deliberately. No boot was run. This is desk work.

…ith a real reduce STRUCTURE/PRIO, root D of the 0516 wedge. This lands the DECISION and its red-first repro; the scheduler wiring is the next commit and is called out below rather than implied. WHAT EXISTS AND WHY IT IS NOT ENOUGH. prefetch_ballot (#791b) carries a CRC digest of the waiting-queue head on the packed MIN-reduce and voids itself when the group's min and max digest differ. sgl-project#823 already gave that detector onset, persistence and recovery-edge logging (scheduler.py:5061). Its own docstring states the limit: "On mismatch the ballot is void for the pass and the caller falls back to the rank-local verdict -- the status quo ante". Surface, then fall back to rank-local. Nothing makes the ranks agree. The detector is untouched here; the enforcer goes beside it. WHERE THE DIVERGENCE IS BORN. SchedulePolicy.calc_priority (schedule_policy.py:197) orders waiting_queue by req.num_matched_prefix_tokens under a CacheAwarePolicy (_sort_by_longest_prefix, :229-232). That number comes from the RANK-LOCAL radix tree and each TP rank's prefix cache evolves independently -- the #616B family. Same queue, same policy, different ORDER. THE RULE, transplanted from sgl-project#791. sgl-project#791 made PP admission uniform with an asymmetric local/told rule: a locally computed value may only be truncated toward what the anchor said, never used to extend it. The TP sibling: the group's match length is the MIN across ranks and every rank sorts by the GROUP number. MIN is the safe direction for the same reason as #616B's evict floor and the ballot itself -- the agreed length is <= every rank's own, so no rank is ever told to reuse a prefix it does not hold. Worst case a rank recomputes a prefix it had cached: slower, never wrong. Capacities stay rank-local; only the DECISION is uniform, per kein-bindender-rang. THE CIRCULARITY, AND WHY THE SLOTS ARE NOT QUEUE POSITIONS. Per-rid values cannot be reduced by queue position when the positions are what diverge -- slot i is a different request on different ranks and a MIN over that is meaningless. Slots are indexed by a CANONICAL rid order (sorted rid strings), which depends only on the rid SET, the replicated part. A rank that does not hold a rid contributes -1, which MIN-reduces to -1 if ANY rank lacks it, so the group drops that rid rather than admitting a request a peer cannot form: delay, never force, the ballot's own safety property. sorted() is deterministic across processes; hash() is not and must never touch this path. TWO BEHAVIOUR CHANGES, each with its own can-fail arm. The second is the one easy to leave implicit: a digest mismatch must stop falling back to rank-local. The group order is derived from the canonical set and the MIN, so it is still computable in exactly the pass where the digest says the orders disagree -- the mismatch case IS the wedge case, and improving only the agreeing case would leave it untouched. head_decision() carries that branch explicitly and reports which rule ran. PURE ON PURPOSE, per sgl-project#823's own lesson in uniform_floor_scope.py:45: inline behind a real all_reduce the only thing a test can check is whether the source still mentions a branch, and a mutant that disabled the recovery edge once survived a whole suite on exactly that. TESTS (hermetic, CVD="", CPU only, real gloo, no CUDA) test/registered/unit/managers/test_tp_head_congruence_823.py 9 passed. Pure arms plus a REAL gloo MIN all_reduce at world=2 and world=3. The premise is asserted, not assumed: test_todays_local_rule_really_does _diverge shows the rank-local rule giving three different orders on the same queue (rank 0 leads with charlie, rank 1 with alpha). Every rank feeds the enforcer ITS OWN diverged queue order, not a shared fixture list. That is load-bearing: with a shared list the canonical-order step is never exercised and mutant 2 below survives. Mutants killed: enforcer switch ignored -> 1 failed (the can-fail arm) slots indexed by queue order -> 2 failed (canonical independence and the real-gloo uniformity) Regression: 38 passed across test_tp_head_congruence_823, test_prefetch_ballot_divergence_823 and test_pp_prefetch_ballot_791b -- the detector suites are unchanged and still green. NOT YET WIRED. get_new_batch_prefill still calls calc_priority's rank-local sort; this commit adds no slots to _update_uniform_pool_budget's packed reduce. That wiring is next and needs care: the layout comments there record that #639b's appended pair silently moved what the host floor read, and the harness modelling that reduce had drifted six times (repaired in the preceding commit). WINDOW-QUEUE ticket stays preflight_pass N until it is wired and green. Second known gap, recorded so it is not mistaken for covered: scheduler.py :7542/:7547 break the candidate loop on get_num_allocatable_reqs() and req_to_token_pool.available_size(), neither of which rides the sgl-project#610/#616g uniform floor. Equal order with unequal counts still yields unequal batches, so the count needs the same treatment as the order. No boot was run. This is desk work.

W9's second variable, and the same defect in it. The candidate loop stops on a RANK-LOCAL count -- scheduler.py:7542 `get_num_allocatable_reqs` and :7547 `req_to_token_pool.available_size()` -- and neither rides the sgl-project#610/#616g uniform floor that already covers `PrefillAdder`'s token budget. So two ranks can agree perfectly on the ORDER and still build different batches by taking a different number of requests off the same head. That is the other half of the 0516 specimen. The order arm explains "#cached-token 0 vs 16384"; this arm is what puts "#new-seq 1 vs 3" next to it. THE RULE IS THE ONE ALREADY IN THIS FILE, applied to the count: MIN across ranks, so the agreed number is <= every rank's own and no rank is ever asked to seat more than it can. It can only admit FEWER than it would have -- delay, never force, the third time that direction is the safe one here after the order key and the ballot itself. The rank-local capacities remain the INPUTS; only the decision built from them is uniform, which is the kein-bindender-rang line: a binding rank shortens THIS pass, it does not own a permanent share. `batch_decision` now returns the whole decision -- which requests, in which order, how many -- from one place, because two call sites deciding the two halves are two call sites that can drift apart. An unpriced group (no allocator to ask on any rank) rides a sentinel and leaves the local limit untouched, so such a configuration behaves exactly as it does today rather than collapsing to a zero-sized batch. TESTS (hermetic, CVD="", CPU only) test_tp_head_congruence_823.py 15 passed (9 order + 6 count). The premise is asserted, not assumed: with the enforcer off, two ranks with the same order and different free pool admit 3 and 1 -- the specimen's own numbers. `test_both_arms_are_required_for_a_uniform_batch` pins why W9 needs both: order enforced with the count left local STILL yields different batches. Mutants killed on this arm: count uniformisation off -> 1 failed (the can-fail arm) MAX instead of MIN -> 1 failed (uniformity) overshoot the bound (+2) -> "rank 1 was told to admit 3 with room for 1" That third mutant was added after the second one taught me something: MAX can never exceed the local limit (the group MIN is already <= local), so the safety assertion could not catch it and I had not actually proven that assertion was live. It is now. Regression: 55 passed across this suite, both detector suites and the sgl-project#610 collective-family harness. The detectors are untouched. STILL NOT WIRED, so W9 stays preflight_pass N: no slots added to _update_uniform_pool_budget's packed reduce yet, and neither calc_priority nor the candidate loop consumes these decisions. That is the next commit. No boot was run. This is desk work.

W9 wiring, part 1 of 2: the group now COMPUTES the uniform decision every TP-loop iteration. Consuming it in batch formation is part 2. PLACEMENT. The block sits between the mamba pair and the ballot, at the same seam the sgl-project#794 corridor width uses and for the same reason its comment gives: everything above is indexed from the HEAD and the ballot is indexed from the TAIL (`len(vals) - (PREFETCH_BALLOT_SLOTS + 2)`), so an insertion here leaves both readings intact. Both new blocks capture an explicit index BEFORE appending and are read back by it; nothing is read by a negative offset, which is the mistake #639b's note records -- "the `t[-2]`/`t[-1]` the host floor used to read would have silently started reading MAMBA availability". The full payload order is written out in the comment so the next person does not have to reconstruct it. THE PULL-FORWARD, and it is not a new idea in this function. The sort key `num_matched_prefix_tokens` is populated by `calc_priority`, which runs LATER in the pass, so at reduce time it is zero or last pass's value and reducing it would make the group agree on a stale number. #791b already solved this exact shape here for the prefetch verdicts: pull the RANK-LOCAL computation forward to the reduce site -- no collective, once per TP-loop iteration -- and memoise it for the batch formation to consume. `_local_head_prefix_matches` does the same, bounded to the canonical head, against a tree `calc_priority` was about to walk in full anyway. THE COUNT VOTE GETS ITS OWN SLOT rather than being derived from the availability floor, because `get_num_allocatable_reqs` is bounded by `admission_limiter.current` (:6526-6529) -- rank-local floating state the availability reduce does not capture, so a count derived from the uniform avail would still diverge. Neither vote can break the reduce: both are wrapped, and a rank that cannot price contributes the absent/unpriced sentinel, which can only delay admissions or leave the local limit untouched. Never a collapse to bs=0. THE DRIFT GUARD CAUGHT ME, WHICH IS THE POINT OF HAVING REPAIRED IT FIRST. Widening the payload made `TheHarnessTracksTheProductionSurface` fail with BudgetHarness has drifted behind Scheduler._update_uniform_pool_budget: ['_local_admit_limit', '_local_head_prefix_matches'] -- the seventh drift of that harness after #616g, sgl-project#639, #639b, #791b, sgl-project#794 and sgl-project#701, and the FIRST caught in the same change that caused it rather than a quarter later. Then it named `get_num_allocatable_reqs` behind them, the transitive member, exactly the cascade the file warns about. Both votes are BOUND from Scheduler so the harness keeps modelling the real contract, and both ride neutrally there: the harness's waiting_queue is empty, so the head vote is an empty canonical set of absent sentinels. For `get_num_allocatable_reqs` the guard's own message offers a stand-in as the alternative to binding, and a stand-in is right: the shipped method is bounded by the admission limiter and carries the sgl-project#677 parking branch, so binding it would oblige this harness to model a limiter and a phase policy to answer a question it is not asking. Constant and EQUAL on both ranks deliberately -- a divergent count is test_tp_head_congruence_823's subject, and making it diverge here would put a second unrelated variable into the budget cases. TESTS (hermetic, CVD="", CPU only, real gloo where the suite uses it) 55 passed across test_collective_family_siblings_610 (the harness that models this reduce), test_pp_prefetch_ballot_791b (the tail-indexed ballot, i.e. the reading most at risk from a widening), test_prefetch_ballot _divergence_823 and test_tp_head_congruence_823. That the ballot suite still passes IS the evidence that the tail indexing survived the insertion. STILL PART 2, so W9 stays preflight_pass N: `calc_priority` and the candidate loop do not yet consume `_uniform_head_match_lens` / `_uniform_admit_limit`. The decision is computed and published; nothing acts on it yet, so this commit changes no batch. No boot was run. This is desk work.

…art 2) Part 1 (6f8a385) published the uniform decision and nothing acted on it. This makes the ranks actually form the same batch. ORDER. `_apply_uniform_head_order` runs immediately after `calc_priority`, so it overrides exactly the rank-local `_sort_by_longest_prefix` result that diverges (schedule_policy.py:225-232) and nothing else. It REORDERS and never drops: requests the group did not name -- deeper than the head, or not held on every rank -- keep their relative order behind the group head, so such a request waits for a later pass instead of disappearing. COUNT. The candidate loop's stop becomes `_uniform_allocatable_reqs`, the group's MIN, so every rank stops at the same candidate count. Equal order with an unequal stop is still an unequal batch -- it is what put "#new-seq 1 vs 3" in the 0516 specimen beside the "#cached-token 0 vs 16384" the order arm explains. MIN can only stop EARLIER than the local number: admit fewer, never more. Both go through `tp_head_congruence`, whose `batch_decision` keeps the two halves in one place; two call sites deciding them separately are the drift W9 exists to heal. `SGLANG_TP_HEAD_CONGRUENCE=0` is the kill switch and restores the pre-sgl-project#823 rank-local formation exactly. The enforcer also stays off for a TP world of one, which has nothing to agree with. A DEFECT OF MY OWN, caught by the suite rather than reasoned away: `_tp_head_enforcer_enabled` dereferenced `self.ps` directly and raised on harnesses that model a Scheduler without a ParallelState. `self.ps` is now read with getattr like its members are -- absent means no group to agree with, which is the rank-local path. This predicate sits on the admission path, so it may not raise for any reason. TESTS (hermetic, CVD="", CPU only, real gloo where the suite uses it) 55 passed across test_collective_family_siblings_610 (the harness that models the reduce), both ballot suites and test_tp_head_congruence_823. Battery over the 7 files that carried every failure, before and after this commit: 12 failed / 49 passed, UNCHANGED. This commit adds none. The 3 test_scheduler_chunked_req_gate failures are pre-existing and are raised from scheduler.py's VRAM flight-serving mark, not from this path. No boot was run. This is desk work. A full managers run follows with no concurrent tree edits, per the contamination rule this strand already paid for once.

…gl-project#821, sgl-project#823, sgl-project#824, sgl-project#770/sgl-project#812, sgl-project#815; close the section 18 breach Merge-checklist duty for the five wave-2 stages on this branch (fix/801, fix/824, feat/770, feat/822, fix/815). Mechanism text is taken from the commits' own measured evidence, not restated from ticket titles. WHAT WAS ALREADY THERE, checked before writing rather than after. feat/822 catalogued ITSELF mid-train -- 21ff075 carries +49 lines of this file -- so sgl-project#822 already has a complete section 3 entry including the sgl-project#816 clamp firing-rate baseline. No second sgl-project#822 entry was written. sgl-project#801, sgl-project#815, sgl-project#821, sgl-project#823, sgl-project#824 and sgl-project#770 had zero prior mentions; sgl-project#812 was name-checked inside the KV-backing-relief bullet as a consequence of the cap, with no entry of its own. SECTION 12, five new families: Said-once (sgl-project#823) -- a fact that changes during a process, reported once at the moment it first became true, is a stale label afterwards. Both instances live in _update_uniform_pool_budget: the floors' coverage said three times at startup and never again through four cutovers in 55 s, and a divergent TP queue head logged as an event when it is a duration. The mutant that makes the case: disabling the recovery edge left all 14 tests green before the streak was extracted. Blind-instrument (sgl-project#821) -- the scheduler watchdog's activity predicate reads exactly the two values a rank parked in the dict receive freezes, so the instrument best placed to see the wedge is the one structurally unable to. Non-monotone-source (sgl-project#824) -- a monotone guarantee (the protected prefix) derived from a non-monotone tracker (mamba_last_track_seqlen). Landed as a measured assertion, cache_protected_len=16384 against page_aligned_len=8192. Invariant-for-funding (sgl-project#770, sgl-project#812) -- a group-derived per-rank floor asking PP1 for 102.9% of its own cap, and the withdrawal of a floor clamp that bought funding by breaking the levelling invariant (1 failed with, 12 passed without). Stub-drift (sgl-project#815) -- ten helpers drifted behind named commits, and the rule the repair follows: faithfully or not at all. This entry also RECORDS why seven cases in test_pp_flip_slot_hold_631.py are deliberately left red -- the faithful stub is a real rank identity, which drags the whole admission-receive path in with it. That is documentation of a known state, not a claim that it is fixed. Plus a NUMBER-COLLISION note, because the log is ambiguous without it: 9d13bf0 is subject-tagged [sgl-project#824] and names its test file _824.py but is canonically register-sgl-project#823 (floor-scope reporter, strand 16f, arriving on fix/801-admission-recv-liveness); the canonical sgl-project#824 is 658d0c1 on fix/824-chunked-prefill-protected-len. No rebase, so the titles stand and the mapping is written down instead. SECTION 7: sgl-project#801's dead-peer measurement, which REFUTES the premise under the sgl-project#816 survivor analysis -- an exited or SIGKILLed peer makes the admission receive raise after 1.02 s, while alive-never-sends and SIGSTOP both block unbounded, so a rank silent for 119.7 s was not waiting on a corpse. SECTION 2: the funding authority (sgl-project#770/sgl-project#584/sgl-project#819) -- one question, posts declared by name, the three priced losses (L1 the unreachable 2638 MiB, L2 a 3437-row ask against 8192-row granularity returning a silent zero at 65% of the shortfall, L3 claimed=0 against an accounted 107 MiB), and the arming floor solved rather than asserted (512 MiB reserve needs 1523 against a 1229 ceiling = structurally unreachable; 218 is the largest that fits). sgl-project#819's break-even carries its provenance because X and P are env-overridable defaults that are never runtime-measured. SECTION 18 -- THE BREACH, CLOSED. Rule 5 says a PR adding a reusable module adds its entry in the SAME PR. Three modules had landed without one: kv_row_ownership.py (sgl-project#822), funding_authority.py (sgl-project#770), uniform_floor_scope.py (sgl-project#823). prefetch_ballot.py (#791b) had the same gap from an EARLIER merge and is added here labelled as backfilled -- the provenance is stated rather than quietly absorbed, because the checker only tests citations that EXIST, so a missing entry costs nothing and is exactly the failure mode rule 5 cannot catch. Each entry says what is NOT wired, in those words. Both authorities are so far consulted only for the arming floor, the census and the retire/audit pair; the clamp firing-rate metric, format_violations, diagnose_floor_band, uniform_absolute_floor and slack_above_uniform_floor have no production caller at this commit. Recording an unwired export as wired is how a catalog becomes a liability. One trap recorded: funding_authority's docstring at :56 names a solve_funding function that does not exist -- the solve is can_fund at :378. TWO DEFECTS FOUND WHILE WRITING, both fixed here: 1. Section 18 cited `watchdog.py:88`, which resolves to python/sglang/srt/watchdog.py -- a file that does not exist in this tree. The intended target is turnkey/watchdog.py:88 (the retired generation probe), now qualified. This was a live red test, not a cosmetic issue. 2. The sgl-project#822 section 3 entry cited kv_backing_relief.py:504 for exposure_over_backing; :504 is a docstring line and the def is at :547. The entry landed already drifted. Corrected, with the drift noted in place per the header rule that the code wins. TESTS. test/registered/unit/docs/test_building_blocks_catalog_538.py, the mechanical section-18 pointer checker: 91 passed / 1 failed BEFORE (the watchdog.py:88 breakage), 101 passed / 0 failed AFTER. The ten new cases are the ten new citations, and every one resolves. The indicator is proven in both directions without needing an injected mutant: it was red on a real bad pointer of exactly the class it claims to detect, and is green now. codespell on the file: clean.

…eciding who DESK BUILD of TICKET_943_REISSUE_REACHABILITY.md. sgl-project#937 refuses to publish a prefetch whose binding generation went stale across a cutover -- correctly: the sgl-project#943 bisection put the garbage fix at exactly that commit, and every pin that publishes stale spans returns 1/7 coherence. What the refusal leaves behind is a request owed its prefix, and the only correct way to return it is a FRESH fetch under the binding that is current now. THE SHAPE, and every piece of it is chosen to avoid inventing a second version of something that exists: * At the refusal, the req_id is recorded as owed -- a NAME and a COUNT, never the operation, the span or the indices. Keeping any of those is what would tempt the re-stamp that `StaleStampRewrite` (a882e64) already refuses. * `take_agreed_reissue` picks ONE request the whole group agrees on, using the shape `drain_retired_prefetch` already proved: MIN over `[d, -d]` yields the group min and max in one pass, and only `min == max != 0` is agreement. * The candidate set is the INTERSECTION of "owed" and "present in this rank's waiting queue". Voting on "owed" alone could agree on a request some rank cannot act on, and that rank would then sit out the collective its peers entered -- the failure, not a smaller version of it. * The re-issue itself runs through the ordinary `Scheduler._prefetch_kvcache`, so it inherits the existing participation vote, the rank-local eligibility handling and the symmetric-mode branch instead of reproducing them. * The count is REPORTED, never gated. `_MAX_PREFETCH_REISSUES` already carried that rule in its own docstring; a cap would be a rank-local predicate deciding collective participation. I WROTE THE sgl-project#580 FAILURE INTO THE CODE MEANT TO PREVENT IT, and it is recorded at the site rather than quietly corrected. The first draft of the scheduler block read `if self.tree_cache._reissue_pending:` before calling the gate -- a rank-local predicate in front of a collective, so a rank with nothing owed would skip the all_reduce its peers had already entered. The comment on the reap two lines up states the rule verbatim for the same reason. The call is now unconditional and the vote answers 0 for an empty candidate set precisely so it needs no guard. WHY THE LIVE MEASUREMENT WAS NOT ACCEPTED AS THE PROOF. Boot a810ef6 measured the refusal verdict rank-uniform (DIVERGES 0, AGREES 3, over 111 cutovers and 48 refusals). That is one boot on one rig at TP=3. Building a collective on it would make the uniformity load-bearing and checked nowhere, so the sgl-project#580 direction is held by a TEST that INJECTS the divergence the boot never showed. TESTS, hermetic (CUDA_VISIBLE_DEVICES=""), /spinning/htsglang-gpu/.venv. GLOO FALSIFIER, three spawned processes, the real gate and the real `_all_reduce_attn_groups` in each child, both arms bounded by a deadline so a hang is a REPORTED timeout: * split verdict (ranks disagree WHICH request): guarded -> no rank takes, no rank enters the follow-on vote, all three finish. Ungated -> rank 0 takes req-A while ranks 1 and 2 take req-B and all three act, i.e. the ranks re-register DIFFERENT requests. That is the assertion, measured: `distinct taken == {req-A, req-B}`. The first version of this arm was a three-way OR that any outcome satisfied; it was sharpened after checking what the mutant actually did. * lonely verdict (only rank 0 owes): ungated -> one rank enters the collective alone, the literal wedge. Guarded -> nobody enters. ANCHOR SURVIVES A CUTOVER, the coverage the bisection exposed as missing -- no test pinned what happens to a prefix across a cutover at all. Pinned as BOTH halves at once, because pinning only the first is satisfied by the very defect sgl-project#937 removed: the prefix is RECOVERABLE (an agreeing round hands the request back to be re-fetched) and the old span is UNRECOVERABLE (the stale operation cannot be re-stamped; the retained state is a count, asserted). RED-FIRST against the pre-#943b tree: 7 failed, 5 passed; extracted FAILED-name count 7 == summary "7 failed". THE DRIFT GUARD CAUGHT ME, which is the system working. The frozen A/B went 45 -> 47 failures: `test_collective_family_siblings_610` noticed that the drain grew a member (`_prefetch_kvcache`) its harness did not carry. That guard's own comments count the previous times it paid for itself (#616g, sgl-project#639, #639b, #791b, sgl-project#794, sgl-project#701, sgl-project#823); this is the next. Fixed by giving the harness a STAND-IN -- it exercises the budget reduce, not the prefetch path, and with `enable_hicache_storage` False the shipped drain returns {} before reaching the re-issue -- and the stand-in raises if it is ever actually called, so a moved early-return cannot leave these cases silently pinning nothing. FAMILY, FROZEN A/B over the same 66 files, base vs this tree, after the fix: base 45 failed, 1472 passed, 865 skipped (extracted names 45) built 45 failed, 1472 passed, 865 skipped (extracted names 45) diff of the FAILED-name sets: IDENTICAL. New + touched files together: 31 passed. ruff: scheduler 103 findings before and after (none introduced), everything else clean. black clean. NOT PROVEN, AND IT IS THE WHOLE POINT OF THE NEXT WINDOW: that this actually returns the prefix on metal. The acceptance is encoded in devtools/bisect_869b_anchors.sh as a CONJUNCTION -- cached>0 on a repeated prompt AND 7/7 coherence, never one alone -- and verified to FAIL both known states: the pre-sgl-project#937 pin (cached=0, 1/7) and the current tree (cached=0, 7/7). It can only pass in the state no pin in f1a3391..dd0e3bc has reached.

… was in the tree ROOT, per-rank coverage of window-958 boot 2 (pin 78d030e) plus both boot logs. Under pure PP the HiCache storage-prefetch veto is a RANK-LOCAL verdict: prefetch_ballot.prefetch_done_under_ballot returns the local value when the ballot is None, and the ballot rides _update_uniform_pool_budget's reduce on tp_cpu_group -- a group the PP loop never passes, and which has world 1 under pp_size>1 in any case. The ranks therefore decide independently. scheduler.py :9048 (`continue` after _note_skip("prefetch_pending")) is the LAST line PP0 and PP1 both ran; :9050 is the first PP0 ran ALONE. Everything downstream -- add_one_req, `self.chunked_req = adder.new_chunked_req`, the unconditional stash at :7010-7011 -> stash_chunked_request:5449 -> mem_cache/common.py:169 -> cache_unfinished_req -- is PP0-only. PP0's radix tree gains a 1250-token prefix PP1's and PP2's never received. From there it is self-sustaining. PP0 matches its own tree and offers told=1250; PP1 measures local=0 HONESTLY against its own; sgl-project#791 retracts, sgl-project#797 voids, the requeue resets the REQUEST and nothing resets the TREE. PP1 can only acquire the prefix by running the batch it is refused for lacking the prefix. WHY THE EXISTING BOUND COULD NOT REACH IT. PPAdmissionCongruenceGuard is, in its own words, "RID-SCOPED, ONE-SHOT", and its termination argument is per rid: each new retraction for THAT rid lowers THAT rid's floor. The argument is sound and silent about the POPULATION. The shortfall is a property of the TREE, so every fresh rid over the same prefix starts unclamped and buys its own voided pass -- six distinct rids in one second, all told=1250 local=0, which is why _learned_floor was measured RUNNING and LOWERING on PP0 and never bound. FIX: the same actuator, scoped to the prefix the offer was made over. No new mechanism and no new collective -- the recorded fatal (the 2026-08-17 HiCache ack-count reduction) rules out a collective on this path, and a group-uniform INSERT is structurally impossible: the radix value is a tensor of that rank's own KV slot ids, dereferenced as real memory by four consumers (allocator free at unified_radix_cache.py:1713, evict at full_component.py:96, HiCache backup at :2284, write-back into req_to_token at :1325), and the GDN mamba component donates a live state slot a non-computing rank does not have. offered_prefix_key() names the prefix by a blake2b fingerprint of its tokens -- NOT hash(), which is PYTHONHASHSEED-salted and would disagree between the very ranks this keeps congruent (tree_congruence's constraint 3, same lesson). Length is mixed in so a prefix and its extension cannot share a floor. prefix_key=None leaves the pre-sgl-project#963 rid-scoped path byte-identical. PRIOR ART, gated rather than duplicated. #616g's group-MIN reduces available_size(), a different quantity, and is switched OFF in the PP phase (tp_cpu_group world=1). sgl-project#823's tp_head_congruence MINs the right quantity but only REORDERS, and its enforcer_gate returns GATE_OFF_TP_WORLD_OF_ONE here. sgl-project#825's tree_congruence detects exactly this divergence and emitted 0 in BOTH boots: in the PP phase on_round is reached only via scheduler_pp_mixin.py:2477 with require_armed_and_parked=True, so it samples only at an armed flip -- which the livelock prevents. That is this instance's compensator-reachability gap. RED-FIRST, and the first red was rejected as worthless: all six cases failed on the SIGNATURE (TypeError), not on behaviour. Four mutants were then run against the finished fix and their errors READ: consultation withdrawn -> assert 6 == 1 (boot 2's six voided passes, exactly) learn withdrawn -> assert 6 == 1 clear-on-serve withdrawn-> assert 0 == 1250 (permanent loss on a healed prefix) key-blind clamp -> assert 0 == 1250 (loss on a prefix nobody reported) The last two are the danger direction: this fix must never discard a prefix every rank holds. It clamps only against an OBSERVED shortfall on a SPECIFIC prefix, and clears the moment the group serves that prefix, so the cost is one voided pass once rather than one per rid for ever. Double-prefill law: the requeue loses 0 today and still loses 0; what changes is that it terminates. An UNRESOLVED miss (observed_local=None) teaches the prefix floor NOTHING -- the stakes are strictly higher than for the rid scope, since a floor invented from a number nobody measured would cap every request over that prefix. The sgl-project#944 consumer ratchet CAUGHT this file on its first gate run and is now registered. NUMBERS, desk gate scripts/gate_tier2_partitioned.py, CVD="" : BEFORE 2 genuine (test_collective_family_siblings_610.py x2), 685.00 s AFTER 2 genuine (the same two), plus 14 new tests green count probe: 2 named == 2 summary, SUBFAILED and ERRORS included ruff clean on both touched files. NOT CLAIMED: no boot, no metal. The divergence SOURCE (the rank-local prefetch verdict under PP) is named here and left open -- closing it needs the sgl-project#791 ring lap to carry a prefetch-pending fact home, which is its own posten. This makes the system self-healing against a divergence however caused.

…e steht, aber der Umbau loest den Stall NICHT -- zwei Korrekturen an meiner eigenen Skizze Desk-Arbeit, kein Boot. Jeder Slot am Pin relokalisiert. Volltext: /spinning/gpu-arb/DESIGN_968_budget_verdict_to_pp0.md == DIE KLASSIFIKATION IST DREI-, NICHT ZWEIWERTIG == Die Order fragt "Divergenz-Patch vs echtes physisches MIN". Die acht Slots zerfallen tatsaechlich in drei Klassen, und die dritte ist der Grund, warum der Umbau kleiner ausfaellt als gedacht und den Stall nicht behebt: (a) DIVERGENZ-PATCH -- ein VERDIKT, das jeder Rang rechnet und ueber das man sich einigt: head_match (sgl-project#823 W9), admit_limit (:6167), Prefetch-Ballot (#791b). DIESE DREI SIND DIE LOESCHLISTE. (b) PHYSISCHE EINGABE -- eine per-Rang-Tatsache, die PP0 NICHT SELBST WISSEN KANN: local_avail (#616g), admission (sgl-project#610), host (sgl-project#639), mamba (#639b), corridor (sgl-project#794). Muessen weiter reisen, aber als EINGABE in PP0s Verdikt. (c) DIVERGENZ-DETEKTOR -- die `x, -x`-Paare, die aus einem MIN zugleich Max liefern, also Uneinigkeit ERKENNEN. Nach dem Rang-Gesetz ist die einzig legale Reaktion CRASH/STOP; heute speisen sie Kompensation. == KORREKTUR 1 AN MEINER SKIZZE: die Kadenz-Deckel fallen NICHT == §4 Punkt 5 meiner eigenen Skizze behauptete, die sgl-project#1027/sgl-project#1028-Deckel wuerden mit dem Umbau gegenstandslos. FALSCH. Die drei teuren Calls (memory_snapshot, memory_stats, mem_get_info) haengen unter Slot 5, und Slot 5 ist eine PHYSISCHE EINGABE -- PP0 kann PP1s freies VRAM nicht selbst messen. Das Verdikt zu PP0 zu verschieben nimmt den Followern die ENTSCHEIDUNG ab, nicht die MESSUNG. Der Stall liegt damit auf einer ANDEREN ACHSE als die Verdikt-Platzierung: die physische Messung darf nicht auf dem kritischen Pfad zwischen Rundenbeginn und Barriere liegen (Off-Thread-Sampling; der `corridor-trace`-Thread existiert bereits). Beides kombinierbar, aber keines folgt aus dem anderen. == KORREKTUR 2 / TRANSPORT-VERDIKT: der Ring-Lap traegt es NICHT == pp_admission_congruence.py:235 woertlich: eine Entscheidung braucht BIS ZU `pp_size - 1` Runden (hier 2) ueber den Lap. Und die sgl-project#1027-Sicherheitsanalyse UEBERTRAEGT SICH NICHT -- die Asymmetrie ist der Kern: * `trapped` (sgl-project#1027) wird ABGEZOGEN -> ein zu alter, zu grosser Wert VERENGT einen Cut. Sichere Richtung. * `corridor_width` (Slot 5) ist eine OBERGRENZE -> ein zu alter, zu grosser Wert WEITET den gewaehrten Chunk. UNSICHERE Richtung, und exakt der Fall, vor dem der sgl-project#856-F6-Kommentar an spendable_bytes:606-619 warnt. Ein Ein-bis-Zwei-Lap-altes Budget ist also fuer mindestens einen Slot in der unsicheren Richtung. Der Lap traegt das Verdikt in seiner heutigen Phase nicht. XL-Fork, geht zur Meldung statt in einen stillen Umbau. Alternativen benannt: (1) frueherer Lap-Punkt -- aendert die Rundenphasen, Risiko bei den Lockstep-Familien; (2) Piggyback auf bestehendem Kollektiv -- kein neuer Draht, aber PP0s Verdikt raeste auf demselben Reduce, den der Umbau abschaffen soll. Dritte Moeglichkeit (konservative Marge gegen die unsichere Richtung) NICHT empfohlen, aber benannt, damit sie nicht spaeter als neu auftaucht -- sie waere eine vierte Kompensationsschicht. == AUFWAND, PRAEZISIERT == Loeschliste Slots 6-8 zu PP0 M (Verdikte, kein Transportproblem) Slots 1-5 als Eingaben behalten S (Payload schrumpft, Semantik bleibt) Stall-Behebung (Messung vom Pfad) M-L (eigene Achse) Slot 5 unter PP0-Autoritaet XL-Risiko, UNGELOEST EMPFEHLUNG: Loeschliste und Mess-Achse sind unabhaengig und beide OHNE den XL-Fork machbar. Der volle PP0-Umbau von Slot 5 wartet auf die Transport-Entscheidung. BELEG-STUFE: DESK-BEWIESEN (Slots am Pin relokalisiert, Archaeologie aus den Einfuege-Kommentaren, Lap-Latenz aus pp_admission_congruence.py:235). Kein Boot, kein Code geaendert.

… group STOP, not a void Root (HANDOVER_1153_0902, PROVEN, pre-existing at the pin 228a66d): scheduler.py _pp_refuse_forwarded_schedule answered PPScheduleRefused ('sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE', PP1 reached 1 of the 2 rids PP0 named, batch_full_break) with a rank-local compensation: _pp_admission_pass_voided = True + emptied decision dicts -> mixin _pp_void_own_batch -> 'sgl-project#631 ROW-DELIVER BATCH NULLED slot=0 pass_voided=True' (boot_855_weg1b2 log 65000-65004) -> PP1 sent no proxy while PP0's slot stayed set -> nothing carried the void upstream (sgl-project#797 return trip: zero call sites since CUT V; sgl-project#1072 deleted the void relay) -> PP0's blocking _do_recv consumed PP2's NEXT output under this slot's label (log 65119) -> one output ahead for the rest of the boot -> the pp_to_tp arm at 21:42:05 turned the debt into an unproducible output -> sgl-project#980 ObjectRecvStalled 60 s / sgl-project#1071 PpChainRecvStalled 90 s. Same form sgl-project#1071 (169f53c) deleted for _pp_void_retracted_pass; this was the second writer of the same flag. F1 ROOT: the compensation is deleted, the detector is kept, the refusal is a group STOP. scheduler.py get_new_batch_prefill's except now does `raise self._pp_forwarded_schedule_stop(refusal) from refusal`; _pp_refuse_forwarded_schedule (flag write, dict emptying, sgl-project#971 re-home) is removed; the new _pp_forwarded_schedule_stop logs the kept 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank N' line and returns a RuntimeError formatted by pp_admission_congruence .forwarded_schedule_stop_message: 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP rank={rank} slot={slot} told=[{told}] reached=[{reached}] census={census} local={local} limiter={limiter} running_bs={running_bs} parked={parked} r2t_avail={r2t_avail} headroom={headroom} group_limit={group_limit} batch_full_setter={batch_full_setter} batch_full_at_loop_entry={batch_full_at_loop_entry}: {refusal}' Every probe is guarded (n/a on an unreadable value); the reached rids are recorded after the admission loop before any of the three raises; the batch_is_full setter site is recorded at each writer of the pass (no_allocatable_reqs_gate, count_arm, disagg_prefill_r2t_avail, add_one_req_NO_TOKEN). Group stop mechanism (existing, no new collective): the RuntimeError leaves run_event_loop; run_scheduler_process (scheduler.py 'except Exception') logs 'Scheduler hit an exception' and parent_process.send_signal(SIGQUIT) (+ killpg/kill_process_tree under SGLANG_KILLPG_ON_SCHEDULER_EXCEPTION); peers end in the bounded sgl-project#980 / sgl-project#1071 receives or the barlink dead-peer probe. F2 TRIGGER CLASS (PP0 order): on a forwarded schedule the follower's rank-local seat-count veto is not a verdict. pp_admission_congruence .rank_local_count_veto_applies(scheduled_extents) is True on PP0 and on every non-PP boot (unchanged expression) and False on a rank > 0 executing PP0's decision; the sgl-project#823 count arm, its batch_full_break, and the three pre-loop count gates of the same arithmetic (batch_full_or_empty_queue's batch_is_full half, min_free_slots_delay, no_allocatable_reqs) are gated on it. The physical allocator still refuses (NO_TOKEN -> membership refusal -> STOP naming the numbers). F3 SIBLINGS: mixin _pp_void_pass_without_upstream_launch's writer of the flag is provably unreachable (pp_upstream_void_pending returns False on every path; its final statement is `return False`) -- left with the proof in a comment. The row-authority _row_skip_plan exit now nulls a slot through _pp_null_frameless_slot, which applies the same named sgl-project#1020 'VOID REFUSED ON A LAUNCHED SLOT' guard as the void path; the guard is factored into _pp_slot_holds_unconsumed_launch(mb_id, site) and used by both sites. Matched check (error class: a follower still ends a PP0-launched pass silently): grep -rn '_pp_admission_pass_voided = True' python/sglang/srt -> only scheduler_pp_mixin.py (the unreachable sgl-project#801 writer); chain scheduler.py raise PPScheduleRefused (:11633/:11646, :11606) -> except PPScheduleRefused (:9441) -> raise self._pp_forwarded_schedule_stop (:9472) -> RuntimeError (:9623); the flag's only readers are _event_loop_pp_body (:4519 -> _pp_void_own_batch) and the scheduler.py void guard (:8955), neither reachable from the refusal. Tests (hermetic, CUDA_VISIBLE_DEVICES=""): new test_pp_forwarded_refusal_stop_1153.py (T1 STOP form + no void, T2 the count-veto helper + source pin, T3 the sgl-project#1020 guard on the frameless null, + the mixin:9389 unreachability proof): 17 passed after; on the parent ca4c6b7 (git worktree) it is a collection error (new names). test_pp_refused_pass_keeps_continuation_971.py: 12 tests that pinned the old compensation INVERTED with the withdrawal named in each docstring (24 passed after; the inverted 12 are red on the parent: 16 failed / 8 passed / 1 error across both files). Mutants: M1 flag write restored in the STOP builder -> 2 T1 red; M2a helper returns True -> 1 T2 red; M2b count-arm gate dropped in the loop -> 1 T2 red; M3 guard dropped from _pp_null_frameless_slot -> 1 T3 red. Bounded suite (7 unit/managers files + test/registered/scheduler): before 103 failed / 527 passed / 1 skipped / 27 errors; after 103 failed / 544 passed / 1 skipped / 27 errors -- per-file identical except the new file (+17). ruff check clean on all touched files; ruff format --diff hunk count unchanged vs parent (15/14/3, pre-existing) and the two test files formatted. Evidence tier: DESK-PROVEN. Boot-3 acceptance: grep -F 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' in the same second as 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank', and zero 'ROW-DELIVER BATCH NULLED ... pass_voided=True' lines.

…rwarded schedules; STOP line names pp_max_mb and in-loop reached Four non-blocking reviewer items on fa14571 (sgl-project#1153, PASS/PASS), each a rank-disagreement generator or a diagnosability gap on the path Boot 3 exercises (weg1b2 pp_to_tp arm, follower executing PP0's schedule). 1. THE FOURTH SITE OF THE SAME COUNT ARITHMETIC IS GATED. scheduler.py `_get_new_batch_prefill_raw`: the `_maybe_yield_parked_carrier` gate (`get_num_allocatable_reqs(running_bs) <= 0 and chunked_req is None and not enable_priority_preemption`) ran ungated on a follower executing a FORWARDED schedule, and it is an ACTUATOR: it retracts a parked decode carrier (`_retract_decode_and_requeue`) on this rank's own seat count, a rank-local state change the peers do not make (RAENGE-NIE-UNEINS, construction half). Now `_count_veto and ...`, the same `rank_local_count_veto_applies(self._pp_scheduled_extents())` the sgl-project#1153 fix put on the three pre-loop count gates, the sgl-project#823 count arm and its batch_full_break. PP0 and every non-PP boot: unchanged expression. 2. STOP LINE COMPLETENESS. `get_num_allocatable_reqs` is min(pp_max_micro_batch_size, admission_limiter.current) - max(0, running_bs - parked); the STOP line printed local/limiter/running_bs/ parked but not the first min() term (=2 under the flip override on weg1b2, the adversarial reviewer's likeliest trigger). pp_admission_ congruence.FORWARDED_SCHEDULE_STOP_FORMAT gains `pp_max_mb={pp_max_mb}` after `limiter=`; `forwarded_schedule_stop_message` takes the kwarg; the scheduler builder probes `get_server_args().pp_max_micro_batch_size` (guarded, n/a on an unreadable value). The prefix 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' is byte-identical (the Boot-3 acceptance greps it). Final format: 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP rank={rank} slot={slot} told=[{told}] reached=[{reached}] census={census} local={local} limiter={limiter} pp_max_mb={pp_max_mb} running_bs={running_bs} parked={parked} r2t_avail={r2t_avail} headroom={headroom} group_limit={group_limit} batch_full_setter={batch_full_setter} batch_full_at_loop_entry={batch_full_at_loop_entry}: {refusal}' 3. THE LOAD-BEARING MEMBERSHIP LINE IS PINNED (reviewer mutant MC survived). With the count veto off, the post-loop `if missing:` is the only thing turning a follower's physical inability (add_one_req NO_TOKEN -> a told rid not reached) into a STOP. New T5 source pin: the membership block between `scheduled_extents = self._pp_scheduled_ extents()` and the `extra` check contains `if missing:` verbatim, no `_count_veto`, and the raise naming `missing rid(s)=`. Source pin, not a driven pass: nothing in this tree drives the ~700-line loop (same precedent as T2). No source change for this item. 4. reached=[] WART. `_pp_admission_reached_rids` was reset at pass entry and recorded only after the loop. New `Scheduler._pp_record_reached_ rids(can_run_list)`; called at the loop's in-loop `except PPScheduleRefused` (before `schedule_refusal = exc; break`) and at the existing post-loop site, so the STOP line names what the loop actually reached at the moment of a mid-loop raise. Note for the record: on this tree the add_one_req raise was already carried to the post-loop record (break -> alloc_group_end -> record -> raise); the only path that escapes both records is the pre-loop `adder.add_chunked_req` -> `_add_scheduled_req` raise (schedule_policy.py :1628), where can_run_list is genuinely empty, so reached=[] is truthful there. Matched check (error class: a count site left ungated on a follower): grep -n "_count_veto" python/sglang/srt/managers/scheduler.py lists the helper import (:227), the derivation (:10281), the three pre-loop gates (:10350 batch_full_or_empty_queue, :10363 min_free_slots_delay, :10413 no_allocatable_reqs), the parked-carrier yield (:10382), the sgl-project#823 count arm (:11076) and its batch_full_break (:11090). Tests (hermetic, CUDA_VISIBLE_DEVICES=""): test_pp_forwarded_refusal_stop_1153.py: red-first on the unedited source 7 failed / 15 passed (T1 pp_max_mb x3 incl. format string, T1 in-loop reached x2, T2 fourth-site pin, T4 follower-never-yields); after the edits 22 passed. T4 drives the REAL `_get_new_batch_prefill_raw` from its first line to the yield site on a SimpleNamespace stand-in (PP0 with 0 seats: yields once, then declines at no_allocatable_reqs_gate; follower on a non-empty told map: never yields, runs past both gates into a sentinel at policy.calc_priority). Both files: 46 passed (22 + test_pp_refused_pass_keeps_continuation_971.py 24). Mutants (each applied and reverted by string replace, sha1-verified): M1 drop the yield gate -> 2 red (T2 pin, T4 follower); M2 `if missing and _count_veto:` -> 1 red (T5); M3 drop the in-loop record -> 1 red (T1 source pin). ruff check parity with parent (pre-existing E402/E731/F811), no new findings; ruff format --diff hunk count unchanged vs parent (15/3) and the test file formatted (0 hunks). Evidence tier: DESK-PROVEN. Boot-3 acceptance unchanged: grep -F 'sgl-project#791 FORWARDED SCHEDULE UNEXECUTABLE STOP' beside 'sgl-project#791 PP-ADMISSION forwarded schedule REFUSED on rank', zero 'ROW-DELIVER BATCH NULLED ... pass_voided=True' lines; the STOP line now carries pp_max_mb=.

…llot digest mismatch is a group STOP; the cold-build opener honours the cap Boot weg1b3 @ 6980c75 (log /spinning/evidence-665-f1/boot_855_weg1b3_6980c75eac_0902_234752.log): a /health_generate probe reached all three ranks at 23:54:18; PP0 was busy and dropped it in its dispatch loop, PP1/PP2 took the sgl-project#631 row-authority branch of the same loop and enqueued it (queue=6 vs 7, +1 through five seams). The #791b ballot saw the disagreement on every TP pass from 23:56:17 (18 mismatch lines, cadence 1..32, 0 restored) and fell back to the rank-local prefetch verdict, until at 23:59:54 the verdicts split: PP0/PP2 admitted 8f31846b and entered the forward, PP1 declined and sat in the request broadcast. The opener-side cold build window (x40, uncapped) held the hung forward until the 300 s watchdog. F2a ONE DISPOSAL, AT THE ORIGIN, BEFORE RELAY (upstream-minimal). request_receiver.recv_requests: on the request origin only, health-check requests meet the existing idle gate BEFORE the policy hook, the TP broadcast and the PP chain forward (busy -> ipc answered from the origin + dropped; idle -> kept and relayed like any request). The rank-conditional block in scheduler.process_input_requests (upstream idle gate + the sgl-project#631 follower-enqueue special case, 8da4ec5) is deleted: every rank dispatches the identical list and waiting_queue stays replicated. Drop line: 'sgl-project#1158 HEALTH-CHECK dropped at origin before broadcast rid=%s busy queue=%d running=%d' (the gate's own reading). The PP chain forward (_pp_forward_and_process_input_requests) sends the list recv_requests RETURNED, and the DP-attention work/control split runs inside _broadcast_reqs_across_ranks after the disposal, so no second path can deliver a probe the origin dropped. F2b A BALLOT DIGEST MISMATCH IS A STOP (raenge-nie-uneins). prefetch_ballot.unpack_prefetch_ballot raises PrefetchBallotDigestMismatch (RuntimeError) on min != max: '#791b PREFETCH-BALLOT DIGEST MISMATCH STOP rank=%d digest=%d group_min=%d group_max=%d queue_len=%d head=[%s]'. The (x, -x) pair rides a MIN all_reduce, so every rank of the reduce holds the same min and max and raises on the SAME pass. The scheduler's void ballot, the fallback to the rank-local verdict, the streak counters and the geometric cadence line are deleted from _update_uniform_pool_budget; a missing ballot slice in the TP loop is its own '#791b PREFETCH-BALLOT LAYOUT STOP'. None stays only for the single-rank / PP-loop callers of prefetch_done_under_ballot. Group stop = the existing run_scheduler_process except -> SIGQUIT -> kill_process_tree (_update_uniform_pool_budget is called from get_next_batch_to_run with no except in between); no new collective. advance_mismatch_streak stays for the sgl-project#823 head-congruence COUNT/ORDER degradation counters (its one remaining caller, _note_tp_head_degradation) and says so. F2c THE OPENER HONOURS THE CAP TOO. jit_cold_build.capped_cold_build_deadline(base, cap) = min(base * mult, base + cap) is the ONE formula for both readers: barlink_liveness. wait_timeout_s (host seconds) and jit_cold_build.resolve_timeout_cycles (device cycles, cap expressed at the documented 2 GHz nominal clock). cap = barlink_build_window.build_cap_s(), the same SGLANG_BARLINK_BUILD_WINDOW_ CAP_S the peers honour (60 s on this rig); cap 0 = no extension on both. F2d SIBLINGS. flush_cache on rank-local is_fully_idle: judged and left, documented in place -- with the one disposal at the origin the queues are replicated again and the predicate reads the same on every rank. tp_head_congruence ORDER arm: the 'except Exception -> SOURCE_RANK_LOCAL' degradation in _apply_uniform_head_order was one block and is now the named 'sgl-project#823 HEAD-ORDER APPLY STOP' (re-raised from the cause); it propagates through _get_new_batch_prefill_raw on the same path as the sgl-project#1153/sgl-project#1157 stops. _drain_prefetch_progress's replicated-queue premise holds again by construction (no edit). Tests (hermetic, CUDA_VISIBLE_DEVICES="" PYTHONPATH=<tree>/python): NEW test/registered/unit/managers/test_1158_one_disposal_and_stop.py (2-rank fake of recv_requests over a fake broadcast wire; AST proof that is_health_check_generate_req is consulted at exactly one disposal site and pp_rank no longer appears in process_input_requests; digest-mismatch raise with both digests / none on agreement; head-order STOP) and test/registered/unit/distributed/test_1158_opener_honours_cap.py (window open, mult 40, base b: host b+cap, device b+cap cycles, non-binding cap untouched, cap 0 = no extension, one formula). INVERTED with withdrawal docstrings: test_pp_prefetch_ballot_791b.py test_a_divergent_queue_head_voids_the_ballot_loudly -> ..._stops_every_rank; test_prefetch_ballot_divergence_823.py TheLatchIsGoneAndTheRecoveryEdgeExists (5 probes turned around) and TheFallBackBehaviourIsUnchanged (None = no ballot taken). Red-first on the parent 8a7dd38 (tests run from this tree against the parent's code, never copied in): one_disposal 16 failed / 4 passed, opener 4 failed / 1 passed, 791b 1 failed / 8 passed, 823 4 failed / 16 passed; the survivors pin unchanged behaviour (wrong-width None, agreement verdict, ballot-less local verdict, disabled gate, non-binding cap). Fixed tree: 54 passed. Mutants (each restored, all red): keep the probe when busy 2F; skip the disposal at the origin 3F; raise only when group_min == 0 4F; host reader uncapped 2F; device reader uncapped 3F; except degrades to SOURCE_RANK_LOCAL 2F. Bounded suite test/registered/unit/managers (+ the new distributed file on the fixed side only), pytest -q -rf -n 4 --dist loadscope: parent 8a7dd38 (scratch worktree, new files excluded): 246 failed / 4648 passed / 18 skipped / 8 errors (280.69 s) fixed: 246 failed / 4673 passed / 18 skipped / 8 errors (274.84 s) comm over the FAILED ids: 242 unique FAILED ids on each side, 0 only-on-parent, 0 only-on-fixed (-rf lists FAILED only; the 8 errors are collection errors on both sides, not listed by id) Harness stand-in: test_collective_family_siblings_610.py BudgetHarness gains `ps = SimpleNamespace(tp_rank=...)` because the reduce now reads `self.ps.tp_rank` for the STOP line (the file's own drift guard caught it, eighth drift of that harness); 11 passed. ruff 0.15.1 on the touched files: parent 103 errors / fixed 103 errors (parity; the two new files: 0). Evidence tier: DESK-PROVEN. Boot 4 proof lines: 'sgl-project#1158 HEALTH-CHECK dropped at origin' on PP0 only; every 'sgl-project#788 PP-ADMISSION ... queue=' triple equal across ranks after each health probe; grep -c '#969C READMIT-PREFETCH.*rid= HEALTH' = 0; 'Ballot void for this pass' = 0; every 'PHASE-FLIP SEAM RE-ADMISSION' triple identical; every '#1033c CUTOVER FORWARD WARMUP begin' matched by a done or a named abort within base+60 s; 'watchdog timeout' = 0. This commit audits and completes the partial staged work of a previous sgl-project#1158 builder (r2_partial_0903.patch); the drop line now prints the gate's own reading, and the opener test collects on a tree without the fix.

…hichever rank asked first A1, A4, A5 of the WEG-1 review's family A: three consumers where the group branches on a number one rank computed. A2 is deliberately NOT here and A3 is verified-and-recorded, both for reasons stated below. A1 -- THE SEAM-TRANSPORT PREMISE. `phase_purity.seam_transport_premise_holds` decides whether the TP layout may build a transport prefill batch; True builds an extend batch, False yields no batch at all under strict:3 with drain mode, so the two branches carry different collectives. Its whole read set was rank-local: the store witness reads the tree cache's prefetch records, whose reduced half exists only under `UnifiedRadixCache.tp_world_size > 1`, and that field is bound once at cache construction (unified_radix_cache.py:522) and never rebound at the cutover -- while `scheduler.tp_cpu_group` IS rebound to world N (phase_flip_runtime.py:3201-3202). On the shipping form (--tp-size 1 --pp-size 3) the cache's world is 1 in BOTH phases. The divergent input is on this rig's metal at the parent commit, one second before the cutover of boot_855_weg1b9_1116175f6d_0904_164023: log:1900 PP0 `absent=67` against log:1977 / :2067 PP1 and PP2 `assembling=67` -- same 67 stems, same second, two answers about presence. The cut is one AND-slot on the packed MIN `_update_uniform_pool_budget` already takes once per iteration, head-indexed and placed ahead of the tail-indexed ballot so both readings stay intact. No new collective. The read site is `local AND group`, in that order: a published verdict can only ever REFUSE, so a stale or wrong 1 cannot license a rank whose own reading says no, and `None` (PP loop, single-rank group, the kv-session-offload branch that takes no reduce of its own) is byte-identical to the pre-sgl-project#1203 path. The verdict is cleared, not left standing, on both early returns -- sgl-project#823 W9b's rule for `_uniform_head_inputs`. A4 -- `check_prefetch_progress` published `completed_tokens`, the TRANSFER, as this rank's completion; PP0 MINs those into a group floor and CLAMPS the told prefix to it. A rank whose insert declined the fetched tail moved the bytes and retained none, so the floor licensed a prefix no rank holds. Now `insert_result.prefix_len + loaded_from_storage`, the retention. Both sides of the comparison read the same accessor, so the unit stays consistent. A5 -- the writeback, host-headroom and seam-budget defer bounds counted on rank-local counters while the abandon they bound is group-unanimous; three ranks taking turns objecting never spend a budget between them (the 411-abandon decode wedge, through the mechanism that exists to prevent it). All three now read `self._seam_abandons_in_a_row.get(direction, 0)`, booked from the already-reduced fit verdict -- the form the seam-margin term twelve hundred lines up has used since it was written. Read-site change only; the local counters stay as per-rank instruments. FALSE TEXT, corrected in the same edit rather than left for the next reviewer: * `prefill_blocked_here`'s docstring claimed rank-uniformity "by construction" and `get_next_batch_to_run` repeated it. Claimed twice, enforced nowhere. Both now state what A1 makes true (the seam term is a group fact wherever the reduce runs) and what is STILL unenforced (`seam_transport_exempt` and `running_bs` are read locally; the offload branch is a named gap; nothing walks the input set and asserts). * `store_witness`'s "RANK UNIFORMITY of the inputs (review N3)" paragraph rested on the packed MIN that does not run on the shipping form -- retracted with the measurement that falsifies it. * `unified_radix_cache.py` "N1: the reduced value, never the local stamp" -- same falsification, corrected in place. A3 -- VERIFIED, NOT FIXED, and why is recorded at the site. `_admission_held_for_deferred_prefetch`'s `pp_size > 1` follower exemption is still inert in the TP phase (phase_flip_runtime.py:3193 writes pp_size=1 on every rank), so all N ranks withhold on their OWN mark, which is set from this rank's prefetch budget and this rank's monotonic clock. The group machinery that exists -- the prefetch ballot -- covers only the queue head and answers a different question; conflating them would make a wrong predicate uniform, which is exactly the failure recorded for A2. The honest close is a group-uniform defer budget or a second per-rid ballot arm, both larger than this commit. A2 IS NOT ATTEMPTED. Moving the decode-stall SLO stamp out from behind scheduler.py's empty-batch branch fixes the stamp and leaves family C's divergent residency underneath -- a uniform reading of a wrong state. Left to the sgl-project#1202/sgl-project#1203 follow-up. WHAT WAS RED, at parent 4a1da47: test_1203_group_branches_on_one_rank.py -- 11 failed, 3 passed. The 3 that passed are the can-fail floors (the fixture's local premise holds; no group verdict leaves today's answer alone; the seam-margin term still reads the abandon book). Green after the cut: 14 passed. MUTANTS, planted in a scratch copy at /tmp/w1mut, four on the corrupting direction: M1 ignore the group verdict (the pre-cut behaviour) -> test_a_peer_without_a_restore_refuses_for_the_whole_group M2 group verdict alone, local check bypassed -> test_a_group_yes_can_never_override_a_local_no M3 publish a constant yes instead of the reduced slot -> test_one_peer_voting_no_pulls_the_group_to_no M4 publish the transfer again instead of the retention -> test_the_published_completion_is_the_retained_prefix, test_the_transfer_count_is_not_what_is_published M5 writeback bound back on the rank-local counter -> test_the_writeback_bound_reads_the_group_book M6 append the AND-slot AFTER the tail-indexed ballot -> test_a_unanimous_yes_survives_the_reduce, test_one_peer_voting_no_pulls_the_group_to_no None survived. VERIFIED: the reduce is unconditional and pre-branch in `get_next_batch_to_run` (scheduler.py:8617) and precedes the prefill gate in the same pass; all four callers of the premise now take the group wrapper; ruff parity with the parent (103 + 1 pre-existing findings before and after, 0 in the new test file); siblings run once -- 7 failed / 263 passed against the parent's identical 7 failed / 249 passed (test_1189_sweep_a2, a hardcoded line-number census, and two test_tp_decode_formation_861j cases were already failing at the parent). NOT VERIFIED: nothing was run on metal. A4 and A5 are pinned by AST tests, not execution -- the A4 write site sits inside `check_prefetch_progress`, which needs a cache controller, a host pool and a live storage backend, and the A5 sites sit inside `_execute_body`. Their arithmetic is argued, not measured. The co-occurrence rate of A4's PP0-adopts/follower-declines lap is still unmeasured, as the review recorded. test_1157_seam_witness.py's premise-source pin follows the half of the split that still holds the check, and now asserts the delegation too, so it cannot be evaded by dropping it.

Ying1123 changed the title ~~Adjust mem-fraction to avoid OOM~~ Adjust default mem-fraction to avoid OOM Jul 30, 2024

Ying1123 changed the title ~~Adjust default mem-fraction to avoid OOM~~ Adjust default mem fraction to avoid OOM Jul 30, 2024

Ying1123 added 3 commits July 30, 2024 01:43

adjust mem frac

4eb13bd

update flashinfer_useragged

470b2d0

lint

394b308

merrymercy force-pushed the mem-frac branch from e3bd206 to 394b308 Compare July 30, 2024 08:43

Ying1123 merged commit e7487b0 into main Jul 30, 2024

Ying1123 deleted the mem-frac branch July 30, 2024 08:58

Ying1123 mentioned this pull request Jul 30, 2024

[Bug] OOM when benchmarking #810

Closed

3 tasks

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Adjust default mem fraction to avoid OOM (sgl-project#823)

c87c4eb

Navigation Menu

Uh oh!

Adjust default mem fraction to avoid OOM - #823

Adjust default mem fraction to avoid OOM#823
Ying1123 merged 3 commits into
mainfrom
mem-frac

Ying1123 commented Jul 30, 2024

Uh oh!

zhyncs commented Jul 30, 2024 •

edited

Loading

Uh oh!

zhyncs commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Ying1123 commented Jul 30, 2024

Uh oh!

zhyncs commented Jul 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhyncs commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhyncs commented Jul 30, 2024 •

edited

Loading