Support disable_ignore_eos in bench_serving.py by Ying1123 · Pull Request #824 · sgl-project/sglang

Ying1123 · 2024-07-30T08:36:10Z

No description provided.

modify 27b and 35b param

…nges, said once ROOT OF THE 0516 WEDGE, named (COORD-strand16f-801-build.md B.10). All three rank-uniformity floors -- evict, host, mamba -- are published from ONE MIN-reduce over `tp_cpu_group` in `_update_uniform_pool_budget`. When that group holds one member the reduce is a no-op and every floor switches OFF. scheduler.py's own comment at that site already says what that costs and calls it "the measured cause of a pipeline deadlock": "With pp_size>1 the ranks that must agree are NOT in this reduce group." And #616g's docstring states the chain that follows from unfloored eviction: the radix trees stop being replicas, `match_prefix` returns a rank-dependent prefix, `prepare_for_extend` derives extend_num_tokens from it, and every per-layer TP all_reduce of that forward is entered with a rank-dependent token count. Specimen /spinning/evidence-816-18f/wedge_0823_055757 (boot 0516) carries the scope line exactly three times, once per rank, all reading `world=1 -> floors OFF ... pp_size=3 tp_size=1`. 19 s later the queue-head digests had parted (#791b); at 05:56:18 the three ranks built prefill batches with #cached-token 0 against 16384 -- rank-dependent prefix match, the exact fingerprint #616g predicts; at 05:57:57 two ranks sat in the spec verify arm and one in the extend arm, all GPUs at 100%. WHY A LATCH WAS THE WRONG SHAPE. Under --enable-phase-flip the scope is not a boot constant: phase_flip_runtime rebuilds the TP group per phase (`want_tp_size = n if tp_phase else 1`), so the floors are ON through TP decode and OFF through PP prefill, switching at every cutover. Boot 0516 completed FOUR cutovers in 55 s and reported its coverage ONCE, at the first PP iteration. "Off for the whole run" and "off for the prefill half of every cutover" are different situations with different fixes and read identically; coverage coming BACK was never reported at all. Now: report on transition, both directions, first observation always. The decision is `uniform_floor_scope.scope_transition`, a pure function, for the sgl-project#823 reason -- inline, a test could only grep this file for a branch, and a grep cannot tell a live branch from `elif False:`. ENFORCES NOTHING, deliberately. The reduce group is not widened and no admission or eviction decision changes. Closing the gap means changing what the group agrees on; that needs metal and is B.10's 18-lane ticket. A MODULE FUNCTION, NOT A Scheduler METHOD, and that was measured. The first cut was a method and broke 61 tests across six files at once: the uniformity-floor suites drive `_update_uniform_pool_budget` bound onto `_FakeScheduler` stand-ins that bind only what they name, so `self._report_...` raised AttributeError on every one. Same binding trap `_pp_wait_for_proxy_readiness`' alias comment documents. A module function reaching the holder via getattr needs no binding; a `getattr(self, ..., None)` guard at the call site was rejected as the silently-skip shape. ALSO FIXES A REGRESSION THIS BRANCH SHIPPED IN sgl-project#823. `BudgetHarness` in test_collective_family_siblings_610.py drifted behind the reduce, which now writes `_prefetch_ballot_mismatch_streak`/`_total`. sgl-project#823's regression scope did not include that file, so the drift shipped. It is the FOURTH time this harness has fallen behind the reduce (after #616g, sgl-project#639, #639b) and the fourth time the file's own drift guard is what named it -- that guard earned its line count again here, catching my change rather than a stranger's. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): 16 passed. RED-FIRST: the module is new, so on unchanged 587e4c2 the file is a collection error rather than per-test reds. Stated as the weak evidence it is; the mutants carry the weight here. Mutants, each killed by the arm that owns it: M1 latch: only ever report the first observation DEAD (3) M2 a singleton group counted as covering DEAD (8) M3 never quiet: report every iteration DEAD (4) M4 coverage coming back swallowed DEAD (4) M5 ON-side call site dropped DEAD (1) -- SURVIVED until an arm was added for the CALL SITES; every other test reads the reporter's own body and never its callers. Same shape as sgl-project#823's M2. Regression: 10 files covering every registered unit test that touches the uniform floors or the ballot -- 54 failed / 98 passed on branch against 54 failed / 62 passed on unchanged base (the +36 are this branch's two new files). comm -13 EMPTY. Zero regressions. black clean; ruff (F401,F821,E731) and codespell clean on the new files; 0 import lines touched in scheduler.py.

Four commits from 16f, each tested and pushed separately: [sgl-project#801] a dead PP peer is not the wedge (the transport already reports it), [sgl-project#821] the watchdog could not see the wedge it was best placed to diagnose, the ballot-divergence observability commit, and the uniformity-floor scope reporter. NUMBERING, recorded because the titles do not match the register: the head commit 9d13bf0 is titled '[sgl-project#824]' but canonically it is the register's sgl-project#823 artefact (the floor-scope reporter). Register sgl-project#824 is 18f's chunked-prefill protected-len crash fix, merged as its own stage below. No rebase -- the commit titles stand as written and the mapping is documented here, in the COORD and in the catalog. Clean merge. Shares test_collective_family_siblings_610.py with fix/815 (not yet merged): checked at hunk level, sgl-project#801 adds two class-level counters to BudgetHarness (:465) while sgl-project#815 fixes _budget_state_stub (:540/:557) -- different symbols, different drift instances, no collision.

18f's metal-proven crash fix; register sgl-project#824 (see the numbering note on the fix/801 stage merge -- 16f's '[sgl-project#824]'-titled commit is register sgl-project#823). Clean merge, production surface disjoint from the other wave-2 stages (mamba_ckpt_utils / mamba_radix_cache / mamba_component).

…he ring with the existing drain W4a made the PP chain receive bounded, named and resumable, but left its automatic abort off (SGLANG_PP_CHAIN_RECV_STALL_S=0), so on metal it could diagnose nothing and recover nothing. This wires it. THE DEFAULT STAYS OFF, AND THAT IS THE POINT. An idle PP rank legitimately blocks in this receive until a request arrives, so there is no duration that separates "idle" from "wedged"; a wall-clock default would SIGQUIT a healthy idle server, the same trap sgl-project#821's marker documents for its own arm. The recovery is therefore armed by EVIDENCE. THE PREDICATE. bump_attempted publishes that a rank has ENTERED a send BEFORE it posts it (phase_flip_counters.py:220-226) -- the only counter whose timing can witness a peer parked INSIDE a send rather than one that has finished. When this rank's CHAN_DICT upstream has entered more dict sends than this rank has taken off that wire, the peer is parked in a send only this rank can drain while this rank is parked in a receive that peer will never feed. That is boot_827's ring stated in counters: PP0 in _pp_commit_admission_send_work on the typed-dict channel, PP1 and PP2 in the request-relay chain receive. abort_check is consulted only at the yield, i.e. only once a receive is already overdue, so a healthy pass never reaches it. THE RING-CUT REUSES sgl-project#757 RATHER THAN INVENTING A CONSUMPTION PATH. pp_flip_drain_leftover_dicts already demultiplexes, stashes a wrong-kind message in _pp_tensor_dict_inbox where its real consumer looks, and discards only a provably void proxy -- which is exactly what makes taking the dict off the wire out of the pass's normal order safe. It runs on every disarm route already. request_receiver catches PpChainRecvStalled, runs one drain turn, and resumes the SAME posted receive, which is what ParkedWait's resumability is for. Servicing that does not clear the stall re-raises rather than spins: at that point the ring is closed for a reason this code does not model, and retrying would turn a diagnosable wedge into an invisible one. Follows sgl-project#789's shape deliberately. _pp_wait_for_dict_readiness argues the false-positive direction is the safe one for the mirror gate, and it is here too: a spurious fire costs one drain turn and a resumed receive, and the receive stays posted and framed throughout, so the late message still arrives intact. Missing a real one costs the boot. sgl-project#789 also declines to invent a new protocol for its case; this declines likewise and cuts the ring on the arc waiting for a message nobody posted. Also: _pp_flip_pass_tick publishes _pp_live_mb_id, which the drain needs to tell an owed proxy from a leftover one. TESTS (hermetic, CVD="", CPU only, no gloo, no scheduler construction) test/registered/unit/managers/test_pp_chain_abort_check_824.py 9 passed; all 9 fail pre-fix. Covers both directions: the boot_827 counter state aborts, and an idle rank is NOT aborted at 0 s or at 3600 s. Mutants killed, one per edge: entered >= taken -> the idle rank is aborted (1 failed) service-turn cap raised -> an unclearable stall stops being re-raised service hook never called -> the ring is never cut (2 failed) Run under /spinning/htsglang-gpu/.venv (torch 2.11.0+cu130, datasets 5.0.0) with PYTHONPATH leading to this worktree, verified by `import sglang; sglang.__file__`. My earlier runs used /usr/bin/python3, which lacks datasets and silently collected only part of the suite; see the corrected note in WINDOW-QUEUE.md. No boot was run. This is desk work.

…gl-project#821, sgl-project#823, sgl-project#824, sgl-project#770/sgl-project#812, sgl-project#815; close the section 18 breach Merge-checklist duty for the five wave-2 stages on this branch (fix/801, fix/824, feat/770, feat/822, fix/815). Mechanism text is taken from the commits' own measured evidence, not restated from ticket titles. WHAT WAS ALREADY THERE, checked before writing rather than after. feat/822 catalogued ITSELF mid-train -- 21ff075 carries +49 lines of this file -- so sgl-project#822 already has a complete section 3 entry including the sgl-project#816 clamp firing-rate baseline. No second sgl-project#822 entry was written. sgl-project#801, sgl-project#815, sgl-project#821, sgl-project#823, sgl-project#824 and sgl-project#770 had zero prior mentions; sgl-project#812 was name-checked inside the KV-backing-relief bullet as a consequence of the cap, with no entry of its own. SECTION 12, five new families: Said-once (sgl-project#823) -- a fact that changes during a process, reported once at the moment it first became true, is a stale label afterwards. Both instances live in _update_uniform_pool_budget: the floors' coverage said three times at startup and never again through four cutovers in 55 s, and a divergent TP queue head logged as an event when it is a duration. The mutant that makes the case: disabling the recovery edge left all 14 tests green before the streak was extracted. Blind-instrument (sgl-project#821) -- the scheduler watchdog's activity predicate reads exactly the two values a rank parked in the dict receive freezes, so the instrument best placed to see the wedge is the one structurally unable to. Non-monotone-source (sgl-project#824) -- a monotone guarantee (the protected prefix) derived from a non-monotone tracker (mamba_last_track_seqlen). Landed as a measured assertion, cache_protected_len=16384 against page_aligned_len=8192. Invariant-for-funding (sgl-project#770, sgl-project#812) -- a group-derived per-rank floor asking PP1 for 102.9% of its own cap, and the withdrawal of a floor clamp that bought funding by breaking the levelling invariant (1 failed with, 12 passed without). Stub-drift (sgl-project#815) -- ten helpers drifted behind named commits, and the rule the repair follows: faithfully or not at all. This entry also RECORDS why seven cases in test_pp_flip_slot_hold_631.py are deliberately left red -- the faithful stub is a real rank identity, which drags the whole admission-receive path in with it. That is documentation of a known state, not a claim that it is fixed. Plus a NUMBER-COLLISION note, because the log is ambiguous without it: 9d13bf0 is subject-tagged [sgl-project#824] and names its test file _824.py but is canonically register-sgl-project#823 (floor-scope reporter, strand 16f, arriving on fix/801-admission-recv-liveness); the canonical sgl-project#824 is 658d0c1 on fix/824-chunked-prefill-protected-len. No rebase, so the titles stand and the mapping is written down instead. SECTION 7: sgl-project#801's dead-peer measurement, which REFUTES the premise under the sgl-project#816 survivor analysis -- an exited or SIGKILLed peer makes the admission receive raise after 1.02 s, while alive-never-sends and SIGSTOP both block unbounded, so a rank silent for 119.7 s was not waiting on a corpse. SECTION 2: the funding authority (sgl-project#770/sgl-project#584/sgl-project#819) -- one question, posts declared by name, the three priced losses (L1 the unreachable 2638 MiB, L2 a 3437-row ask against 8192-row granularity returning a silent zero at 65% of the shortfall, L3 claimed=0 against an accounted 107 MiB), and the arming floor solved rather than asserted (512 MiB reserve needs 1523 against a 1229 ceiling = structurally unreachable; 218 is the largest that fits). sgl-project#819's break-even carries its provenance because X and P are env-overridable defaults that are never runtime-measured. SECTION 18 -- THE BREACH, CLOSED. Rule 5 says a PR adding a reusable module adds its entry in the SAME PR. Three modules had landed without one: kv_row_ownership.py (sgl-project#822), funding_authority.py (sgl-project#770), uniform_floor_scope.py (sgl-project#823). prefetch_ballot.py (#791b) had the same gap from an EARLIER merge and is added here labelled as backfilled -- the provenance is stated rather than quietly absorbed, because the checker only tests citations that EXIST, so a missing entry costs nothing and is exactly the failure mode rule 5 cannot catch. Each entry says what is NOT wired, in those words. Both authorities are so far consulted only for the arming floor, the census and the retire/audit pair; the clamp firing-rate metric, format_violations, diagnose_floor_band, uniform_absolute_floor and slack_above_uniform_floor have no production caller at this commit. Recording an unwired export as wired is how a catalog becomes a liability. One trap recorded: funding_authority's docstring at :56 names a solve_funding function that does not exist -- the solve is can_fund at :378. TWO DEFECTS FOUND WHILE WRITING, both fixed here: 1. Section 18 cited `watchdog.py:88`, which resolves to python/sglang/srt/watchdog.py -- a file that does not exist in this tree. The intended target is turnkey/watchdog.py:88 (the retired generation probe), now qualified. This was a live red test, not a cosmetic issue. 2. The sgl-project#822 section 3 entry cited kv_backing_relief.py:504 for exposure_over_backing; :504 is a docstring line and the def is at :547. The entry landed already drifted. Corrected, with the drift noted in place per the header rule that the code wins. TESTS. test/registered/unit/docs/test_building_blocks_catalog_538.py, the mechanical section-18 pointer checker: 91 passed / 1 failed BEFORE (the watchdog.py:88 breakage), 101 passed / 0 failed AFTER. The ten new cases are the ten new citations, and every one resolves. The indicator is proven in both directions without needing an injected mutant: it was red on a real bad pointer of exactly the class it claims to detect, and is green now. codespell on the file: clean.

Wave 3, stage 3 of the batched-window tree. WINDOW-QUEUE tickets W4 (PP ring wedges immediately after an abandoned flip) and W5 (sgl-project#821 did not instrument the receive path that actually wedged), both preflight_pass=Y. Four commits, base 500be7e -- which is EXACTLY this tree's stage-1 branch, so this stage sits directly on top of W1/W2 with no divergence to reconcile: 2efb933 [sgl-project#824 W4a] Bound the PP chain receive without tearing the stream it guards dd92de4 [sgl-project#824 W5] Name the arm that tripped the watchdog, and instrument the path that wedged 430e468 [sgl-project#824 W4b] A zero-iteration armed window must not move the slot ceb79d6 [sgl-project#824 W4a] Arm the chain-recv recovery on STATE, and cut the ring with the existing drain GATED AGAINST ceb79d6, NOT 430e468. The branch head moved while this stage's first battery was already running, so that run was measuring a superseded commit and was DISCARDED rather than reported -- a gate that certifies a commit the tree does not carry is worse than no gate. The in-flight battery was stopped by explicit PID (no pkill), py-spy dumped first per standing rule and found healthy mid-test rather than wedged, and its partial log kept aside as w3c_stale430.log. A foreign battery belonging to another session was running on the box at the same time and was NOT touched. The final W4a commit arms the recovery on STATE (`bump_attempted`, entered >= taken as the evidence predicate) and cuts the ring through the existing sgl-project#757 drain, with the wall-clock default left at 0. W4b is the ROOT of the pair, and it lands in the same mechanism as this branch's own known-red neighbours: an armed window that ran ZERO slot iterations must not advance mb_id. The boot_827 specimen recorded exactly that -- "rank 0 ran 0 slot iteration(s) (armed at mb_id=0, disarmed at mb_id=1)" -- one line before the ring stopped dead and stayed silent for 31 s until the health check noticed. That is the sgl-project#631 defect class with the spread reduced to a single rank: arm on one slot, leave on another, then re-enter the pipeline wherever you happened to stop. TOUCH SURFACES, computed rather than assumed. This stage shares scheduler_pp_mixin.py with both earlier wave-3 stages and with wave 2, and it extends sgl-project#821's own suite (test_pp_wedge_watchdog_is_honest_821.py) rather than duplicating it -- W5 is the continuation of sgl-project#821, by the same argument sgl-project#821 made, so touching that file is intended and not a collision. It also adds to mem_cache/hicache_collective.py, the file carrying sgl-project#734's `waited < timeout_s * 0.95` discriminator that strand 17c's 622 falsification identified as the fragile part. That last point is what makes this gate non-vacuous rather than merely green: `test_pp_sync_rendezvous_630.py` -- the suite that went red under the 622 merge for precisely this threshold, and the reason 622 is still off the line -- sits INSIDE the battery. If this stage disturbed that discriminator, the gate would say so in the same way it said so for 622. NOTE ON A PATH, because it looks like a contradiction and is not: this stage modifies python/sglang/srt/utils/watchdog.py, while commit 9f1af20 fixed a catalog citation that resolved against python/sglang/srt/watchdog.py. Those are different files; the catalog's intended target was turnkey/watchdog.py:88 (the generation probe retired by user order), which is where it now points. GATE. Battery test/registered/unit/{managers,planner,server_args,mem_cache}, hermetic under CUDA_VISIBLE_DEVICES="", one battery at a time. baseline (tip 4f2072a) 7 failed, 8585 passed, 1852 skipped, 887 s stage 1 (W1+W2) 7 failed, 8638 passed, 1852 skipped, 868 s stage 2 (W3) 7 failed, 8653 passed, 1852 skipped, 922 s this stage 7 failed, 8673 passed, 1852 skipped, 915 s NEW failure ids NONE (comm against the baseline list is empty) fixed vs baseline 0 THE +20 IS ACCOUNTED FOR EXACTLY, not assumed. Four new suites adding only 20 passes is low enough to be worth checking, because a suite that silently skips looks identical to a suite that passes cheaply. Two facts settle it: the skipped count is 1852 in every run of this train -- baseline, stage 1, stage 2 and this one -- so nothing new is being skipped; and running the four suites standalone reports exactly 20 passed, matching the battery's delta to the case. They run, and they are collected. COVERAGE, and this stage is the reason to state it explicitly. The W4/W5 strand reported that its OWN worktree cannot collect the managers suite -- 92 modules import `datasets`, which is not installed there. That limitation is NOT present in this gate, verified in the positive direction rather than inferred from a missing error line: datasets 5.0.0 is importable in this venv, the battery logs carry zero collection errors and zero ModuleNotFoundError, and an explicit --collect-only over test/registered/unit/managers reports 3436 tests collected. So this stage's no-regression claim rests on a fully collected managers suite, which the strand's own local run could not provide. Its four new suites land inside the battery directories and are therefore actually collected: test_pp_ring_abort_recovery.py, test_pp_wedge_watchdog_names_the_arm_824.py, test_pp_zero_iteration_window_slot_824.py and test_pp_chain_abort_check_824.py, all under managers. ON PROVENANCE, if this stage ever shows a new failure id: the branch is PRE-sgl-project#815 (its base 500be7e sits on 21ff075), so the last_batch-631 failures visible in the strand's own worktree are expected to DISAPPEAR on this line, which carries fix/815. A new id here would therefore have to be checked against that provenance before being attributed to the strand's change. In this run none appeared, so the question stays hypothetical. No boot. This tree is the input to ONE batched window.

…h setters WHAT THE FIELD MEANS, stated once because two sites were deriving it and only one of them was right: `cache_protected_len` is HOW MANY LEADING ROWS OF THIS REQUEST'S KV THE TREE OWNS. It is not a length of anything the request owns. Two consumers depend on exactly that reading -- `_insert_helper`'s duplicate free (`dup_start = max(0, prev_prefix_len - total_prefix_length)`) and sgl-project#824's `retention_shrinks_protected` -- and both are unsafe if it under-reports. THE HAZARD, REPRODUCED BEHAVIOURALLY rather than argued. On a prefix HIT `req.prefix_indices` ARE the tree's row ids; the request reuses them, it does not copy them. So in the prefix region `_insert_helper`'s `value_slice` holds the TREE's ids, and `prev_prefix_len` is the only thing standing between them and `token_to_kv_pool_allocator.free`. With `prev_prefix_len=0` and a full prefix hit, every row of the prefix ends up in the free list AND in the tree at once -- counted, ids compared, not read off the source. That set is precisely the `double_owned` population (`free_rows & cached_rows`) the on-idle ledger reports as `src=live`. THE ROOT: the two setters guessed differently, because neither was told. `MatchResult.cache_protected_len` defaults to None and `UnifiedRadixCache` never populates it, so on this rig the field's value depended on which site touched the request last: * `Req.init_next_round_input` (schedule_batch.py:1351-1354) -- has an `else` and falls back to `len(self.prefix_indices)`. CORRECT. * `match_prefix_for_req` (schedule_policy.py:148-149) -- had NO else, so the branch never fired and the field kept its previous value, which is 0 for a fresh Req (schedule_batch.py:1677). Meanwhile the same function assigns `req.prefix_indices = match_result.device_indices` UNCONDITIONALLY. A request could therefore carry the tree's rows while claiming none of them were tree-owned. * `UnifiedRadixCache.cache_unfinished_req` (:1261) -- `len(new_indices)`. CORRECT. The sibling is given the same fallback here, so the two cannot diverge again. Both callers pass `include_req=True` over the WAITING queue, so the value can only be (re)derived for requests that are not yet in flight -- it cannot unprotect anything mid-prefill. AND THE VALUE THE LIVE LOG SHOWS IS RIGHT, WHICH MATTERS MORE THAN THE FIX. 2g-1 reads `cache_protected_len 8192` with a mamba `tracked position 4096`, and sgl-project#824 declines the anchor. That decline is CORRECT and the 8192 is not a symptom: * with `--chunked-prefill-size 4096` and a ~9447-token prompt, `cache_unfinished_req` publishes the protected length at each chunk boundary -- 4096, then 8192, then 9447. At the chunk-2 boundary the tree genuinely owns 8192 leading rows, so 8192 IS the true value at that instant. * `tracked position 4096` is a DIFFERENT AXIS: it is mamba's `cache_len` after the ReplaySSM `write_pos` subtraction, i.e. the last FLUSH boundary of the recurrent state. The mamba state lags the KV by a chunk. * so sgl-project#824 is refusing to file a state captured at 4096 under a key of 8192, which is the sgl-project#767 pairing direction exactly. Refusing is right. THE WARNING THAT FOLLOWS, for whoever owns the anker/decline chain: do NOT "fix" the decline by lowering `cache_protected_len` to meet the tracked position. That would re-open the duplicate-free hazard above AND pair a recurrent state with a depth it was not captured at -- both directions of the same corruption at once. The number is right; the lag is the defect. Tests, hermetic, CUDA_VISIBLE_DEVICES="": test_insert_dup_free_927.py, 3 passed. Combined mem_cache+managers lane: 17 failed / 7473 passed, and all 17 are NAME-IDENTICAL at f1a3391 (arena_high_water_631 x7, restore_never_rebuild_677 x4, phase_flip_rotation_wiring_809 x4, acceptance_emitters_758 RefillTiming x2) -- zero new failures. Genuinely red-first this time, and checked with the mutant shape that defeated the last suite: reverting the `else` turns `test_match_prefix_for_req_states_the_protected_len` RED while the other two stay green, then restored. The hazard case is a CHARACTERISATION (it asserts the tree's rows ARE freed at prev_prefix_len=0) rather than a red-first pin, and is labelled as such. THE RE-ADMIT PATH, TRACED, because the observed crashes all run through it (PP prefill -> retract at the cutover -> re-admission in TP as a full prefix hit, `ADMIT prefix_lens=9447 phase=tp #cached-token: 9447`): * the hazard condition IS created there. `Req.reset_for_retract` (schedule_batch.py:1611) sets `prefix_indices = empty`, `last_node = None` and `cache_protected_len = 0`, and the request is requeued at the front of the waiting queue. A request that then takes a FULL prefix hit is exactly `prev_prefix_len=0` + full hit -- this file's characterisation case. * but it is CLOSED again before the insert. `get_new_batch_prefill` calls `req.init_next_round_input(self.tree_cache)` (scheduler.py:8723) on every admitted request, and that is the sibling that HAS the `len(prefix_indices)` fallback. So the value reaching `_insert_helper` on the observed path was already correct, and the crash is NOT this hazard firing. * SO sgl-project#927 IS NOT CLOSED BY THIS COMMIT. What this closes is the window where `match_prefix_for_req` is the last setter -- real, but not the observed instance. Said plainly so the ticket is not marked done on it. ONE ADJACENT GAP FOUND WHILE TRACING, recorded rather than fixed blind: under `pp_size > 1`, `scheduler.py:8749-8773` truncates `req.prefix_indices` to the PP-agreed `told` (sgl-project#791 admission uniformity) and does NOT update `req.cache_protected_len` with it -- zero mentions of the field in that block. After `init_next_round_input` set them equal, the truncation leaves `cache_protected_len > len(prefix_indices)`. That direction is SAFE for the duplicate free (a larger `dup_start` frees less), which is why it has not shown up as a double-claim; it is the direction that feeds `assert req.cache_protected_len <= len(new_indices) + page_size - 1` (unified_radix_cache.py:1231). Not touched here because the safe direction does not warrant a blind edit on the admission path, and because it wants its own red-first. NOT CLAIMED: that this closes the 2f/2g crash. The live value is already correct via the sibling that had the fallback, so this closes a WINDOW -- the path where `match_prefix_for_req` is the last setter -- not necessarily the observed instance. What it does settle is the reachability question and the meaning of the field, and it removes the disagreement so the next reader is not choosing between two answers.

…ypothese WIDERLEGT, Detektor existiert mit DECKUNGSLUECKE -- der Fix ist S Reiner Lese-Pass, kein Boot, kein Code. Drei Fragen in der vorgegebenen Ordnung, beide Quellen je Zeile. == (1) SCHLEIFENKOERPER: HYPOTHESE WIDERLEGT == `dispatch_event_loop` schickt auf einer Flip-Form JEDEN Rang in `run_phase_flip_event_loops` (:14251-14253, unbedingtes return). Dort wird zwischen `event_loop_normal` (TP-Phase) und `event_loop_pp` (PP-Phase) nach aktivem Stack gewaehlt -- ein echtes Divergenz-RISIKO, aber NICHT das, was im Specimen passierte: alle drei Raenge standen in `event_loop_normal` (PP0 :2681/:2711, PP1+PP2 in recv_requests aus :2668). GLEICHER Schleifenkoerper. Die richtige Beschreibung des Zustands ist damit nicht Divergenz, sondern: PP0 ist eine Iteration VORAUS und arbeitet seinen Durchlauf noch ab, waehrend die Follower schon im Broadcast der naechsten warten. Also wieder die STRAGGLER-Form -- nur an einer anderen Barriere und mit PP0 als Langsamem. SCHRANKE, ausdruecklich: ich habe von PP0 ZWEI Proben, beide in billigem Code an verschiedenen Stellen. Das schliesst "PP0 haengt in einem Aufruf, den ich nicht getroffen habe" nicht aus. Um "die ganze Iteration ist langsam" von "eine ungesehene Stelle haengt" zu trennen, braucht es ein PROFIL (py-spy record ueber Sekunden), keine Dumps. Nicht behauptet, benannt. == (2) DETEKTOR: ER EXISTIERT, UND DIE LUECKE IST PRAEZISE == `on_blocked_recv` (sgl-project#824 W5(b), request_receiver.py:89-95) ist genau der Detektor fuer diese Klasse. Sein eigener Kommentar nennt den Fehlermodus woertlich: "Without it that call is a blocking PP receive that records nothing, SO THE WATCHDOG CANNOT NAME IT -- the same blind spot sgl-project#821 left on the chain_receiver path, which is where two of three ranks wedged on boot_827." Er wird um den DIREKTEN KETTEN-EMPFANG gestampft (`point_to_point_pyobj`, :323-326). Er wird NICHT um den TP-BROADCAST gestampft (`broadcast_pyobj`, :380-404) -- und genau dort standen PP1 und PP2. Das ist keine fehlende Idee und kein Detektor ohne Aktionsarm, sondern eine DECKUNGSLUECKE eines vorhandenen Fixes. Und sie erklaert, warum dieser Wedge nur per py-spy zu finden war: der Watchdog konnte die Stelle nicht benennen, exakt wie der Kommentar es vorhersagt. Im ganzen File: kein timeout, keine deadline auf dem Broadcast-Pfad. Das Warten ist heute unbegrenzt. == (3) PRIOR-ART-GATE: TREFFER, ALSO LUECKE STATT NEUBAU == Dieselbe Stelle hat Wedge-Historie: `kill_window_lines.txt` (2026-08-01) zeigt einen Traceback ueber `recv_requests -> _broadcast_reqs_across_ranks`, und BEFUND-pp3solo-ring-wedge-1013.md haelt die Geschwister-Form fest (geschlossener Ring, alle in SEND). Kein Neubau abgeleitet. == VORGESCHLAGENER SCHNITT, ZWEIGETEILT == (a) DECKUNG, S: `on_blocked_recv` um den Broadcast stampfen, wie es um den point-to-point-Empfang schon geschieht. Wirkung: der Watchdog kann die Stelle benennen; der naechste Wedge dieser Art braucht kein py-spy. Reine Deckungserweiterung eines bestehenden Mechanismus, kein neuer. (b) AKTIONSARM, getrennt: heute ist `on_blocked_recv` ein AUFZEICHNENDER Haken, kein Aktuator. Der Rang-Gesetz-Teil (begrenztes Warten -> CRASH/STOP mit benannter Diagnose beider Seiten) braucht zusaetzlich die "was tut PP0 gerade"-Haelfte, und die ist ohne das Profil aus (1) nicht sauber formulierbar. EMPFEHLUNG: (a) jetzt, (b) nach dem Profil. (a) ist S und macht (b) ueberhaupt erst diagnostizierbar. BELEG-STUFE: DESK-BEWIESEN (beide Quellen je Zeile am Pin gelesen, Kommentar-Zitate woertlich). Die Stall-Beobachtung aus dem Vorlauf bleibt BOOT-BEOBACHTET mit der oben benannten Proben-Schranke.

…ein selbst verursachter Boot-Killer und was er ueber meinen Werkzeug-Check sagt Schnitt (a) wie beauftragt: reine Deckungserweiterung, keine Verhaltens- oder Timeout-Aenderung. == WAS GEBAUT WURDE == `_stamped_broadcast(site, *args)` -- `broadcast_pyobj` unter dem vorhandenen sgl-project#824-W5(b)-Stempel, exakt nach dem Muster der point-to-point-Stampfung (:323-340), inklusive `finally`-Freigabe mit (None, None) aus demselben Grund (ein toter Peer darf keinen stale Zeitstempel hinterlassen). Alle SECHS Broadcast-Aufrufe der Funktion stampfen jetzt, je mit eigenem Site-Namen; null nackte `broadcast_pyobj` bleiben in `_broadcast_reqs_across_ranks`. == ABNAHMEBEWEIS == Boot stamp1028e, health 200, greedy kohaerent, und die Zeile nennt genau die Stelle, an der PP1 und PP2 im Specimen standen: "#1028d request-broadcast stamping ACTIVE ... (first site: request-broadcast/tp<-reqs). Coverage only -- no deadline is armed" Der Konsument ist verdrahtet (scheduler.py:3210 `on_blocked_recv=self._note_pp_chain_blocked`) und behandelt den Site-String opak (er praefixt `chain-recv/`). Der Wedge dieser Klasse braucht ab jetzt kein py-spy mehr, um lokalisiert zu werden. == EHRLICHE GRENZE DIESES SCHNITTS == `_note_pp_chain_blocked`s eigener Docstring sagt: "a healthy idle rank blocks here continuously, so a flag would SIGQUIT an idle server". Der Broadcast IST der normale Idle-Wartepunkt. Die Stampfung macht die Stelle BENENNBAR, unterscheidet aber NICHT Idle von Wedge -- diese Unterscheidung ist Aufgabe des Aktionsarms (Stueck b), der weiter als Meldung geparkt ist. == SELBST VERURSACHTER BOOT-KILLER, und die Lehre ist die wichtigere Haelfte == Der erste Versuch (stamp1028d) toetete den Boot auf allen drei Raengen: TypeError: super(type, obj) ... at request_receiver.py:370 self._bcast_stamp_logged = True Diese Klasse ist `@dataclass(kw_only=True, slots=True, frozen=True)` -- die Deklaration steht in ZEILE 49 DERSELBEN DATEI, die ich in diesem Zug gelesen hatte. Ich habe die Einschraenkung gelesen und dann dagegen geschrieben. Ein-Schuss-Zaehler jetzt auf Modulebene, wo er ohnehin hingehoert. UND DER WERKZEUG-CHECK WAR SCHULD: `py_compile` kann eine Frozen-Dataclass-Zuweisung strukturell nicht sehen. Ein gruener Compile hat mir Sicherheit vorgetaeuscht, die er nicht liefern kann. Ersetzt durch einen Check AUF DIE FEHLERKLASSE -- ein AST-Scan auf `self.<x> = ` in den Methoden dieser Klasse, der jetzt "class frozen+slots: True / self-attribute WRITES: 0" meldet. Das ist die Form, die der naechste Schnitt an dieser Datei braucht. BELEG-STUFE: BOOT-BEWIESEN (Stempel feuert mit der Specimen-Site, Serving gesund). Der Wedge selbst ist damit NICHT behoben -- nur benennbar. NAECHSTER SCHRITT (b): Repro Pack 4->5 mit py-spy record, um "PP0-Iteration insgesamt langsam" von "ungesehene Stelle haengt" zu trennen; die neue Watchdog-Zeile ist dabei die Gegenprobe.

…n-Grundlage: es ist ein LIVELOCK, und der Detektor dafuer EXISTIERT SCHON Leser gebaut wie beauftragt, AST-bewiesen kollektiv-frei, gebootet, Repro gefahren. Er feuerte NULL mal. Der Grund ist der eigentliche Ertrag. == WAS GEBAUT WURDE == `_read_collective_waits` im admission-wedge-watchdog-Thread: liest BEIDE publizierten Zeitstempel (_uniform_reduce_since aus #1028g, _pp_blocked_recv_since aus sgl-project#824/#1028d) und druckt sie in EINER Zeile. Werkzeug-Check fehlerklassen-passend: AST-Beweis, dass weder der Leser noch `format_local_history` ein Kollektiv nimmt (0/0) -- die Bedingung, an der der Census-Vergleicher scheitert, weil er die Gruppe braucht, deren Uneinigkeit er melden soll. Vorher korrigiert, bevor es Schaden anrichtete: ich publizierte `_uniform_reduce_since` mit `perf_counter()`, waehrend der Leser in `monotonic()` rechnet -- VERSCHIEDENE EPOCHEN, das haette Muell ergeben, nicht leichte Ungenauigkeit. Auf monotonic vereinheitlicht, Begruendung im Code. == WARUM ER NICHT FEUERTE: ES IST EIN LIVELOCK == Derselbe Rang, drei py-spy-Proben 4 s auseinander: all_reduce -> all_reduce -> broadcast Er WANDERT zwischen Kollektiven. Ein zweiter Rang stand in einer Probe im all_reduce und in der naechsten in transformers-Config. Die Raenge blockieren also NICHT in einem Kollektiv -- sie durchlaufen laufend welche, ohne dass das System Fortschritt macht (health 503, GPU 0 %). Eine Schwelle auf die Dauer EINER Wartezeit kann das per Konstruktion nicht sehen. #1028g (0 Samples >30s) und #1028h (0 Zeilen) sind beide korrekt still. == SELBSTKORREKTUR ZUR SCHWELLEN-HERLEITUNG == Die 60 s stammten aus "gesundes DE-06 = 35 s vs Strandung 81,6/95,6 s". Das sind FRAGE-Gesamtzeiten, nicht Kollektiv-Wartezeiten -- zwei verschiedene Groessen, die ich verglichen habe, als waeren sie eine. Quantity-Identity- Fehler, dieselbe Falle, die die Uebergabe als Nr. 2 fuehrt. Die Phase-1-Messung (0 Wartezeiten >30 s gesund) betraf die richtige Groesse; die Strandungs-Seite hatte nie einen gemessenen Wert. == UND DER BEFUND, DER DIE ARBEIT NEU RAHMT == Der VORHANDENE Watchdog hat den Stall die ganze Zeit erkannt -- 9 Alarme im selben Log: "ADMISSION-WEDGE: 1 queued, 0 running, and NO first token for 22.6s (>= 20.0s) and no prefill chunk either. Work is admissible and nothing is serving it." Das ist die exakte Diagnose, und die Schwelle (Fortschritts-Uhr, nicht Wartezeit) ist die richtige Groesse fuer einen Livelock. DAMIT WAR MEINE LUECKEN-DIAGNOSE FALSCH: es fehlt KEIN Detektor. Es fehlt der AKTIONSARM (dieser Watchdog ist per Design log-only) und die Beider-Seiten-Diagnose. Ich habe zwei Instrumente gebaut, bevor ich geprueft habe, ob ein vorhandenes bereits alarmiert -- mein Prior-Art-Gate deckte die Census-Frage ab, aber nicht die Frage "schlaegt der Wedge-Watchdog schon an". == WAS DARAUS FOLGT (kein Bau ohne GO) == Der richtige naechste Schnitt ist klein und haengt am VORHANDENEN Alarm: den kollektiv-freien `format_local_history`-Dump an die ADMISSION-WEDGE- Alarmstelle haengen (sie feuert bereits korrekt und kennt den Zustand), statt an eine eigene Schwelle. Der Aktionsarm (CRASH/STOP) waere dann eine Promotion DIESES Alarms, nicht eines neuen. Meine beiden Instrumente bleiben vorerst drin: sie sind still, kollektiv-frei und liefern die gesunde Verteilung. Sie zu behalten oder zu entfernen ist ein Operator-Entscheid -- als Loesch-Kandidaten benannt, weil sie eine Groesse messen, die den Zielfehler nicht traegt. BELEG-STUFE: Livelock BOOT-BEWIESEN (drei Proben eines Rangs, wandernde Kollektive). Vorhandener Detektor BOOT-BEWIESEN (9 Alarme, Zitat oben). Meine Instrumente BOOT-GEBOOTET und still. Serving wieder oben, health 200.

…ot end a pass PP0 launched `_pp_void_retracted_pass` let ONE downstream rank decide that the GROUP's pass ran nowhere. Its own docstring named the return trip that made that safe -- "the void output carries the observed local match home, and PP0's guard learns it as a floor" -- and sgl-project#969 CUT V had already deleted that emitter (`_PP_VOID_OUTPUT_KEY`: zero originating senders at ca0ee3a). The verdict therefore travelled downstream only. PP0's `mbs[slot]` stayed set while the last rank's did not, so no output was ever sent and PP0 blocked in `_do_recv` until the deadman -- the exact invariant `_do_recv`'s own comment relies on ("sender and receiver ask one question of one batch"). Measured, twice, and the second is on the stall second itself: 1068cap 07:34:02-09 sgl-project#797 void on rank 1 ONLY (no void/retract line on PP0 or PP2); PP0 parked, PP2 spinning. 1069cohort 08:00:54/55 sgl-project#791 unhonourable on PP1, told=12493 local=8397 then told=13399 local=12493 -- `local` exactly one pass behind `told`. After 08:00:55 only ranks 1 and 2 emit at all (slot_occupant / output_fill / width_agreement run to the takedown at 08:13:32, 4177 and 7973 lines; rank 0 emits nothing). That last measurement also refutes the occupant-sleep node as the halting member: ranks 1 and 2 are alive and turning; the `sgl-project#1000 SLOT-OCCUPANT reasons={'no-statement'}` spin is an INERT probe whose carrier sgl-project#1015 EDIT-F made permanently None. The only halting member is PP0's unbounded output receive. Repairing the return trip would repair a compensation layer for a rank-local verdict, which is the arc the sgl-project#968 order forbids continuing; under upstream-minimal the repair carries the burden of proof and the deletion does not. So the verdict is deleted and the disagreement is DETECTED instead (RAENGE-NIE-UNEINS: a detected divergence stops the group, never a compensating wait): * `_pp_assert_told_honourable` replaces it -- an unhonourable told names rank, slot, rid, told and local and raises. No clamp: clamping to this rank's own local match is rank-local geometry, i.e. sgl-project#631. * The chain-receive throttle arm gets a horizon (SGLANG_PP_OCCUPANT_HORIZON_S, default 90 s, 0 disables) and a named stop. Taking the arm is legitimate and frequent; outliving it never is. * `PpChainReceiver.recv` is bounded the way its sibling `consume_up_to` already was (runaway guard + reported counter), and the launcher now sets SGLANG_PP_CHAIN_RECV_STALL_S=60 -- the sgl-project#824 mechanism has existed since 2026-08-24 and shipped disabled by default, which is why it never fired in either stall. (A-i) is WITHDRAWN rather than built: its counter proof is blind at that site (the rendezvous bumps `sent` only on recv entry) and the DEFER one-shot is itself compensation for the rank-local verdict this commit deletes, so its trigger is removed at the source. Its two red-first tests are replaced by zombie tests for the deletion. Desk evidence: hermetic import + AST (deleted verdict absent, watchman and horizon present, recv bounded); test_968_deletion_falsifiers 24 passed, 1 failed -- test_C_the_void_relay_is_wired_or_deleted_but_never_half_built, which stays red until the relay SYMBOLS are swept out too. The relay is already unreachable at runtime (no rank originates a void any more, and `_pp_absorb_void_output` has no caller in production), so that sweep is a dead-code deletion scheduled beside this commit, not a runtime dependency of it. Belegstufe: DESK-BEWIESEN. Boot pending.

support ignore_eos

b12f470

zhyncs approved these changes Jul 30, 2024

View reviewed changes

Ying1123 merged commit ae5c0fc into main Jul 30, 2024

Ying1123 deleted the ignore_eos branch July 30, 2024 08:42

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Support disable_ignore_eos in bench_serving.py (sgl-project#824)

bab8514

JinyanYi pushed a commit to JinyanYi/sglang that referenced this pull request Jun 23, 2026

Merge pull request sgl-project#824 from Sugar920/testcases

a27d896

modify 27b and 35b param

Navigation Menu

Uh oh!

Support disable_ignore_eos in bench_serving.py - #824

Support disable_ignore_eos in bench_serving.py#824
Ying1123 merged 1 commit into
mainfrom
ignore_eos

Ying1123 commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Ying1123 commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants