Simplify mem state by Ying1123 · Pull Request #622 · sgl-project/sglang

Ying1123 · 2024-07-15T08:24:46Z

No description provided.

… attribution — barlink_abort_gate gains a rank-local replay tag (kind, key, ordinal, seq) written host-side at replay launch on all four graph runners (full, breakable x2 per segment, multi-layer-eagle per rung); Bar1CollectiveAborted now prints REPLAY WINDOW and dumps the capture census in the abort path (first-dying rank never reaches the scheduler-tick dump — closes that half of sgl-project#619, expiry path still open); no device read, no sync, no .item(), key stored by reference with identity assert; 16 hermetic tests + 5 single-line can-fail reverts + 14/14 real-GPU replay smoke on the 5090; suite deltas exactly +16 vs base, 20 pre-existing fails unchanged; family root deliberately NOT claimed — 03:23 wedge dump shows all ranks on the identical owner.py:548 host stack, censuses byte-identical, NOTE_622 records hypotheses and confounds)

…ort dump + evidence chain (H1 refuted)

… decidable, and refute the collective-stream fix The sgl-project#622/sgl-project#649 family is an ordering hazard, not a transport fault. Five production hangs on 2026-08-07 park all three ranks on the identical host line, GPUs at 100 %, never a divergence -- but on three DIFFERENT lines across the specimens (dcp/owner.py:566 twice, the draft kv_indptr .cpu() twice, a host-path abort once). What they share is the shape: a blocking host sync in the out-of-graph metadata-prep phase is ordered behind a barlink collective from the previous step. CUDA streams are in-order, so a collective that stalls for D blocks the host thread for D, and a host thread that is blocked cannot enqueue, cannot service the abort gate, and cannot be the rank that unwedges its peers. A bounded stall becomes an unbounded cluster hang. Enumerating the syncs does not close it, and that is now measured rather than argued. sgl-project#623 removed the .item() at owner.py:548 by threading total_tokens through all five call sites; the 15:55 specimen wedges at owner.py:566, five lines later in the same function, on boolean-mask indexing that has no host-derivable form. The callsite at flashinfer_backend.py:7237 does pass total_tokens, so :548 was correctly skipped and the wedge relocated. That is NOTE_622 section 3's prediction observed in production, before and after on one line pair, the same day. This commit adds the seam that makes the property decidable, and reports what it decides. barlink_stream_policy states the placement of forward work -- stream role plus the cross-stream ordering edges it implies -- as inspectable data instead of a scatter of torch.cuda.stream contexts across the attention backends, the graph runners and the transport. It holds no torch import and allocates nothing, so the decision is testable on a host with no CUDA device. The falsifier builds the ordering graph a production step would produce under a given policy and asks whether any collective is reachable backwards from a prep sync. It is a property test, not a call-site test: a test built around any one of the five specimens would pass while the class stayed open. THE RESULT, WHICH IS NEGATIVE FOR THE PROPOSED FIX Giving the collectives their own stream does NOT satisfy the property. A collective whose result the model consumes must be joined back onto the compute stream, and that join is itself a compute-stream node that the next step's sync waits for. Forking without also isolating the sync side moves the kernel and keeps the hazard. This is asserted explicitly (test_collective_stream_alone_violates_the_property) so the limitation cannot be forgotten and re-proposed. The placement that does satisfy it is ISOLATED_PREP: the prep phase on its own stream, ordered after the host-driven input copies and nothing else. Its soundness obligation -- that prep inputs are host-written -- is encoded rather than documented: a DEVICE_INPUT is joined into prep and breaks the property check, so adding a device-produced prep input fails a test instead of hanging production. Worth noting for whoever wires this up: the fork/join pattern at barlink.py:1028-1067 is the gloo host-staged fallback, which contains ev.synchronize() and a host dist.all_reduce and therefore cannot be captured. The in-graph BAR1 path (barlink.py:1007 -> barlink_all_reduce) has no stream context at all and lands on whatever is current. Today's placement is exactly LEGACY. TESTS test_collective_stream_sync_isolation_622.py: 12 passed, 202 subtests, on CUDA_VISIBLE_DEVICES=99. Can-fail proof by mutation, both executed: - PREP_SYNC moved back onto the compute stream in ISOLATED_PREP -> 107 failed, 11 passed. - the join modelled as free (no node on the joined stream) -> 8 failed, 9 passed, killing both the collective-stream refutation and the consumer-ordering control. Full test/registered/unit/distributed: 2645 passed, 12 skipped, 0 failed. Baseline before this commit was 2633 passed / 12 skipped / 0 failed; the 12 new tests are the entire delta. The 18 pre-existing failures attributed to sgl-project#627 in the briefing did not reproduce on this tree -- the named 615/580/603 files are 82/82 green. ruff and codespell clean. NOT VALIDATED ON A GPU. This branch was produced on a host with no CUDA device. ACTIVE is pinned to LEGACY and the module has zero production importers, so merging changes no runtime behaviour; flipping the default is a separate, GPU-evidenced change.

…raph replay ISOLATED_PREP moves out-of-graph attention prep onto a private stream. That prep WRITES buffers the captured graph READS -- cuda_graph_kv_indices and kv_indptr. Today the write-after-read ordering is guaranteed for free, because prep and replay share one FIFO compute stream: prep for step N cannot begin before replay for step N-1 has retired. Moving prep off that stream is precisely what removes the guarantee. The resulting failure does not crash and does not hang. It writes attention indices while the previous replay is still reading them, producing silently wrong output. A stability test scores such a build as a complete success, because the wedge it was built to remove is genuinely gone. Stability evidence is therefore necessary and not sufficient, and this bracket is the missing half. Uses the existing sgl-project#616 index_race_guard: snapshot() before the replay launch, check_stable() after. Both are enqueued on the current stream, so they are ordered around the graph by construction; per the guard's own contract a same-stream comparison must report zero, and any non-zero count is positive proof that another stream wrote the tensor in between. Counting is device-side, so the instrument adds no host sync and cannot perturb the ordering it is measuring. Expected readings, the middle arm being the one that makes the instrument trustworthy rather than merely reassuring: baseline, no isolated prep -> 0 (no false positives) isolated prep, WAR event omitted -> >0 (proves it can fail) isolated prep, WAR event in place -> 0 (the fix is ordered) An instrument never shown to fire is not evidence. Inert unless SGLANG_INDEX_RACE_GUARD=1, which defaults to False. Buffer resolution degrades to guarding fewer tensors on any missing attribute rather than raising, since a falsifier that can take down serving is worse than no falsifier. Tested (CPU, no GPU held; all three cards were held by another session): - AST parse + py_compile of the modified module: OK - helper returns [] for no backend and for a backend with no attributes - single-backend shape yields cuda_graph_kv_indices + kv_indptr - multi-backend shape additionally yields per-step kv_indptr[i] - zero-numel tensors and non-tensor attributes are skipped - no degradation path raises NOT yet exercised on a GPU: the three-arm table above is unrun, so this commit adds the instrument only and claims no result from it.

…e capture census overwriting itself Two instrument defects, both of which cost a forensic answer today. 1. roundDev was never printed anywhere. Every reading of the abort flag snapshots has had to ASSUME which round the spin was waiting for, because the counter is not in any dump. That assumption is load-bearing: the claim "the aborting spin's exit condition was already satisfied in its own flag region" (ANALYSE_622_replay_abort.md:151-160) is derived from it and has never been measured. It is the same class of error as the retracted per-rank-maximum reading (41d76e7), which this very function's own closing sentence warns against. _abort_flag_snapshot now copies the 8-byte counter through the SAME ctypes memcpy path as the flag region, deliberately not via .item(): a tensor read would enqueue on the compute stream and, on the wedge this runs inside, queue behind the stuck kernel and hang exactly when the evidence is wanted. It stays out of the host-only sibling dump, which takes no device access at all and where the existing device copy already cost 55 s in the 06:12 specimen. With the counter printed, "was the exit condition satisfied?" stops being an inference and becomes a subtraction against the per-topology watermarks. 2. The per-rank capture census overwrote its own evidence. dump_to_file wrote a fixed capture_census_rank<N>.txt. Reading the 16:08 wedge today, the ordered per-segment collective list was the one datum that would have separated "the replay stopped at a segment boundary" from "a transport was frozen behind another" -- and three later boots had each rewritten those files, the last with barlink disabled, so every file read "0 collectives". The question was unanswerable because the instrument clobbered itself, not because it failed. Only counts and digests survived, in the log, and those carry no ordering. The record is now also written boot-scoped, preferring the boot id the launcher already publishes and falling back to the pid. The VRAM flight recorder solved this same problem the same way. The stable un-suffixed name is still written so existing tooling and the log line pointing at it keep working. Tested (CPU only; the cards were held by another session for this work): - py_compile of both modified modules: OK - roundDev read exercised over its real source text in four states: _round_dev None -> "unavailable" _cuda None -> "unavailable" memcpy raises -> "unreadable (RuntimeError)" happy path -> correct little-endian decode of a known value No exception escapes any state, so the flag words are never suppressed by a failure to read the counter -- the flag dump is the primary evidence. - boot-id sanitiser: "../../etc/passwd" -> "etcpasswd" (traversal removed), "a/b;rm -rf" -> "abrm-rf", over-long ids truncated to 40 chars. - the boot-scoped copy is guarded separately, so a failed copy still returns the primary path rather than nothing. NOT yet exercised on a GPU: no abort has been produced against this build, so no claim is made about what the counter will show.

…ult injector First-divergence attribution for the rank-async wedge family. The census compares counts and has proven counts byte-identical in wedged specimens; this records the per-rank event stream POSITIONALLY (host collectives at the census bump, so coverage is census-identical and transport-agnostic; graph replays with their full selection key at note_replay, the only point that knows the per-step selection) and cross-compares rolling FNV chains at the group-min position over a dedicated gloo sidecar group with bounded timeout. First mismatch is named as rank/seq/expected/got, ring dumps written per rank. Payload sizes excluded from the position hash (uneven TP shard bytes differ legitimately). Env-gated (SGLANG_LOCKSTEP_SENTINEL), off by default, zero-cost disarmed. V1 scope: single-node pure TP (world == tp), PP=1. Can-fail proof: SGLANG_SENTINEL_FAULT=rank:seq:mode injector; test drives RED first over 3 real gloo processes: an injected skip and dup are both named with the right rank inside the stream's repeat-period ambiguity window; identical streams stay green; all ranks reach the same verdict. Tests: test/registered/unit/distributed/test_lockstep_sentinel_622.py 4 passed in 23.53s (venv pytest, PYTHONPATH=worktree, CUDA hidden).

…ntinel The sentinel's first live specimen (farm round 20260808T051733Z, ~90 s of amplified load) named the family's injury: at ring seq 9902 rank 0 selected the full decode graph with ShapeKey size=5 while ranks 1/2 selected size=6, and STAYED there — one request left rank 0's running batch that the peers kept, with the entire preceding host-op stream still in lockstep (which is exactly why count-based census stayed byte-identical in every wedged specimen). The abort then lands whenever the differing tiers imply differing barlink round counts, tens of thousands of replays later. Batch membership changes only through filter_batch (drops) and merge_batch (admissions), and under pure TP both MUST be rank-uniform — so record them in the position chain with rid and finish-reason class. The next divergence names the diverging DECISION and its reason (finish / abort / retract / admit) instead of its replay-selection symptom. No-op unless the sentinel is armed (module-level None check).

…ated FINISH-TRACE Two divergence specimens narrowed the family injury to divergent FINISH_MATCHED_TOKEN decisions (specimen 1: rank 0 drops early; specimen 2: ranks 1+2 drop and rank 0 misses the match — rank 0 odd both times, refuting a 5090-vs-3080 arch split). The on-device accept path is rank-0 broadcast (sgl-project#50/#616c fused), so the divergent input is CPU-side or temperature-dependent; the greedy A/B arm is running. - filter_batch drop tags now carry the matched token/str and output_ids length: the diverging VALUE lands in the sentinel dump itself. - SGLANG_FINISH_TRACE=1 logs one line per finish transition (rid, reason, matched, accept window len, output tail) so the three ranks' lines for the same rid diff to show WHICH finish input diverged.

…very rank mid-wedge Specimen 20260808T052850Z reproduced the production replay-abort class WITH the sentinel armed: all three ranks abort at replay sgl-project#22382 while the rings are chain-IDENTICAL and positions sit one event apart ([236584, 236583, 236583]) — no selection divergence, no membership divergence. The desync is BELOW op granularity (flags/rounds inside the transport). On a stall the sidecars therefore now exchange and dump every rank's ring tail: notes are written before launch, so each rank's last entry names the op it entered and is stuck in. Sidecar threads keep running while the main threads hang, so the exchange works mid-wedge.

…eadlock wedge shape Specimen 20260808T060219Z (13 min sync-ON load, zero divergences, then Bar1CollectiveAborted at replay sgl-project#33494 on all ranks): every ring froze at the same seq. The leader-ahead stall check is structurally blind to this — there is no leader in a mutual deadlock. The sidecar now dumps every rank's ring tail after 8 s of global freeze (gated on prior traffic so an idle server stays silent). Count-level abort histories cannot separate the ranks here (both end in identical broadcast runs); the positional tails name the exact op each rank entered last. 4/4 sentinel tests still green (21.8 s).

…rst-token divergence root ROOT. Upstream's base sampler skips the cross-rank token-id sync by default (_sync_token_ids_across_tp, opt-in env), resting on "the last all-reduce, the last lm_head matmul, and all sampling kernels" being cross-rank deterministic. On this fork that assumption is violated three ways at once: mixed GPU architectures (5090+3080: near-tie argmax flips), uneven-TP shard geometry (per-rank reduction order -> per-rank logit bits even between same-arch ranks), and per-rank sampling RNG at temperature > 0. The verify accepts (sgl-project#50) and draft picks (sgl-project#185) already have rank-0 broadcasts; the base sampler — which produces exactly the FIRST token of every request — did not. PROOF (crash farm, 2026-08-08, /spinning/622-farm rounds 0523..0548). Six lockstep-sentinel divergence specimens: ranks read DIFFERENT first-token values for the same request, always surfacing at output length 1 with a genuine EOS id (248046, generation_config eos [248046, 248044], vocab 248320) on exactly one side — the only single-token flip that changes batch membership. Membership divergence (bs N vs N-1) then yields divergent graph-tier selection and the group wedges when the tiers' barlink round counts differ — tens of thousands of replays after the injury, which is the sgl-project#622 replay-abort signature. Axes: reproduced under barlink AND NCCL (transport-independent, n=2 each), at temperature 0.7 AND 0.0 (greedy does not suppress), MTTD 60-180 s under amplified load. Falsifier arm: SYNC_TOKEN_IDS_ACROSS_TP=1 on the unfixed tree ran ≥30 min clean under the same load (≥10x baseline MTTD). Explains sgl-project#649's shape (bs-divergent verify -> DCP owner bool-mask sync at owner.py:566 parks before the group collective) and plausibly sgl-project#634 (same wait-for-peer abort at an eager host-path all_reduce); both confirmed only by their absence in the acceptance soak. NOT explained: the sub-op flag-level wedge with chain-identical rings (specimen 20260808T052850Z) — stays open as a separate defect. DESIGN DECISION. Sync is ON by default for every tp>1 group, not only mixed-arch groups: the farm's odd rank was NOT arch-predicted (rank 0 odd twice against an agreeing 5090+3080 pair; an arch split would isolate rank 1), i.e. uneven-TP reduction order breaks the determinism assumption even between same-arch ranks, so an arch-gated default would leave the proven injury reachable. Opt-out: SGLANG_SYNC_SAMPLED_TOKENS=0 (the legacy opt-in MIN-allreduce and the grammar-forced sync are preserved under opt-out). Mechanism: rank-0 broadcast via capture_safe_tp_broadcast — the established hetero pattern (sgl-project#50/sgl-project#185), authoritative sample semantics (rank 0's true distribution, unlike the MIN-allreduce), works under barlink (coordinator dispatch; census/sentinel-visible) and pynccl, capture-safe. COST. One bs-sized int broadcast per base-sampler call (extend/prefill under spec_v2; spec decode rounds use the verify broadcast and are unaffected). Priced via CollectiveClock in the acceptance soak (ms/round compute-vs-wait per rank), reference: pre-fix farm prefill lines. Falsifier-first test: test/registered/unit/distributed/ test_sampler_token_sync_622.py — red on the unfixed tree (default path leaves an injected rank-2 EOS flip divergent / fix function absent), green on the fixed one (all ranks end with rank 0's tokens), over 3 real gloo processes.

… a2a — the mutual-deadlock root ROOT. The mesh and a2a kernels wait for an EQUALITY conjunction over all peers' flag lines — lines that are single per (topology,step,sender) and reused across collectives under a round counter that is GLOBAL across topologies. A rank entering its next same-topology collective overwrites a line a slow peer still awaits; the conjunction (short-circuit, must see ALL peers simultaneously) makes a lead of ONE round sufficient under per-path BAR1 propagation asymmetry. Flags only grow: the awaited equality never returns — permanent group deadlock, surfacing as the replay-abort-clean production class. EVIDENCE (farm 2026-08-08): sentinel FREEZE ring tails (all ranks frozen at one seq, source one op ahead); roundDev 612939 vs peer flag watermarks 612936; mid-wedge py-spy naming BOTH historical callchains — rank 0 at dcp/owner.py:566 (sgl-project#649) and ranks 1+2 at flashinfer_backend.py:7557 (sgl-project#622) — as prep-syncs stuck behind the spinning streams, which is why per-site fixes only relocated the wedge. Wedge MTTF 5-15 min amplified. FIX. Each rank acks its completed round (after its receive phase) into every peer's ack bank; a writer entering its next same-topology collective first waits — monotonic >=, deadline-guarded, distinct abort status 2 for attribution — until all peers acked its previous round. All device-side, capture/replay-safe (watermark and acks are device-resident, re-evaluated per replay). Ack banks appended at the flag-region end so every pre-existing offset stays byte-identical; _round_dev grows to 3 words (round, mesh watermark, a2a watermark). Ring is untouched with a proof comment: its 2(R-1) single-peer waits chain through every rank, so no rank can lead by a collective. Rejected alternatives documented in the strand log: parity flag banks (defeated by the global round counter via interleaved ring rounds) and monotonic flag waits (turn the deadlock into silent data corruption). NEW-EDGE AUDIT (in the kernel comment): the awaited ack is produced by a receive phase that depends only on the PREVIOUS round's flags, never on anything the writer does after it — the new edge closes no cycle; a dead peer hits the deadline and status 2, never a hang. FALSIFIERS, red-first: - test_barlink_ack_protocol_622.py: host-side discrete-event automaton of the flag/round/ack state machine; the OLD protocol deadlocks via reader-visible overshoot in the farm-proven one-round-lead scenario, the NEW protocol completes it; peer death yields named deadline aborts. 4 test functions, 0.05 s, no GPU. - test_barlink_ack_layout_622.py: 16 test functions pinning region arithmetic (banks strictly after all pre-existing lines incl. pipe, non-overlapping, +2*world*256 exactly) and kernel-source invariants (ack fields, >= entry wait, status 2, ring body ack-free). - On-card A/B pending: pre-fix wedge MTTF 5-15 min under amplified load; acceptance = zero FREEZE wedges at >= 8x that, with capture-safety proven via capture census + spaced SIGUSR1 round/ack monotonicity probes under replay load. Battery: 122 test items green across the 8 protocol/instrument files (60 s); full distributed suite 2670 passed / 1 pre-existing unrelated failure (live-process pgrep test); nvcc compile smoke of the CUDA source rc=0 (sm_86); ruff clean.

… probe The SIGUSR1 launch dump must never touch device memory (its documented no-sync constraint), so it could not print the ack barrier's round/ watermark words — and the on-card monotonicity probe had nothing to read. The watchdog abort poll (sgl-project#517) already stages device words through a private stream every 10 ms; it now also stages (round, mesh watermark, a2a watermark) into a pinned mirror, and the dump prints that mirror. Two spaced SIGUSR1 probes under replay load showing strict growth of all three words per rank are the ack barrier's capture-safety proof through the real replay path. 46 test items green (launch-diff + abort suites).

…-async wedge family — both roots fixed Defect 1 (bcbe31d): base sampler never synced first sampled token ids across TP ranks; under mixed archs / uneven-TP reduction order the ranks read different tokens and batch membership diverges at EOS edges. Sync is now default-on for every tp>1 group (opt-out SGLANG_SYNC_SAMPLED_TOKENS=0). Defect 2 (b42405c, sgl-project#632): barlink bar1 mesh/a2a peer barrier was an equality spin on flag lines reused across collectives with one global round counter — a one-round lead overwrites a line a slow peer still awaits, the conjunction never assembles, mutual deadlock inside graph replay (the production replay-abort-clean wedge). Replaced by a consumption-ack barrier: each rank acks after its receive phase; writers wait monotonically for all peers' previous-round acks before re-entering the same topology. Device-side, capture/replay-safe; ring path untouched (2(R-1) single-peer waits, proof comment in-tree). Test results (documented in /spinning/622-farm/staging/MERGE_BLOCK_FINAL.md): - 22/22 falsifier tests green (sampler sync red-first 2, ack protocol automaton red-first 4, ack layout 16), re-run by the operator on the merge candidate. - 2h26m+ amplified-load soak on f2154ba: zero divergences, zero FREEZE, zero aborts, zero OOB (>=9.7x pre-fix wedge MTTF); load continuity proven per-minute; 9.2M ack-barrier rounds with strict cross-rank watermark growth through graph replay (capture-safety probes). - Cost: sampler sync +2.0% on 2048-token prefill chunks (wait-share); ack barrier within measurement noise; zero added collectives in spec decode.

…t stubs The abort tests build BarlinkBar1Transport via __new__ and hand-set the attributes under test; that list silently diverges from the real __init__ every time the transport gains state (the historically recorded 7-fail drift of the deferred-517 file; sgl-project#622's ack barrier repeated the mechanism and was hand-patched again). At audit time __init__ assigns 103 attributes, the stub sets ~30. Fix shape: not a bigger stub — an exact two-way audit. Every __init__ attribute must be stub-set or listed in a reviewed exclusion table with a reason (bring-up plumbing, byte-proof/sensor state, dispatch thresholds, result-relay slots, log-dedupe flags, the None-guarded poll mirror); a new attribute turns the audit RED naming it, a removed one turns its stale exclusion RED, and an attribute both set and excluded is rejected. Red-first record: the first run with the example-only table failed naming all 59 uncovered attributes; the table was populated from that output. Battery: 42 test items green across drift-audit + deferred-517 + abort-431; ruff clean.

…me the wedged rank THE GAP. The census heartbeat is a scheduler-thread collective and dies with the hung rank — structurally blind in exactly the case it exists for — and the abort dump spoke only rank-local state: a survivor's Bar1CollectiveAborted could not say where the wedged peer was. THE FIX. The lockstep sentinel's sidecar keeps exchanging while a main thread hangs (proven mid-wedge on-card during the sgl-project#622 hunt). Its last successful gather (every peer's ring position) and its last anatomy exchange (every peer's tail = last op entered) are now RETAINED, and peer_statement() formats them: rank, ring seq, age of the observation, last op, plus a SYNC LOST clause when a peer's sidecar has since died — the listed positions are then its last life sign. The barlink abort raise appends the statement as 'PEER POSITIONS (sgl-project#650)', never-raise guarded. FALSIFIER, red-first (executed record in the test docstring): on the pre-fix tree, module-level peer_statement absent, instance method absent, raise text without PEER POSITIONS — all three checked False by direct import of the pre-fix module. Green: synthetic hang over 3 real gloo processes (one rank stops noting at seq 500, its sidecar keeps participating — the real wedge shape); both survivors' statements name 'rank 2 at ring seq 500' and its last op from the retained tails; disarmed path safe; source invariant pins the raise wiring. Counting basis: 3 new test functions; 7 test items green with the sentinel regression suite (34.8 s); ruff clean; barlink import clean.

…icrobatch iteration Measured wedge (first serving flip boot, 2026-08-08 11:06Z, log preserved as server.log.attempt9-WEDGE-kvpressure-consensus): right after the warmup prefill, PP0+PP1 sat in the flip runtime's bounded MIN-reduction while PP2 sat in recv-from-PP1; barlink liveness caught the no-progress after 120 s on every rank and the tree self-terminated cleanly (the sgl-project#622 liveness machinery working as designed). Root: the rank-local-state-feeds-collective family in PP form. The hook ran inside get_next_batch_to_run -- the TOP of the pp iteration, before this rank's sends are issued -- so a rank could enter the blocking world-reduction still owing the send its successor needs to reach ITS reduction. Under event_loop_normal (lockstep TP rounds) the placement is safe and stays; under event_loop_pp the hook is now deferred (_defer_flip_round_to_pp_loop) and runs at the END of each microbatch iteration, after every send of the iteration is flushed -- the reduction becomes the last blocking op of the iteration and no recv/reduction cycle can close. PhaseFlipLoopExit now raises from that quiescent boundary. Extracted _phase_flip_on_round() serves both call sites; the pp loop resets the defer flag on any exit so the post-flip TP loop runs the hook inline again. Falsifier-first: TestPpLoopConsensusOrdering drives the REAL on_round consensus through a bounded barrier channel in the measured composition (last stage recvs mid-iteration before its hook, middle stage's send trails its hook): top placement deadlocks (negative control, broken barrier), end placement completes. Family suite 222 passed. Also books the flip-build VRAM transient (TP originals + arena coexist at arena allocation; measured deficits 4.4/1.7 GiB at the old budgets) into the 3.4a ledger as its own term.

…own copy of the ship env route_a_631_prod_boot.sh maintained a private, hand-written copy of the environment and had drifted from the capture in seven keys: SGLANG_UNEVEN_TOKEN_VECTOR 28,26,20 against the ship process's 14,10,8; SGLANG_CORRIDOR_FLOOR_MIB, SGLANG_CORRIDOR_REBALANCE, SGLANG_KV_BACKING_RELIEF, SGLANG_SEAM_ENTRY_DELAY_BUDGET and SGLANG_SEAM_ENTRY_MARGIN_MIB dropped; PYTORCH_CUDA_ALLOC_CONF added. The instance that env booted came up, answered /model_info with 200 and never answered /generate. Patching seven keys leaves the class intact, so the private copy is what goes: * deploy/turnkey/ship_env.capture is the captured ship env, in the repo and under review rather than in an evidence directory. * scripts/turnkey_539_export_env.py renders a capture as sourceable shell (shlex-quoted, so a value with a space, a quote or a $ survives -- the replay path in val-r4/restore_ship.sh:17 exports unquoted and works only by luck), and --check compares an environment against the capture and names every unsanctioned divergence as CHANGED / MISSING / EXTRA. * the boot script sources that render and runs --check immediately before exec, refusing on any divergence. The only way past it is override_env / set_tunable / drop_env: named per key, printed to stderr with a reason, and passed to the gate as --allow <that key>. There is no blanket bypass. Two genuine overrides remain, both now visible: LD_LIBRARY_PATH (venv cu13 before system cuda-12.2) and PYTORCH_CUDA_ALLOC_CONF (the corridor knob). * DRY_RUN=1 assembles env and argv, gates them, prints both and launches nothing, so the assembled environment is testable on the desk. * the sanctioned per-boot keys are defined once, in the tool; turnkey_539_parity_proof.py imports them instead of keeping a second list. Argv is untouched: the DRY_RUN token list was compared against the pre-change script token for token, 60 tokens identical, on the default arm and on SPEC=off, HICACHE=1, KV_LADDER, POLICY=auto, RANK_MIB/CTX and a trailing passthrough argument containing a space. Tests: test/registered/unit/turnkey/ 116 passed (56 before this branch). Red observed first for every one: the two new files failed to collect (FileNotFoundError on the tool), and the boot-script file failed 10 of 13, including "'SGLANG_A_STRAY_KNOB' not found in 'REFUSE: a serving instance for port 30030 is already running.'". bash -n clean on both shell scripts; ruff --select=F401,F821 clean. Pre-existing on the base commit and not touched here: test_no_bare_pytest_main (test_prefill_graph_barlink.py:272) and test_module_state_ratchet (layers/moe/utils.py).

The order was "retire the 1-token generation liveness prober; keep crash detection, passive only". Retiring the obvious thing would have retired the wrong thing, so the finding comes first: **`/health` is not a passive endpoint on this stack.** `http_server.py` binds BOTH `/health` and `/health_generate` to the same `health_generate` handler, which submits a real `max_new_tokens: 1` generation, gated by `SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION` -- which DEFAULTS TO TRUE. Every plain `curl /health` poller on this box, including this shift's own progress marker, was therefore a generation prober. That variable is now set to 0 in the serving boot environment, and the marker reads liveness from process presence, boot-log mtime and NVML instead. This commit covers the other half, the turnkey watchdog's own periodic probe: * `generation_probe_enabled` is added to `Policy` and `WatchdogSpec` defaulting to **False**, so omission cannot reintroduce the probe -- the failure mode `[watchdog].enabled` already demonstrated, being parsed and never read; * the state machine short-circuits so a reachable lane is HEALTHY on passive evidence alone; * `_cmd_watch` now honours the previously-dead `enabled` flag. WHAT THIS GIVES UP, stated rather than left to be rediscovered: with generation retired, a sgl-project#622 wedge (HTTP 200, no tokens) is not detectable by ANY periodic check, because detecting it requires generating. It is covered by the one-shot real generation at teardown/restore and by boot-log age. The blind spot is written into the code comment, the tests and the report. Tests: 24 -> 28, none deleted. The former falsifier `test_http_200_alone_never_reaches_healthy` is RENAMED to `..._WHEN_PROBING` and keeps asserting exactly what it did, scoped to the probing configuration; every other wedge test opts in with `generation_probe_enabled=True` so the probe machinery keeps its coverage. Four new tests pin the shipped default: the default is retired, passive evidence alone is healthy, no tick ever schedules a generation, and -- the half that must not be given up -- crash detection still fires. 120 passed in the turnkey arm.

…flip, and sgl-project#722/sgl-project#634 stay closed I reproduced a RETRACTED verdict. My previous commit (1b6b989) rooted the barlink-BAR1 family in a teardown-ordering race in Bar1Transport.close(). That is task sgl-project#722, which was already filed with the same mechanism -- OPERATOR-STATE :9053-9057 records it almost word for word ("poll_status_word liest stale _ctl_dev-Mapping ... Guard prueft is-None = IDENTITAET statt VALIDITAET") -- and then RETRACTED on live evidence: the abort-poll is REPORTER not perpetrator, the real root was sgl-project#717's backing mismatch, and the revert was proven over 216+33 flips with 0 faults. WHY I MISSED IT: I gated the prior-art search on the repository. Both sgl-project#722 and sgl-project#634 live in OPERATOR-STATE.md, outside it. The subagent's "NOT FOUND in the repo" was correct and the repo was the wrong place to look. The gate is only as wide as the corpus you point it at, and I pointed it at one. THE REAL ROOT, from S2's own log: 48743 19:30:31 PHASE-FLIP armed: pp_to_tp 48749 19:30:31 FLIP EXTENT PROBE: seqlen=51311 kv_allocated_len=51310 48740 19:30:31 CENSUS at-arm: cached=127182 cur_slot_reqs=4, backing=309464, highest live row=183998 48931 19:30:40 KV-BACKING EVICTED 127731 recomputable rows to bring the high-water mark below 61303 (resident ceiling -1) 48935 19:30:40 backing 61303 instead of 116736, highest live row 0 48955 19:30:4x CUDA illegal memory access Twenty-four lines from eviction to fault. _nothing_resident() returns True when _last_live_split["req_rows"] == 0. During a pp_to_tp flip the requests are PARKED for the consensus round, so req_rows reads 0 while 127182 rows are still cached and the flip's own probe has just measured a request at seqlen=51311. The rung takes the nothing-resident branch, evicts 127731 rows, shrinks backing 309464 -> 61303 -- and the flip then reads the rows it was about to move (phase_flip_runtime:6926 _execute -> _pack_outgoing -> kv_reshard.py:359 _checksum). Those rows are above the new cap. THE SAFETY NET CANNOT COVER THIS BY CONSTRUCTION. Site A's comment says the optimistic pricing "is only safe BECAUSE" _shrink_to re-reads the live set afterwards. It re-reads the live REQUEST set, and the flip's pending pack is not a live request -- it is parked, which is the very state that made req_rows zero. Check and hazard share one blind spot. This is b786858's revert text ("backing shrank to 69,054 rows under a highest live row of 233,289, and the next access above the cap was an illegal address") recurring through the REBUILD 675793c, on a path the rebuild did not consider: not a live request above the cap, but the phase flip. LINEAGE, by git merge-base --is-ancestor against serving commit 9255258: c4e5579 (attempt), b786858 (revert) AND 675793c (rebuild) are all IN. So S2 is post-revert AND post-rebuild -- and that distinction is the answer. Post-revert alone would suggest a barlink reopen; post-rebuild says the reverted defect returned through its own replacement. S3 (2026-08-05) predates sgl-project#622 (08-08) and sgl-project#717 (08-17) entirely: old world, not a trigger. VERDICT: sgl-project#722 and sgl-project#634 do NOT reopen. The trigger is a new post-revert specimen with a BARLINK perpetrator; S2's perpetrator is the KV rung and the abort-poll is again the reporter -- so this note is a second independent confirmation of sgl-project#722's retraction, not a challenge to it. sgl-project#741 reclassifies as a sgl-project#717-rebuild x phase-flip regression: live, rooted, unowned. The barlink stand-down/re-arm change stays UNBUILT per the standing order; my section 5 is superseded. Fix shape for the real root (not built, three options with a preference): make the flip's pending extent visible to the split so req_rows counts parked-for-flip rows (preferred, smallest, closest to the defect); or gate the rung off while a flip is armed (blunt, costs the idle-box win, cannot be wrong); or extend _shrink_to's re-read to the flip's pending set so the net covers what its comment claims. Sections 1-4 of the note are KEPT, marked retracted, as the record of what was claimed and why it is wrong.

wisclmy0611 added 2 commits July 15, 2024 01:18

simplify memory pool

98cc898

simplify constants

253208e

Ying1123 requested review from hnyls2002 and merrymercy July 15, 2024 08:24

Ying1123 closed this Jul 15, 2024

Ying1123 deleted the simplify-mem-state branch July 15, 2024 08:25

wisclmy0611 restored the simplify-mem-state branch July 15, 2024 08:25

TobyMint added a commit to TobyMint/sglang that referenced this pull request Jul 6, 2026

Fix DSV4 decode radix cache: fill_ids refactor + missing env var (sgl…

0fdce66

…-project#622)

efschu added a commit to efschu/htsglang that referenced this pull request Aug 7, 2026

Merge fix/replay-abort-root-622: sgl-project#622 sibling-transport ab…

6d2f4b0

…ort dump + evidence chain (H1 refuted)

Navigation Menu

Uh oh!

Simplify mem state - #622

Simplify mem state#622
Ying1123 wants to merge 2 commits into
mainfrom
simplify-mem-state

Ying1123 commented Jul 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Ying1123 commented Jul 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants