misc: rm rpyc from PACKAGE_LIST by zhyncs · Pull Request #649 · sgl-project/sglang

zhyncs · 2024-07-18T09:34:44Z

Motivation

as titled cc @merrymercy @Ying1123 @hnyls2002
ref #646

Modification

as titled

Checklist

Ensure pre-commit or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

…container A new release of setuptools-scm (v9+, now `vcs_versioning`) treats git's "dubious ownership" error as fatal during version introspection. The checkout is owned by the runner user but the container runs as root, so git rejects the cross-user repository. Add `git config --global --add safe.directory /sglang-checkout` right after the container launches to fix all 16 "Install dependencies" failures in Nightly Test (AMD) #649. Ref: https://github.com/sgl-project/sglang/actions/runs/23555724959

… decidable, and refute the collective-stream fix The sgl-project#622/sgl-project#649 family is an ordering hazard, not a transport fault. Five production hangs on 2026-08-07 park all three ranks on the identical host line, GPUs at 100 %, never a divergence -- but on three DIFFERENT lines across the specimens (dcp/owner.py:566 twice, the draft kv_indptr .cpu() twice, a host-path abort once). What they share is the shape: a blocking host sync in the out-of-graph metadata-prep phase is ordered behind a barlink collective from the previous step. CUDA streams are in-order, so a collective that stalls for D blocks the host thread for D, and a host thread that is blocked cannot enqueue, cannot service the abort gate, and cannot be the rank that unwedges its peers. A bounded stall becomes an unbounded cluster hang. Enumerating the syncs does not close it, and that is now measured rather than argued. sgl-project#623 removed the .item() at owner.py:548 by threading total_tokens through all five call sites; the 15:55 specimen wedges at owner.py:566, five lines later in the same function, on boolean-mask indexing that has no host-derivable form. The callsite at flashinfer_backend.py:7237 does pass total_tokens, so :548 was correctly skipped and the wedge relocated. That is NOTE_622 section 3's prediction observed in production, before and after on one line pair, the same day. This commit adds the seam that makes the property decidable, and reports what it decides. barlink_stream_policy states the placement of forward work -- stream role plus the cross-stream ordering edges it implies -- as inspectable data instead of a scatter of torch.cuda.stream contexts across the attention backends, the graph runners and the transport. It holds no torch import and allocates nothing, so the decision is testable on a host with no CUDA device. The falsifier builds the ordering graph a production step would produce under a given policy and asks whether any collective is reachable backwards from a prep sync. It is a property test, not a call-site test: a test built around any one of the five specimens would pass while the class stayed open. THE RESULT, WHICH IS NEGATIVE FOR THE PROPOSED FIX Giving the collectives their own stream does NOT satisfy the property. A collective whose result the model consumes must be joined back onto the compute stream, and that join is itself a compute-stream node that the next step's sync waits for. Forking without also isolating the sync side moves the kernel and keeps the hazard. This is asserted explicitly (test_collective_stream_alone_violates_the_property) so the limitation cannot be forgotten and re-proposed. The placement that does satisfy it is ISOLATED_PREP: the prep phase on its own stream, ordered after the host-driven input copies and nothing else. Its soundness obligation -- that prep inputs are host-written -- is encoded rather than documented: a DEVICE_INPUT is joined into prep and breaks the property check, so adding a device-produced prep input fails a test instead of hanging production. Worth noting for whoever wires this up: the fork/join pattern at barlink.py:1028-1067 is the gloo host-staged fallback, which contains ev.synchronize() and a host dist.all_reduce and therefore cannot be captured. The in-graph BAR1 path (barlink.py:1007 -> barlink_all_reduce) has no stream context at all and lands on whatever is current. Today's placement is exactly LEGACY. TESTS test_collective_stream_sync_isolation_622.py: 12 passed, 202 subtests, on CUDA_VISIBLE_DEVICES=99. Can-fail proof by mutation, both executed: - PREP_SYNC moved back onto the compute stream in ISOLATED_PREP -> 107 failed, 11 passed. - the join modelled as free (no node on the joined stream) -> 8 failed, 9 passed, killing both the collective-stream refutation and the consumer-ordering control. Full test/registered/unit/distributed: 2645 passed, 12 skipped, 0 failed. Baseline before this commit was 2633 passed / 12 skipped / 0 failed; the 12 new tests are the entire delta. The 18 pre-existing failures attributed to sgl-project#627 in the briefing did not reproduce on this tree -- the named 615/580/603 files are 82/82 green. ruff and codespell clean. NOT VALIDATED ON A GPU. This branch was produced on a host with no CUDA device. ACTIVE is pinned to LEGACY and the module has zero production importers, so merging changes no runtime behaviour; flipping the default is a separate, GPU-evidenced change.

…raph replay ISOLATED_PREP moves out-of-graph attention prep onto a private stream. That prep WRITES buffers the captured graph READS -- cuda_graph_kv_indices and kv_indptr. Today the write-after-read ordering is guaranteed for free, because prep and replay share one FIFO compute stream: prep for step N cannot begin before replay for step N-1 has retired. Moving prep off that stream is precisely what removes the guarantee. The resulting failure does not crash and does not hang. It writes attention indices while the previous replay is still reading them, producing silently wrong output. A stability test scores such a build as a complete success, because the wedge it was built to remove is genuinely gone. Stability evidence is therefore necessary and not sufficient, and this bracket is the missing half. Uses the existing sgl-project#616 index_race_guard: snapshot() before the replay launch, check_stable() after. Both are enqueued on the current stream, so they are ordered around the graph by construction; per the guard's own contract a same-stream comparison must report zero, and any non-zero count is positive proof that another stream wrote the tensor in between. Counting is device-side, so the instrument adds no host sync and cannot perturb the ordering it is measuring. Expected readings, the middle arm being the one that makes the instrument trustworthy rather than merely reassuring: baseline, no isolated prep -> 0 (no false positives) isolated prep, WAR event omitted -> >0 (proves it can fail) isolated prep, WAR event in place -> 0 (the fix is ordered) An instrument never shown to fire is not evidence. Inert unless SGLANG_INDEX_RACE_GUARD=1, which defaults to False. Buffer resolution degrades to guarding fewer tensors on any missing attribute rather than raising, since a falsifier that can take down serving is worse than no falsifier. Tested (CPU, no GPU held; all three cards were held by another session): - AST parse + py_compile of the modified module: OK - helper returns [] for no backend and for a backend with no attributes - single-backend shape yields cuda_graph_kv_indices + kv_indptr - multi-backend shape additionally yields per-step kv_indptr[i] - zero-numel tensors and non-tensor attributes are skipped - no degradation path raises NOT yet exercised on a GPU: the three-arm table above is unrun, so this commit adds the instrument only and claims no result from it.

…e capture census overwriting itself Two instrument defects, both of which cost a forensic answer today. 1. roundDev was never printed anywhere. Every reading of the abort flag snapshots has had to ASSUME which round the spin was waiting for, because the counter is not in any dump. That assumption is load-bearing: the claim "the aborting spin's exit condition was already satisfied in its own flag region" (ANALYSE_622_replay_abort.md:151-160) is derived from it and has never been measured. It is the same class of error as the retracted per-rank-maximum reading (41d76e7), which this very function's own closing sentence warns against. _abort_flag_snapshot now copies the 8-byte counter through the SAME ctypes memcpy path as the flag region, deliberately not via .item(): a tensor read would enqueue on the compute stream and, on the wedge this runs inside, queue behind the stuck kernel and hang exactly when the evidence is wanted. It stays out of the host-only sibling dump, which takes no device access at all and where the existing device copy already cost 55 s in the 06:12 specimen. With the counter printed, "was the exit condition satisfied?" stops being an inference and becomes a subtraction against the per-topology watermarks. 2. The per-rank capture census overwrote its own evidence. dump_to_file wrote a fixed capture_census_rank<N>.txt. Reading the 16:08 wedge today, the ordered per-segment collective list was the one datum that would have separated "the replay stopped at a segment boundary" from "a transport was frozen behind another" -- and three later boots had each rewritten those files, the last with barlink disabled, so every file read "0 collectives". The question was unanswerable because the instrument clobbered itself, not because it failed. Only counts and digests survived, in the log, and those carry no ordering. The record is now also written boot-scoped, preferring the boot id the launcher already publishes and falling back to the pid. The VRAM flight recorder solved this same problem the same way. The stable un-suffixed name is still written so existing tooling and the log line pointing at it keep working. Tested (CPU only; the cards were held by another session for this work): - py_compile of both modified modules: OK - roundDev read exercised over its real source text in four states: _round_dev None -> "unavailable" _cuda None -> "unavailable" memcpy raises -> "unreadable (RuntimeError)" happy path -> correct little-endian decode of a known value No exception escapes any state, so the flag words are never suppressed by a failure to read the counter -- the flag dump is the primary evidence. - boot-id sanitiser: "../../etc/passwd" -> "etcpasswd" (traversal removed), "a/b;rm -rf" -> "abrm-rf", over-long ids truncated to 40 chars. - the boot-scoped copy is guarded separately, so a failed copy still returns the primary path rather than nothing. NOT yet exercised on a GPU: no abort has been produced against this build, so no claim is made about what the counter will show.

…rst-token divergence root ROOT. Upstream's base sampler skips the cross-rank token-id sync by default (_sync_token_ids_across_tp, opt-in env), resting on "the last all-reduce, the last lm_head matmul, and all sampling kernels" being cross-rank deterministic. On this fork that assumption is violated three ways at once: mixed GPU architectures (5090+3080: near-tie argmax flips), uneven-TP shard geometry (per-rank reduction order -> per-rank logit bits even between same-arch ranks), and per-rank sampling RNG at temperature > 0. The verify accepts (sgl-project#50) and draft picks (sgl-project#185) already have rank-0 broadcasts; the base sampler — which produces exactly the FIRST token of every request — did not. PROOF (crash farm, 2026-08-08, /spinning/622-farm rounds 0523..0548). Six lockstep-sentinel divergence specimens: ranks read DIFFERENT first-token values for the same request, always surfacing at output length 1 with a genuine EOS id (248046, generation_config eos [248046, 248044], vocab 248320) on exactly one side — the only single-token flip that changes batch membership. Membership divergence (bs N vs N-1) then yields divergent graph-tier selection and the group wedges when the tiers' barlink round counts differ — tens of thousands of replays after the injury, which is the sgl-project#622 replay-abort signature. Axes: reproduced under barlink AND NCCL (transport-independent, n=2 each), at temperature 0.7 AND 0.0 (greedy does not suppress), MTTD 60-180 s under amplified load. Falsifier arm: SYNC_TOKEN_IDS_ACROSS_TP=1 on the unfixed tree ran ≥30 min clean under the same load (≥10x baseline MTTD). Explains sgl-project#649's shape (bs-divergent verify -> DCP owner bool-mask sync at owner.py:566 parks before the group collective) and plausibly sgl-project#634 (same wait-for-peer abort at an eager host-path all_reduce); both confirmed only by their absence in the acceptance soak. NOT explained: the sub-op flag-level wedge with chain-identical rings (specimen 20260808T052850Z) — stays open as a separate defect. DESIGN DECISION. Sync is ON by default for every tp>1 group, not only mixed-arch groups: the farm's odd rank was NOT arch-predicted (rank 0 odd twice against an agreeing 5090+3080 pair; an arch split would isolate rank 1), i.e. uneven-TP reduction order breaks the determinism assumption even between same-arch ranks, so an arch-gated default would leave the proven injury reachable. Opt-out: SGLANG_SYNC_SAMPLED_TOKENS=0 (the legacy opt-in MIN-allreduce and the grammar-forced sync are preserved under opt-out). Mechanism: rank-0 broadcast via capture_safe_tp_broadcast — the established hetero pattern (sgl-project#50/sgl-project#185), authoritative sample semantics (rank 0's true distribution, unlike the MIN-allreduce), works under barlink (coordinator dispatch; census/sentinel-visible) and pynccl, capture-safe. COST. One bs-sized int broadcast per base-sampler call (extend/prefill under spec_v2; spec decode rounds use the verify broadcast and are unaffected). Priced via CollectiveClock in the acceptance soak (ms/round compute-vs-wait per rank), reference: pre-fix farm prefill lines. Falsifier-first test: test/registered/unit/distributed/ test_sampler_token_sync_622.py — red on the unfixed tree (default path leaves an injected rank-2 EOS flip divergent / fix function absent), green on the fixed one (all ranks end with rank 0's tokens), over 3 real gloo processes.

… a2a — the mutual-deadlock root ROOT. The mesh and a2a kernels wait for an EQUALITY conjunction over all peers' flag lines — lines that are single per (topology,step,sender) and reused across collectives under a round counter that is GLOBAL across topologies. A rank entering its next same-topology collective overwrites a line a slow peer still awaits; the conjunction (short-circuit, must see ALL peers simultaneously) makes a lead of ONE round sufficient under per-path BAR1 propagation asymmetry. Flags only grow: the awaited equality never returns — permanent group deadlock, surfacing as the replay-abort-clean production class. EVIDENCE (farm 2026-08-08): sentinel FREEZE ring tails (all ranks frozen at one seq, source one op ahead); roundDev 612939 vs peer flag watermarks 612936; mid-wedge py-spy naming BOTH historical callchains — rank 0 at dcp/owner.py:566 (sgl-project#649) and ranks 1+2 at flashinfer_backend.py:7557 (sgl-project#622) — as prep-syncs stuck behind the spinning streams, which is why per-site fixes only relocated the wedge. Wedge MTTF 5-15 min amplified. FIX. Each rank acks its completed round (after its receive phase) into every peer's ack bank; a writer entering its next same-topology collective first waits — monotonic >=, deadline-guarded, distinct abort status 2 for attribution — until all peers acked its previous round. All device-side, capture/replay-safe (watermark and acks are device-resident, re-evaluated per replay). Ack banks appended at the flag-region end so every pre-existing offset stays byte-identical; _round_dev grows to 3 words (round, mesh watermark, a2a watermark). Ring is untouched with a proof comment: its 2(R-1) single-peer waits chain through every rank, so no rank can lead by a collective. Rejected alternatives documented in the strand log: parity flag banks (defeated by the global round counter via interleaved ring rounds) and monotonic flag waits (turn the deadlock into silent data corruption). NEW-EDGE AUDIT (in the kernel comment): the awaited ack is produced by a receive phase that depends only on the PREVIOUS round's flags, never on anything the writer does after it — the new edge closes no cycle; a dead peer hits the deadline and status 2, never a hang. FALSIFIERS, red-first: - test_barlink_ack_protocol_622.py: host-side discrete-event automaton of the flag/round/ack state machine; the OLD protocol deadlocks via reader-visible overshoot in the farm-proven one-round-lead scenario, the NEW protocol completes it; peer death yields named deadline aborts. 4 test functions, 0.05 s, no GPU. - test_barlink_ack_layout_622.py: 16 test functions pinning region arithmetic (banks strictly after all pre-existing lines incl. pipe, non-overlapping, +2*world*256 exactly) and kernel-source invariants (ack fields, >= entry wait, status 2, ring body ack-free). - On-card A/B pending: pre-fix wedge MTTF 5-15 min under amplified load; acceptance = zero FREEZE wedges at >= 8x that, with capture-safety proven via capture census + spaced SIGUSR1 round/ack monotonicity probes under replay load. Battery: 122 test items green across the 8 protocol/instrument files (60 s); full distributed suite 2670 passed / 1 pre-existing unrelated failure (live-process pgrep test); nvcc compile smoke of the CUDA source rc=0 (sm_86); ruff clean.

…-async wedge family — both roots fixed Defect 1 (bcbe31d): base sampler never synced first sampled token ids across TP ranks; under mixed archs / uneven-TP reduction order the ranks read different tokens and batch membership diverges at EOS edges. Sync is now default-on for every tp>1 group (opt-out SGLANG_SYNC_SAMPLED_TOKENS=0). Defect 2 (b42405c, sgl-project#632): barlink bar1 mesh/a2a peer barrier was an equality spin on flag lines reused across collectives with one global round counter — a one-round lead overwrites a line a slow peer still awaits, the conjunction never assembles, mutual deadlock inside graph replay (the production replay-abort-clean wedge). Replaced by a consumption-ack barrier: each rank acks after its receive phase; writers wait monotonically for all peers' previous-round acks before re-entering the same topology. Device-side, capture/replay-safe; ring path untouched (2(R-1) single-peer waits, proof comment in-tree). Test results (documented in /spinning/622-farm/staging/MERGE_BLOCK_FINAL.md): - 22/22 falsifier tests green (sampler sync red-first 2, ack protocol automaton red-first 4, ack layout 16), re-run by the operator on the merge candidate. - 2h26m+ amplified-load soak on f2154ba: zero divergences, zero FREEZE, zero aborts, zero OOB (>=9.7x pre-fix wedge MTTF); load continuity proven per-minute; 9.2M ack-barrier rounds with strict cross-rank watermark growth through graph replay (capture-safety probes). - Cost: sampler sync +2.0% on 2048-token prefill chunks (wait-share); ack barrier within measurement noise; zero added collectives in spec decode.

misc: rm rpyc from PACKAGE_LIST

a66d74c

zhyncs requested review from Ying1123, hnyls2002 and merrymercy July 18, 2024 09:34

wisclmy0611 approved these changes Jul 18, 2024

View reviewed changes

wisclmy0611 merged commit 6a4dc99 into sgl-project:main Jul 18, 2024

zhyncs deleted the misc branch July 18, 2024 09:35

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

misc: rm rpyc from PACKAGE_LIST (sgl-project#649)

d81c6ce

Navigation Menu

Uh oh!

misc: rm rpyc from PACKAGE_LIST - #649

misc: rm rpyc from PACKAGE_LIST#649
wisclmy0611 merged 1 commit into
sgl-project:mainfrom
zhyncs:misc

zhyncs commented Jul 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

zhyncs commented Jul 18, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants