Fix rid state map leak + Refractor .finished by Qubitium · Pull Request #505 · sgl-project/sglang

Qubitium · 2024-06-05T06:58:00Z

HIGH IMPACT: @ZhouXingg Fixed a bug/memory leak of rid state map when stream=False. Rid state map cleanup code after yield is not executed since the receiver of the generator early returns and never calls it again. Qubitium@3407263
MEDIUM IMPACT: The rest of the changes deal with merging all structures/properties related to req.finished from a simple bool into a BaseFinishReason object.
LOW IMPACT: added type hints for some zmq event receivers so ide/coders can actually see real type even if runtime doesn't use them.

Reason for .finished refractor:

Consolidate the following into a single finished_reason state/property: output_hit_stop_str, hit_stop_str, finish_reason, finished. They are all related to the same Finished state. Having 4 separate vars separate and passing them around makes the code more complex than necessary.
FinishReason.EOS enum is now a more generic FINISH_MATCHED_TOKEN object which now can handle any future changes where any token/tokens are matched, not just for single EOS. For example, EOT tokens in structured chat models in future code. Some existing chat models already have multiple EOS defined in model/config.json. This allows the FinishReason object to hold the actual token hit instead of fixed enum which would requires another eos_token var in the current structure to know which of the N possible EOS token was hit.

TESTS:

Tests are performed with vllm 0.4.2 since 0.4.3 requires another PR #487 :

PASS: Matched eos
PASS: Matched stop_str
PASS: Length reached
PASS: dp == 2

…l not be executed

Co-authored-by: ZX <zx@lbx.dev>

* oss_monitor workflow update * update path --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ilent-wrongness sweep Adds docs/dev/AUDIT_505_silent_wrongness.md (1692 lines, 10 commits): axis A1 (warning-instead-of-error in loader/spec/MoE), A2 (warning-instead-of-error in memory/dist/pool), B (comment invariants as testable claims), C (numeric defaults without a binds-proof, the sgl-project#449/sgl-project#493 class generalised), D (catalog §14/§16 against their code predicates). Cross-cutting: the load-completeness check binds in only 2 of 186 model loaders; reconciles with the sgl-project#501 audit's A1-01 finding (two completeness remedies, neither general). Corrects FEATURE_CATALOG.md §14 (Dashboard) and §16 (Measurement/window infrastructure) with file:line-cited predicates in place of the old short prose: wizard refusals cite their source; the anonymization gate covers the rig-artifact share route only, NOT the sgl-project#152 result-share route, which renders argv verbatim (sgl-project#505-D3); energy metering is GPU board power only, not wall-socket (no knee-point probe actually exists, sgl-project#505-D6); self-update switching needs --serve-supervised (sgl-project#505-D8); gpu-arb is a CONVENTION, not an enforcement (sgl-project#438); forward_peak.py is off unless SGLANG_FORWARD_PEAK_PATH is set, and that env var has no environ.py entry (sgl-project#505-D11); CollectiveClock covers plain-prefill-only, cuda-only, pp_size==1-only -- no decode/verify-round equivalent (sgl-project#505-D14); the measured-KV-budget stale-boot trap only applies when the feature flag is on. Real conflict as briefed: HEAD's sgl-project#493 merge had appended a "transient-rate sampler" paragraph to the end of the OLD §16 prose; sgl-project#505's branch rewrote that same prose from base, sentence by sentence, with citations. Resolved by taking sgl-project#505's full rewrite (precision wins) and re-appending the sgl-project#493 paragraph after it unchanged, plus a one-line cross-reference noting forward_peak needs SGLANG_FORWARD_PEAK_PATH set before use (ties sgl-project#493's "use forward_peak's nvml_free_bytes_min" advice to sgl-project#505-D11's off-by-default finding). §14 conflict-free (git auto-merged; HEAD never touched that section). All 17 catalog sections and the four previously-preserved additive paragraphs (two-axis §1, matrix doctrine §2, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified still present. Verified: 1f2b24e (sgl-project#501 fix commit) is confirmed an ancestor of the current line tip via the earlier fix/kvso-reclaim-decline-501 merge -- the sgl-project#505 audit's own warning that sgl-project#501 was absent from ITS base does not apply to this integration line. git diff --stat d653405..HEAD -- '*.py' empty (pure docs). codespell clean on both touched files. Scrubbed for personal email/PAT/private-IP leakage -- no hits in either audit document.

…each with a falsifier B3 (draft-KV-DCP now keys on the installer's own predicate uneven_weighted_dcp_enabled(), admitting the --rank-kv-ratio route the sibling speculation x DCP gate already accepted -- the -67% draft-KV win is no longer unreachable for flag-route boots); B8 (validate_breakable_boot's None arm split into a NO_SERVER_ARGS sentinel that skips vs. an unresolvable backend that refuses by name); B10 (verdict: the kvso x speculation refusal is DELIBERATE, not stale -- reason and the one unobserved round now in the message, KVSO_ALLOW_SPEC surfaced in --enable-kv-session-offload's CLI help for the first time); B2 (--rank-kv-ratio without a placement refuses by name instead of being accepted-and-inert); B18 (the marlin uneven-TP coarsening fold reads a per-backend marlin_packable_linear declaration instead of a class-name list; of the three configs the audit named, only MarlinConfig was a genuine latent gap, W8A8Fp8Config/QuarkConfig reach no marlin repack entry point); planner/flags.py's two inverted registry edges (I-2/I-3) plus a new contract test that drives every declared uneven-TP edge against the live ServerArgs validation instead of trusting the registry by inspection. Adds a status header to AUDIT_500_mechanism_reach.md §5 recording the six fixes and two corrections-from-execution. Manually extended that header with the sgl-project#503 B1-refutation (the sgl-project#503 branch predates B1's discovery entirely, so it couldn't record it): combined board is now B1 REFUTED (sgl-project#503), B2/B3/B8/B10/B18 + the flags.py registry FIXED (sgl-project#504), B4-B7, B9, B11-B17, B19, B20 still OPEN -- one place to read the whole status rather than two disjoint headers. Clean auto-merge, no conflict: FEATURE_CATALOG.md's touched regions (near the preamble, and in §1/§3/§12-adjacent prose) were untouched by every merge since the ac84f5d base; server_args.py was untouched by sgl-project#503/sgl-project#505/sgl-project#506/ sgl-project#509 (its only other change on this line, sgl-project#493, is already inside the base). All 17 catalog sections and the four previously-preserved additive paragraphs (two-axis §1, matrix doctrine §2, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present; the sgl-project#505 §16 corrections (gpu-arb convention-not-enforcement, forward_peak off-by-default, CollectiveClock prefill-only) verified present and unaffected. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_flag_registry_contract_500.py alone: 15 passed (exact target). - test/registered/unit/server_args/: 532 passed, 86 subtests (up from 526 pre-merge -- the new/extended test files account for the delta). - test/registered/unit/planner/ (HTSGLANG_TEST_MODEL_DIR set): 2223 passed, 1 skipped, 352 subtests -- crossing-check against sgl-project#503 (which also worked the planner edge): both merges coexist without collision, count is above the sgl-project#503-alone baseline of 2205 as expected from sgl-project#504's added tests. - test/registered/unit/layers/moe/: 153 passed, 395 subtests (includes the new test_breakable_boot_gate_500.py). - test/registered/unit/distributed/ (touched via test_marlin_unit_coarsening.py): 24 failed, 2134 passed, 8 skipped, 721 subtests. Diffed the failing-test list against the sgl-project#503-merge state line by line: BYTE-IDENTICAL set of 24 failures, only the passed/subtests counts grew (from marlin coarsening's new cases) -- confirmed non-regression, not just a count match. - ruff check --select=F401,F821,UP037: 2 findings (planner/flags.py:1717 `"Profile"`, test_rank_kv_ratio_args.py's unused `math` import), both confirmed pre-existing at the ac84f5d base and outside sgl-project#504's diff hunks. No ruff format run (CLAUDE.md: do not reformat planner/* and other listed subtrees -- unrelated churn buries the semantic diff). - codespell clean on all 18 touched/added files.

…-wrongness sweep Eight commits, each closing or correcting a sgl-project#505 finding: - B-01: offload_movement splits the wave-in error path -- a destination release that already landed is not a failed retrieval; the item is only marked PARKED and its memtier booking only held when the bytes genuinely are still at the park target. - C-01: step 0 of the sgl-project#505-C-01 liveness-timeout task REFUTES its own ranking -- TTFT is outside the 90s budget (the streaming response is only built after generator.__anext__() returns, so the watchdog does not exist during queueing/prefill). The default is deliberately left unchanged; what sgl-project#514 pins instead is the ORDERING that makes that true (TimeToFirstTokenIsOutsideTheBudgetTest, proven can-fail by removing the pre-pull). Same lesson as sgl-project#500-B1: an audit finding is a hypothesis until its falsifier is red. - C-05: CONVENTION_bounding_defaults.md + a value-pinning test convention, with the first three defaults pinned against removal/halving. - A1-01: widens the draft-load completeness check and makes its non-reach audible instead of silent. - A2-04: PD topology returns UNKNOWN instead of fabricating a CUDA->NVML identity mapping when it cannot resolve one. - D3: scrubs the sgl-project#152 GitHub result-share route (was rendering the start command's argv verbatim) and binds the posted body to what the user actually previewed. - A2-03/A2-01: the kv-session-offload spill verdict is now rank-uniform (a group decision), and a carve-out that used to be assumed is now verified against the live state. - A2-05: the barlink transport fallback is a GROUP decision, not a per-rank one. Also PARTIALLY REFUTES the sgl-project#505-A2-05 damage story: the byte proof already reduces its verdict group-wide, so the three named sites desync into a hang, not a silent split; the genuine split window is narrower (a rank-local exception after a transport's last bring-up collective) and is what this fix actually closes. The broader desync class is untouched, recorded as a separate GPU-window ticket. Corrects AUDIT_505_silent_wrongness.md and FEATURE_CATALOG.md in place with the same append-only CORRECTION/REFUTED convention used throughout this line (blockquotes, nothing deleted). Clean auto-merge, no conflicts in either file -- all 17 catalog sections and every previously-preserved additive paragraph (two-axis §1, matrix doctrine §2, sgl-project#510/sgl-project#513/sgl-project#501 §12 families, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. test_kvso_reclaim_decline_501.py (sgl-project#501's own test) picks up a 1-line change from sgl-project#514's kv_session_offload.py work -- sgl-project#514's base already contains sgl-project#501, so this is sgl-project#514 building on top of it, not a collision. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - Targeted run across disaggregation/, the new/changed test files, managers, distributed (barlink/gpu_battery), and liveness: 530 passed, 21 subtests, 1 failure (test_decode_queue_cleanup.py, untouched by sgl-project#514's diff; reproduced byte-identically on the pre-merge base 96c7dc5 -- same generic retry()-wrapped environmental failure seen throughout this line). - test/registered/unit/distributed/ (full): 24 failed, 2147 passed, 8 skipped, 721 subtests -- matches the established 24-failure baseline. - test/registered/unit/model_loader/ (full, since loader.py/weight_utils.py are touched): 32 failed, 229 passed, 9 skipped, 41 subtests on the merged tree vs 32 failed, 226 passed on the pre-merge base -- diffed the two failing-test-ID lists directly: BYTE-IDENTICAL. The three extra passes are sgl-project#514's own new/extended test coverage elsewhere in the directory. - ruff check --select=F401,F821,UP037: 2 findings (offload_movement.py:206 "DeviceOps", loader.py's unused _export_static_state import), both confirmed pre-existing at 96c7dc5 and outside sgl-project#514's diff hunks. No ruff format run (CLAUDE.md sgl-project#514-lesson rule -- ironically the same task number). - codespell: one hit, "unter" in test_gpu_battery_checks.py:1965 (a pre-existing German test-fixture string, `"reason": "unter der Marke"`, already present at 96c7dc5 and untouched by sgl-project#514's diff -- a pre-existing English-only-code violation, not introduced here).

…-cost probe, the F2 instrument sgl-project#462 (the breakable MoE-offload CUDA-graph route) was already merged onto this line long ago; this pulls in two new commits added on top since, building the sgl-project#494 F2 measurement instrument that ticket named as missing. srt/utils/break_cost_clock.py prices ONE CUDA-graph crossing of a breakable capture (segment_end -> eager slot -> segment_start) with a CUDA event pair around every segment and every break slot in BreakableCUDAGraph.replay(). Per crossing: gap_in_ms / slot_ms / gap_out_ms (device wait, device compute, device wait) plus host_ms split into the four F2 terms (rendezvous, planning, publish, fetch), bracketed via break_cost_phase() in MoEExpertOffloadCache.prepare_breakable; per round: compute_ms / wait_ms / span_ms / residual_ms, per rank, as JSONL. Read via SGLANG_BREAK_COST_DEFER_ROUNDS (default 2) rounds late through query() only, so the read never synchronises the round it measures. SGLANG_BREAK_COST_PROBE=1, OFF by default and byte-neutral off (the disabled replay() loop is the original one statement-for-statement -- no event created, none recorded, nothing allocated per round, pinned by a call-count spy). Route-agnostic: any eager_on_graph break point is priced, not only the MoE one. Reader: scripts/dev/494_break_cost/summarise.py. 21 hermetic tests, three executed can-fail arms. No GPU number exists yet -- the probe has never run on a card; it is the instrument F2 was missing, not a result. FEATURE_CATALOG.md §16 conflict as expected: this branch's own base predates every merge from today (sgl-project#493/sgl-project#505/sgl-project#459/etc.), so its diff carried the OLD short §16 base prose alongside its new "Break-cost probe" paragraph. Kept HEAD's full precise §16 (all of today's corrections) and appended the Break-cost-probe paragraph after it, discarding only the superseded old base text the branch's own diff had inherited. §3's small in-place edit ("its instrument now exists (sgl-project#494, §16 break-cost probe)") landed with no conflict at all -- untouched by anything else on the line. All 17 sections and every other additive paragraph verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree; test tree is tests/moe_offload/, not test/registered/): - test_break_cost_probe_494.py + test_breakable_route_462.py: 58 passed (21 + 37, matching the catalog's own count for each file). - tests/moe_offload/ (full): 2 failed, 455 passed, 34 skipped -- both failures in test_gguf_moe_offload.py, a file this branch's diff never touches. Bisected across today's entire merge history AND the prior session's tips back to 9fab358 (before sgl-project#500's audit branch base): the same 2 failures reproduce identically at every single point, including before this session's work began. Confirmed NOT a new regression from sgl-project#494/sgl-project#462 or anything merged today -- a long-standing pre-existing issue, out of scope for this merge. - ruff check --select=F401,F821,UP037: 18 findings in expert_offload.py and break_cost_clock.py (including two F821 undefined-name hits), ALL confirmed pre-existing at cc5a92e and outside this branch's three diff hunks (import addition + two break_cost_phase() wraps). - codespell: clean on all 6 touched/added files.

… of stalling on it Caught live 2026-08-07 08:26 in the degraded window with all three ranks still alive -- not two different SIZES this time, two different COLLECTIVES: TP0 qwen3_5.py:1241 prepare_mlp -> _gather_hidden_states_and_residual -> attention_tensor_model_parallel_all_reduce ALL_REDUCE TP1, TP2 qwen3_5.py:1234 self_attention -> radix_attention -> _forward_extend_dcp (flashinfer_backend.py:5905) -> cp_lse_ag_out_ar_mha_uneven -> _ag_lse ALL_GATHER Seven lines apart in one layer body. TP0 is already past attention; its peers are inside an LSE all-gather it never joins. The exit is _forward_extend_dcp's if not has_prefix: ... return o_cur.contiguous().view(...) guarded only by a comment: "has_prefix is a global (rank-uniform) property, so every rank takes the same branch -> the DCP collectives below stay balanced (no deadlock)". The three branches BELOW that return each carry their own note that they are rank-local but reach the collective identically; the author defended those and left this one undefended. sgl-project#505 class, and sgl-project#131 is the same property failing before. THE PREMISE IS FALSE, and the link to the token-count specimens is an identity rather than a correlation. weightless_has_prefix reduces to any(extend_prefix_lens_cpu) (lockstep.py:143), whose provenance is one line: schedule_batch.py prefix_lens = [len(r.prefix_indices) for r in reqs] schedule_batch.py self.prefix_lens = prefix_lens forward_batch_info.py ret.extend_prefix_lens_cpu = extend_prefix_lens and the line two above the first builds extend_num_tokens from the SAME expression. One rank-local vector decides both the SHAPE of every per-layer collective and WHICH collectives run at all. 912-vs-2048 is that vector differing in magnitude; 08:26 is it differing in whether it is all-zero. Not two roots -- one root at two amplitudes. lockstep.py's docstring defends the claim with "extend_prefix_lens_cpu is a host-side length vector, not a per-rank tensor", which is true about its TYPE and false about its CONTENT. A DETECTOR, NOT A CORRECTION, and the code says so in both modules. Once the vector diverges the attention result is already wrong; refusing only makes the failure uniform, immediate and self-describing. Deliberately NOT an OR-ballot over the derived boolean: OR-ing would adopt the position that a divergent prefix vector is legitimate and should be absorbed, which is the opposite of the position #616B took when it installed a group-MIN floor to make that vector uniform on the paths it reached. Refusal is neutral between "extend the floor to this path" and "divergence is legal here" and forecloses neither. THE OPEN QUESTION THIS EXISTS TO ANSWER: is a divergent len(r.prefix_indices) legitimate under DCP token ownership, or a gap in #616B's floor? The detector's first live firing decides it, because it prints the per-rank vectors. Recorded with it, unresolved: the host-backup mechanism fixed in the previous commit predicts the ROOMIEST rank keeps the LONGEST prefix, and 08:26 has TP0 with NO prefix and its peers with one -- the opposite direction. Either that batch diverged in composition rather than in prefix length, or a second divergence source is still open. This is NOT claimed closed. Cost, named: ONE MIN all_reduce of a 4-element int64 CPU tensor, 32 bytes, on the DCP gloo group, once per EXTEND BATCH -- not per layer, because the sixteen full-attention layers all read a vector that is fixed by then. The alternative considered and rejected, unconditional participation, costs two collectives per full-attention layer on every genuinely prefix-free extend, i.e. 32 per first-chunk prefill on this checkpoint. Gating is replicated by construction (--dcp-size and a kill switch read ONCE at import, so a mid-run environment edit cannot make one rank skip a collective the others enter). No DCP, single rank, or no vector: no collective at all, byte-identical. The ballot carries length and a digest as (x, -x) pairs so one MIN yields min and max of each, making disagreement detectable without a second collective. The digest is a plain integer polynomial, NOT hash(), which is salted and per-process in exactly the situation this compares across processes. The failure path then pays for all_gather_object to print every rank's vector, since the group is about to raise anyway. Tests: 13 hermetic cases (CUDA_VISIBLE_DEVICES=99, register_cpu_ci + CustomTestCase) driving the real assert_prefix_lens_rank_uniform through a real elementwise MIN. The falsifier feeds the 08:26 shape [[0],[2048],[2048]] and pins that every rank raises -- a detector that fired on one rank only would itself be the rank-local-test-before-a-collective defect. Also pinned: the early return still precedes cp_lse_ag_out_ar_mha_uneven, the check sits at the line that materialises the vector, the kill switch is not read per call, and has_prefix and extend_num_tokens still read the same expression, so the link cannot be broken silently. Base-vs-branch by name over test/registered/unit/{distributed,mem_cache} against e9d67ae: zero regressions (see the previous commit for the table; both changes were in the tree for that run). ruff: no new findings.

…es, and pin it THE INVARIANT WAS PROSE. phase_policy's branch says "Declining here is what routes the caller to the evict rung instead of to a cutover." The caller did not: scheduler.py handled every decline identically -- one throttled log line, return. So on 2026-08-16 the instance printed "this is an evict trigger and NOT a flip" 350 times across 54 minutes while no eviction was ever attempted, health answered 200 the whole time, three GPUs sat at 0%, and 10.5M tokens queued behind a pool nothing would free. I wrote that comment in sgl-project#688. It is the fourth counter-vs-actuator member found today, and the first where the actuator existed only as a sentence. WHAT THIS COMMIT DOES: the decline now calls evict_from_tree_cache, bounded to one attempt per 5 s (the decline is evaluated every round, so an unbounded call walks the whole tree in a tight loop on an already-wedged box), and REPORTS WHAT EVICTION RETURNED. THE REPORT IS THE POINT AS MUCH AS THE CALL. "The remedy ran and freed 0" and "the remedy never ran" are the two states this outage could not distinguish, and telling them apart is what turns a silent wedge into a diagnosis. AND ON THIS SPECIMEN IT WOULD HAVE DELIVERED ZERO -- said plainly, because the routing is necessary and NOT sufficient. At 16:22:58-16:23:11 a single request was mid CHUNK-PREFILL (#new-seq 1, #new-token 512, #cached-token 0) while usage climbed 0.95 -> 1.00, with no retract, abort or finish anywhere in the window. A chunked request is resident but sits in NO BATCH (sgl-project#631 defect O), which is why the scheduler read #running-req: 0 while its own protected prefix held the pool. Eviction cannot free a locked chain. The deeper root is that a chunked request's prefix can fill the pool its next chunk must allocate from -- a self-deadlock -- and that is a separate fix on the chunked/admission path, not this one. Also exports BOTH_BLOCKED as a shared constant instead of matching a respelled literal across two modules. TESTS: test_both_blocked_routing_698.py 6 passed -- the sgl-project#505 discipline: the pin fails if the caller stops routing, if the relief stops calling eviction, if it stops reporting a zero delivery, or if the rate limit is removed. managers + mem_cache 64 failed / 1928 passed, the same 64 baseline.

… remedy THE SPECIMEN. The 2026-08-17 speed boot was ordered at cut [31,17,16] and launched with `--pp-stage-ratio 31,17,16`. The coupled derivation balances both layer families together and returned [32,16,16], moving a FULL-ATTENTION layer onto stage 0 -- 8/4/4 instead of 7/5/4 -- which shifted KV mass against a pinned 550000-token pool. The boot died in an NCCL send at 08:53:51, PP0 at 176 MiB free against a 1024 MiB corridor law, `Cuda failure 'out of memory'`. The derived split was already logged. What was missing is any statement that the REQUEST HAD NOT BEEN HONOURED, so the boot record read as the cut that was ordered and the OOM looked like a property of [31,17,16]. It was not: it is a property of [32,16,16], a cut nobody chose. sgl-project#505(a) class -- an unannounced substitution, not a wrong number. WHAT THIS DOES NOT SAY. It does not say the cut is unrealizable. It is: `--pp-attn-stage-ratio` (sgl-project#485) decouples the full-attention family from the linear one, and 7,5,4 gives exactly [31,17,16]. An earlier verdict here that "stage boundaries must be multiples of 4, so [31,17,16] cannot exist" was inferred from the COUPLED path alone and is wrong; the decoupling had already been built. So this warns and proceeds rather than refusing. The advice is computed, not hinted: the full-attention counts of the operator's own requested ranges ARE the --pp-attn-stage-ratio that realizes them, so the warning prints the vector to paste. Fires only when the scores sum to the model depth -- i.e. when the operator is spelling out layer counts rather than capability ratios. Scores are ratios by contract, so a true ratio like the incumbent 14,10,8 -> [28,20,16] must stay silent, and a test pins that it does. TESTS test/registered/unit/server_args/test_pp_stage_ratio_snap_warns_713.py, 7 passed. Pins the mechanism the advice points at (coupled snaps to [32,16,16]; attn_scores 7,5,4 realizes [31,17,16] exactly; the advised vector is derivable from the ranges), plus three can-fail cases: an honoured request, a true ratio, and an already-decoupled request must none of them warn.

…ze, not a desk number Answer to the standing question: nothing in the reserve derivation needs a scheduler. `_admission_reserve_rows` derives from `chunked_prefill_size`, a server_arg fixed before the pool exists -- it was simply not reachable from `self`, which is why the first version passed `None` and silently took the 512 fallback while W22's live value was 4096. The sizer now reads `get_global_server_args().chunked_prefill_size` directly. CONSERVATIVE_ADMISSION_RESERVE_ROWS is demoted to belt-and-braces: a floor under a missing or zero server_arg, where the alternative is a reservation smaller than the pre-sgl-project#851 wall. This removes both problems the constant carried: a desk-chosen number with no proof behind it (sgl-project#505 shape), and the failure mode I flagged myself -- a boot configured with a prefill chunk above 16384 would have rebuilt the wall. On a correctly configured boot the constant now decides nothing. Recorded in NOTE_851_build_caveats.md next to the constant, per the ruling's either/or: the derivation replaced it, so no unknown term needs documenting -- the note says which input it derives from and why the floor survives. Tests: 18 passed / 0 failed across the reservation and exposure suites. ruff clean. Hermetic, CVD="", no boots.

…hat hid it (1), the second boot-blocker, plus (2) -- deliberately in ONE commit because their tests share a file and splitting them would ship a red commit. (2) does not block the boot; landing it early costs nothing and keeps every commit standing on its own. (1) MEASURED, NOT ASSUMED. The refusal in ed09bbe would merely RENAME the wall if the armed serving form never built a semaphore set. It does, and this drives the launcher's own publisher to prove it with no boot and no GPU: prepare_xchg_env(log, nonce, "exchange") launcher.py:3463 -> prepare_xchg_region(...) creates the region and the names -> env[SGLANG_WEG2_XCHG_REGION/_BOOT] the two variables that _weg2_shadow_region reads (weight_updater.py:2162-2165) => _weg2_shadow_region() is not None AND _weg2_xchg_sems() is a SemSet So the phased deposit/collect path is the one an armed boot takes, and the W68 refusal guards the failing case rather than renaming the normal one. The test tears the region down in a `finally`, leaving no /dev/shm behind either way. (2) THE sgl-project#505 SWALLOW AT weight_updater.py:3173, CLOSED. `except BaseException: sems = None` is why a lost handshake could reach a boot as `W74 ... src_resolved=0/N` -- an ADDRESS complaint for a HANDSHAKE cause. The exception is now named with type AND text at the moment it happens: WEG2-XCHG-SEMS-UNAVAILABLE type=<T> text=<msg> armed=<bool> and with the exchange ARMED it is no longer swallowed at all but raised as a named W68: an armed rank is one of the two weg2 groups, and a leg without semaphores is not a degraded observer, it is a flip with no ordering. Unarmed boots (`ring`, the pure `shadow` arm) keep the observer `None` the branch was written for -- but loud. test_serving_env_path_yields_a_real_semaphore_set_1330 test_unavailable_semaphores_are_named_not_swallowed_1330 (injected OSError: armed -> refusal carrying the name; unarmed -> None, name still logged) 245 unit tests, provider 19/19, manifest 22/22, replay self-test MATCH 6/6.

…hat hid it (1), the second boot-blocker, plus (2) -- deliberately in ONE commit because their tests share a file and splitting them would ship a red commit. (2) does not block the boot; landing it early costs nothing and keeps every commit standing on its own. (1) MEASURED, NOT ASSUMED. The refusal in ed09bbe would merely RENAME the wall if the armed serving form never built a semaphore set. It does, and this drives the launcher's own publisher to prove it with no boot and no GPU: prepare_xchg_env(log, nonce, "exchange") launcher.py:3463 -> prepare_xchg_region(...) creates the region and the names -> env[SGLANG_WEG2_XCHG_REGION/_BOOT] the two variables that _weg2_shadow_region reads (weight_updater.py:2162-2165) => _weg2_shadow_region() is not None AND _weg2_xchg_sems() is a SemSet So the phased deposit/collect path is the one an armed boot takes, and the W68 refusal guards the failing case rather than renaming the normal one. The test tears the region down in a `finally`, leaving no /dev/shm behind either way. (2) THE sgl-project#505 SWALLOW AT weight_updater.py:3173, CLOSED. `except BaseException: sems = None` is why a lost handshake could reach a boot as `W74 ... src_resolved=0/N` -- an ADDRESS complaint for a HANDSHAKE cause. The exception is now named with type AND text at the moment it happens: WEG2-XCHG-SEMS-UNAVAILABLE type=<T> text=<msg> armed=<bool> and with the exchange ARMED it is no longer swallowed at all but raised as a named W68: an armed rank is one of the two weg2 groups, and a leg without semaphores is not a degraded observer, it is a flip with no ordering. Unarmed boots (`ring`, the pure `shadow` arm) keep the observer `None` the branch was written for -- but loud. test_serving_env_path_yields_a_real_semaphore_set_1330 test_unavailable_semaphores_are_named_not_swallowed_1330 (injected OSError: armed -> refusal carrying the name; unarmed -> None, name still logged) 245 unit tests, provider 19/19, manifest 22/22, replay self-test MATCH 6/6. (cherry picked from commit 35bc33e)

Qubitium and others added 3 commits June 5, 2024 01:53

refractor .finished bool into .finished_reason BaseFinishReason obj

55739ad

fix issue: If request.stream is false, the code after "yield out" wil…

3407263

…l not be executed

fix field name error

9d18ff0

Qubitium changed the title ~~Fix rid stat leak + Refractor .finished~~ Fix rid state map leak + Refractor .finished Jun 5, 2024

merrymercy merged commit f70f725 into sgl-project:main Jun 7, 2024

Qubitium deleted the refractor-finished branch June 7, 2024 21:23

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Fix rid state map leak + Refractor .finished (sgl-project#505)

49a0ead

Co-authored-by: ZX <zx@lbx.dev>

Lzy17 mentioned this pull request Aug 31, 2026

[AMD][MORI] Bump MoRI to 7c51d18 for ionic RoCE dmabuf fix (#509) #37286

Merged

Navigation Menu

Uh oh!

Fix rid state map leak + Refractor .finished - #505

Fix rid state map leak + Refractor .finished#505
merrymercy merged 3 commits into
sgl-project:mainfrom
Qubitium:refractor-finished

Qubitium commented Jun 5, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Qubitium commented Jun 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Qubitium commented Jun 5, 2024 •

edited

Loading