Bump the version to v0.1.8 by merrymercy · Pull Request #93 · sgl-project/sglang

merrymercy · 2024-01-24T11:31:40Z

ignore warnings

…gl-project#93) * fix & upd range * adjust heuristics for grid dim enumeration * better search statistics display * add verbose flag * set default max_num_threadblock_graphs to 1 for attentions --------- Co-authored-by: Mengdi Wu <mengdiwu@andrew.cmu.edu>

[FEAT] Shared mem pool based cuda ipc for multi-modal data transport

* bugfix for qwen35 * resolve lint * support Support mamba cache transfer for NPU

…um (sgl-project#93) Add --speculative-adaptive-graph-memory {auto,resident,offload}. In offload mode the per-k-state scratch buffers (private flashinfer float workspaces, cuda-graph kv_indices/custom_mask) are tagged via torch_memory_saver so inactive states hold no physical VRAM: pause(tag) unmaps their pages at stable virtual addresses, resume(tag) remaps + zeroes them (the sgl-project#50 boot contract; plan data is host-side and re-planned per forward). A k-swap is sync -> pause(old) -> resume(new)+zero -> pointer swap, ms-scale at the measured ~0.1/s swap rate. Guarantees: - states built largest-first and paused right after build (boot peak ~ one state); finalize_boot verifies free VRAM >= max-state footprint with all states paused => a swap can never OOM (no lazy growth). - segment-isolation audit: every tagged buffer must lie in its own tag's build-window segment; only individually-noted MiB-scale never-freed allocations are tagged, so tags cannot share allocator segments. - request-boundary workspace zeroing skips paused workspaces (their zero happens at resume). - G4 data_ptr isolation assert unchanged: managed aliasing multiplexes physical pages under distinct virtual addresses, so unmanaged sharing stays exactly as fatal as before (docstring updated). - resident mode is first-class and byte-identical to the previous behavior (sum-of-states reserve, ~us pointer swaps); auto resolves to offload only when CUDA + flashinfer + torch_memory_saver prerequisites hold. - rank determinism: swap driven by the rank-invariant step decision; SGLANG_ADAPTIVE_ALIAS_VERIFY_RANK_SYNC=1 all-gathers (ordinal, steps) per swap as a stress check. SGLANG_ADAPTIVE_FORCE_SWAP_INTERVAL=N is a test-only knob cycling states every N verifies. Launcher injects the torch_memory_saver LD_PRELOAD hook for offload even without --enable-memory-saver (the hook is inert outside tagged regions). CPU tests: existing adaptive suites green (46+7+6); GPU validation follows in the T93 window.

…ect#93) --speculative-adaptive-config now also accepts built-in profile names: 'default' (per-algorithm default) and 'high-accept'. The high-accept profile adds k=4 and k=5 rungs for workloads with per-position accept p >~ 0.85 (code boilerplate, structured emission): with graph-memory offload an extra rung costs only boot capture time plus the max-state alias pool, so the rungs are nearly free VRAM-wise. up_hysteresis 0.25 (T75 recommendation) on the bs1/bs8 slots ensures the ladder climbs to 4/5 only on SUSTAINED high acceptance; no step-0 slots so the profile is valid for EAGLE and FROZEN_KV_MTP alike. The DEFAULT candidate set stays [1,2,3]: at p <= 0.8 k=4/5 is net-negative (2 extra serial draft forwards for < 0.6 expected tokens), especially with all-reduce over PCIe x4 per draft step; the GPU-phase measurements will confirm or amend this default.

…er (sgl-project#93) 26 tests: mode resolution (auto/resident/offload, prerequisite degradation, explicit-offload hard errors), manager bookkeeping (tagging, pause-after- build, pause-before-resume ordering, zero-on-resume, idempotent activation, no-OOM reserve check, segment-isolation audit incl. recycled-VA false- positive guard, LD_PRELOAD guard), controller integration (largest-first build order in offload vs historical ascending in resident, activation through the manager, forced-swap stress knob), and the high-accept profile (k=4/5 climb on sustained acceptance, up_hysteresis blocking borderline climbs, no step-0 slots). All CUDA interaction mocked; combined unit/spec run: 120 tests OK.

…ad (sgl-project#93) GPU-validation boot finding (5-state high-accept config): illegal memory access during the k2 state build. Root cause (instrumented boot with CUDA_LAUNCH_BLOCKING): the caching allocator packs 1-10 MiB allocations into shared 20 MiB segments (kLargeBuffer) and serves any large-enough free block regardless of which region entry created the segment — k2's multistep kv_indices landed in the free tail of PAUSED k4's segment (observed at 0x208af00000) and faulted on first touch, since pause(tag) unmaps a tag's segments wholesale. The 2-state boots carried the same latent hazard and survived on allocation-size luck. Fix, three layers: - one private torch.cuda.MemPool per tag: a tag's free space is only visible to that tag's own allocations (which happen only during its own build) — cross-tag free-list reuse becomes structurally impossible. New TorchMemorySaverAdapter.region_config() provides tag interception without TMS's shared primary pool. - size gate enforced (was only documented): allocations < MIN_TAGGED_BYTES never enter a tagged region and stay resident in the default pool. - SGLANG_ADAPTIVE_ALIAS_DEBUG=1 logs noted-tensor pointers and per-tag segment windows at pause time (the instrumentation that found this). GPU results (T93, full log in job tmp): offload==resident 9/9 byte-identical; adaptive-OFF 9/9 byte-identical to pre-sgl-project#93 AND unmodified base; KV 261120 == static 261120 at standard 4800 reserve (T75's 261k->39k adaptive cost recovered 100%); ~2000 forced swaps/rank across 2- and 5-state configs, zero rank-divergence with the per-swap all-gather assert ON, swap avg 14 ms / max 30 ms; 5-state high-accept boot clean after this fix (max 424 vs sum 1633 MiB tagged); high-accept ladder climbs 3->4->5 and yields 120.2 tok/s on a repetitive workload (pinned: k3 111.2 / k4 119.4 / k5 128.6) with mixed-workload parity 92.1 vs 91.3. CPU tests: 27 alias tests green.

…ls + int workspaces (sgl-project#102) Shrink the untagged per-state VRAM residue of adaptive offload states: * Per-state CUDA-graph capture pools: every adaptive build captures into a private torch.cuda.graph_pool_handle() with the state's torch_memory_saver region active (same mechanism as TorchMemorySaver.cuda_graph), so capture-time allocations are unmapped with the state. Private per tag for the same cross-tag free-list-isolation reason as the sgl-project#93 per-tag MemPools. * flashinfer wrapper int workspaces (8 MiB each; graph-mode wrappers exist per captured shape) are swapped for tagged, zero-on-resume buffers via the public reset_workspace_buffer API. Safe because flashinfer re-plans every forward (host state -> pinned -> device copy) and their boot contract is fresh-cudaMalloc zero pages (sgl-project#50). * Reserve check now uses the measured pause-released bytes per state (mem_get_info delta) instead of the sum of noted tensors; per-state logs itemize scratch / int-ws / capture-pool bytes. * Mode surface: 'offload' is now Stage 2; new 'offload-scratch' keeps the exact Stage-1 (sgl-project#93) behavior as a fallback knob; 'auto' degrades offload -> offload-scratch -> resident (Stage-2 prereqs: 'full' decode cuda-graph backend, no SGLANG_MEMORY_SAVER_CUDA_GRAPH). * SGLANG_ADAPTIVE_CAPTURE_CPU_BACKUP=1: exact byte restoration of capture pools through host RAM (fallback falsifier knob). Resident mode and the static path are untouched (all new behavior is gated on an active Stage-2 build scope). Tests: test_adaptive_graph_memory.py 44/44 (17 new: Stage-2 mode ladder, per-tag pool routing, capture region reentrancy guard, measured-footprint reserve check, capture-pool-only swap path, int-ws retagging); full CPU unit/spec suite 242 passed. GPU validation to follow (T102).

… offload+ladder fork) Correction to the prior provenance pass: it examined a HEAD (/spinning/htsglang) that did NOT contain feat/adaptive-draft-len, so the merge-base..HEAD diff showed adaptive_spec_params.py / adaptive_runtime_state.py as untouched and wrongly concluded the fork's adaptive contribution was only rank-determinism. Re-verified against feat/adaptive-draft-len (merged into integration/r2). Precise, two-sided truth: - UPSTREAM: the base adaptive STEP-controller (--speculative-adaptive, EMA AdaptiveStepSlot step-picking, resolve_candidate_steps_from_config, and the adaptive_unsupported_reason constraints topk=1/not-multi-layer-EAGLE/no-DP-attn- TBO-pdmux) — present at merge-base 82e7cdc. - FORK-ORIGINAL, built ON TOP: adaptive_graph_memory.py (new file, absent at merge-base) — graph-memory offload for multi-k capture pools: pauseable capture pools + int workspaces (0195ffb sgl-project#102), reserve max(one state) not sum (a7c58ec sgl-project#93), per-tag mempools fix (9969db7 sgl-project#93); the high-accept k=4/5 ladder (acf6014 sgl-project#93); frozen-MTP adaptive wiring (3816606); state-isolation + broadcast ratchet (c230af7) + bs-axis debounce (5d57785) + candidate- ceiling trim (4e029f7); and rank-0 broadcast for mixed-GPU determinism. The fork also genuinely extended adaptive_spec_params.py (+198) / adaptive_runtime_ state.py (+231). Restores the credit the first pass erased WITHOUT re-introducing the base-controller over-claim. FLAG A (graph-state offload) and FLAG B (high-accept k4/5) from the prior report are hereby confirmed FORK (sgl-project#93/sgl-project#102), not upstream.

…pstream-base prose Apply the governing doc rule across the whole file: list only (1) fork-original features and (2) upstream code the fork MEANINGFULLY changed, framed by the fork's DELTA — not by the upstream feature name. Remove the paragraph-length upstream-base explanations added in the prior provenance passes; a brief '(adapts upstream X)' tag replaces them. - Category-B index: retitled GGUF / HiCache / adaptive-draft / PD entries to name the fork delta, each with a one-line '(adapts upstream X)' tag; dropped the multi-line base descriptions. - §2 / §3 / §4 headers retitled to name the fork delta on top of the upstream base; their 'Provenance' paragraphs compressed to a single italic 'Base: ... everything below is the fork's delta' line. - §3 adaptive-draft bullet retitled to lead with the fork machinery (graph-memory offload sgl-project#93/sgl-project#102, high-k ladder, frozen-MTP wiring, state-isolation/debounce, rank-0 broadcast); cut the half that re-described upstream's step-controller. - 'Flip side' paragraph reworded to name the fork deltas rather than bare upstream capability names. No whole feature sections were deleted: verified the doc contains no standalone bare-upstream feature line (RadixAttention / prefix caching / chunked prefill / CUDA graphs / torch.compile / continuous batching appear only inside fork-extension bullets, never as their own claimed feature). Provenance basis unchanged: merge-base 82e7cdc vs efschu's 74 fork commits (feat/adaptive-draft-len included).

…8 validated Design steps 5-8. No code delta (steps 5+6 are satisfied by construction -- see below); this records the end-state validation that closes Stage-3. Step 5 (fetch || shared-expert overlap): ACTIVE by construction. Under decode capture get_is_capture_mode() is True, so Qwen2MoeSparseMoeBlock.forward takes forward_normal_dual_stream -> the routed path (self.experts -> run_moe_core -> the capturable offload branch, incl. the UVA gather) runs on alt_stream while the shared expert runs on the main stream. The same capture flag gates BOTH my capturable branch and the dual-stream branch, so whenever the captured gather runs, the overlap is recorded into the graph. No ungating needed. Step 6 (address stability, sgl-project#93/sgl-project#102): satisfied by construction + empirically confirmed. The persistent buffers (resident [R+C] buffer, pinned spill pool, its UVA device view, the scratch [R:R+C] dest view) are allocated BEFORE capture and never freed (held by self._resident/_pinned/_cap_* refs), so the caching allocator can never reuse/move them post-capture -- the exact hazard sgl-project#93/sgl-project#102 tagging guards against cannot occur here. Transient per-step index tensors (presence/rank/src_row/remapped) are allocated into the graph's private pool during capture and are stable across replay. Confirmed: 366 (122B, 3 ranks) + 244 (35B static+hotset) decode-graph replays, self-det 5/5 throughout, and the GPU isolation test is exact -- no drift. Heavyweight torch_memory_saver tagging deliberately NOT added (would risk the working path for zero measured benefit). Steps 7-8 -- 122B TP=3/DCP=3 end-state (Qwen3.5-122B-A10B-GPTQ-Int4, f=0.25 static F3, 5090[cuda:0/rank0/28000]+2x3080, --rank-gpu-id 0,1,2, float16, flashinfer; dropped --disable-cuda-graph, added SGLANG_MOE_OFFLOAD_CUDA_GRAPH=1 SGLANG_MOE_OFFLOAD_MAX_GRAPH_BS=2): - BOOTS + coherent: first token id=11751 ' Paris', full text coherent. - rank0 intermediate=512 => 5090 at cuda:0 (device-order gotcha sgl-project#1 satisfied). - Captures decode buckets [1,2] per rank; 366 decode-graph replays across the 3 ranks; self-det 5/5. - Per-rank VRAM fits: 5090 28.9/32.6 GB, 3080s 20.0/18.9 of 20.5 GB. - GATE (revised, coordinator-adopted -- machine-zero end-to-end is unattainable on this stack; the fork already ships CUDA graphs on non-offload paths at the same capture!=eager ~1e-2 floor): 1. Offload machinery Delta=0: GPU isolation test (E=256/R=64/C=16 routing == 122B per-rank routing) proves remap+gather bit-identical eager-vs-captured. 2. End-to-end Delta <= platform floor: graph-static vs residency-matched eager-static (/tmp/g122tp3_f025.json) argmax Delta max 3.342e-2 -- this IS the platform floor measured AT 122B (offload adds 0 by sgl-project#1), and it matches the 35B fraction=1.0 stock control (3.4e-2) to 3 sig figs. 3. Tokens identical (32 pos) + first-flip None. 4. self-det 5/5. - HEADLINE tok/s: 10.61 graph vs 6.97 eager-static baseline = +52% decode throughput. (35B: static 11.75->20.62 +75%; hot-set 14.92->33.27 +123%.) Captures: /tmp/s3_122b_graph.json. Follow-up available (not blocking): a 122B hot-set-from-file run (needs a 48-layer 122B routing trace) would compound the gain as it did on the 35B. tests/moe_offload 49/49.

…um (sgl-project#93) Add --speculative-adaptive-graph-memory {auto,resident,offload}. In offload mode the per-k-state scratch buffers (private flashinfer float workspaces, cuda-graph kv_indices/custom_mask) are tagged via torch_memory_saver so inactive states hold no physical VRAM: pause(tag) unmaps their pages at stable virtual addresses, resume(tag) remaps + zeroes them (the sgl-project#50 boot contract; plan data is host-side and re-planned per forward). A k-swap is sync -> pause(old) -> resume(new)+zero -> pointer swap, ms-scale at the measured ~0.1/s swap rate. Guarantees: - states built largest-first and paused right after build (boot peak ~ one state); finalize_boot verifies free VRAM >= max-state footprint with all states paused => a swap can never OOM (no lazy growth). - segment-isolation audit: every tagged buffer must lie in its own tag's build-window segment; only individually-noted MiB-scale never-freed allocations are tagged, so tags cannot share allocator segments. - request-boundary workspace zeroing skips paused workspaces (their zero happens at resume). - G4 data_ptr isolation assert unchanged: managed aliasing multiplexes physical pages under distinct virtual addresses, so unmanaged sharing stays exactly as fatal as before (docstring updated). - resident mode is first-class and byte-identical to the previous behavior (sum-of-states reserve, ~us pointer swaps); auto resolves to offload only when CUDA + flashinfer + torch_memory_saver prerequisites hold. - rank determinism: swap driven by the rank-invariant step decision; SGLANG_ADAPTIVE_ALIAS_VERIFY_RANK_SYNC=1 all-gathers (ordinal, steps) per swap as a stress check. SGLANG_ADAPTIVE_FORCE_SWAP_INTERVAL=N is a test-only knob cycling states every N verifies. Launcher injects the torch_memory_saver LD_PRELOAD hook for offload even without --enable-memory-saver (the hook is inert outside tagged regions). CPU tests: existing adaptive suites green (46+7+6); GPU validation follows in the T93 window.

…ect#93) --speculative-adaptive-config now also accepts built-in profile names: 'default' (per-algorithm default) and 'high-accept'. The high-accept profile adds k=4 and k=5 rungs for workloads with per-position accept p >~ 0.85 (code boilerplate, structured emission): with graph-memory offload an extra rung costs only boot capture time plus the max-state alias pool, so the rungs are nearly free VRAM-wise. up_hysteresis 0.25 (T75 recommendation) on the bs1/bs8 slots ensures the ladder climbs to 4/5 only on SUSTAINED high acceptance; no step-0 slots so the profile is valid for EAGLE and FROZEN_KV_MTP alike. The DEFAULT candidate set stays [1,2,3]: at p <= 0.8 k=4/5 is net-negative (2 extra serial draft forwards for < 0.6 expected tokens), especially with all-reduce over PCIe x4 per draft step; the GPU-phase measurements will confirm or amend this default.

…er (sgl-project#93) 26 tests: mode resolution (auto/resident/offload, prerequisite degradation, explicit-offload hard errors), manager bookkeeping (tagging, pause-after- build, pause-before-resume ordering, zero-on-resume, idempotent activation, no-OOM reserve check, segment-isolation audit incl. recycled-VA false- positive guard, LD_PRELOAD guard), controller integration (largest-first build order in offload vs historical ascending in resident, activation through the manager, forced-swap stress knob), and the high-accept profile (k=4/5 climb on sustained acceptance, up_hysteresis blocking borderline climbs, no step-0 slots). All CUDA interaction mocked; combined unit/spec run: 120 tests OK.

…ad (sgl-project#93) GPU-validation boot finding (5-state high-accept config): illegal memory access during the k2 state build. Root cause (instrumented boot with CUDA_LAUNCH_BLOCKING): the caching allocator packs 1-10 MiB allocations into shared 20 MiB segments (kLargeBuffer) and serves any large-enough free block regardless of which region entry created the segment — k2's multistep kv_indices landed in the free tail of PAUSED k4's segment (observed at 0x208af00000) and faulted on first touch, since pause(tag) unmaps a tag's segments wholesale. The 2-state boots carried the same latent hazard and survived on allocation-size luck. Fix, three layers: - one private torch.cuda.MemPool per tag: a tag's free space is only visible to that tag's own allocations (which happen only during its own build) — cross-tag free-list reuse becomes structurally impossible. New TorchMemorySaverAdapter.region_config() provides tag interception without TMS's shared primary pool. - size gate enforced (was only documented): allocations < MIN_TAGGED_BYTES never enter a tagged region and stay resident in the default pool. - SGLANG_ADAPTIVE_ALIAS_DEBUG=1 logs noted-tensor pointers and per-tag segment windows at pause time (the instrumentation that found this). GPU results (T93, full log in job tmp): offload==resident 9/9 byte-identical; adaptive-OFF 9/9 byte-identical to pre-sgl-project#93 AND unmodified base; KV 261120 == static 261120 at standard 4800 reserve (T75's 261k->39k adaptive cost recovered 100%); ~2000 forced swaps/rank across 2- and 5-state configs, zero rank-divergence with the per-swap all-gather assert ON, swap avg 14 ms / max 30 ms; 5-state high-accept boot clean after this fix (max 424 vs sum 1633 MiB tagged); high-accept ladder climbs 3->4->5 and yields 120.2 tok/s on a repetitive workload (pinned: k3 111.2 / k4 119.4 / k5 128.6) with mixed-workload parity 92.1 vs 91.3. CPU tests: 27 alias tests green.

…ls + int workspaces (sgl-project#102) Shrink the untagged per-state VRAM residue of adaptive offload states: * Per-state CUDA-graph capture pools: every adaptive build captures into a private torch.cuda.graph_pool_handle() with the state's torch_memory_saver region active (same mechanism as TorchMemorySaver.cuda_graph), so capture-time allocations are unmapped with the state. Private per tag for the same cross-tag free-list-isolation reason as the sgl-project#93 per-tag MemPools. * flashinfer wrapper int workspaces (8 MiB each; graph-mode wrappers exist per captured shape) are swapped for tagged, zero-on-resume buffers via the public reset_workspace_buffer API. Safe because flashinfer re-plans every forward (host state -> pinned -> device copy) and their boot contract is fresh-cudaMalloc zero pages (sgl-project#50). * Reserve check now uses the measured pause-released bytes per state (mem_get_info delta) instead of the sum of noted tensors; per-state logs itemize scratch / int-ws / capture-pool bytes. * Mode surface: 'offload' is now Stage 2; new 'offload-scratch' keeps the exact Stage-1 (sgl-project#93) behavior as a fallback knob; 'auto' degrades offload -> offload-scratch -> resident (Stage-2 prereqs: 'full' decode cuda-graph backend, no SGLANG_MEMORY_SAVER_CUDA_GRAPH). * SGLANG_ADAPTIVE_CAPTURE_CPU_BACKUP=1: exact byte restoration of capture pools through host RAM (fallback falsifier knob). Resident mode and the static path are untouched (all new behavior is gated on an active Stage-2 build scope). Tests: test_adaptive_graph_memory.py 44/44 (17 new: Stage-2 mode ladder, per-tag pool routing, capture region reentrancy guard, measured-footprint reserve check, capture-pool-only swap path, int-ws retagging); full CPU unit/spec suite 242 passed. GPU validation to follow (T102).

…roject#93, Turn-Grenzen-Wechsel hinter Prefill versteckt) — schliesst die 525-MiB-Luecke fuer beide Drafter auf der 5090

… falsified — attention/GDN are NOT uniform, they follow the base rank-tp-ratio at [0.50,0.25,0.25], MORE concentrated on the 5090 than MLP [0.463,0.272,0.265] (divergence: model_config.py:1313 calls get_tp_partition_ratios bare; base vector is already a de-facto attn/GDN axis); time-share fit reproduces all six sgl-project#296 per-rank values to 0.1 ms: no comparative advantage (MLP 6.23-6.64x vs attn+GDN 6.41-6.53x on the 5090), the MLP lever already took 78% of the sgl-project#252 wait (residual 85 ms not 390), independent attention vector lands +0.4-2.2% against the 3.18% floor — REJECTED; clones rejected on six grounds (sgl-project#93 is physical remap 40-51 ms/GB vs 25-30 ms round, real bytes on the capping rank-0 VRAM, lending refuses permanent posts, GDN state sticks, graph ladder + no all_gatherv, no decode sensor); THE FINDING: sgl-project#296's -84% context on the 10,1,1 prefill optimum is an ARTIFACT — kv-ratio stayed 7,3,3; capacity-matched 2,11,10 predicts 431,475 tokens (6.18x) at unchanged prefill, sum conserved to 2.7%; XS follow-up: one boot with matched vector

…ot per-flip User correction, pinned hard because it is a repeat-forgotten class. The first draft priced every graph-addressed knob at 3-6 s recapture and therefore put all of them on the slow side of the tick/regime line. That is wrong: the stack can pre-capture N stages and park the inactive ones FULL-CAPTURED in host RAM, so a flip back to an already-captured stage rebuilds nothing and costs only the graph-state RESTORE -- observed 40-85 ms, two orders of magnitude cheaper. Carriers verified in tree rather than taken on report: * sgl-project#93 physical aliasing/remap keeps VAs stable across the park (offload_register.py, offload_movement.py, input_buffers.py, runtime_context.py) -- this is what leaves a parked capture valid; * sgl-project#286 offload register ALREADY lists graph_rungs as a parkable item class (offload_register.py:17 "cold capture rungs of the K-/algo ladder", class table at :114) -- confirmed, so this is drive-not-build; * sgl-project#89 hibernate for host-side staging; * sgl-project#464 has NO marker in python/sglang/srt -- it is a target, not a mechanism, so restore is priced at the observed band and any sgl-project#464 gain is upside the design must not assume. R9's classification gains two middle classes: restore-flippable (graph- addressed but target already captured and parked -- priced at restore, possibly coarse-tick) and capture-bound (target never captured -- 3-6 s, one-time). The criterion is unchanged in form; what changed is that "graph-addressed" no longer implies "expensive", it implies "expensive ONCE". R9(iii): the planner pre-captures its own sgl-project#363 stair at boot or lazily on first entry, then parks. First entry is a priced one-time investment scheduled deliberately (low load), never a tax paid at the moment a flip is wanted -- a design that pays capture at flip time has turned a startup cost into a serving cost. R9(iv) + R1: holding N stages is nearly VRAM-free while parked (physical pages returned via the sgl-project#330 VMM dial, VA reservation remains) but NOT free in host RAM, so R1 now extends to host bytes and the ledger must carry the parked stages' host footprint as a term. A stack parking eight stages without that term runs exactly the unledgered-demand defect sgl-project#582 removed from VRAM, one tier down. Falsifier R1b: parking N stages must move the host term by N x per-stage. R5 splits its graph cost into two terms (capture one-time, restore per-flip); a single "graph cost" line is wrong by ~2 orders of magnitude in one direction or the other. F5b: a gain against a 60 ms restore must be accepted even at short horizon; refusing it means a restore was priced as a capture. F9d (runtime) asserts on the ABSENCE of a recapture rather than on elapsed time, since a fast machine can hide a recapture inside a generous threshold. F9e (modelling) is the more dangerous error and gets its own test: a model that prices an already-captured flip at capture cost never misbehaves visibly, it just silently refuses flips that were cheap, so the gain is never observed and nothing surfaces the mistake. FOUND IN TREE, same error, today: registry/rungs.py:93-95 prices the WARM rung at "3-6 s / graph recapture" -- a blanket recapture tax on promotion. Correct for a never-captured stage, wrong as a blanket price. Slice 4b must split it or the planner inherits the very tax this correction removes. Slice plan gains 4b (pricing split + WARM rung fix, desk-provable) and 4c (pre-capture/park + host-RAM term). 4c is the one addition needing a card, and it is cheap: capture two stages, park one, flip, time the restore.

… rung that already names it PRIOR-ART GATE FIRST, and it changed the design twice. (1) The spill ladder ALREADY has this rung. phase_flip_spill.py:173 DEPTH_DRAFT_GRAPHS = 4, CLI name "draft+graphs", with IMPLEMENTED_DEPTH = DEPTH_ARENA_TAIL -- so rung 4 parses and is refused, by design, rather than silently under-delivering. No new flag and no new concept was needed; the work is wiring the rung that exists. (2) The park/restore machinery exists too, and its own docstring names this exact case: short_term_offload_register.py:1269 GraphFamilyRegister ("a whole LAYOUT/ALGO family's capture state, pre-captured at boot and parked while its family is inactive"), with offload_family/onload_family, over the sgl-project#93 VMM route. The credit side exists as well -- kv_backing_relief / recover_kv_backing, and the mixin already carries paused rung graph bytes as registry posts feeding the KV budget (model_runner_kv_cache_mixin.py:1488-1603). Nothing here needs inventing. (3) Rung 4 carries a RECORDED REFUSAL that must not be overturned silently: it "buys a phase-local spill of something the next TP phase must re-capture, which is a different and much worse trade than rung 2". That reasoning is about a RUNTIME per-flip spill where every flip pays a recapture. What is proposed is a BOOT-time deferral paid once. The arithmetic that changed the trade is measured, not argued: on boot 735-bal785 rank 0 binds the pool at 525462 tokens and needs 2242 MiB more to fund the 669k reference, against a residual on that same rank of 2294 MiB. Same number. The refusal is therefore LEFT STANDING in phase_flip_spill until the park/restore path is wired and exercised in a real flip cycle; this commit only makes the sizer able to express the credit, so the next step is wiring rather than arithmetic. The credit is bounded by what was MEASURED, not by what is needed -- a term that grows to close whatever gap it is shown is not a measurement, and that has its own can-fail test. The draft WEIGHTS are never credited: they are needed in the phase the draft actually runs in, and crediting them would be the sgl-project#678 OOM one rung deeper. Also: the pool-pin guard now distinguishes a recipe from prose about a recipe. argv_hc_interval8192.HOSTRAM-NOTES.txt sits under the same glob, was touched after the order, and quotes the flag it documents. Tests: 6 new for the credit, 4 for the guard. mem_cache runs 8 failed / 1441 passed on this branch and 8 failed / 1408 passed at the merge base 332f083 -- same failures, 33 more passing. ruff clean.

update

19a53ee

merrymercy merged commit c70b3cf into main Jan 24, 2024

merrymercy deleted the fix branch January 24, 2024 11:33

CSEEduanyu mentioned this pull request Jan 26, 2025

[Bug] NCCL Crash with SIGSEGV Frequently when deploying deepseek v3 #2803

Closed

5 tasks

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Bump the version to v0.1.8 (sgl-project#93)

c60799d

CaoE referenced this pull request in CaoE/sglang Oct 28, 2025

Optimize chunk_gated_delta_rule kernel (mingfeima#93)

b0ce63e

sammysun0711 pushed a commit to sammysun0711/sglang that referenced this pull request Dec 16, 2025

Merge pull request sgl-project#93 from zhyajie/cuda_ipc_qwen_dev

14299a1

[FEAT] Shared mem pool based cuda ipc for multi-modal data transport

vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026

feat: use env to control cutlass fp4 gemm (sgl-project#93)

b93f79b

cen121212 pushed a commit to cen121212/sglang that referenced this pull request Mar 14, 2026

[NPU] Support mamba cache transfer for NPU (sgl-project#93)

c1b937e

* bugfix for qwen35 * resolve lint * support Support mamba cache transfer for NPU

amd-bot mentioned this pull request Jun 4, 2026

[CI Monitor] Daily Report - 2026-06-04 bingxche/sglang-ci-bot#93

Open

Navigation Menu

Uh oh!

Bump the version to v0.1.8 - #93

Bump the version to v0.1.8#93
merrymercy merged 1 commit into
mainfrom
fix

merrymercy commented Jan 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

merrymercy commented Jan 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant