Add max_prefill_num_token into server arguments by Ying1123 · Pull Request #133 · sgl-project/sglang

Ying1123 · 2024-02-03T10:34:49Z

Example:

python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --max-prefill-num-token 8192

* gitignore * init * [Draft]Complete basic structure, still met with type conversion problem * Update core.pyx * Visualizer python interface * support newly added optypes * Restore cutlass submodule to commit cc3c29a * Repair minor issues including replace magic type numbers, useless var in func and remove block_dim * fit in kn output node * Visualizer doc --------- Co-authored-by: Mengdi Wu <48128384+wmdi@users.noreply.github.com> Co-authored-by: Jianan Ji <jji2@catalyst-0-9.eth>

* enhance trace tools

* opt:ascendc_conv1d_update support * bugfix * bugfix * add ascendc conv1dfn for prefill

…el README DESIGN.md is the spec source of truth. Removed docs whose content duplicates DESIGN, narrates an incident now closed, or refers to the pre-round-9 schema: - NOTES.md (incident narratives + env table duplicated in CLAUDE.md memory) - scenarios/MIGRATION_PLAN.md (migration done) - scenarios/_shared/methodology.md (N>=3 protocol now in PLAN.md §5) - scenarios/experiments_notes/{GAPS, runaway_tail, ttft_analysis, swa_assert_hypothesis, instrument_chain, algo_baselines_sim}.md - scenarios/1_swebench_default/_archive_N3_matrix_SUMMARY.md - scenarios/2_hbm_pressure/PLAN.md (duplicates top-level PLAN.md) Top-level README rewritten as a 1-page index pointing at DESIGN/PLAN/RUNBOOK and per-subdir READMEs. Methodology block added to PLAN.md §5 (N>=3 cycles, B/O alternation, fixed knobs, acceptance rule). Lingering dead links in active docs fixed; per-arm RESULTS.md frozen records left intact. Per-verify-T cleanup happens in tasks sgl-project#133-sgl-project#144 (verify/t1 ... t11 + g10 audit-and-refresh).

…, bit-identical, ~5x decode

Add a framing section 'Performance-per-watt (theoretical, ~constant board power)' after the homogeneous-hardware analysis: - Principle: a GPU under inference draws ~constant board power, so a throughput gain of X% at constant power is a ~X% perf/Watt gain (adaptive MTP as the clean deterministic example). Explicitly labelled THEORETICAL / first-order (constant-power assumption); the Dashboard's measured energy is named as the ground truth. - Per-feature table mapping each already-MEASURED throughput gain to its theoretical perf/Watt gain, baselines named precisely: adaptive MTP / EAGLE3 (net gain), GGUF K-quant kernel (prefill ~5-8x), CUDA-graph decode sgl-project#133 (+385% vs eager-weightless), MoE-offload Graph+Hotset (+134% vs eager-offload), solo-prefill PD (2-5x TTFT), rank-kv-ratio capacity (neutral decode). - Three hard honesty splits: (1) capacity/enabling features (offload to run bigger, host-tier KV-spill sgl-project#134) ADD streaming overhead -> tps/Watt can be LOWER than a fits-in-VRAM run; value is 'runs at all'; the +134% is a gain WITHIN the offload path vs eager-offload, not absolute efficiency. (2) spec-decode perf/Watt never above the net throughput gain; low acceptance can lower it -> adaptive MTP avoids that. (3) all theoretical, ~constant-power assumed, Dashboard = ground truth. Section numbers unchanged; TOPOLOGIES.md untouched.

…decode (bucketed rung ladder) Lifts the B0 eager-only gate: the weightless lane's streaming block-decode (B0 block loop + B1 host-spill H2D streaming) is now CUDA-graph captured, using the mechanism de-risked byte-identical in b2_graph_derisk.py: - Per-bs-bucket pool of persistent cuda-graph-mode flashinfer block wrappers (fixed indptr/indices buffers), shared by every rung; all per-block .plan() + index/staging-map construction hoisted OUT of the graph into init_forward_metadata_out_graph (once per STEP, vs per layer x block on the eager path -- that hoist, plus baking out the per-collective guard-gloo, is where the speedup comes from). - Captured fixed-count block loop per rung: {H2D staging copy (live pinned- host gather via transfer_kv_per_layer) + pre-planned wrapper.run + merge_state} x R; no .plan(), no host sync in-graph. Empty blocks are planned zero-length at replay and sanitized in-graph to the (o=0, lse=-inf) empty-attention contract from the live device indptr (merge_state folds them in as identity; all-empty prefixes re-pinned after each merge). - Bucketed capture LADDER over the block count R = ceil(seq/(B*dcp)) (graphs cannot branch): dense step-1 rungs to 8, then ~x1.5 geometric to the context max. Replay picks the smallest covering rung. Dense low rungs matter: one wasted trailing block region halves the streaming rate (measured 26 -> 11 tok/s), because captured H2D copies are fixed-count. - Spill graphs anchor blocks to SLOT WINDOWS (block j = owned slot values [jB,(j+1)B)), so the per-block host/device copy template is exact for any OFFSET-LINEAR allocation (fresh request, radix-cached prefix). Admission (wl_graph_can_replay) verifies bs==1 + contiguous req_to_token; any miss or over-ladder seq falls back rank-uniformly to the eager block loop (guard-free for decode under graphs-enabled, sgl-project#133 rule). - Graph-safe owner-write (_wl_spill_owner_write_graph): fixed-shape masked device write + unconditional stage of all T rows + fixed-count D2H gather whose dst is the real host slot or a reserved dump slot (one past the logical host tier, never read) -- the eager .any()/.nonzero() split is unrecordable. - Head + weightless workers capture/replay SYMMETRICALLY (sgl-project#133): identical (bs, rung) capture sequence, rung chosen from the shared global seq len -> the 4 DCP collectives/layer pair up by construction. Graph key carries the rung via the ShapeKey variant label ("wlblk{R}"). - capture_bs capped (SGLANG_WL_GRAPH_MAX_BS, default 1; spill forces 1); server_args eager-only gate removed; --disable-cuda-graph still selects the pure eager path unchanged. Default (non-lane) paths untouched -- all hooks are gated on the lane's block-decode flags. VALIDATION (TP=3/DCP=3, head=5090 + 2x3080, Qwen3.6-27B Q3_K_M GGUF, block 1024, spill 64000, device-cap 2048 forcing host streaming, ctx 16384; ladder [1..6], capture 4.2 s): - BYTE GATE: graph-decode trajectory vs eager-boot baseline (same config, --disable-cuda-graph), 8118-token prompt + 70 greedy tokens: argmax-IDENTICAL, 0 flips. - SELF-DET: 5/5 identical graph trajectories. - NO-HANG: 1200-token in-graph decode crossing rung 4->5 at seq 12289 (coherent, deterministic); separate 1200-token over-ladder run exercising the rank-uniform graph->eager fallback (radix offset pushes the slot-window rung past the ladder) -- also clean. - tok/s (streaming decode, host spill active): eager boot 9.05-9.15; same-boot guard-free eager fallback 10.1-10.6; GRAPH exact-rung 24.7-26.1 (+173% vs eager boot at seq 8.1-8.4k, +134% vs guard-free eager at 12.2k); freshly-entered rung (template blocks still mostly empty) 13.0 -- still above eager. PCIe H2D wall remains the ceiling -> #136b prefetch.

…weightless lane The weightless-KV workers hold a meta-device model (zero weights, the VRAM win). Decode-graph capture crashed because decode_cuda_graph_runner captures model.forward directly, bypassing _forward_raw's is_weightless_worker interception -> the worker ran the full meta-forward and dereffed an uninitialized meta embed_tokens. Fix (symmetric, decode-only): the worker captures the stripped per-full-attention-layer DCP dispatch (forward_decode_weightless_worker) as its OWN decode graph, so head + worker both replay graphs and their capture-time DCP collectives (KV all-gather x2, Q all-gather, LSE-merge) stay in lockstep. The head cannot capture alone -- its capture-time DCP collectives need worker co-participation. - decode_cuda_graph_runner: _capture_one_shape_weightless records the per-layer dispatch (metadata prepped out-of-graph via init_forward_metadata_out_graph(in_capture=True); sentinel None output the backend stores/replays); execute() maps a None replay output to a logits-free result. Guard is disabled for the whole capture (its per-step gloo handshake is not capturable). - model_runner: worker replays its decode graph in _forward_raw when the batch is graph-eligible (else eager dispatch). Rank-uniform guard rule: graphs-enabled => DCP guard OFF for decode on both ranks (a captured graph is guard-free by construction); guard stays ON for eager prefill/extend. Weightless lane routes PREFILL to the eager runner (decode-graph-only). - All new behavior strictly gated on is_weightless_head/is_weightless_worker; head-only and normal paths byte-for-byte unchanged. Validation (Qwen3.6-27B-Q3_K_M GGUF, TP=3 DCP=3, head on 5090 + 2x3080): - Graphs ON: clean capture on all 3 ranks + serve (previously crashed at capture). - LONG decode 512 tokens: NO HANG (sustained head<->worker graph-replay lock-step held). - Decode-class byte gate vs eager-weightless: output_ids IDENTICAL, max contender |d|=0.000e+00, argmax-clean (no flips over 48 steps). - self-det 5/5 PASS (graph and eager). - Decode tok/s: graph 63.5 vs eager 13.1 = +385%.

…decode (bucketed rung ladder) Lifts the B0 eager-only gate: the weightless lane's streaming block-decode (B0 block loop + B1 host-spill H2D streaming) is now CUDA-graph captured, using the mechanism de-risked byte-identical in b2_graph_derisk.py: - Per-bs-bucket pool of persistent cuda-graph-mode flashinfer block wrappers (fixed indptr/indices buffers), shared by every rung; all per-block .plan() + index/staging-map construction hoisted OUT of the graph into init_forward_metadata_out_graph (once per STEP, vs per layer x block on the eager path -- that hoist, plus baking out the per-collective guard-gloo, is where the speedup comes from). - Captured fixed-count block loop per rung: {H2D staging copy (live pinned- host gather via transfer_kv_per_layer) + pre-planned wrapper.run + merge_state} x R; no .plan(), no host sync in-graph. Empty blocks are planned zero-length at replay and sanitized in-graph to the (o=0, lse=-inf) empty-attention contract from the live device indptr (merge_state folds them in as identity; all-empty prefixes re-pinned after each merge). - Bucketed capture LADDER over the block count R = ceil(seq/(B*dcp)) (graphs cannot branch): dense step-1 rungs to 8, then ~x1.5 geometric to the context max. Replay picks the smallest covering rung. Dense low rungs matter: one wasted trailing block region halves the streaming rate (measured 26 -> 11 tok/s), because captured H2D copies are fixed-count. - Spill graphs anchor blocks to SLOT WINDOWS (block j = owned slot values [jB,(j+1)B)), so the per-block host/device copy template is exact for any OFFSET-LINEAR allocation (fresh request, radix-cached prefix). Admission (wl_graph_can_replay) verifies bs==1 + contiguous req_to_token; any miss or over-ladder seq falls back rank-uniformly to the eager block loop (guard-free for decode under graphs-enabled, sgl-project#133 rule). - Graph-safe owner-write (_wl_spill_owner_write_graph): fixed-shape masked device write + unconditional stage of all T rows + fixed-count D2H gather whose dst is the real host slot or a reserved dump slot (one past the logical host tier, never read) -- the eager .any()/.nonzero() split is unrecordable. - Head + weightless workers capture/replay SYMMETRICALLY (sgl-project#133): identical (bs, rung) capture sequence, rung chosen from the shared global seq len -> the 4 DCP collectives/layer pair up by construction. Graph key carries the rung via the ShapeKey variant label ("wlblk{R}"). - capture_bs capped (SGLANG_WL_GRAPH_MAX_BS, default 1; spill forces 1); server_args eager-only gate removed; --disable-cuda-graph still selects the pure eager path unchanged. Default (non-lane) paths untouched -- all hooks are gated on the lane's block-decode flags. VALIDATION (TP=3/DCP=3, head=5090 + 2x3080, Qwen3.6-27B Q3_K_M GGUF, block 1024, spill 64000, device-cap 2048 forcing host streaming, ctx 16384; ladder [1..6], capture 4.2 s): - BYTE GATE: graph-decode trajectory vs eager-boot baseline (same config, --disable-cuda-graph), 8118-token prompt + 70 greedy tokens: argmax-IDENTICAL, 0 flips. - SELF-DET: 5/5 identical graph trajectories. - NO-HANG: 1200-token in-graph decode crossing rung 4->5 at seq 12289 (coherent, deterministic); separate 1200-token over-ladder run exercising the rank-uniform graph->eager fallback (radix offset pushes the slot-window rung past the ladder) -- also clean. - tok/s (streaming decode, host spill active): eager boot 9.05-9.15; same-boot guard-free eager fallback 10.1-10.6; GRAPH exact-rung 24.7-26.1 (+173% vs eager boot at seq 8.1-8.4k, +134% vs guard-free eager at 12.2k); freshly-entered rung (template blocks still mostly empty) 13.0 -- still above eager. PCIe H2D wall remains the ceiling -> #136b prefetch.

…ogy SVGs; add runtime-feature diagrams TOPOLOGIES.md 2.7: TP=5 co-location was booted+validated on the 3-card rig (--rank-gpu-id 0,0,0,1,2, 3 ranks time-slicing the 5090 via MPS, 1 rank/3080, bit-identical across two boots); kv-boundary-aware auto planner (sgl-project#116). Replaces the incorrect "TP=5 not booted here / TP=4 first boot" wording. gen.py: rework diagram 07 to the booted TP=5 layout (3 co-located ranks, MPS time-slice). Neutralize upstream-comparison wording across all paired diagrams (stock->upstream; drop WASTED/OOM/impossible/refuses/throttles/fails) to match the neutral prose. Update weightless diagram 06 (graph-decode landed sgl-project#133/sgl-project#136, 262k measured, not eager-only/"4x"). Align offload correctness/perf wording with the corrected prose (self-deterministic, not bit-identical; factors not benchmarked). Add four runtime-feature diagrams + Part 3 sections: - 17 adaptive drafter routing (NEXTN/DFLASH dual residence, policy vs bandit, ctx-gate) (§5) - 18 session KV spill (newest-first host offload, keeps decoding; S1 measured vs modeled curves) (§20) - 19 fast-lane priority scheduling (reserved-heavy floor, aging, spill interplay) (§16) - 20 measured VRAM budget (component registry, KV as measured remainder, corridor rule) (§10) All 20 SVGs regenerated and validated as well-formed XML.

…e weightless-KV lane The lane's decode step is graph-captured and guard-free (sgl-project#133, +385%), and costs the same four collectives per full-attention layer no matter how many tokens it commits. Chain spec turns one such step into 1..k+1 committed tokens for that same collective count. Composed, not rebuilt. The lane and --speculative-draft-placement solo are the same shape with different names (weightless_kv_head_rank plays _spec_solo_rank), and model_runner already routes both through the same weight-TP=1 override and the same meta-model build. So spec is admitted only with solo placement hosted on the lane's head rank: the draft phase is then one fixed-shape broadcast per round and zero draft forwards on the workers, with no new mechanism. The verify that follows is a plain extend over bs*(k+1) rows, which the lane's extend dispatch already mirrors. The lockstep invariant, now stated in one place (layers/dcp/lockstep.py): the number, order and op-tags of a weightless worker's DCP collectives are a function of (forward_mode, has_prefix, layer count) only -- accept_len is not an input. It enters as VALUES (seq_lens += accept_lens, kv_committed_len += n), never as control flow, and reaches the workers by broadcast so none of them can derive a different schedule. Verify's op-tag tuple is therefore identical to decode's, which is what makes a symmetric head+worker verify capture legal. Six gaps closed: - server_args: the Stage-1 blanket "no speculative decoding" reject becomes a set of named rejects (_reject_unsupported_weightless_spec), and the blanket CUDA "DCP + spec" reject in _handle_dcp_validation gains a weightless disjunct -- relaxing only the first would have been useless since the lane forces dcp_size == tp_size and is never uneven_WEIGHTED. The sgl-project#139 topk>1 and tree-verify guards are untouched, as their comments instructed. Admission runs after handle_speculative_decoding so it reads the resolved alias (NEXTN -> EAGLE) and the resolved solo rank. - model_runner: is_weightless_head/worker are never set on a DRAFT runner. They name a role in the TARGET forward; leaving them set crossed the two and would have routed a draft forward into the stripped weightless dispatch, whose first line asserts a weightless_kv backend. Latent until now only because spec was rejected at boot. - tp_worker: the is_verify early return now precedes the weightless gloo receive. On a verify step the head skips sampling and never reached the matching send, so every verify step would have left the workers blocked in broadcast_pyobj forever. Under spec the tokens come from the accept broadcast instead; the non-spec path is unchanged. - eagle_sample: receive-only mode for a weightless worker (no logits, so no accept kernel) -- it allocates the three results at the boot-fixed shapes and enters the SAME rank-0 broadcast, deliberately not a second channel. src is generalised from a hard-coded 0 to the lane's head rank; with --weightless-kv-head-rank != 0 the constant would have published garbage accepts silently. SGLANG_SIMULATE_ACC_LEN is refused on the lane (its rewrite happens after the broadcast and cannot be reproduced). - verify(): the worker skips the nan/inf probes, the logprob computation and the mamba commit (it never runs GDN); everything else is tensor/CPU bookkeeping over replicated inputs and runs identically on both sides. - the guard rule keys on is_decode_or_idle() OR is_target_verify(), and the worker's captured body follows capture_forward_mode -- a verify capture must record the EXTEND dispatch, which reads prefill_wrappers where the decode dispatch reads decode_wrappers. Also de-duplicated: the verify-first prefix rule lived as two verbatim copies in flashinfer_backend (head and worker). Two copies of a rule whose two answers must agree is the drift surface of the D5 defect family; one expression, two callers, pinned by test. V1 exclusions, each rejected by name rather than hoped around: topk > 1, rejection sampling, FROZEN_KV_MTP, non-EAGLE families, split placement, a solo rank that is not the head rank, adaptive k (one capture shape per boot), and the block-decode / host-spill tier (its graph ladder is captured over block count for a decode-shaped step; a verify's bs*(k+1) rows are a second axis nobody has built). Tests (CPU, no GPU available this session): - test_weightless_chain_spec.py, 16 new cases: collective-sequence invariance across accept=0..k, verify==decode in schedule, op-tags pinned against comm.py's source, the prefix gate answering forward_mode first, accept-broadcast geometry, and source pins on the four wiring decisions. - test_uneven_tp_args.py, 9 new cases: the admitted chain config boots through both handlers; split placement, a mismatched solo rank, non-EAGLE families, adaptive k, the block loop and topk>1 each raise with their reason. - Regression: unit/server_args + unit/distributed + unit/spec = 1218 passed against 1193 at integration/r3-probe, with the SAME 7 pre-existing environment failures (vmm LOCAL_RANK, MPS pipe dir, kv_session_offload bookkeeping allowlist) on both sides. unit/planner + unit/managers identical to baseline (787 passed / 12 pre-existing failures). - layers/dcp/test_weightless_kv_math.py green. NOT validated on hardware: no GPU window this session. The symmetric verify capture has never been captured on a device, and it is the one piece the feature's value depends on -- an eager verify at ~13 tok/s times an accept length of ~2.5 is a regression against the 63.5 tok/s graph decode it replaces. GPU recipe R0-R6 (token identity at temp 0, accept length, self-determinism, tok/s, the 1-token D5 provocation, EOS soak) is in docs_new/weightless_chain_spec.md, which is the authoritative feature file.

…y-swa-20261014 [Cherry-pick] Bring unified-memory and SWA fixes into dev

Ying1123 added 3 commits February 3, 2024 09:45

debugging

6071503

add server arg max_prefill_num_token

0fb5550

update

12f7a82

Ying1123 merged commit e095b16 into main Feb 3, 2024

Ying1123 deleted the ying-debug branch February 3, 2024 10:35

CSEEduanyu mentioned this pull request Jan 26, 2025

[Bug] NCCL Crash with SIGSEGV Frequently when deploying deepseek v3 #2803

Closed

5 tasks done

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Add max_prefill_num_token into server arguments (sgl-project#133)

6ddcf66

pi314ever pushed a commit to pi314ever/sglang that referenced this pull request Sep 5, 2025

fix bucket info differences (sgl-project#133)

a4a98d6

Garrybest pushed a commit to Garrybest/sglang that referenced this pull request Jan 9, 2026

fix (sgl-project#133)

5659664

* enhance trace tools

McZyWu added a commit to McZyWu/sgl-sglang that referenced this pull request Apr 7, 2026

opt:ascendc_conv1d_update support (sgl-project#133)

af1b6e9

* opt:ascendc_conv1d_update support * bugfix * bugfix * add ascendc conv1dfn for prefill

efschu added a commit to efschu/htsglang that referenced this pull request Jul 20, 2026

[docs] Weightless-KV lane: CUDA-graph decode landed (sgl-project#133)…

bfa4e50

…, bit-identical, ~5x decode

Jiminator added a commit that referenced this pull request Oct 16, 2026

Merge pull request #133 from radixark/codex/cherry-pick-unified-memor…

8f94384

…y-swa-20261014 [Cherry-pick] Bring unified-memory and SWA fixes into dev

This was referenced Nov 4, 2026

Linhu/kvcr 45650 land ziqifan617/sglang#6

Merged

KVCR linker: update README for the KeyAdapter split and Guard CLI ziqifan617/sglang#8

Merged

Navigation Menu

Repository navigation

Add max_prefill_num_token into server arguments - #133

Add max_prefill_num_token into server arguments#133
Ying1123 merged 3 commits into
mainfrom
ying-debug

Ying1123 commented Feb 3, 2024 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Ying1123 commented Feb 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Ying1123 commented Feb 3, 2024 •

edited

Loading