[Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path by YAMY1234 · Pull Request #45450 · sgl-project/sglang

YAMY1234 · 2026-10-20T22:30:08Z

Motivation

DeepSeek-V4.1-Flash low-latency decode on GB300 (TP4, DSpark speculative decoding). With the NVIDIA megakernel plugin enabled (--enable-nvidia-megakernel), the GPU part of each decode step gets short enough that SGLang's host-side work between graphs (input preparation, verify metadata, accept bookkeeping, KV commits) becomes a large share of the step. This PR moves that work into the existing CUDA graphs, into fewer launches, or off the critical path.

Most changes are behind environment flags that default to off. Two DSpark-only changes are on by default: the draft sampling-param restaging (1) and the draft runner's metadata prep through the glue graph where it applies (2). The two changes that reach beyond DSpark are opt-in: the MoE output-shape fix (3, SGLANG_OPT_MXFP4_DEFERRED_FINALIZED_SHAPE) and the CUDA graph fill fast paths (7, SGLANG_OPT_CUDA_GRAPH_FILL_FAST_PATH).

Modifications

One commit per change:

DSpark draft: restage the draft graph's sampling params only when they change.
DSpark draft runner: prepare its attention metadata through the glue graph.
MXFP4 trtllm-gen MoE (SGLANG_OPT_MXFP4_DEFERRED_FINALIZED_SHAPE): pass the finalized output shape on deferred-finalize calls above 32 rows, so decode batches hit the tuned tactic instead of the default tile. Off keeps main's 97-384-row window.
Fused static accept tail (SGLANG_DSPARK_FUSED_ACCEPT_TAIL): the static greedy accept and its bookkeeping in one kernel.
One-launch step inputs (SGLANG_OPT_DSV41_DSPARK_PREPARE_INPUTS_ONCE, SGLANG_OPT_DSV41_DSPARK_MID_FOLD): the step inputs in one launch; the verify ids come from the draft graph's tail.
Deferred draft context-KV commit (SGLANG_DSPARK_DEFER_CONTEXT_KV): the static verify's draft context-KV commit moves to the next draft graph.
CUDA graph replay fill fast paths (SGLANG_OPT_CUDA_GRAPH_FILL_FAST_PATH): skip the padding reset of an unpadded fill; copy one dense integer dtype pair with one foreach launch.
Target verify metadata through the glue graph (SGLANG_OPT_DSV41_TARGET_VERIFY_METADATA_GLUE).
Engram history commit and block accept lengths inside the verify graph (SGLANG_OPT_DSV41_POST_FORWARD_CUDA_GRAPH).
Draft graph metadata on a side stream (SGLANG_OPT_DSV41_DSPARK_DRAFT_PREP_FORK).
Seq lens and bonus relayed from the verify graph (SGLANG_OPT_DSV41_DSPARK_POST_FOLD).
The deferred context KV projected inside the verify graph, so that only its stores are deferred.
Simulated bonus: reads the shared accept length as a stride-0 view. Simulated acceptance only.
The deferred commit's staging in one launch (SGLANG_OPT_DSV41_DSPARK_ONE_LAUNCH_COMMIT_STAGE).
Target verify metadata built beside the draft graph (SGLANG_OPT_DSV41_TARGET_METADATA_EARLY; enabled only when the runner's conditions hold, including the metadata glue graph).
A pre-planned verify replay skips its input re-copies (SGLANG_OPT_DSV41_DSPARK_SKIP_PREFILLED_INPUTS).
Skip the replay ForwardBatch view once the metadata glue graph is captured (SGLANG_OPT_SKIP_GLUED_REPLAY_VIEW).
Draw the simulated acceptance before the draft launch (SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD). Simulated acceptance only.
Block-wise rejection sampling for the sampled verify accept (SGLANG_OPT_DSV41_BLOCKWISE_REJECTION).
A CUDA graph for the static verify's temperature-sampling accept (SGLANG_OPT_DSV41_SAMPLING_ACCEPT_GRAPH): the sampling accept, its TP broadcast and the KV / engram commits replay as one graph launch.

Accuracy Tests

GB300, TP4, DeepSeek-V4.1-Flash, megakernel plugin on, all flags above on, real acceptance (commands under Reproduce below):

check	all 20 commits on (incl. 19-20)	commits 19-20 off
GSM8K-200 c8, greedy	196 / 200, accept length 4.26	198 / 200, accept length 4.29
GSM8K-200 c8, temperature 1.0	198 / 200, accept length 4.17	196 / 200, accept length 4.07
needle at ~155K tokens	26 / 27	24 / 27

Both commits were confirmed active in the first run (block-wise rejection on sampled verifies, the sampling-accept region captured, 4 / 4 ranks). The needle test varies between 22 and 27 / 27 from run to run on identical code.

Speed Tests and Profiling

GB300, TP4, DeepSeek-V4.1-Flash. AgentX (InferenceX agentic traces), concurrency 1, 3,600 s, synthetic acceptance 3.51 with the stock accept sampling (the simulated-acceptance shortcut is reverted in every arm). All arms run on SGLang main f0b65907227 with the same image and settings, and are compared on the same 220 requests. TPS/User = 1 / ITL.

configuration	TPS/User P50	TPS/User P90	decode step p50 (> 256K)
main	694.0	676.1	5.06 ms
main + this PR (commits 1-18)	709.7 (+2.3%)	694.1 (+2.7%)	4.95 ms
main + megakernel plugin (sgl-project/sgl-nvidia-mega#18, default slots)	762.2 (+9.8%)	717.4 (+6.1%)	4.57 ms
main + this PR (commits 1-18) + plugin (#18, 42 slots)	952.0 (+37.2%)	902.2 (+33.4%)	3.66 ms
main + this PR (all 20 commits) + plugin (#18, 42 slots)	995.7 (+43.5%)	972.8 (+43.9%)	3.54 ms

Rows with this PR were measured with the changes listed above enabled, including SGLANG_OPT_MXFP4_DEFERRED_FINALIZED_SHAPE=1 and SGLANG_OPT_CUDA_GRAPH_FILL_FAST_PATH=1 (on by default at measurement time). With the plugin held fixed, this PR adds +30.6% P50 / +35.6% P90 (rows 3 and 5). Rows 4 and 5 also enable three opt-in plugin slots. Without the plugin, the host work this PR removes is hidden behind the longer GPU step, so the PR alone moves little. Rows 1-3 repeat on a second cluster within 0.3% at P50.

Known interaction: with NCCL_CUMEM_ENABLE=1 and NCCL_MNNVL_ENABLE=1 set in the environment, the deferred context-KV commit (SGLANG_DSPARK_DEFER_CONTEXT_KV) adds a 45-105 ms stall before the last streamed token of requests above 256K tokens (87 / 209 such requests in one AgentX run). The stall grows with context length. With both NCCL variables at 0, no request stalls (0 / 209) and P90 rises 4.5%. The table above was measured without these variables.

Reproduce

Code and setup. One GB300 node (4 GPUs), image lmsysorg/sglang:nightly-dev-cu13-20261020-ab580b2e, model deepseek-ai/DeepSeek-V4.1-Flash (revision df42c109f1d). Every tree is on main at f0b65907227 with #45421 reverted (git revert 7d9bc104932): under SGLANG_SIMULATE_ACC_LEN that commit skips the accept sampling on CUDA.

tree	rows
`f0b65907227` + the revert	Speed 1 and 3
commits 1-18 on the same base (YAMY1234/sglang `98891329ced`)	Speed 2 and 4
all 20 commits on the same base (YAMY1234/sglang `227d2a4934e`)	Speed 5, Accuracy

The current head carries the same commits on a newer main plus review fixes. On it, also set SGLANG_OPT_MXFP4_DEFERRED_FINALIZED_SHAPE=1 and SGLANG_OPT_CUDA_GRAPH_FILL_FAST_PATH=1 (commits 3 and 7 had no switch when measured). Plugin for Speed 3-5 and Accuracy: sgl-project/sgl-nvidia-mega 547bf4486e9, installed over the image's copy with pip install --no-deps -e .; it stays inactive without --enable-nvidia-megakernel.

Server (Speed Tests).

# all rows
export SGLANG_SPEC_TP_SYNC=rng,-dspark-accept-sample
export SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank
export SGLANG_OPT_DSV41_MEGA_MHC_PREFILL=1
export SGLANG_SIMULATE_ACC_LEN=3.51 SGLANG_SIMULATE_ACC_METHOD=match-expected SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token
export SGLANG_DEFAULT_THINKING=1 SGLANG_DSV41_REASONING_EFFORT=high SGLANG_TIMEOUT_KEEP_ALIVE=900
# rows 2, 4 and 5: this PR's switches
export SGLANG_DSPARK_FUSED_ACCEPT_TAIL=1 SGLANG_DSPARK_DEFER_CONTEXT_KV=1 \
  SGLANG_OPT_DSV41_DSPARK_PREPARE_INPUTS_ONCE=1 SGLANG_OPT_DSV41_DSPARK_MID_FOLD=1 \
  SGLANG_OPT_DSV41_DSPARK_POST_FOLD=1 SGLANG_OPT_DSV41_DSPARK_DRAFT_PREP_FORK=1 \
  SGLANG_OPT_DSV41_DSPARK_ONE_LAUNCH_COMMIT_STAGE=1 SGLANG_OPT_DSV41_DSPARK_SKIP_PREFILLED_INPUTS=1 \
  SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD=1 SGLANG_OPT_DSV41_TARGET_VERIFY_METADATA_GLUE=1 \
  SGLANG_OPT_DSV41_TARGET_METADATA_EARLY=1 SGLANG_OPT_DSV41_POST_FORWARD_CUDA_GRAPH=1 \
  SGLANG_OPT_SKIP_GLUED_REPLAY_VIEW=1
# row 5 also
export SGLANG_OPT_DSV41_BLOCKWISE_REJECTION=1 SGLANG_OPT_DSV41_SAMPLING_ACCEPT_GRAPH=1
# rows 4 and 5: the plugin's 39 default slots + target_lm_head, hop_b_quad_trigger, lm_head_pinned_tactic
export SGLANG_NVIDIA_MEGAKERNEL_SLOTS=hop_a,moe,gate,attn,embed,two_shot,markov,draft_lm_head,aux_mean_stats,gate_wide,shared_ll,hop_a_batch2,engram,attn_qkv_a,attn_qkv_a_bf16,index_q_mxfp8_in,attn_indexer,hc_prenorm,hop_b,hop_b_wide,early_trigger,hop_b_start_trigger,draft,draft_moe,draft_gate,draft_tail,draft_post,draft_seam,draft_attn,draft_qfront,draft_wo_a,draft_swapab,draft_qkv_a,draft_prenorm,draft_qkv_a_ll,draft_wo_b_ll,sched_ahead,req_pos_once,kv_before_wait,target_lm_head,hop_b_quad_trigger,lm_head_pinned_tactic

python3 -m sglang.launch_server --model-path <DeepSeek-V4.1-Flash path> \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash --host 0.0.0.0 --port 8000 \
  --trust-remote-code --tp 4 --ep-size 1 --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.8 \
  --chunked-prefill-size 8192 --context-length 1048576 --max-running-requests 1 \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-decoder-swa-bounded-replay \
  --dsv4-attn-backend flashmla --cuda-graph-max-bs-decode 64 --cuda-graph-backend-prefill breakable \
  --cuda-graph-bs-prefill 128 256 512 1024 2048 3072 4096 6144 8192 --prefill-decode-interval 0 \
  --tokenizer-backend fastokens --reasoning-parser auto --tool-call-parser auto \
  --enable-cache-report --enable-metrics --watchdog-timeout 7200
# rows 3-5 add --enable-nvidia-megakernel

AgentX client (SemiAnalysisAI/InferenceX ea0db0384, benchmarks/srt_agentic.sh, run with CONC=1 DURATION=3600):

pip install agentx-harness==1.0.11 "datasets>=4.7.0"
AIPERF_HTTP_TCP_USER_TIMEOUT=900000 AIPERF_EXPERIMENTAL_FAST=0 AIPERF_FAILED_REQUEST_THRESHOLD=0.10 \
agentx profile --url http://<server>:8000 \
  --model deepseek-ai/DeepSeek-V4.1-Flash --tokenizer deepseek-ai/DeepSeek-V4.1-Flash --tokenizer-trust-remote-code \
  --concurrency 1 --benchmark-duration 3600 --warmup-grace-period 3600 \
  --server-metrics http://<server>:8000/metrics --output-artifact-dir <out> \
  --public-dataset semianalysis_cc_traces_weka_062126

TPS/User P50 / P90 = 1 / p50 and 1 / p90 of the per-request ITL in profile_export.jsonl (profiling phase, failed requests dropped). Every row is read on the same 220 requests, matched by conversation and turn; over each whole run (391-492 requests) the values are within 0.8% of the table. Decode step p50 (> 256K) comes from TP0's Decode batch lines with one running request: accept length / gen throughput, lines above 262,144 tokens.

Accuracy Tests. Tree 227d2a4934e, the plugin with the 42 slots above, real acceptance (no SGLANG_SIMULATE_ACC_*), the server args above with --max-running-requests 256. The two columns differ only in SGLANG_OPT_DSV41_BLOCKWISE_REJECTION and SGLANG_OPT_DSV41_SAMPLING_ACCEPT_GRAPH (1 vs 0). These runs used image nightly-dev-cu13-20261012-e6072935 with FlashInfer 0.7.2 and sglang-kernel 0.4.10, the versions in the image above.

GSM8K-200: the first 200 openai/gsm8k (main) test questions, 5-shot from the first five train examples, DeepSeek-V4.1 chat format with thinking off, sent to /generate with max_new_tokens 512 and 8 requests in flight, temperature 0.0 or 1.0. An answer is correct when its last number equals the reference's. Accept length = Δsglang:generation_tokens_total / Δsglang:spec_verify_calls_total over the pass.
Needle: a ~155K-token haystack of numbered records and 27 greedy questions (depths 2% to 94% in 4% steps, plus 10%, 50% and 90%), sent to /v1/completions with max_tokens 16 after a cache flush.

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): 🚫 Run #39872814449
Latest PR Test (Extra): ❌ Run #39872814456
Latest PR Test (AMD ROCm 10): ⏳ Run #39872814641

YAMY1234 · 2026-10-21T02:14:05Z

@Oasis-Git Two more commits are now appended to this PR (7036147, 02d5caf):

Block-wise rejection sampling for the sampled verify accept (SGLANG_OPT_DSV41_BLOCKWISE_REJECTION, default off).
A CUDA graph for the static verify's temperature-sampling accept (SGLANG_OPT_DSV41_SAMPLING_ACCEPT_GRAPH, default off). The sampling accept, its TP broadcast and the KV / engram commits replay as one graph launch instead of about 25 eager kernels.

AgentX c1, GB300 TP4, SGLang main f0b65907227 with the megakernel plugin (sgl-project/sgl-nvidia-mega#18, 42 slots), same image and settings in every arm, compared on the same 220 requests, TPS/User P50 / P90:

configuration	P50	P90
main	694.0	676.1
main + this PR without the two commits	952.0 (+37.2%)	902.2 (+33.4%)
main + this PR with the two commits enabled	995.7 (+43.5%)	972.8 (+43.9%)

The two commits add +4.6% P50 / +7.8% P90 on top of the rest of the PR. CPU tests and GPU kernel tests pass on both commits (GPU: 168 / 168).

Note: on a second cluster, long-context requests showed an extra 45-105 ms stall before the last streamed token; it follows that cluster's NCCL cuMem / MNNVL settings, not these commits, and the numbers above are from the cluster without it.

…is captured A captured metadata glue graph reads runner slots and backend buffers by address, so a replay whose key it has captured needs neither the replay ForwardBatch view nor the out_cache_loc re-point; building the view costs host time on every step, in the target verify and the draft runner alike, between the draft and the verify. SGLANG_OPT_SKIP_GLUED_REPLAY_VIEW (default off): the decode graph runner's replay metadata prep, now _init_replay_attn_metadata, replays SGLang's own captured glue graph without the view and builds the view only for the glue's warm-ups and capture or the eager prep. A staged early verify metadata step only reinstalls. The routing guards and keys are unchanged; a plugin-supplied glue still receives run(view, key).

…launch SGLANG_OPT_DSV41_DSPARK_SIM_DRAW_AHEAD (default off; inert without SGLANG_SIMULATE_ACC_LEN): the step's simulated correct length and, under the simulate-fold bonus, its temperature / seed staging run before the draft graph launch instead of between the draft and the target verify. At small batches the host is the critical path in that window, while before the draft it waits on the previous verify graph anyway. Neither write reads anything the draft produces: the draw uses the host RNG, the bonus staging the batch size and request temperatures; the draft path draws no host randomness, so fixed seeds give the same values in either order. The sim bonus buffers are read only by a folded verify accept, so a step that turns out not fold-eligible ignores them. Real acceptance (the ACC gate) is unchanged.

… verify accept A sampled DSpark verify step softmaxed the full draft and target logits and ran the single-CTA classic rejection kernel. The block-wise sampler reads per-vocab- block max / sum-exp stats instead, accepts with one warp per request, and resamples the rejected row's residual with a block-parallel Gumbel-max. It draws (accept length, bonus) from the exact chain speculative-sampling law; only the random stream differs. Its draws are counter-based: a function of the server seed, a device step counter, row and token. Every TP rank holds the same seed, counter and logits, so the ranks agree without an accept broadcast, and a CUDA graph replay advances the counter like an eager call. Scope: CUDA, no top-k / top-p, no block verification; anything else keeps the classic kernel. Behind SGLANG_OPT_DSV41_BLOCKWISE_REJECTION (default off).

…re-sampling accept The static verify graph folds only the greedy accept. On an all-sampled step the rest ran eagerly after the replay: draft and target softmax, the rejection kernel, the bonus gather, the accept-site TP broadcast, the engram history commit, the future relay and the target-hidden KV commit. With the megakernel verify shorter than that host enqueue, the GPU idled on the host after every verify. SGLANG_OPT_DSV41_SAMPLING_ACCEPT_GRAPH (default off) captures that region once per verify-graph batch size and replays it right after the verify graph. It reads the verify graph's static inputs and outputs (StaticVerifyIO) and the draft sampler's static buffers, so a replay stages nothing, and its outputs land in the epilogue's accept buffers like the greedy fold. The region and the eager accept share one RejectionNoise, so a replay draws what the eager block-wise accept draws at the same counter. Greedy steps, top-k / top-p, logit adjustments, grammar and a simulated sampled bonus keep the eager accept.

nvpohanh

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

[by Claude Code] Severity: style | Confidence: High

Issue

The Accuracy Tests and Speed Tests sections report results but give no commands, so others cannot rerun them. "All flags above on" does not list the env vars and values used, and the DSpark speculative arguments and megakernel plugin settings are not shown. The isolated A/B and DEP2 comments have the same gap.

Fix

For each table, add the exact server launch command (model path, TP, DSpark draft and speculative arguments, every env var with its value, and the plugin setup), plus the GSM8K, needle, and AgentX benchmark commands with their settings.

nvpohanh · 2026-10-22T07:14:09Z

+        self._skips_prefilled_inputs = self._resolve_skips_prefilled_inputs()
+        if self._draft_metadata_glue or self._target_metadata_glue:
+            self._metadata_glue = MetadataGlueGraph(self.device)
+            atexit.register(self._log_draft_glue, " at shutdown")


[by Claude Code] Severity: functional | Confidence: Medium

Issue

atexit.register(self._log_draft_glue, ...) holds a strong reference to this runner until the process exits. The draft glue is on by default for the DSpark draft runner, so that runner, its CUDA graphs and its buffers can never be freed.

Example

A weight update with recapture_cuda_graph=True calls ModelRunner.recapture_decode_cuda_graph, which drops the old runner and captures a new one. The old runner stays alive through the atexit entry, so each recapture keeps another set of graphs in GPU memory and adds another shutdown log.

Fix

Remove the atexit hook; _maybe_log_draft_glue already logs the counts every 1000 steps. If the shutdown line is needed, register through a weakref.WeakMethod.

nvpohanh · 2026-10-22T07:14:09Z

+            return False
+        self.discard()
+        matches = staged.key == key and all(
+            getattr(forward_batch, name) is not None


[by Claude Code] Severity: style | Confidence: High

Issue

take reads the staged inputs with getattr(forward_batch, name). SGLang does not allow getattr here. The field names come from the STAGED_SLOTS strings, so type checks and grep cannot see which ForwardBatch fields this compares.

Fix

Store the four sources as named fields, for example a msgspec.Struct with positions, out_cache_loc, req_pool_indices and seq_lens, and compare forward_batch.positions and the others directly. The new tests repeat the pattern (test_dspark_accept_tail.py:196, test_dspark_sampling_accept_graph.py:186 and :208); msgspec.structs.asdict(...) can replace those calls.

nvpohanh · 2026-10-22T07:14:09Z

        self.out_tokens_buf = torch.zeros(
            (self.max_bs, self.stride), dtype=torch.int64, device=device
        )
+        # SGLANG_DSPARK_FUSED_ACCEPT_TAIL. With the accept-site broadcast selected, only


[by Claude Code] Severity: style | Confidence: Medium

Issue

Many new comments and docstrings run three to six dense lines and are hard to read. This one packs two conditions and a reason into one sentence. Other examples: the run_static docstring (dspark_verify.py:745), stage_verify_metadata_early (decode_cuda_graph_runner.py:1857) and _install_early_verify_metadata (deepseek_v4_backend.py:2628). The repo's .claude/rules/comment-style.md asks for one or two lines.

Fix

Keep one short line of rationale, for example # The fused tail cannot run the greedy accept's TP broadcast; use it only without the broadcast or under simulate_fold. Put longer contracts in the module docstrings, which already describe them.

…aging - Drop the glue-count atexit hook, which kept the draft runner and its graphs alive across a CUDA graph recapture; the periodic debug line stays. - Stage the early verify inputs as a named VerifyInputSources struct instead of reading ForwardBatch fields by name with getattr; tests compare structs via msgspec.structs.asdict. - Shorten the long comments and docstrings added by this PR to one or two lines.

Oasis-Git · 2026-10-22T23:35:00Z

CI pass with three known issues: https://github.com/sgl-project/sglang/actions/runs/39830030045/job/120003967874

Oasis-Git · 2026-10-22T23:38:23Z

[Codex]: As requested, run only the three previously failing tests on the updated PR head b69cdf6: DSpark decode/prefill logprob consistency, dynamic LoRA update consistency, and UNO CUDA graph parity. This branch now includes the upstream lm_head precision fix. Use /rerun-test with the three specific test methods; this is targeted validation, not a full CI rerun.

Oasis-Git · 2026-10-22T23:38:24Z

/rerun-test test/registered/core/test_basic_sanity_dspark.py::TestBasicSanityDSpark.test_logprob_decode_match_prefill test/registered/lora/test_lora_update.py::TestLoRADynamicUpdate.test_dynamic_lora_update_server test/registered/spec/uno/test_uno.py::TestUnoCudaGraph.test_ar_parity_and_nontrivial_tpf

github-actions · 2026-10-22T23:38:46Z

🚀 1-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/core/test_basic_sanity_dspark.py TestBasicSanityDSpark.test_logprob_decode_match_prefill
cd test/ && python3 registered/lora/test_lora_update.py TestLoRADynamicUpdate.test_dynamic_lora_update_server
cd test/ && python3 registered/spec/uno/test_uno.py TestUnoCudaGraph.test_ar_parity_and_nontrivial_tpf

Oasis-Git · 2026-10-23T00:21:54Z

[Codex]: Targeted CI passed on b69cdf6: all three requested H100 tests executed and passed—DSpark decode/prefill logprob consistency, dynamic LoRA update consistency, and UNO CUDA graph parity. DSpark maximum logprob differences were 0.077889 and 0.059973. This validates only these three tests, not full CI.

Run and logs.

Oasis-Git · 2026-10-23T02:07:37Z

Failed CI Pass

…ck too sgl-project#45450 added block_accept_lens_buf, written by the static accept tail and handed to GenerationBatchResult as a view of the graph buffer; the overlap readback copies it on copy_stream after the next replay may have rewritten it, the same hazard snapshot_accept covers for the other accept outputs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

github-actions Bot added deepseek npu jit-kernel labels Oct 20, 2026

YAMY1234 mentioned this pull request Oct 20, 2026

[Perf] [GB300] deepseek_v41: hop B to 48 rows, the decode indexer and shared expert in the plugin, the batch-1 LM head and hop B trigger; runs on current SGLang main sgl-project/sgl-nvidia-mega#18

Merged

3 of 5 tasks

YAMY1234 force-pushed the dsv41-mega-perf branch from 469d319 to 6ded689 Compare October 21, 2026 00:30

YAMY1234 changed the title ~~[Perf] [NVIDIA] DSv4.1 DSpark: Move the host work between decode graphs into the graphs for the megakernel path~~ [Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path Oct 21, 2026

YAMY1234 force-pushed the dsv41-mega-perf branch from 02d5caf to 2fc44e4 Compare October 21, 2026 03:17

YAMY1234 marked this pull request as ready for review October 21, 2026 04:30

YAMY1234 added 4 commits October 21, 2026 21:11

YAMY1234 force-pushed the dsv41-mega-perf branch from 88f6731 to 6410850 Compare October 22, 2026 04:24

YAMY1234 added bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed labels Oct 22, 2026

nvpohanh reviewed Oct 22, 2026

View reviewed changes

Merge upstream main and resolve decode graph capture conflicts

b69cdf6

Oasis-Git removed run-ci CI: run the baseline test suite on this PR bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed labels Oct 22, 2026

Oasis-Git merged commit c82f714 into sgl-project:main Oct 23, 2026
140 of 210 checks passed

weireweire mentioned this pull request Oct 23, 2026

[Fix] docker: Pin sgl-nvidia-mega to 3333e4c so the megakernel starts on main #46030

Merged

akhilg-nv mentioned this pull request Oct 23, 2026

[Perf] DSpark: Replay the static verify epilogue only on steps that take its accept #46064

Closed

4 of 5 tasks

bingxche mentioned this pull request Oct 24, 2026

[CI Monitor] Daily Report - 2026-10-24 bingxche/sglang-ci-bot#243

Open

feng397 mentioned this pull request Oct 26, 2026

[Fix] dspark: Avoid host reads during draft metadata glue capture #46618

Closed

3 of 5 tasks

chaohu2000 mentioned this pull request Oct 27, 2026

feat(dsv41): enable DeepInfer features on PPU SGLang (v0.5.22) flytiger-eco/sglang-for-sail#165

Open

5 tasks

YAMY1234 mentioned this pull request Oct 28, 2026

[Feat] [NVIDIA] runtime: Add a master switch for megakernel runtime optimizations #47105

Merged

5 tasks done

Navigation Menu

Repository navigation

[Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path - #45450

[Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path#45450
Oasis-Git merged 22 commits into
sgl-project:mainfrom
YAMY1234:dsv41-mega-perf

YAMY1234 commented Oct 20, 2026 •

edited by github-actions Bot

Loading

Uh oh!

YAMY1234 commented Oct 21, 2026 •

edited

Loading

Uh oh!

nvpohanh left a comment

Uh oh!

nvpohanh Oct 22, 2026

Uh oh!

nvpohanh Oct 22, 2026

Uh oh!

nvpohanh Oct 22, 2026

Uh oh!

Oasis-Git commented Oct 22, 2026

Uh oh!

Oasis-Git commented Oct 22, 2026

Uh oh!

Oasis-Git commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

Oasis-Git commented Oct 23, 2026

Uh oh!

Oasis-Git commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

YAMY1234 commented Oct 20, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Reproduce

Checklist

CI States

Uh oh!

YAMY1234 commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvpohanh left a comment

Choose a reason for hiding this comment

Uh oh!

nvpohanh Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

nvpohanh Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

nvpohanh Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Oasis-Git commented Oct 22, 2026

Uh oh!

Oasis-Git commented Oct 22, 2026

Uh oh!

Oasis-Git commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Oasis-Git commented Oct 23, 2026

Uh oh!

Oasis-Git commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YAMY1234 commented Oct 20, 2026 •

edited by github-actions Bot

Loading

YAMY1234 commented Oct 21, 2026 •

edited

Loading

github-actions Bot commented Oct 22, 2026 •

edited

Loading