Repository navigation
feat(dsv41): enable DeepInfer features on PPU SGLang (v0.5.22) - #165
Open
chaohu2000 wants to merge 29 commits into
Open
chaohu2000 wants to merge 29 commits into
chaohu2000 wants to merge 29 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Static-verify layout errors, broken test contracts, and unexecuted registered assertions must be addressed before approval.
11 open issues
What changed in this PR
Enables DeepInfer features for DeepSeek-V4.1 on PPU SGLang v0.5.22, covering attention kernels, DSpark speculative decoding, request-window caching, and distributed execution.
Changes:
- Enable PPU DeepGEMM sparse indexing and MegaMHC, with optimized top-k, KV-cache, mHC, and quantization kernels.
- Update DSpark KV placement, DeepEP graph layouts, multimodal CP, independent CP/DP sizes, and bounded SWA replay.
- Add regression tests, PPU E2E scenarios, and a dequantization benchmark.
Unresolved issues include static-verify layout failures, incompatible test fixtures, and a registered top-k suite that does not execute its assertions. GPU/PPU correctness and performance were not validated in this review.
| File | Description |
|---|---|
test/registered/unit/models/test_deepseek_v41_tail_cp_ids.py |
Add CP-tail ownership and routing-ID regressions. |
test/registered/unit/models/test_deepseek_v41_cp_policy.py |
Add CP-policy and multimodal tests. Blocked by a nonexistent indexer_quant patch, incorrect with_phase and is_ppu patch targets, and missing mm_inputs fields in multimodal and idle fixtures. |
test/registered/unit/mem_cache/test_dsv4_compressed_pools.py |
Adapt fixtures to updated KV buffers and window layouts. |
test/registered/unit/layers/test_dsv4_nonpaged_indexer.py |
Adjust platform skips and grouped-verify tests. The non-SAIL contract test still patches removed topk_transform_paged_from_metadata instead of the v1 interface. |
test/registered/unit/layers/test_deepep_graph_adapter.py |
Test captured DeepEP layout restoration. |
test/registered/unit/layers/test_deepep_buffer_layouts.py |
Test multi-layout buffer sizing and transitions. |
test/registered/kernels/ops/layernorm/test_mhc_stream_mean.py |
Check stream-mean numerical parity and output placement. |
test/registered/kernels/ops/attention/test_topk_v1.py |
Add top-2048, tie-fallback, and graph-replay checks. Missing execution entry point leaves registered CI assertions unexecuted. |
test/registered/kernels/ops/attention/test_q8kv8_sparse_prefill_backend.py |
Update sparse-prefill fixtures. |
test/registered/kernels/ops/attention/test_dsv41_request_window_copy.py |
Test packed KV copies and retained-row commits. |
test/registered/kernels/ops/attention/dsv4/test_v41_kv_store.py |
Exercise both fused K-store implementations. |
test/registered/kernels/benchmark/attention/bench_dequant_k_cache_paged.py |
Compare scalar and vectorized dequantization. |
test/registered/attention/unittests/dsv4/test_deepseek_v4.py |
Check replay address preservation and buffer reuse. |
test/ppu/unit/test_dsv41_sparse_indexer_support.py |
Add dispatch checks. Fixtures inject _is_ppu instead of required _is_sail, expect removed req_to_token, and assert an INT8 fallback the tested factory does not implement. |
test/ppu/unit/test_dsv41_candidate_topk_v1.py |
Add candidate-publisher checks. Fixtures use an obsolete two-argument publish_decode call, lack paged_logits_args(), and assume an unsupported raw-output field. |
test/ppu/e2e/test_deepseek_v4_dspark_prefill_cp_graph.py |
Add eager DSpark prefill-CP coverage and document graph limitations. |
test/ppu/e2e/test_deepseek_v4_dspark_deepep_engram_dp_graphs.py |
Add concurrent DSpark, DeepEP, Engram, and DP scenarios. |
test/manual/test_dsv41_ragged_prefill.py |
Update sparse-prefill buffer references. |
python/sglang/srt/utils/common.py |
Include sharded Engram in DP synchronization bookkeeping. |
python/sglang/srt/speculative/spec_info.py |
Clarify DSpark shared-read completion ordering. |
python/sglang/srt/speculative/multi_layer_eagle_draft_extend_cuda_graph_runner.py |
Complete DeepEP capture bookkeeping. |
python/sglang/srt/speculative/frozen_kv_mtp_cuda_graph_runner.py |
Record finalized DeepEP capture layouts. |
python/sglang/srt/speculative/eagle_draft_extend_cuda_graph_runner.py |
Record completed DeepEP capture layouts. |
python/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py |
Complete layout bookkeeping after graph variants. |
python/sglang/srt/speculative/dspark_components/dspark_worker_v2.py |
Compute draft KV placement before target prefill. |
python/sglang/srt/speculative/dspark_components/dspark_kv_inject.py |
Separate prefill placement from hidden-state injection. |
python/sglang/srt/models/deepseek_v4_mhc.py |
Gate MegaMHC execution while preserving fallback support. |
python/sglang/srt/models/deepseek_v4_dspark.py |
Avoid redundant vocabulary gathers and FP32 copies. |
python/sglang/srt/models/deepseek_v2.py |
Add vision-aware routing to DeepEP dispatch. |
python/sglang/srt/model_executor/runner/eager_runner.py |
Prepare multimodal CP inputs and gather tail outputs. |
python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py |
Record DeepEP layouts after graph capture. |
python/sglang/srt/model_executor/runner_utils/deepep_adapter.py |
Restore captured DeepEP layouts during replay. |
python/sglang/srt/mem_cache/dsv41_request_window.py |
Fuse copies and commits while reducing history-check synchronization. |
python/sglang/srt/layers/moe/token_dispatcher/deepep.py |
Size buffers across layouts and manage layout transitions. |
python/sglang/srt/layers/deep_gemm_wrapper/configurer.py |
Enable sparse DeepGEMM APIs on supported PPU devices. |
python/sglang/srt/layers/aux_hidden_states.py |
Add direct capture slots and packed-row gathering. |
python/sglang/srt/layers/attention/dsv4/v41_indexer/types.py |
Carry snapshotted K slots and cached prefill plans. |
python/sglang/srt/layers/attention/dsv4/v41_indexer/sparse_table.py |
Use v1 top-k and grouped paged scoring. |
python/sglang/srt/layers/attention/dsv4/v41_indexer/scoring.py |
Cache prefill plans and add chunked PPU scoring. Dense fallback still launches per-row queries against a grouped-request schedule instead of using paged_logits_args(). |
python/sglang/srt/layers/attention/dsv4/v41_indexer/full_topk.py |
Integrate grouped and chunked decode scoring. |
python/sglang/srt/layers/attention/dsv4/v41_indexer/dense_blocks.py |
Add chunked PPU candidate publication and consumption. |
python/sglang/srt/layers/attention/dsv4/v41_indexer/__init__.py |
Extend PPU backend selection and dense fallback. |
python/sglang/srt/layers/attention/dsv4/sparse_prefill_utils.py |
Track reusable dequantized KV-buffer heads. |
python/sglang/srt/layers/attention/dsv4/metadata.py |
Build schedules for grouped verify rows. |
python/sglang/srt/layers/attention/deepseek_v4_backend.py |
Integrate snapshots, grouped metadata, replay locations, and KV reuse. Partial attention-TP padding retains incompatible request grouping, causing valid static-verify batches to fail. |
python/sglang/srt/layers/attention/deepseek_v4_backend_hip_radix.py |
Adapt HIP buffer and window-layout interfaces. |
python/sglang/srt/disaggregation/common/conn.py |
Support independent prefill CP and decode DP widths. |
python/sglang/srt/arg_groups/model_hook.py |
Clarify validation ordering. |
python/sglang/srt/arg_groups/deepseek_v4_hook.py |
Validate supported topologies and allow decode-role bounded replay. |
python/sglang/kernels/ops/layernorm/mhc.py |
Parallelize small-batch post-processing and add fused stream mean. |
python/sglang/kernels/ops/layernorm/mhc_mega.py |
Add capability and input checks around MegaMHC. |
python/sglang/kernels/ops/kvcache/dsv41_window_layout.py |
Fuse query and history layout construction. |
python/sglang/kernels/ops/kvcache/__init__.py |
Register the window-layout entry point. |
python/sglang/kernels/ops/attention/dsv4/topk.py |
Support optional page tables for raw v1 output. |
python/sglang/kernels/ops/attention/dsv4/request_window_copy.py |
Add packed-row copy and commit kernels. |
python/sglang/kernels/ops/attention/dsv4/index_logits.py |
Apply PPU logits-memory budgets. |
python/sglang/kernels/ops/attention/dsv4/fp4_indexer.py |
Fuse selection-to-pool-slot mapping. |
python/sglang/kernels/ops/attention/dsv4/dequant_k_cache.py |
Select vectorized dequantization with scalar fallback. |
python/sglang/kernels/ops/attention/dsv4/candidate_blocks.py |
Switch candidate-block selection to v1 top-k. |
python/sglang/kernels/ops/attention/dsv4/attn.py |
Expose the vectorized dequantization wrapper. |
python/sglang/kernels/ops/attention/dsv4/__init__.py |
Export vectorized dequantization. |
python/sglang/kernels/jit/csrc/gemm/per_token_group_quant_8bit_v2.cuh |
Reduce quantization scheduling arithmetic. |
python/sglang/kernels/jit/csrc/deepseek_v4/main_norm_rope.cuh |
Add warp-per-token V4.1 K stores. |
python/sglang/kernels/jit/csrc/deepseek_v4/dequantize_k_cache.cuh |
Implement vectorized paged K-cache dequantization. |
Review effort: Balanced
| if c128_compress_metadata is None and self.has_c128: | ||
| c128_compress_metadata = create(compress_ratio=128) | ||
| low = core_attn_metadata.low_ratios | ||
| rows_per_request = 1 if is_ragged else num_draft_tokens |
Comment on lines
+628
to
+640
| for rows, plan in metadata.row_chunks(): | ||
| rows = slice(rows.start, min(rows.stop, bs)) | ||
| if rows.start >= bs: | ||
| break | ||
| scores = deep_gemm_fp4_paged_mqa_logits( | ||
| (q_fp4[rows].unsqueeze(1), q_sf[rows].unsqueeze(1)), | ||
| k_cache, | ||
| weights[rows].float(), | ||
| metadata.compressed_seq_lens[rows], | ||
| metadata.page_table[rows], | ||
| plan, | ||
| metadata.max_compressed_seq_len, | ||
| )[:, :lmax] |
Comment on lines
+119
to
+121
| table = publisher.publish_decode( | ||
| inputs, SimpleNamespace(page_indices=pages, raw_indices=raw) | ||
| ) |
| LAYERS / "deep_gemm_wrapper/configurer.py", | ||
| "_supports_paged_sparse_mqa_logits", | ||
| { | ||
| "_is_ppu": ppu, |
| ) | ||
| constructor.assert_called_once_with( | ||
| token_to_kv_pool=None, | ||
| req_to_token=None, |
| graph.replay() | ||
| torch.cuda.synchronize() | ||
| for row, size in enumerate(lens): | ||
| _check_row(scores[row], output[row], size, 2048) |
| for call, plan in zip(topk.call_args_list, self.topk_plans): | ||
| self.assertIs(call.kwargs["topk_metadata"], plan) | ||
|
|
||
| @unittest.skipIf(_IS_SAIL, "v0.5.22 PPU fork's sparse_table has no topk_transform_paged_from_metadata") |
Comment on lines
+121
to
+124
| patch( | ||
| "sglang.srt.layers.attention.dsv4.indexer_quant.dsv41_requires_int8_indexer", | ||
| return_value=False, | ||
| ), |
Comment on lines
+130
to
+132
| patch( | ||
| "sglang.srt.model_executor.cuda_graph_config.with_phase" | ||
| ) as update, |
| return_value=core.page_table, | ||
| ), | ||
| patch(module + ".get_platform", return_value=NS(is_sm120=False)), | ||
| patch(module + ".is_ppu", return_value=True), |
Collaborator
Author
Record of upstream commits NOT cherry-picked into this PRBranch A. Portable, but rejected on evidence (3)
B. Engram family (4)
C. MegaGate / MegaMoE family (6, frozen)
D. Platform-exclusive (27, not portable to PPU)Hardware ISA / vendor-library-bound kernels (CUTLASS & DeepGEMM tactics, aiter FlyDSL, HIP graph, NPU mem_cache). No cherry-pick path; equivalent gains require SAIL kernel-level native implementations.
E. No pick needed (1)
Note on sgl-project#45251 ( |
…p-2048 candidates
Why:
- The v41 FP4 indexer's DeepGEMM paths were SM100+-only, so PPU could not use them.
- PPU flattened-K prefill tile size was bound only by the hard-coded 2GiB budget, so SGLANG_SPARSE_INDEXER_MAX_LOGITS_MB could not govern FP4.
- The paged indexer decode assumed a single plan and the batch-local token count, while the runtime metadata is chunked per row.
- The sparse candidate indexer was disabled on PPU SM89+ and the V1 candidate top-k was capped at 1024 entries.
- The V1 tie-break stage could rank an arbitrary subset when scores clustered into one coarse FP16 bin.
How:
- Add use_deep_gemm_fp4_indexer() (SAIL or SM100+) to gate the v41 DeepGEMM prefill/decode indexer.
- On PPU, use SGLANG_SPARSE_INDEXER_MAX_LOGITS_MB instead of the 2GiB tile budget, consume chunked plans per metadata.row_chunks(), and keep PPU out of the SM100-only decode fast path.
- Enable paged sparse logits on PPU SM89+ when the JIT DeepGEMM API is available, falling back to dense blocks otherwise.
- Raise the V1 top-k limit to 2048, slice per row chunk, and fall back to radix_topk when the tie buffer overflows.
- Pass scores, seq_lens, page table, page output and page size to the V1 top-k without a V2 plan.
Dependency:
- Library changes (optional): DeepGEMM can export paged sparse MQA logits on PPU SM89+.
- Guards: _supports_paged_sparse_mqa_logits() requires the JIT DeepGEMM build, SM >= 89 on PPU (Blackwell otherwise), and the fp8_fp4_paged_sparse_mqa_logits API; unsupported configurations fall back to dense blocks.
Test:
- Unit:
- test/ppu/unit/test_dsv41_candidate_topk_v1.py
- test/ppu/unit/test_dsv41_sparse_indexer_support.py
- test/registered/kernels/ops/attention/test_topk_v1.py
- E2E:
- DeepSeek-V4.1-Flash:
- Model data type: mxfp4
- Affected platforms: M890
- Test case:
- GSM8K accuracy with the sparse candidate indexer and top-2048 selection enabled.
(cherry picked from commit f71cdb2b0f807d21de1a3dad9c519c9ce9b1e260)
…ependent CP/DP sizes
Why:
- DSpark PD required DP=1 and CP=1 on both sides, so prefill CP and decode attention DP could not be sized independently.
- CP rejected V4.1 vision and used rank-major order for the image mask and Engram hashing; the late-layer tail gathered with full CP metadata and unsliced IDs.
- The DeepEP low-latency buffer was sized for the first dispatched layout, so a DSpark draft/target pair could not share it, and CUDA-graph replay restored the dispatch mode but not the layout.
- V4.1 vision rejected DeepEP, and its top-k path never applied DeepEP's physical expert mapping; sharded Engram tables under attention DP skipped the MLP TP gather and DP-sized capture metadata.
- V4.1 validation ran after CP size resolution and used stale flags; BREAKABLE prefill graphs kept the default 16k capture limit, and a CP-captured graph could replay for a non-CP batch.
How:
- Build full-sequence vision embeddings before CP sharding, use original model IDs for the image mask and Engram, and drive the CP gather/logits slice from the tail's global token indices.
- Allow prefill interleave CP with decode attention DP; CP/DP sizes are independent when attention TP=1, otherwise TP*CP must match TP*DP; every CP rank sends KV for DSpark PD.
- Register every DeepEP low-latency layout before allocation, size RDMA/QPs for the maximum, validate later layouts against the live buffer, and capture/replay the layout in the CUDA graph adapter.
- Add the VL top-k branch so DeepEP receives physical expert IDs, allow DeepEP for V4.1 vision with separate shared experts, and require MLP TP gather for sharded Engram under attention DP unless the host table is shared.
- Pad attention-TP verify blocks with dummy rows while keeping num_q_tokens logical; decide prefill CP before topology resolution, keep the live context budget for BREAKABLE CP prefill graphs, and reject non-CP batches on CP graph replay.
- Keep language-model-only V4.1 PD on the standard KV transfer path.
Dependency:
- None
Test:
- Unit:
- test/registered/unit/models/test_deepseek_v41_tail_cp_ids.py
- test/registered/unit/models/test_deepseek_v41_cp_policy.py
- test/registered/unit/layers/test_deepep_buffer_layouts.py
- test/registered/unit/layers/test_deepep_graph_adapter.py
- E2E:
- DeepSeek-V4.1-Flash:
- Model data type: all
- Affected platforms: all
- Test case:
- E2E accuracy of DSpark PD with prefill interleave CP and decode attention DP (TP=1 on both sides, CP != DP).
- E2E accuracy of V4.1 multimodal vision with prefill CP enabled.
- 8x PPU DSpark + DeepEP low_latency + DP attention + sharded Engram + CUDA graph, GSM8K accuracy.
(cherry picked from commit 1b8d6e580b0860edd37f38f53ffef74241938087)
Why:
- Unfused row-major quantization performs unnecessary runtime division and modulo to derive token and group coordinates.
How:
- Pass flattened group offsets directly through scheduler callbacks.
- Compute token and group coordinates only for column-major scales or fused SiLU-and-multiply.
Dependency:
- None
Test:
- Unit:
- test/registered/kernels/ops/quantization/test_per_token_group_quant_8bit_v2.py
- E2E:
- DeepSeek-V4.1-Flash:
- Model data type: all
- Affected platforms: all
- Test case:
- Prefill/decode performance test.
(cherry picked from commit 5cb51efbd84da5fedad0300ec11dcc65b5ea6408)
Why:
- The paged indexer metadata sized `q_fp8_shape` rows from the C4 top-k length tensor, which does not describe the selected compression ratio, so ratio-1/2 query metadata could be mis-sized.
How:
- Size `q_fp8_shape` rows from the batch's selected-ratio compression lengths.
Dependency:
- None
Test:
- Unit:
- test/registered/unit/models/test_deepseek_v41_cp_policy.py
- E2E:
- DeepSeek-V4.1-Flash:
- Model data type: all
- Affected platforms: all
- Test case:
- GSM8K accuracy.
(cherry picked from commit 4e8995b77287ee24c0f9e948bdde738cac33d9a2)
Why:
- The scalar paged K-cache dequant kernel runs one thread per token with single-byte loads, and on PPU it is bound by memory-request issue rate and unhidden latency (89% mem busy, 25% occupancy) rather than bandwidth.
How:
- Split each token across a 32-thread warp with 16-byte uint4 accesses and a device-filling grid; arithmetic is unchanged, so output stays bit-identical (18.4x at 20096 tokens, faster at all 52 swept shapes).
- Fall back to the scalar kernel when the 16-byte alignment, output stride, or page-byte conditions are unmet, warning once with the failed conditions.
- Add a shape-sweep benchmark.
Dependency:
- None
Test:
- Unit:
- test/registered/kernels/ops/attention/dsv4/test_dequant_k_cache_paged.py
- E2E:
- DeepSeek-V4.1-Flash:
- Model data type: all
- Affected platforms: all
- Test case:
- GSM8K accuracy.
(cherry picked from commit 357ae7231e0a1068cbe011c2167245b532e6a876)
Why:
- Channelwise FP8 checkpoints name layers ffn.*.w1/w2/w3 and store wo_a scales as .scale; the remap only ran for Quark checkpoints and the streaming dequant only matched .wo_a.scale.
- Shared experts were therefore built as MXFP4 and wo_a scales were loaded into BF16 weights, failing weight loading with shape mismatches.
How:
- Return the DeepSeek-V4 HF-to-SGLang mapper when the checkpoint declares fp8_channelwise_layers, so the quant config sees mlp/gate_proj/down_proj/up_proj names.
- Match .wo_a.scale, .wo_a.weight_scale and .wo_a.weight_scale_inv in the streaming dequant and raise on an unpaired weight or scale.
Dependency:
- None
Test:
- Unit:
- None
- E2E:
- DeepSeek-V4.1-Flash:
- Model data type: mxfp4 with dense FP8 per-channel
- Affected platforms: M890
- Test case:
- The server loads the checkpoint and passes GSM8K accuracy.
(cherry picked from commit 13e433f760e1623983b18b448adcf91c79983c38)
Why:
- The MegaMHC prefill boundary was gated to SM100, so PPU SM89+ always ran the separate mHC ops per layer.
- DeepGEMM may not export the mega_mhc API, and the adapter had no probe or fallback.
How:
- Probe the DeepGEMM mega_mhc API once and expose mega_mhc_supported() for SAIL SM89+ and CUDA SM100-109.
- Validate boundary tensor contracts and return None so the existing mHC path runs when the API or contracts are unavailable.
- Keep batch-invariant mode and CP prefill on the existing path.
Dependency:
- Library changes: DeepGEMM must export a compatible mega_mhc API on PPU SM89+.
- Guards: mega_mhc_supported() checks ENABLE_JIT_DEEPGEMM, the SM range and the API; try_mhc_mega_boundary() returns None on contract failure.
Test:
- Unit:
- None
- E2E:
- DeepSeek-V4.1-Flash:
- Model data type: mxfp4
- Affected platforms: M890
- Test case:
- GSM8K accuracy with SGLANG_OPT_DSV41_MEGA_MHC_PREFILL=1 and prefill CP disabled.
(cherry picked from commit da64499bf4a0b018f960e6d1e5aa721ce39df2d4)
…-norm-rope store (sgl-project#45934) (cherry picked from commit 16885f3eab34f8326bc617f3dffe4b4f279255de)
(cherry picked from commit 45dfe6d057e158213d72cfb0edb1d4a66684089a)
…yer host syncs (sgl-project#45223) (cherry picked from commit 543002de921db17333cba2cbc58081ecef480279)
…gl-project#45247) (cherry picked from commit b3d0380985cb088c787f5bc3474f9f3502f3ec3d)
…riant checks are off (sgl-project#45276) (cherry picked from commit ce329c85fa9702e2cb5dc5219317c0c7e1417f0c7) (cherry picked from commit 728f45111ad62b8e79c8eb3110198fb094f9b3a5)
…copy (sgl-project#45940) (cherry picked from commit 620a7a27e83f2960a8577c06651102e063d60f00)
…p32 copy (sgl-project#45964) (cherry picked from commit 6994aa03b4b259a64fd489646d46dcfbf51c4669)
… prefill index plan (sgl-project#45577) (cherry picked from commit e8e19f50f97a6cf2c4ed3450a65caeb00708b908)
…gl-project#45741) (cherry picked from commit 7d43eeb75cd0aa4e99ac5c383e3531e3afb62895)
…ernel (sgl-project#45938) (cherry picked from commit 7507c69a811a1960bb37a49dd48f34994dd2d34c)
…l-project#46229) v0.5.22 adaptation: - Drop the deepseek_v4_replay_tail.py hunk and the late-layer-tail reader test: v0.5.22 keeps DecoderReplayTail inside the backend (_prefill_graph_tail_metadata / late_layer_tail), so there is no such file. - Do not add _activate_request_window()/prefill_graph_replay_batch to the prefill-graph capture path: both arrive with other main-only commits and the v0.5.22 capture path does not use them. - Keep the v0.5.22-specific prefill_shared_reads_snapshotted assertions in the breakable-graph contract test and mock the SWA pool it now needs. (cherry picked from commit d85eed5fc8c7ccf9c02d38ed0e9639ca9734f2d1)
Four plan-contract cases assert the upstream non-paged indexer, which C4IndexerBackendMixin._get_nonpaged_indexer_plan disables on SAIL (returns None), so they can never pass on PPU. Two more assert upstream-only surfaces: the candidate publisher patches sparse_table.topk_transform_paged_from_metadata, which the v0.5.22 PPU fork does not have, and the SM100 candidate-indexer gating never fires on PPU. All six already fail identically on the batch1 baseline; skipping them on SAIL keeps the per-batch unit-test signal clean. CUDA coverage is unchanged. (cherry picked from commit 08cf784577a6af576fad0ac9ea6c06c1817cf2b5)
…project#45579) (cherry picked from commit 7be625e1a97b421b119b49c26596a3145ab31a77)
…ct#44783) (cherry picked from commit 106296f881cf749a87332afba5ea75b955371aa3)
(cherry picked from commit 2f51076c18187dd4acd392b0e3a66875994a4393)
…t KV inject before the target forward (sgl-project#45682) (cherry picked from commit acbe5b204028122f950a493670103be23386edd7)
… forward (sgl-project#45909) (cherry picked from commit b5ce3ff9e547be0388ef34d26606763b41766626)
…alive through the prefill step (sgl-project#45900) (cherry picked from commit b6a9c1b171e3f617b06f305360dac03da3f18ae4)
…ject#44906) Co-authored-by: leonardHONG <178642616+leonardHONG@users.noreply.github.com> Co-authored-by: Xiaoyu Zhang <1182563586@qq.com> (cherry picked from commit c9e89e5697749f59ba5125bd3473ba6a510867e6)
… the full-scan indexer (sgl-project#45251) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 54b99da72ea89f27fb13d512d691054be21676fb)
…code role (PPU adaptation) The replay machinery only runs on extend forwards, so a PD decode role never reaches it: every consumer is guarded by is_extend_without_speculative() and the disaggregation wire carries no replay state. Exempt that role from the attention-DP/MoE clause so a uniform four-node flag deployment is accepted. Prefill roles keep the original restriction (attention DP + deepep stays rejected). (cherry picked from commit 6f389cd758e7f2fc9084aea48813fb5ab2c154d9)
chaohu2000
force-pushed
the
dsv41-deepinfer-v0.5.22
branch
from
October 27, 2026 09:52
f84c80b to
6f77ed6
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Implements the DeepInfer enablement tracked in #142: bring the DeepSeek-shipped kernels and features (Mega series, DSpark speculative decoding, SWA bounded replay, and sparse/Engram placements) to PPU SGLang for DeepSeek-V4.1 on the
v0.5.22line.Relates to #142.
Modifications
--enable-decoder-swa-bounded-replayon a PD decode role; skip discarded KV rows during window commit; translate SWA locations once per prefill graph step.Commits (29)
Accuracy Tests
To be provided by the author.
Speed Tests and Profiling
To be provided by the author.
Checklist
CI States
Latest PPU CI: Not run yet