[kpham-sgl]

Motivation

DeepSeek-V4.1 checkpoints with vision_n_layers > 0 reject every MoE A2A backend at initialization, including text-only serving with DP attention and MegaMoE. Simply removing the guard would also route image tokens using the text router bias. Flash additionally fails the old MegaMoE intermediate-size check because its expert intermediate dimension is 2304.

This enables DeepSeek-V4.1-Flash text and image serving with DP attention and CUDA MegaMoE using SGLang's existing sgl-deep-gemm==0.2.1.post1 pin.

Modifications

• Admit CUDA MegaMoE for V4.1 vision-capable checkpoints; retain the CP, PP, and unsupported A2A restrictions.
• Route local image/text rows through vision-aware top-k on MegaMoE, preserving EPLB mapping, expert recording, home-rank shared slots, routed scaling, and padded/idle rank handling.
• Require 128-element intermediate alignment for the pinned FP8/FP4 path, allowing Flash's 2304 dimension. Retain the input-scale alignment requirement and the other MMA paths' restrictions.
• Add constructor admission and numerical routing regressions covering image/text bias, shared slots, scaling placement, padding, and idle ranks.
• Skip DSv4 DSPARK attention on idle CUDA ranks during DP speculative prefill coordination. Idle ranks have no local attention metadata or KV rows, while their later FFN still participates in the shared MoE draft.

Accuracy Tests

• 22 focused unit tests passed: MegaMoE DeepGEMM API and DeepSeek-V4 shared-expert/admission tests. Six existing fused shared-weight scaling tests also passed.

• On 2x GB300 with DeepGEMM 0.2.1.post1: 16 CUDA routing cases passed, covering image/text bias, EPLB remapping, shared slots, scaling placement, and padded rows.

• Two-rank real DeepGEMM dispatch/combine checks passed at Flash's exact hidden/intermediate dimensions (5120/2304), with shared slots, scaling placement, and idle ranks. Six CUDA graph replay cases passed while switching captured inputs between text and image tokens.

• Full deepseek-ai/DeepSeek-V4.1-Flash checkpoint (revision 2cba9e42aa026125f3ed06c6d98c1db82f7ca027) served successfully on 2x GB300 with TP2 / attention DP2 / EP2 / MegaMoE. Four semantic smoke requests returned HTTP 200 and the expected answers: arithmetic 5, red image Red, concurrent capital question Paris, and concurrent blue image Blue.

• Native DSPARK with coordination enabled on DSv4.1 Flash/MegaMoE passed the registered mixed-rank greedy output comparison on 2x GB300. Fresh ~26.7K-token contexts were prefilled on each rank while both ranks decoded; ongoing and subsequent outputs matched isolated execution. Simulation was disabled. The idle-attention regression fails before the fix and all four coordination unit cases pass after it, including with CUDA hidden.

Native GSM8K accuracy

Validated head ee4aa862 on the public Flash checkpoint above, 2x GB300, TP2 / attention DP2 / EP2, MegaMoE and TRTLLM sparse attention. Followed the five-shot chat validation protocol: first five test rows are demonstrations and excluded from scoring, 1,314 scored questions, thinking off, temperature 0, top-p 1, output cap 4096, concurrency 64, no retries. Prompt and scoring helper ASTs match the reference. Simulation is disabled (SGLANG_SIMULATE_ACC_LEN=-1); interval is 0 in every case.

Mode	Correct / 1314	Accuracy	Request errors	Length finishes
Ordinary decoding	1277	97.18%	0	1
DSPARK, coordination off	1284	97.72%	0	1
DSPARK, coordination on	1276	97.11%	0	1
DSPARK, coordination off, fresh repeat	1276	97.11%	0	1
DSPARK, coordination on, fresh repeat	1278	97.26%	0	0

Each run used a fresh server on immutable image sha256:c00e5f291408213c2c95bf59e893c5b5c14076b6c95b1660f5628d9c00273afc; full package manifests match. Independent raw-response audits confirm 1,314 unique scored indices and every reported count. Capped responses remain in the denominator. Coordination-on exercised eager mixed steps without crashing.

The initial eight-answer coordination-on deficit did not persist: the fresh repeat comparison favors on by two answers, while repeating off alone loses eight. These runs show no consistent coordination-specific accuracy loss, but do not establish numerical equivalence or batch invariance. The reference protocol's historical scores use different weights/hardware and are not matched accuracy baselines. GSM8K client wall times are not AgentX throughput measurements.

Full-model launch configuration:

export SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1
export SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4.1-Flash --trust-remote-code \
  --tp 2 --attn-dp-size 2 --ep-size 2 --enable-dp-lm-head \
  --moe-a2a-backend megamoe --moe-runner-backend flashinfer_mxfp4 \
  --max-total-tokens 8192 --context-length 4096 --max-running-requests 4 \
  --chunked-prefill-size 2048 --mem-fraction-static 0.70 \
  --cuda-graph-max-bs-decode 4 --cuda-graph-backend-prefill disabled \
  --reasoning-parser auto --tool-call-parser auto \
  --model-loader-extra-config '{"enable_multithread_load":true}' \
  --watchdog-timeout 1200 --host 127.0.0.1 --port 30000

Four-GPU native DSPARK accuracy

Validated head 2b484a5142668f1229dce705a234e931ee3cc27a on dsv41-megamoe-eval (4× B300), using the public deepseek-ai/DeepSeek-V4.1-Flash checkpoint at revision 2cba9e42aa026125f3ed06c6d98c1db82f7ca027. Ran both accuracy cases in TestDSV41FlashDP4EP4DSpark from a locally modified test/registered/e2e/models/test_deepseek_v41_flash_gb300.py, with the DP class changed to CUDA MegaMoE A2A and flashinfer_mxfp4.

Configuration: TP4 / attention DP4 / EP4, DP LM head, native DSPARK with block size 5, chunked prefill 4096, concurrency 64, and output cap 4096. Simulation and evaluator retries were disabled (SGLANG_SIMULATE_ACC_LEN=-1, SGLANG_TEST_MAX_RETRY=0); MegaMoE's per-rank token cap was 4096.

Evaluation	Correct / total	Accuracy	Required threshold	Request errors	Truncations
GSM8K (zero-shot chat)	196 / 200	98.0%	95.0%	0	0
MMLU	458 / 500	91.6%	90.0%	0	0

Both registered tests passed, including native DSPARK draft-acceptance assertions. Per-rank average acceptance after GSM8K was 4.21–4.37; cumulative server averages after MMLU were 3.34–3.48. This GSM8K run uses the registered test's zero-shot protocol, separately from the five-shot results above.

Runtime: lmsysorg/sglang:dev on x86_64, sgl-deep-gemm==0.2.1.post1, flashinfer-python==0.7.0.post1, and sgl-eval==0.1.2. Built the required Rust TreeCore extension from the exact PR source before the successful run. The test edit remains local and uncommitted.

Speed Tests and Profiling

AgentX cc128 on 2x GB300, TP2 / attention DP2 / EP2 / MegaMoE / TRTLLM sparse attention. Each controlled run used a 900-second admission window after the same 1,422-request warmup, with synthetic DSPARK acceptance 3.51. Results use InferenceX latency/throughput calculations; throughput includes cached prompt tokens. Memory, HiCache, router, chunk size, and graph limits were fixed.

Prefill/decode interval	DP spec/prefill coordination	p90 interactivity (tok/s)	Total tok/s/chip	Output tok/s/chip	p90 TTFT (s)	Cache hit
0	Off	18.70	157,336	1,465	7.03	94.18%
8	Off	38.75	186,951	1,822	14.95	94.48%
0	On	20.65	160,275	1,532	8.31	94.10%

All three had zero request errors, OOMs, and memory retractions. Interval 8 improves interactivity and throughput with a TTFT cost. Coordination provides a smaller gain: decode round counts across ranks were nearly unchanged (18,931 vs 18,936), with 2,184 eager rounds replacing part of normal graph replay. Matched common turns confirm the latency gains; raw cache rates also reflect different completed request mixes. Cache parity with the published vLLM result has not been established. The coordination performance run includes the idle-attention fix; native correctness was checked separately with simulation disabled.

Checklist

• Format code: Ruff formatting/lint, isort, registered-test lint hooks, and git diff --check passed.
• Add unit tests.
• Update documentation (no new flags or dependency changes).
• Provide accuracy and speed benchmarks (native correctness plus controlled AgentX measurements above).
• Follow SGLang code style guidance.

---

CI States

Latest PR Test (Base): ✅ Run #38080797660
Latest PR Test (Extra): ❌ Run #38080797623
Latest PR Test (AMD ROCm 10): ❌ Run #38080797709

[kpham-sgl]

/rerun-test test_deepseek_v41_flash_gb300.py test_deepseek_v4_flash_fp4_megamoe_b200.py

[github-actions]

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300.py

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py