[Qwen3.8] Enable NVIDIA NVFP4 on DGX Spark with file-backed PLE and PDL router fix

[Jiminator]

Motivation

Combining these changes into one PR also saves CI resources by avoiding three separate PRs running overlapping CI suites.

Enable single-node DGX Spark / GB10 deployment of nvidia/Qwen3.8-Flash-Next-NVFP4 from upstream main, with or without its embedded MTP head. Main still lacks the NVIDIA mixed-precision loader handling and file-backed PLE storage needed for this deployment, and both PDL routers read bias before waiting for the preceding producer.

Consolidates and supersedes #38568, #38569 and #38570. Preserves their commits and authorship, upstreaming the original PDL fix #38290, NVIDIA loader work #38121 / #37752, file-backed PLE backend #37068 by @hashd1ve, and TP prefetch correction #38123 by @yhyang201.

Modifications

• Move the PDL dependency wait before bias and score loads in both Triton and radix routers. Preserve register zeros for the no-bias path.
• Resolve ModelOpt mixed-precision checkpoint prefixes, normalize omitted exclusion lists, dispatch FP8_BLOCK_SCALES layers, and select FP8 PLE storage from the mixed config. Preserve existing FP8_PB_WO support.
• Retain mixed quantization for checkpoints with quantized mtp.* layers; preserve BF16 MTP behavior otherwise. Add the original mixed-expert runner selection and use the Triton FP8 MoE fallback when the global NVFP4 runner is FlashInfer CUTLASS/CuteDSL.
• Add --ple-offload-backend {pinned,file} and --ple-offload-dir, using a sparse file-backed table on supported unified-memory devices. Add prefetching and resident-memory trimming.
• Filter prefetch IDs to the local TP vocabulary shard and subtract the shard start before computing file offsets.
• Include the original ports' regression tests and CLI/environment documentation.

Based on main@45715e7f20629d1813ff8f543a23bb437128db2e. All three changes merge cleanly, and the combined tree is identical to merging the empirically tested integration branch onto this base. No additional source edits were introduced during consolidation. Direct source review confirms all three gaps remain in this main revision.

Accuracy Tests

Empirical host deployment on 1x DGX Spark / GB10 (sm121), TP=1; no patched Docker image. SGLang and its native extensions were installed from the combined source checkout. NVIDIA checkpoint revision: fab0aecb760cec45227f6656abcaafa11abca87a.

September 11 tests used main@165d8dd17736fa4938f2e73018f26835cd6e0778 plus these exact component heads:

• [Kernel] Wait for the PDL dependency before loading router bias (Triton + radix) #38568: 626f614f7bcd5dc531de5c268a694ff1a9f31f28.
• [Qwen4] Load nvidia/Qwen3.8-Flash-Next-NVFP4 (ModelOpt MIXED_PRECISION) #38569: 825663897b1fd9ee5c341eeb113b795e4ecfc3f8.
• [Qwen4] File-backed PLE offload for GB10 / DGX Spark with TP prefetch fix #38570: b14d2c55f37f8d3dc2b5fe21175b56fbe62771e1.

Tested combined revision: 69d38ccbd8a03646cae9168c971b3783883c2903. The newer PR base received source and clean-merge review; the numbers below are not relabeled as a GPU run on that newer base.

GSM8K first 200 test questions, greedy chat API, thinking disabled, 8,192-token generation limit:

Configuration	Accuracy	Request errors / invalid / truncated	Ready	Effective max input
MTP, concurrency 8	195/200 (97.5%)	0 / 0 / 0	763.26 s	180,282 tokens
Non-MTP, concurrency 24	194/200 (97.0%)	0 / 0 / 0	633.17 s	262,138 tokens

Both arithmetic smoke tests passed and CUDA graphs were active. MTP selected the intended FP8 Triton expert fallback and reported average draft acceptance 3.52925. No serving NaN/OOM/CUDA-error/exception reports appeared before intentional shutdown. This is log-level evidence, not internal NaN instrumentation. Cancellation tracebacks during intentional SIGTERM were recorded separately; both servers stopped after evaluation.

Environment: Python 3.12.3, PyTorch 2.13.0+cu130, CUDA 13.0, driver 580.173.02, FlashInfer 0.6.18, Triton 3.7.1, sglang-kernel 0.4.6.post1, Transformers 5.12.1. The isolated host environment was updated from main and native extensions rebuilt. Runs reused previously compiled host kernels with MAX_JOBS=2; startup timings exclude cold kernel compilation. The NVIDIA cuSPARSELt package reports a platform-tag warning on aarch64; GPU imports and both deployments succeeded.

Common serving flags (using the local NVIDIA checkpoint copy and a fresh local-NVMe PLE directory per boot):

sglang serve \
  --model-path nvidia/Qwen3.8-Flash-Next-NVFP4 \
  --tp 1 \
  --moe-runner-backend flashinfer_cutlass \
  --fp4-gemm-backend flashinfer_cutlass \
  --page-size 64 \
  --chunked-prefill-size 4096 \
  --context-length 262144 \
  --reasoning-parser qwen3 \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 --port 30000 \
  --ple-offload-embedding --ple-offload-backend file

MTP adds --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --max-running-requests 8 --max-mamba-cache-size 40. Non-MTP adds --mamba-radix-cache-strategy extra_buffer_lazy --max-running-requests 24 --max-mamba-cache-size 96.

Prior component checks recorded in the superseded PRs: 15 router GPU tests; 18 mixed-loader tests; 3 FP8 fallback tests; 34 fusion-gate tests; 2 Qwen MoE override tests; 26 file-table/Qwen argument tests; 13 embedding tests. Changed-file pre-commit hooks passed on the component ports. Consolidation passes git diff --check and exact merged-tree comparison; these component tests were not rerun merely to combine branches.

Scope: TP=1 text/chat validation, not a deterministic PDL-race reproduction, TP>1 prefetch validation, other-hardware validation, or the full 1,319-question benchmark. No full-length 262K prompt was tested; the configured MTP memory budget limits effective input to about 180K.

Known PLE startup limitation: populated files are rewritten on boot and can be much slower than fresh sparse files. Stop all servers using the directory before removing its ple_table_*.bin files for a fresh boot. Automatic cache lifecycle changes are not included.

Speed Tests and Profiling

Configuration	Generated tokens	Evaluation wall time	Aggregate output
MTP, concurrency 8	79,489	855.91 s	92.87 tokens/s
Non-MTP, concurrency 24	79,954	893.02 s	89.53 tokens/s

These are end-to-end chat-evaluation aggregates including prefill and the lower-concurrency tail, not a controlled before/after speed comparison or an isolated measurement of the router fix. The final non-MTP response discussed salary timing for 4,632 tokens, finished normally, and was scored incorrect against the dataset label.

Checklist

• Preserve component changes, regression tests, documentation and authorship.
• Record component pre-commit and test results, and verify the combined diff/tree.
• Provide combined single-node deployment accuracy and throughput results with exact revisions.
• Run the full 1,319-question evaluation and validate other hardware/TP configurations.
• Complete CI and maintainer review for the combined PR.

---

CI States

Latest PR Test (Base): ⏳ Run #34746850917
Latest PR Test (Extra): ❌ Run #34746850845
Latest PR Test (AMD ROCm 10): ⏳ Run #34746850940

[Jiminator]

/tag-and-rerun-ci

[BBuf]

Could we preserve the supported CuTe DSL v2 W4A16 path here?

On SM100, a Qwen mixed checkpoint containing W4A16_NVFP4 experts now rejects an explicit --moe-runner-backend flashinfer_cutedsl, even with --moe-a2a-backend none and SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16=1. The existing CuTe DSL runner already selects w4a16 and passes BF16 activations directly.

A focused comparison of the override function confirms that the base preserves this explicit backend, while this head raises ValueError. This verifies the configuration regression; I have not run this mixed-checkpoint combination on a GPU.

Could we keep auto -> marlin, allow the supported explicit CuTe DSL W4A16 configuration, and add a regression case to test_model_overrides.py? This does not affect the GB10 + flashinfer_cutlass configuration tested in this PR.

[BBuf]

https://github.com/sgl-project/sglang/actions/runs/34648716970/job/103467110642?pr=39126