[Qwen4] Load nvidia/Qwen3.8-Flash-Next-NVFP4 (ModelOpt MIXED_PRECISION)

[Jiminator]

Motivation

nvidia/Qwen3.8-Flash-Next-NVFP4 uses ModelOpt MIXED_PRECISION: NVFP4 routed experts, an FP8 PLE n-gram table, and FP8_BLOCK_SCALES MTP experts. After #37500, main has the generic mixed-precision infrastructure but still lacks the dispatch and PLE storage selection required by this export.

Upstreams the remaining loader changes from #38121 / #37752. The loader garbage collection already present in main is retained.

Modifications

• Resolve both model.* and model.language_model.* checkpoint prefixes and normalize omitted exclusion lists.
• Dispatch FP8_BLOCK_SCALES linear/MoE layers to dynamic 128x128 block FP8 methods, preserving the existing FP8_PB_WO path.
• Preserve the mixed config for checkpoints with quantized mtp.* layers, and port the original sm100 mixed-expert runner selection (including W4A16 routing to Marlin).
• Select FP8 PLE storage from the mixed config before allocating the offloaded table.
• Fall back to the Triton FP8 MoE runner when the global runner is FlashInfer CUTLASS/CuteDSL, allowing the FP8 MTP head to coexist with NVFP4 target experts.
• Port mixed-loader and FP8 runner regression tests.

Load this export without --quantization, and explicitly use --moe-runner-backend flashinfer_cutlass. At TP=1 its in-checkpoint MTP head can load directly. At TP=2, the MTP width 640 gives 320 per shard, which is not divisible by the 128 block; use --speculative-draft-model-path RadixArk/Qwen3.8-Flash-Next-NVFP4 --speculative-draft-model-quantization modelopt_fp4 when enabling speculation.

Accuracy Tests

September 11, 2026: current-main combined deployment validation

Tested #38568 + #38569 + #38570 together on main@165d8dd17736fa4938f2e73018f26835cd6e0778, combined merge 69d38ccbd8a03646cae9168c971b3783883c2903. PR heads: #38568 626f614f7bcd5dc531de5c268a694ff1a9f31f28, #38569 825663897b1fd9ee5c341eeb113b795e4ecfc3f8, #38570 b14d2c55f37f8d3dc2b5fe21175b56fbe62771e1. All merged cleanly; no additional source fixes were needed.

Native host installation on 1x DGX Spark / GB10, TP=1, rebuilt from the merged checkout without a patched Docker image. Model: nvidia/Qwen3.8-Flash-Next-NVFP4, checkpoint revision fab0aecb760cec45227f6656abcaafa11abca87a. GSM8K first 200 test questions, greedy chat API, thinking disabled, 8,192-token generation limit:

Configuration	Accuracy	Request errors / invalid / truncated	Ready	Aggregate output
MTP, concurrency 8	195/200 (97.5%)	0 / 0 / 0	763.26 s	92.87 tokens/s
Non-MTP, concurrency 24	194/200 (97.0%)	0 / 0 / 0	633.17 s	89.53 tokens/s

Both smoke responses were correct and CUDA graphs were active. MTP used the expected FP8 Triton expert fallback and reported average draft acceptance 3.52925. No serving NaN/OOM/CUDA-error/exception reports appeared before intentional shutdown; shutdown cancellation diagnostics were recorded separately.

Serving flags: TP=1, FlashInfer CUTLASS MoE/FP4 backends, page size 64, chunked prefill 4096, requested context 262144, static memory fraction 0.85, and file-backed PLE with a fresh local-NVMe file per boot. MTP: NEXTN, 3 steps, top-k 1, 4 draft tokens, 8 maximum requests, 40 Mamba slots. Non-MTP: extra_buffer_lazy, 24 maximum requests, 96 Mamba slots.

The isolated host environment was updated from current main and native extensions rebuilt (PyTorch 2.13.0+cu130, FlashInfer 0.6.18, Triton 3.7.1, sglang-kernel 0.4.6.post1, Transformers 5.12.1). Runs reused previously compiled host kernels with MAX_JOBS=2; startup timings exclude cold kernel compilation. Throughput includes prefill and the lower-concurrency tail and is not an isolated measurement of this PR's speed impact.

Scope: combined TP=1 deployment validation, not independent deterministic reproduction of the PDL race, TP>1 prefetch validation, or the full 1,319-question benchmark. Effective max input was 180,282 tokens with MTP and 262,138 without MTP; no full-length 262K prompt was tested. Both servers were stopped and their generated PLE files removed after evaluation.

Latest-main necessity check: source-reviewed main@45715e7f20629d1813ff8f543a23bb437128db2e after the deployment run. ModelOpt mixed precision still lacks FP8_BLOCK_SCALES dispatch, the Qwen PLE mixed-config dtype handling, and the conditional quantized-MTP gate; the relevant loader/runner files are unchanged from the tested base. The combined PRs merge cleanly onto this newer revision. The accuracy numbers above remain measurements of 165d8dd177, not a GPU rerun on this newer base.

Earlier validation

Unit checks: mixed-loader 18 passed, FP8 fallback 3 passed, fusion gates 34 passed, Qwen MoE overrides 2 passed. All changed-file pre-commit hooks pass.

Fresh host-side Python 3.12 environment on 1x DGX Spark / GB10 (sm121); no patched SGLang Docker image. NVIDIA checkpoint revision fab0aecb760cec45227f6656abcaafa11abca87a; PyTorch 2.13.0+cu130, FlashInfer 0.6.18, Triton 3.7.1, sglang-kernel 0.4.6.post1, Transformers 5.12.1. Native extensions were built from the checked-out source.

Combined #38568 + #38569 (head 062c0f956f) + #38570 on main 52fecfdf09, local merge a1fe796d71:

Configuration	Ready	GSM8K chat, first 200 test questions	Invalid / request errors / capped
MTP, concurrency 8	728 s	196/200 (98.0%)	0 / 0 / 0
Non-MTP, concurrency 24	618 s	194/200 (97.0%)	0 / 0 / 0

Greedy chat API, thinking disabled, 8,192-token generation limit. Both smoke responses were correct; CUDA graphs and file-backed FP8 PLE gathers were active. The MTP head used the expected FP8 Triton fallback, and server-reported average draft acceptance was 3.5427.

Upstream advanced during testing. Both commands were also booted and served on newer main db272201a2 + the same PR heads, local merge 29b665fc55: MTP ready in 728 s, 23/24 sanity questions correct; non-MTP ready in 623 s, 24/24 correct. All 48 requests completed with no errors, invalid answers, or truncation. These smaller checks are distinct from the 200-question results above. No serving NaN/OOM/exception reports occurred before intentional shutdown in any final run; no internal NaN instrumentation was used.

Common flags: --tp 1 --moe-runner-backend flashinfer_cutlass --fp4-gemm-backend flashinfer_cutlass --page-size 64 --chunked-prefill-size 4096 --context-length 262144 --reasoning-parser qwen3 --mem-fraction-static 0.85 --ple-offload-embedding --ple-offload-backend file, local checkpoint path, and a fresh local-NVMe --ple-offload-dir per boot. MTP adds --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --max-running-requests 8 --max-mamba-cache-size 40; non-MTP adds --mamba-radix-cache-strategy extra_buffer_lazy --max-running-requests 24 --max-mamba-cache-size 96.

Cold-build note: default parallel FlashInfer JIT compilation alongside the resident model drove swap past 82 GiB, so that attempt was stopped. The remaining CUDA module was compiled with the model unloaded; successful runs used that freshly built cache and MAX_JOBS=2. Autotuning remained enabled. The successful startup times exclude the one-time isolated kernel build.

Effective maximum input on the newer base was 172,730 tokens with MTP and 262,138 without MTP. The configured 262K context is not a claim of tested 262K prompt accuracy or sufficient MTP KV capacity.

Speed Tests and Profiling

On the 200-question runs against 52fecfdf09 plus the PRs, aggregate generated throughput was 97.39 tok/s with MTP (76,729 tokens / 787.82 s) and 89.91 tok/s without MTP (80,776 / 898.40 s). These are chat-evaluation aggregates including prefill and low-concurrency tail time, not a dedicated bench_serving comparison.

Checklist

• Format changed code with pre-commit.
• Port and run loader/runner regression tests.
• Validate both single-node GB10 deployment configurations from upstream main plus the PRs.
• Run the full 1,319-question evaluation and validate other hardware/TP configurations.

---

CI States

Latest PR Test (Base): ❌ Run #34592883757
Latest PR Test (Extra): ❌ Run #34592883721
Latest PR Test (AMD ROCm 10): ❌ Run #34592883572

[Jiminator]

Superseded by combined draft PR #39126, which preserves all three component changes and includes the DGX Spark MTP/non-MTP accuracy and throughput results. Closing this separate PR in favor of the combined review.