[Model][SM70] Adapt Qwen3.8 Flash Next NVFP4

[yangzhuxinyzx]

Public migration note (2026-08-27): this Draft is the public continuation of private PR #17, mirrored at exact head 5abeaa6 after public main resumed through #342. The original implementation notes and evidence are preserved below.

This work substantially overlaps public #338. Keep both Drafts independent until a line-by-line consolidation chooses one integration path; this PR does not replace or overwrite the external contributor branch.

Purpose

Bring up RadixArk/Qwen3.8-Flash-Next-NVFP4 on 4x V100-SXM2-32GB with TP4, V2, FP16 compute, SM70 TurboMind W4A16 routed experts, and direct pinned-host PLE/n-gram residency.

Base and model

• Integration: private/main
• BASE_SHA: d63e949
• Hugging Face model revision: 7b719225242aacd3dbd3f9407468c2ee9a9d2594
• ModelScope master was verified against that fixed revision: 419 file sizes and 208 comparable LFS SHA-256 values match. The completed local snapshot then passed a full 419-file SHA-256 verification with zero missing, size-mismatched, or hash-mismatched files.
• Upstream implementation references: vLLM PR 53896 and SGLang PR 36497.

Implemented route

• Adds Qwen4Exp config, registry, model, QSA, GDN, hyperconnection, PLE, and weight-loading support.
• Defaults the Qwen4Exp quantized MoE route to V2.
• First-pass SM70 contract is language-model-only, TP4/PP1, MTP/speculative decoding off, prefix caching off, and FP16 KV cache.
• Routes ModelOpt NVFP4 routed experts through the existing SM70 TurboMind W4A16 kernel for the real (2560, 640, 512, top-10) geometry and TP1/TP2/TP4.
• Allocates and loads each PLE shard directly in pinned host memory. The full table remains host-resident; selected FP8 rows are read through a stable UVA view, transferred asynchronously, and converted to FP16.
• Adds V2 QSA circular-cache grouping, one-block circular tables, scheduler-manager conversion, compressed-page reshaping, warmup support, and committed-token PLE state.
• Rejects unvalidated multimodal, prefix-cache, and speculative routes with actionable errors instead of silently selecting incompatible private APIs.

Test result

• Static: Ruff, Ruff format, compileall, and git diff checks pass for all 55 changed Python files.
• Focused CPU/config suite: 76 passed, 7 CUDA-only skips.
• The real checkpoint config constructs as Qwen4Exp in V2 with ModelOptNvFp4Config, TP4, FP16, pinned-host PLE, and language-model-only mode.
• Full real-config 48-layer meta construction succeeds for QSA/GDN/HC/PLE and selects ModelOptNvFp4SM70MoEMethod(use_a16=True). TP4 expert selection and geometry are covered separately.
• Pre-final real V100 snapshot: 63 focused tests passed, including Triton slot mapping, pinned-host UVA lookup, compressed QSA pages, committed-token PLE state, and NVFP4 route gates.
• Real V100 QSA pre-indexer and HC kernels match PyTorch references.
• CUDA-graph captured operator probe: 95.81 us/replay for random 16x160 PLE row lookup plus FP16 conversion from a 228.9 MiB pinned FP8 table. This is not an end-to-end throughput claim.
• General KV-cache suite: 69 pass; one unchanged integration-base fixture failure omits max_in_flight_tokens.

Memory estimate

• Total checkpoint payload: about 125.910 GiB.
• PLE: about 47.684 GiB total, 11.921 GiB pinned host memory per TP rank.
• Estimated post-repack device weights: about 21.3 GiB/rank; estimated transient weight peak: about 22.1 GiB/rank before runtime allocations.
• Exact cache planning estimate: 0.448 GiB/rank at 32K, 1.649 GiB/rank at 128K, and 3.241 GiB/rank at 262144 tokens.
• These are planning estimates, not measured full-engine peaks.

Pending acceptance gates

• Final real-V100 focused rerun after a device is released.
• Full TP4 load at 32K, followed by 128K and 262144 memory boundaries.
• Deterministic output-quality checks.
• Matched eager/CUDA-graph benchmark: 8192 input, 512 output, pure decode over emitted tokens 33-512, target at least 100 emitted tokens/s.
• MTP remains a separate follow-up and must report accepted length and emitted throughput separately.

The PR remains Draft. Full-model memory, output quality, and the 100 tokens/s target are not claimed yet.

[yangzhuxinyzx]

公开迁移后的并发增量已同步：私有 #17 新增提交 cf5c44a1aa65（[Bugfix] Load fused Qwen4Exp MTP experts）已无改动快进到本 Draft。该增量含 DCO sign-off，git diff --check 5abeaa6..cf5c44a 通过；仍需在本 PR 内按当前 main 完成源码/质量审计后再决定合并。

[yangzhuxinyzx]

Latest-main audit update for locked implementation head ceb543c05573c9bcbf1cc9563bab0c2e8c746ae3:

• The 79-file Qwen4Exp adaptation merges cleanly over public main, but the original head did not pass the changed-file gate: one format drift, 13 source mypy findings, four test typing findings, and four forbidden torch.cuda API uses.
• Source review also found two concrete runtime defects behind those static findings: the non-QSA full-attention path called the inherited attention module without its required output argument, and the new recursive stacked-weight loaders used the module itself as a missing-attribute default before calling weight_loader.
• I repaired these plus heterogeneous Mamba annotations, speculative-config narrowing, PLE accelerator device handling, QSA output-buffer compatibility, test typing, and accelerator synchronization in stacked repair PR [Bugfix][SM70] Repair #345 Qwen4Exp integration gates #359. No model identity gate, numerical policy, or performance default was changed.
• The repaired 14-file delta passes all changed-file pre-commit hooks and Python compilation. Focused CPU/source contracts: 74 passed, 3 CUDA-only skipped.
• All eight V100s are currently owned by an unrelated PP2/TP4 service, so I did not stop or preempt it. The existing recorded real-V100 evidence remains the GPU acceptance record for this source audit.

Next: merge #359 after its latest-head remote gate, update #345 onto the then-current main, verify the exact resulting tree, and merge if unchanged.