Load exact GLM Spark projections and NVFP4 MTP from the QAD source

[voipmonitor]

Load the GLM-5.3-Flash QAD mtp-bf16 checkpoint with exactly the Spark projection precision selection: 520 main-model and eleven MTP BF16 matrices become MXFP8, and the MTP routed experts become NVFP4 W4A16. Conversion happens during loading; source checkpoint files remain unchanged. Strict coverage rejects missing targets, incomplete fused groups and independently enabled conversions outside the manifest.

Loader ownership releases replaced BF16 storage before kernel packing, and the NVFP4 encoder writes directly into its destination without intermediate BF16/FP16 scaling. Draft configuration retains target recurrent-state planning. Each MoE carries its own native/CSF scale policy through deferred preparation; matching main/draft CSF shapes share reconstruction buffers.

Canonical integration and compatibility

This is the canonical counterpart of beta #965/#966/#967. Its base is feat/csf-online-scales (#964); that depends on #949 -> #956. The branch also includes the existing #841 checkpoint-ownership commits unchanged because online conversion needs their direct-loading lifetime contract. Merge #841 and the CSF parents, then retarget this PR to dev/karmic-kraken. These prerequisite commits are not a second implementation of #841.

Deploy with B12X #446 -> #450, then #458 and #459. B12X #465 is required for environment-selected deterministic NVFP4 quality checks; its beta counterpart is #464. Native MXFP8 MTP expert PR #957 is a separate weight format and is not required by this NVFP4 W4A16 MTP recipe. No new B12X kernel implementation is introduced here.

• strict_targets remains opt-in; the example enables it.
• nvfp4_a16 requires BF16 activations and a supported W4A16 backend.
• Existing nvfp4_per_token keeps its SM100 restriction. Its encoded values may change because the encoder no longer rounds globally scaled BF16/FP16 weights before quantizing. The packed format is unchanged.
• VLLM_B12X_MOE_FP4_CSF=1 compresses FP4 expert weight scales losslessly. MXFP8 scale compression remains deferred.
• The existing VLLM_B12X_MOE_FP4_LAYER_MAX_INPUT_SCALE=w13 is a separate, opt-in activation calibration change. It shares gate/up ranges, retains down ranges per expert and preserves trained packed weights; the default remains source calibration (0).

Qualification and measurements

Implemented: exact projection selection, online MXFP8, NVFP4 W4A16 MTP, bounded loading/encoding, independent scale policy, generator and full launch guide. Qualified: the canonical source passes the 29 selected tests below and all applicable pre-commit hooks, including Ruff and mypy. Its loader/encoder/selection implementation matches the tested beta source; native libraries for the component tests come from the pinned image below. A complete model was not rerun on this canonical composition.

The full-model beta evidence uses installed code without source overlays in ghcr.io/local-inference-lab/vllm@sha256:ed1fe72b2c733492ebeda264f9c49906c677b901ff135997c1abfb042425e287, vLLM 58d05bc7626dd87ee401155e9b52e6c6fd92bd5d, B12X 66dba3aca0ab32e197bacb2eea120a067de767cc. TP2 ranks resolve the exact target set, start, generate and prefill. The checkpoint-header audit independently verifies Spark's eleven MXFP8 MTP projections and 288 NVFP4 W4A16 routed experts.

Research-only measurements on two RTX PRO 6000 Blackwell 96 GB GPUs, 600 W, 16365 MHz memory and dynamic graphics clocks:

Configuration / measurement	Result
No-MTP main loader allocation, selected BF16 -> MXFP8	92.071 -> 88.548 GB/rank; approximately 7.047 GB saved across TP2
Additional MTP CSF saving with shared scratch	217,253,376 bytes across TP2
TP2/DCP1, no MTP, chunk 3072, 3 GiB FP8 KV/rank: shared-w13 QAD versus paired Spark controls	+0.175% generated 32K; -0.054% WikiText 32K
Same workload: shared-w13 versus QAD source-scale policy	+4.122% generated; +1.601% WikiText
Shared activation-scale quality, 16 x 1024 WikiText tokens	KLD 0.100458 nats/token; NLL 1.084481 -> 1.085628; zero repeat KLD for both NVFP4 policies

No-MTP measurements do not describe MTP throughput. Short prompts did not benefit from sharing. The English quality sample does not establish broad capability equivalence; BF16 expert-reference runs failed repeat invariance and are excluded. Unsupported: byte parity with Spark, an exported checkpoint, MXFP8-CSF, maximum-context qualification or a universal speed/quality claim.

Launch guide, precision audit, paired prefill report, quality report. Each report links machine-readable receipts and exact source/configuration identities.

Tests

Canonical source in the pinned image's test environment, GPU10, OMP_NUM_THREADS=1:

/opt/venv/bin/python -m pytest --import-mode=importlib -p no:cacheprovider -q \
  tests/quantization/test_quantization_config_args.py \
  tests/quantization/test_online.py \
  tests/model_executor/test_eagle_quantization.py \
  tests/models/test_glm5next_pooled_indexer.py \
  tests/model_executor/model_loader/test_reload.py \
  tests/kernels/moe/test_b12x.py \
  -k 'strict_online or nvfp4_a16 or quantizes_original or uses_caller_workspace or releases_replaced or draft_quantization or selector_lazily_caches_fp32_head_projection or selector_dequantizes_mxfp8_weights_before_fp32_projection or reuses_kernel_when_weights_are_reprocessed or online_csf_releases_original_scale_tensors'

Result: 29 passed, 256 deselected. Existing PyTorch deprecation and Triton logical-operator warnings are retained. The test runner uses identical copies of tracked canonical Python files and the image's compiled extensions, including registry subprocesses. Checkpoint/target JSON and both launch shell levels are validated separately. Broader original encoder and loader coverage remains in #965.

Duplicate-work review: #964 compresses native FP4 scales; #957 loads already-MXFP8 MTP experts; neither implements this BF16 conversion recipe. The beta PRs are merged; there was no canonical counterpart. Tracked with dependencies and bounded serving evidence in #906. AI assistance was used for implementation, audit and validation. Human maintainer review remains pending.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• dev/*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 0f3ae35d-c53b-47bc-bc03-2ff558b44f7e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[voipmonitor]

Closing: this change is already in dev/karmic-kraken, through Luke's karmic integration. The remaining diff is only formatting or test annotations.