Quantize exact GLM Spark projections and MTP experts on load

[voipmonitor]

GLM's NVFP4 mtp-bf16 checkpoint contains BF16 attention/shared projections and BF16 MTP routed experts. This adds an opt-in loading recipe that selects exactly the 531 projections in the pinned Spark checkpoint, converts them to MXFP8, and converts only the MTP routed experts to NVFP4 W4A16. Source checkpoint files are unchanged.

Strict target coverage checks the constructed model before loading. Draft quantization preserves target recurrent-state planning, loader ownership releases replaced BF16 weights before backend packing, and the NVFP4 encoder writes directly into preallocated outputs. Each MoE retains its own scale-compression policy across deferred preparation; the example inherits main-model CSF and shares its reconstruction buffers with MTP.

Compatibility

Existing target matching remains permissive unless strict_targets=true. nvfp4_a16 requires BF16 activations and a supported W4A16 backend. Existing nvfp4_per_token retains its SM100 restriction; its encoded values can change because the encoder now avoids an intermediate BF16/FP16 rounding. The packed format is unchanged. MXFP8 scale compression is not enabled, and no B12X source change is required.

Implementation and qualification

• Implemented: exact selection, online MXFP8 projections, online NVFP4 W4A16 MTP, bounded encoding/loading, independent main/draft scale policy, configuration generator and serving guide.
• Qualified: both TP2 ranks resolve exactly 520 main targets and 11 MTP projections plus the routed MoE; startup, C1/C8 decode, 4K/32K prefill, and native/CSF draft-scale selection pass on RTX PRO 6000 Blackwell 96 GB.
• Research-only: three short decode repetitions and a 16,384-token English WikiText quality sample. No broad capability or long-context capacity claim.
• Unsupported by this qualification: MXFP8-CSF, checkpoint export, byte parity with the pre-encoded Spark checkpoint, and untested hardware/parallelism configurations.

The pinned public image is ghcr.io/local-inference-lab/vllm@sha256:2230db60afb4fd06dc2ef2b7f7f27dc7f78d7a50be70a08461b4df9fa5e90732, with explicit frozen Python overlays, vLLM base 93dabce32fd4d5355662608296e64d720711cdcc and unchanged B12X a77b3f85e5e2a81315ff4a90088912f187708005. This does not claim the unmodified image already contains the feature.

Measurements

TP2, 600 W limits, FP8 KV, 3 GiB KV per rank, 16,365 MHz loaded memory clocks and dynamic graphics clocks:

Comparison	Result
Main loader allocation, no MTP	92.071 -> 88.548 GB/rank; 7.047 GB saved across TP2
No-MTP decode, BF16 -> MXFP8 projections	C1 121.705 -> 145.071 tok/s; C8 511.332 -> 558.631 tok/s
No-MTP 32K prefill, 3072-token budget	12119 -> 12681 tok/s on the same GPUs; online reload repeat 12760 tok/s
No-MTP 32K prefill, 1024-token budget	BF16 7796; online MXFP8 6022 (repeat 6031); serialized Spark 6009 tok/s
MXFP8 + NVFP4 MTP, inherited CSF, 1024-token budget	C1 215.569; C8 707.298 tok/s; 32K prefill 5792 tok/s
MXFP8 + NVFP4 MTP, inherited CSF, 3072-token budget	C1 212.874; C8 704.562 tok/s; 32K prefill 12295 tok/s, TTFT 2.68 s
Additional MTP CSF memory saving with shared scratch	217,253,376 bytes across TP2, verified by buffer addresses and loader allocations
Source-to-MXFP8 KL	0.099063 nats/token; next-token NLL increase 0.001885

The serving example recommends --max-num-batched-tokens 3072. A 1024-token budget amplifies per-chunk work and cannot enter the existing B12X large-prefill regime, which requires at least 2048 rows. Serialized Spark reproduces the same approximately 6K tok/s slowdown at 1024 and measures 13360 tok/s at 3072. This isolates the scheduler-budget effect from online weight preparation; no encoder or B12X source change is required. The two source checkpoints have different weight values, so their remaining throughput difference is not an isolated offline-versus-online encoding comparison.

Quality uses deterministic MoE reduction: same-model repeat KL is exactly zero for both models. The normal fast path has nondeterministic MoE accumulation and is used for the performance measurements. The measured KL includes the MXFP8 backend's activation arithmetic; it is against the mixed NVFP4 source checkpoint, not a fully BF16 model. MTP acceptance is 52.59%; a BF16-draft quality comparison is not claimed.

The validation report records conditions, limits, artifacts and rejected measurements. Machine-readable results include all accepted repetitions and SHA256 identities. The serving guide provides a complete command.

QAD preservation and activation

The TP2 Docker recipe loads the QAD mtp-bf16 source, preserving its main expert weights, router weights and expert-specific activation calibration. Exactly the Spark projection selection is converted to MXFP8 in memory; MTP experts become NVFP4 W4A16. It does not export another on-disk checkpoint or substitute saved Spark calibration. The guide provides the complete beta-container command, 3072-token prefill budget, fixed KV allocation, MTP controls and generated strict target configurations. Both shell levels parse successfully, and generated JSON matches the qualified recipe exactly.

Controlled Spark/QAD compositions distinguish activation calibration from weight encoding. Replacing only main NVFP4 expert weights and weight scales gives a paired 32K throughput change of -0.015%. Replacing only input activation scales gives -1.191%, versus -1.599% for complete online QAD, on fixed token inputs with bracketing Spark controls. The preceding generated-prompt benchmark gives approximately -3.4% and -4.2%, respectively. Uniform Spark activation scales permit shared input quantization; QAD uses per-expert calibration and distinct legal MoE configurations. These are bounded performance measurements; hybrid model quality is unqualified. The linked report contains source identities, code contracts, raw result objects and limits.

Tests

All pre-commit hooks passed, including Ruff, mypy, Markdown and configuration checks. The final combined source check passed 29 cases; the deferred-policy reproducer fails six of eight cases before the correction and passes all eight afterwards. Four activation/encoder/kernel-reuse tests also pass. Broader overlapping checks cover 32 loader unit tests, 43 online/configuration tests and TP-consistent NVFP4 encoding.

Pytest selections executed in the pinned development environment (repository paths shown relative to the checkout):

python -m pytest --import-mode=importlib -q \
  tests/quantization/test_quantization_config_args.py \
  tests/quantization/test_online.py \
  tests/model_executor/test_eagle_quantization.py \
  tests/models/test_glm5next_pooled_indexer.py \
  tests/model_executor/model_loader/test_reload.py \
  tests/kernels/moe/test_b12x.py \
  -k 'strict_online or nvfp4_a16 or quantizes_original or uses_caller_workspace or releases_replaced or draft_quantization or selector_lazily_caches_fp32_head_projection or selector_dequantizes_mxfp8_weights_before_fp32_projection or reuses_kernel_when_weights_are_reprocessed or online_csf_releases_original_scale_tensors'

python -m pytest --import-mode=importlib -q tests/kernels/moe/test_b12x.py \
  -k online_csf_releases_original_scale_tensors

python -m pytest --import-mode=importlib -q tests/quantization/test_online.py \
  -k 'a16_rejects or nvfp4_quantizes_original or reuses_kernel_when_weights_are_reprocessed'

Duplicate-work review: #964 concerns compression of existing native FP4 expert scales, and #957 concerns native MXFP8 MTP MoE weights. This PR adds BF16 online conversion, strict Spark projection coverage and the separate NVFP4 W4A16 draft recipe; it does not replace those PRs or modify the deferred MXFP8-CSF prototype.

AI assistance: implemented and tested with OpenAI Codex.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• dev/*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 8d5287c9-f35f-4273-b8b5-dbe0909e7d7d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}