quant: add native mixed NVFP4/MXFP4 plus NF3 MoE serving - #104
Conversation
A quantization method for per-expert mixed-precision MoE checkpoints: high-saliency experts in NVFP4 (or MXFP4), a low-saliency tail in NF3 3-bit, both served through the b12x W4A16 fused MoE kernel as two preplanned launches per layer sharing one route geometry and one topk_sum (CUDA-graph-safe; TC-decode direct-topk path at M<=8). Detection: a modelopt-tagged NVFP4 checkpoint whose quantization config carries `hybrid_bit_map` (per-layer, per-expert bit widths; 4 = kept, 3 = NF3) selects this method automatically. `kept_format` optionally switches the kept tier to MXFP4. No environment variables; scratch is sized from max_num_batched_tokens. Replaces the out-of-tree runtime graft that has served this configuration in production since 2026-07-05. Requires b12x with the nf3_2p1 W4A16 layout (local-inference-lab/b12x#31). Co-authored-by: Claude (Fable 5) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Canonical B12X pairing: merge local-inference-lab/b12x#31 (already on master) and #34 before production use of the hybrid checkpoint. #34 preserves the exact preplanned W4A16/NF3 tile geometry across the custom-op boundary. The current B12X master plus #28/#32/#34/#35 combined suite passed 120 tests (1 skipped) on SM120. |
3 participants
Summary
Add native serving for per-expert mixed-precision MoE checkpoints such as
madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid: a high-saliency NVFP4 or MXFP4 tier and an NF3 tail selected by the checkpoint'shybrid_bit_map.The method auto-detects modelopt-tagged hybrid checkpoints, demultiplexes expert weights into compact tier-local storage, and executes both tiers with preplanned B12X W4A16 launches sharing routing and scratch state. Layers absent from the map, including an MTP head, remain uniform NVFP4. An explicit incompatible
--quantizationselection is respected.No environment variable or runtime monkey patch is required. This is the current-FF replacement for #92 and requires B12X master with merged NF3 support from local-inference-lab/b12x#31.
Validation
python -m pytest -q tests/quantization/test_nvfp4_nf3_hybrid.py: 5 passedruff check,ruff format --check, andgit diff --check