fix(quantize): add NVFP4 default type mapping and scale tensors - #22897
fix(quantize): add NVFP4 default type mapping and scale tensors#22897t-timms wants to merge 2 commits into
Conversation
llama_ftype_get_default_type() was missing LLAMA_FTYPE_MOSTLY_NVFP4, causing undefined behavior when quantizing to NVFP4. Also adds the NVFP4 option to the quantize tool. NVFP4-quantized weights now emit per-tensor .scale and .input_scale (F32 scalar) tensors alongside the quantized blocks. These are required by the CUDA MMA dequant path for correct output. Scale values default to 1.0 for freshly quantized weights since per-block UE4M3 scales already capture magnitude during quantization optimization. Verified: 168 NVFP4 weights + matching scale pairs, Qwen2.5-0.5B loads and infers correctly (341.5 t/s on RTX 5070 Ti).
Replace hardcoded 1.0f .scale values with MSE-optimal correction factors computed from the original F32 weights and dequantized NVFP4 blocks: scale = sum(f32 * dequant) / sum(dequant^2). This corrects global reconstruction bias that per-block UE4M3 scales cannot capture. Verified 119/168 weight tensors received non-trivial scale corrections (e.g. 0.9976-1.0076 range), 462.4 t/s inference on RTX 5070 Ti.
|
Hi @t-timms, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
|
I found this bug while testing NVFP4 quantization on my RTX 5070 TI - llama-quantize NVFP4 was crashing because the function that maps file types to the quantization types had no case for NVFP4. The fix adds that missing mapping and emits .scale/.input_scale tensors that the CUDA MMA dequant path requires for correct output, computing the optimal scale from sum(original × dequantized) / sum(dequantized²). I verified this on Qwen2.5-0.5B F16 to NVFP4 - all 168 weight tensors got matching scale pairs, and the model loads and infers correctly on Blackwell hardware. AI tools were used for code exploration, but I directed the changes and understand every line. |
|
Friendly ping — this one has been quiet for a couple of months. The branch still merges cleanly against master. Happy to rebase, add test coverage, or rework the approach if a maintainer has a preferred direction. |
1 participant
Summary
llama-quantize NVFP4by adding the missingLLAMA_FTYPE_MOSTLY_NVFP4case tollama_ftype_get_default_type().scaleand.input_scale(F32 scalar) tensors alongside NVFP4-quantized weights — required by the CUDA MMA dequant path for correct output.scaleas the MSE-optimal correction factor (sum(f32 * dequant) / sum(dequant^2)) rather than hardcoding 1.0 — corrects global reconstruction bias that per-block UE4M3 scales cannot captureChanges
src/llama-quant.cpp: default type mapping + scale tensor infrastructure + optimal scale computation via dequantizationtools/quantize/quantize.cpp: NVFP4 option in the quantize toolVerification
llama-quantize --dry-run NVFP4no longer triggers undefined behaviorRelated: #22858 (closed — this replaces that partial fix)