Add NVFP4 four-over-six converter - #8
Draft
wolfcomos wants to merge 9 commits into
Draft
Conversation
Wires TorchAO's NVFP4 four-over-six training prototype (torchao.prototype.moe_training.nvfp4_training.four_over_six, the adaptive map-to-4/map-to-6 block-scaling recipe with optional row-scaled activations) into the model-converter pattern: - NVFP4FourOverSixLinear: pure Linear leaf swap, forward through four_over_six_linear. No RHT and no stochastic rounding (the recipe targets RL and post-training); gradients use standard NVFP4, or bf16 GEMMs in row-scaled mode (a row-scaled four-over-six tensor has no columnwise form for the wgrad operand). - NVFP4FourOverSixLinearConverter: opt-in via converter config with fqns / err_mode / e4m3_scale_bound / row_scaled_activation knobs. - llama3_debugmodel_nvfp4_four_over_six recipe (fqns=['layers'], the lm_head stays bf16 since the vocab dim is not divisible by 128). Verified on 2x GB200 (FSDP=2, 50 steps, llama3 debugmodel): loss 8.15->2.881 vs the bf16 baseline's 8.21->2.880. Depends on the TorchAO branch nvfp4-four-over-six-rowwise (wolfcomos/ao#7). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extends the four-over-six integration to the miles NVFP4 RL recipe shape
(routed experts only, row-scaled activations, 1x16 weights, backward
overrides):
- NVFP4FourOverSixLinear / NVFP4FourOverSixLinearConverter gain
backward_override ('quantized' | 'high_precision' | 'dequantized',
mirroring TransformerEngine's NVTE_BACKWARD_OVERRIDE; None keeps the
previous defaults) and weight_block ('1x16' mirrors
NVTE_NVFP4_DISABLE_2D_QUANTIZATION=1). Knob combinations are validated
at config time.
- NVFP4FourOverSixGroupedExpertsConverter: class-factory _grouped_mm
override calling torchao four_over_six_grouped_mm (like the MXFP8
grouped-experts converter, with fqns include-list filtering). Four-
over-six needs no RHT sign vector and no stochastic-rounding seed, so
the stateless hook carries everything. Token dispatchers swap to
padded variants (pad_multiple=128).
- nvfp4_bf16_first_last_fqns: the miles --first-last-layers-bf16 analog.
- deepseek_v3_debugmodel_nvfp4_four_over_six[_dequantized] recipes: the
miles base point (row-scaled + MSE + bound 256 + 1x16 weights +
high_precision backward, experts-only allow-list) and the GLM-5.2
analog (dequantized backward, first/last decoder layer bf16).
- rl_grpo_qwen3_30b_a3b_varlen_nvfp4_four_over_six: the RL wiring. The
trainer and vLLM generator share one model_spec, so both actors run
the identical four-over-six forward -- the train/inference-consistency
point of the miles recipe, with bf16 master weights syncing as usual.
Unit tests: 35 passed (TE 2.19 devel container), covering converter
targeting, knob plumbing, first/last-bf16 windows, and config-time
validation. GPU smoke on GB200 verifies the hook's (E, K, N) -> (E, N, K)
weight orientation, ragged-group padding, and both grads end to end.
Depends on the TorchAO branch nvfp4-four-over-six-rowwise
(wolfcomos/ao#7, through commit c47af4a7e).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TorchAOTokenDispatcher swapped in by the grouped-experts converter already 128-aligns every expert group, but it over-allocates the activation buffer past offs[-1], while torchao's four_over_six_grouped_mm requires offs[-1] == A.shape[0]. Passing the buffer through with pad_token_groups_for_grouped_mm=True violates that contract, and simply disabling padding would let the unwritten tail rows feed the per-group amaxes and silently poison the last expert's quantization in per-tensor mode. Slice the logical rows before the op, skip the op's own padding (the dispatcher's alignment already satisfies it), and zero-extend the output so downstream shapes match; pad routes zero gradients to the tail and the unpermute never reads those rows.
The quantized grouped GEMM reads expert group offsets on the host and the row-scaled mode loops dense GEMMs per group, both of which CUDA-graph capture forbids (cudaErrorStreamCaptureUnsupported in the debugmodel's captured fwd+bwd). miles runs its quantized recipes with CUDA graphs off for the same class of reason.
The row-scaled grouped forward host-reads the group offsets and loops dense GEMMs per group; under torch.compile fullgraph (nonstrict_trace + capture_scalar_outputs) the offsets become unbacked SymInts and the loop bounds guard on data-dependent expressions with no graph-break escape. Fail at config time instead of at trace time.
has_quantization enumerated only the Float8/MXFP8/NVFP4Linear configs and the float8/mxfp8 grouped-experts caches, so a model quantized by the four-over-six converters reported has_quantization=False and MFU was computed against the bf16 peak.
Covers the config-time ValueError for model compile + row-scaled grouped four-over-six, and has_quantization returning True for models converted by either four-over-six converter (and False for the stock debugmodel).
Mirror the four-over-six grouped converter knobs on the MXFP8 one for the recipe-ablation arms: - Config.fqns: allow-list of module FQN substrings; when non-empty, only matching GroupedExperts are converted (and only their dispatchers swapped), same semantics as MXFP8LinearConverter and the NVFP4 converters. - Config.backward_override: None/"quantized" keeps the quantized MXFP8 backward, "high_precision" and "dequantized" select the torchao grouped-GEMM backward overrides; validated at converter init and threaded through the dynamic experts Config into MXFP8TrainingOpConfig. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reduced-scale replication of the lmsys MXFP8/NVFP4 RL recipe ablation (10-step GRPO on dapo-math-17k, 4 GB200: trainer TP2/EP2 + vLLM generator TP2/EP2) exercising the MXFP8 and NVFP4-4over6 grouped-experts converters end to end. No divergence in any arm; quantization shows up only in rollout-vs-trainer logprob diff and PPO clip fraction, with no backward-mode (high_precision vs dequantized) effect on either. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Wires TorchAO's NVFP4 four-over-six training prototype into torchtitan via
the model-converter pattern, opt-in through converter config.
Four-over-six is an adaptive NVFP4 block-scaling recipe: every FP4 block is
encoded twice — the standard map-to-6 scale and a 1.5x-expanded map-to-4 scale
— and the candidate with the lower MAE/MSE dequantization error is stored.
Forward GEMM operands are four-over-six (activations 1x16, optionally
row-scaled with one FP32 global scale per row; weights 16x16). Gradients use
standard NVFP4, or bf16 GEMMs in row-scaled mode (a row-scaled four-over-six
tensor has no columnwise form for the wgrad operand). No RHT and no stochastic
rounding: the recipe targets RL and post-training.
NVFP4FourOverSixLinear: pureLinearleaf swap (likeMXFP8Linear),forward through
four_over_six_linear. TP is not wired up.NVFP4FourOverSixLinearConverter: config knobsfqns,err_mode(
mae/mse),e4m3_scale_bound(256/448),row_scaled_activation.llama3_debugmodel_nvfp4_four_over_sixrecipe (fqns=["layers"]; thelm_head stays bf16 since the vocab dim is not divisible by 128).
Evidence (2x GB200, FSDP=2, llama3 debugmodel, 50 steps)
The underlying torchao quantizer is verified bitwise (384/384 comparisons)
against the reference CUDA kernels for this recipe; see the TorchAO PR for the
full parity evidence.
🤖 Generated with Claude Code