Skip to content

[MXFP8] Add fqns filtering and a guarded backward_override to the grouped experts converter - #15

Draft
wolfcomos wants to merge 1 commit into
mainfrom
4over6/titan3-mx-override
Draft

[MXFP8] Add fqns filtering and a guarded backward_override to the grouped experts converter#15
wolfcomos wants to merge 1 commit into
mainfrom
4over6/titan3-mx-override

Conversation

@wolfcomos

@wolfcomos wolfcomos commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Independent PR (no stack). Companion torchao PR: wolfcomos/ao#10.

Summary

Two additions to MXFP8GroupedExpertsConverter:

  • fqns: like the Linear converters, only GroupedExperts.Config entries whose FQN contains a match are converted; an empty list keeps the previous convert-everything behavior.
  • backward_override: plumbs torchao's MXFP8 grouped-GEMM backward override through the converter (None/"quantized" keeps the quantized MXFP8 backward, "high_precision" computes both gradients with plain grouped GEMMs on the saved high-precision operands, "dequantized" computes them from the dequantized forward operands). The value is validated at converter build.

Design notes

  • The dataclasses.replace() only runs when the knob is set, and is guarded: the op config is replace()d with backward_override only when config.backward_override is not None, behind a field-presence check that raises an actionable ValueError ("the installed torchao does not support backward_override for MXFP8 grouped GEMMs") when the installed torchao's MXFP8TrainingOpConfig predates the field. Stock MXFP8 recipes therefore keep working on every released torchao; only actually setting the new knob requires a torchao build with the matching field.
  • Runtime dependency is opt-in: the feature requires the ao-side MXFP8 backward_override PR at runtime ([moe training] Add backward_override to the MXFP8 grouped GEMM ao#10), but this titan change merges safely ahead of any torchao pin bump because the default path never touches the field.
  • fqns filtering follows the exact substring-match convention of the Linear converters and the NVFP4 four-over-six converters, so per-layer mixed-precision recipes (e.g. first/last decoder layers in bf16) work for MXFP8 grouped experts too.

Test plan

CPU-only (container without --gpus):

python -m pytest tests/unit_tests/cpu/test_quantization.py -q
32 passed, 14 warnings in 28.91s

New tests: test_mxfp8_backward_override_guard_on_older_torchao (older-torchao guard: unset knob builds, set knob raises the actionable error), test_mxfp8_grouped_converter_fqns_filtering (targeted layer converts, others stay stock; empty fqns converts all), and test_mxfp8_grouped_converter_backward_override_plumbs (invalid value rejected at build; valid value reaches every converted expert config). Import checks of torchtitan.components.quantization{,.mx} pass. Lint: ufmt (black 22.12.0 / usort 1.0.5), flake8 with the repo plugin pins, pydoclint, and codespell all clean on the changed files.

GPU results for the dequantized override come from the assembled stack tip on GB200, since CI has no SM100 runner for MXFP8 grouped GEMMs.

Draft on the fork to stage the upstream submission; supersedes the MXFP8 portions of the exploratory branch

Review pass (2026-08-30)

  • backward_override is typed Literal["quantized", "high_precision", "dequantized"] | None on the converter Config (config-parse-time validation, matching recipe_name's idiom).
  • "quantized" now normalizes to the stock quantized backward before the torchao field probe, so it works on torchao builds that predate the knob (tested against a simulated older torchao).
  • The converter __init__ fail-fasts when an actual override is set but the installed torchao's MXFP8TrainingOpConfig lacks the field; the guard message is capitalized and names the remediation.
  • mxfp8.md documents the grouped converter's fqns and backward_override options.
  • CPU suite re-run in the vLLM 26.08 container: 32 passed / 0 failed.

…ed experts converter

Two additions to MXFP8GroupedExpertsConverter:

- fqns: like the Linear converters, only GroupedExperts.Config entries
  whose FQN contains a match are converted; an empty list keeps the
  previous convert-everything behavior.
- backward_override: plumbs torchao's MXFP8 grouped-GEMM backward
  override (None/"quantized" keeps the quantized MXFP8 backward,
  "high_precision" uses the saved high-precision operands,
  "dequantized" uses the dequantized forward operands). The value is
  validated at converter build.

The op config is only dataclasses.replace()d when the knob is actually
set, and the replace is guarded by a field check with an actionable
error when the installed torchao's MXFP8TrainingOpConfig predates
backward_override -- stock recipes keep working on every released
torchao, and only setting the new knob requires a torchao build with
the matching MXFP8 backward_override support.

Add CPU unit tests for the older-torchao guard, fqns filtering, and
backward_override plumbing/validation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@wolfcomos
wolfcomos force-pushed the 4over6/titan3-mx-override branch from 74661f3 to 054c4d4 Compare August 31, 2026 02:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant