Skip to content

Fix large FP8 and MXFP8 performance repros - #387

Closed
hwanseoc wants to merge 2 commits into
NVIDIA:developfrom
hwanseoc:hwanseoc/fix-fp8-perf-repros
Closed

Fix large FP8 and MXFP8 performance repros#387
hwanseoc wants to merge 2 commits into
NVIDIA:developfrom
hwanseoc:hwanseoc/fix-fp8-perf-repros

Conversation

@hwanseoc

Copy link
Copy Markdown
Member

@coderabbitai ignore

Why

--perf measures graph execution and should not construct full attention references. FP8 and MXFP8 performance repros still allocated or compared references, causing avoidable OOMs after successful cuDNN execution.

Large MXFP8 inputs also exceed Transformer Engine's CUDA grid-y limit when quantized as one flattened 2-D tensor. TE 2.15 passes at 4,194,240 rows and fails at 4,194,272 rows.

Changes

  • Skip FP8 and MXFP8 forward/backward reference construction and comparison under --perf
  • Keep deterministic backward exact-equality checks enabled
  • Use GPU forward output and stats as backward inputs in MXFP8 perf mode
  • Avoid expanded FP32 MXFP8 scale references in perf mode
  • Quantize oversized MXFP8 inputs in aligned chunks, concatenate compact results, then apply the global TE GEMM scale swizzle
  • Add an MXFP8 perf integration test that fails if either reference function is called

This is tester-only. It does not change graph JSON, tensor identity, extraction, or cuDNN graph construction.

Verification

NVIDIA B200, cuDNN backend 9.30:

  • FP8 forward --perf: passed
  • MXFP8 perf integration: passed; forward and backward references were not called
  • MXFP8 deterministic backward: dQ/dK/dV and all three amax outputs matched bitwise
  • MXFP8 non-perf reference comparison: passed
  • Exact DSV3 sequence-16384 repro that previously exceeded TE's launch limit: passed, zero CUDA OOMs

MXFP8 tests used the supported PyTorch 26.05 container with PyTorch 2.12.0a0, CUDA 13.2, and Transformer Engine 2.15. During the original investigation, the same patch moved the supplied set from 2/60 to 60/60 passes and the deduplicated raw-log set from 385/445 to 445/445 passes.

@hwanseoc

Copy link
Copy Markdown
Member Author

Superseded by #386. Its two focused commits were preserved there, so #386 now contains all architecture-independent repro fixes shared by #280 and #383.

@hwanseoc hwanseoc closed this Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant