Skip to content

tests: xfail moe quantization classes mxfp8_bf16 UTs on sm103 - #1754

Merged
yzh119 merged 2 commits into
flashinfer-ai:mainfrom
jimmyzho:jimmzhou/fusedmoe_fp4bf16
Sep 22, 2025
Merged

yzh119 merged 2 commits into
flashinfer-ai:mainfrom
jimmyzho:jimmzhou/fusedmoe_fp4bf16

Conversation

@jimmyzho

Copy link
Copy Markdown
Contributor

📌 Description

Temporarily marking test_trtllm_gen_fused_moe mxfp8_bf16 cases as xfail until we converge on fix without causing regression on B200.

🔍 Related Issues

🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete.

✅ Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit (or used your preferred method).
  • I have installed the hooks with pre-commit install.
  • I have run the hooks manually with pre-commit run --all-files and fixed any reported issues.

If you are unsure about how to set up pre-commit, see the pre-commit documentation.

🧪 Tests

  • Tests have been added or updated as needed.
  • All tests are passing (unittest, etc.).

Reviewer Notes

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello @Quackens, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a temporary measure to mark certain Mixture-of-Experts (MoE) quantization unit tests as expected to fail (xfail) when executed on SM103 compute capability hardware. This action is taken to prevent known regressions on B200 GPUs, allowing development to proceed without being blocked by an issue that is currently being addressed.

Highlights

  • Test Temporarily Disabled: Specific unit tests for MoE quantization classes (mxfp8_bf16 cases within test_trtllm_gen_fused_moe) have been temporarily marked as xfail.
  • Targeted Architecture: This xfail condition is applied exclusively to systems with compute capability sm103 (e.g., B200 GPUs).
  • Reason for Change: The tests are being temporarily disabled to prevent regressions on B200 GPUs while a permanent fix is being developed.
  • Affected Test Case: The xfail specifically targets FP4Moe implementations using QuantMode.FP4_MXFP4_Bf16.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request temporarily disables a test case for mxfp8_bf16 on sm103 to prevent regressions on B200. The change is straightforward and correct. I've suggested a minor refactoring to improve code style and robustness by using isinstance for type checking and comparing the compute capability tuple directly.

Comment on lines +2038 to +2043
if (
type(moe_impl) is FP4Moe
and moe_impl.quant_mode == QuantMode.FP4_MXFP4_Bf16
and compute_capability[0] == 10
and compute_capability[1] == 3
):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For better code style and robustness, it's recommended to use isinstance() for type checking instead of type() is. isinstance() correctly handles inheritance. Additionally, comparing the compute_capability tuple directly with (10, 3) is more concise and readable.

    if (
        isinstance(moe_impl, FP4Moe)
        and moe_impl.quant_mode == QuantMode.FP4_MXFP4_Bf16
        and compute_capability == (10, 3)
    ):

@yzh119
yzh119 enabled auto-merge (squash) September 22, 2025 17:30
@yzh119
yzh119 merged commit 7c8bcdb into flashinfer-ai:main Sep 22, 2025
2 checks passed
feih-nv added a commit that referenced this pull request Jul 29, 2026
Expose the existing TRTLLM-gen `MxFP4xMxFP8` (W4A8) and `MxFP4xBf16`
(W4A16) kernels through the unified MoE API.

## 📌 Description

This is PR 3 in the unified MoE quantization series for FP8 support:
1. #4026 — unified block-scale FP8 (merged)
2. #4091 — unified per-tensor FP8 (merged)
3. This PR — unified TRTLLM MXFP4×MXFP8/W4A8 and MXFP4×BF16/W4A16

### Changes
- Generalize `TrtllmFp4RoutedRunner` beyond NVFP4:
  - `QuantVariant.MXFP4`: `MxE2m1` weights × `MxE4m3` activations
  - `QuantVariant.W4A16`: `MxE2m1` weights × BF16 activations
- Add variant-aware TRTLLM FP4 preparation:
  - MXFP4 weights with 32-element UE8M0 scales
  - MXFP8 activation preparation for W4A8
  - BF16 activation preparation for W4A16
- Add shape, dtype, and scale-layout validation.
- Add unified conformance and fuzzer coverage for packed and
`FromLogits` routing.

### Support matrix
- NVFP4 and MXFP4/W4A8: SM100, SM103
- W4A16: SM100 only (remains disabled on SM103, matching upstream xfail
#1754)
- SM120/121: separate b12x backends where available
- SM107 is unsupported after #4171 

### Scope
- No CUDA/C++ kernel changes; both modes already exist in the TRTLLM
flat API.
- CUTLASS W4A8 is out of scope.

## 🚀 Pull Request Checklist

### ✅ Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

## 🧪 Tests

SM100, CUDA 13 CI container:
- `tests/moe/test_unified_moe_mxfp4.py`
  - 21 passed
- Unified fuzzer, packed routing:
  - seeds `900017,900018`
  - 2 passed
- Unified fuzzer, `FromLogits`:
  - seeds `900019,900020`
  - 2 passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants